This pile of beans and this chart help explain this or why polls that are supposed to predict election results keep missing the mark. To really understand the problems with modern polling, we’ve got to start at the beginning. In 1936, the US presidential election pitted incumbent Democrat Franklin Roosevelt against Alf Landon, a Republican governor from Kansas. Now, up to this point, polls have been informal. Newspapers used to gauge voter sentiment by surveying their readers, but it wasn’t based on any formal survey methodology. In 1916, a magazine called the Literary Digest started conducting national polls. They sent out millions of postcards to their mailing list mostly drawn from phone books and vehicle registration records. The 1936 survey, was their biggest one yet. Meanwhile. From office of the Princeton, New Jersey a famous statistician, Dr George Gallup tells Washington from week to week what the nation is thinking. George Gallup had a background in consumer research. He accurately predicted that Franklin Roosevelt would defeat Alf Landon. The Digest had predicted a Landon victory. Gallup called that too. He knew the Digest would miss the answer because he saw the flaw in their method. Remember how they got their mailing list from phone and auto records, people with landlines and cars tended to be older more affluent and more conservative. Therefore, the Digest results undercounted FDR supporters, and overestimated support for Landon. Gallup got it right because he used a method from statistics called quota sampling. He said that sampling public opinion is like sampling soup. One spoonful can reflect the taste of the whole pot if the soup is well stirred. In other words, you don’t have to survey every single voter. You just need to survey the right subgroups of voters. Gallup used beans to explain his method. You’ve got a barrel of beans, millions of beans. Half of them are white beans and half of them are black beans. Now, count out 3,000 beans. Those beans are your sample. Write down what percentage of white beans you’ve got and what percentage of black beans you’ve got and repeat the process, 1,000 times. What you’ll find is that 997 out of those 1,000 times your sample will have the same make up of beans as the whole barrel. So, you’ll have 50 percent white beans 50 percent black beans. Well, roughly the same within three percent. So, In every Gallup poll, we include people from all walks of life, and in the right proportion. The farmer, skilled workers, white collar workers, business executive If you’re trying to figure out which presidential candidate Americans prefer, you want your sample to be the American voting public in miniature. The sample is the key to the whole thing. So, here’s where the trouble starts. Look at this chart. We just don’t use landlines as much as we used to. More than 60 percent of us primarily use our cell phones, and this demographic tends to be younger more diverse and more liberal leaning. The people who tend to still use landlines are older and more conservative. But if you don’t get in touch with the people who mostly use their cell phones, you’re going to throw off your sample in the same way the Digest did back in 1936. So, can’t we just call people on their cell phones? It’s not so easy. There’s a law against auto dialing cell phone numbers. It’s from 1991 and it was supposed to protect us from invasive telemarketing practices. But opinion surveys were designed to be conducted over the phone using automated dialing. Hello. This is a nationwide radio survey. Were listening to your radio just now? No I am not. Were you listening to your radio just now? Oh yes I am. It can be twice as expensive to reach poll respondents on cell phones because you have to hire and pay real live humans to dial the numbers. Whether we’re talking landline or cell phone, there’s a societal shift that’s endangered Gallup’s method. We’ve been trained by telemarketing and caller ID to screen our calls. We just don’t answer random numbers. That means low response rates. In the 1970s, a good response rate for a national poll was in the 80 percent range. Today it’s an abysmal eight percent. That’s bad news for polling because the smaller the sample size, the less accurate the poll. Pollsters can try to compensate for low response rates using weighting. But that’s another place where things can go off the rails. Let’s say you can’t get any Hispanic millennial men to answer the phone and respond to your survey. You finally get this one guy. But demographically, you should have three times as many Hispanic millennial men. So, you weight his answer three times. One guy standing in for the whole Hispanic millennial male population, and if he’s enough of an outlier that he actually answered the phone, he could be an outlier in other ways including his answers to the survey. If you’re just trying to measure preferences opinion surveys offer valuable insight From America which do you think it’s more important to do? Keep out of war ourselves or helping them win even at the risk of war. It’s when we try to predict human behavior that we get into trouble. While it’s easy to get people to tell you which candidate they prefer. It’s a lot harder to predict who’s actually going to get off the couch and go cast a ballot. Gallup himself had some notable misses. On election night 1948, Republican campaign leader Herbert Brownell and a legion of news men continued to echo public opinion polls that said Thomas E. Dewey would be the next president of the United States. A real poll taker polled in 1948 that the poll guardian valuable. But at the same time, we know that no better method has yet been found for measuring political trend. Ironically, he didn’t think predicting election outcomes was all that important. He only got into the election prediction game because he wanted to demonstrate the value of his main product. His opinion research surveys. What we’re seeing today is it’s harder to get a good sample thanks to changes in how he use the phone, and it’s just as hard as it ever was to predict human behavior. Thanks for watching. Hit the comments with your thoughts, questions, hot takes. Like, subscribe, and we’ll see you next time.