October 18, 2016

Nate Silver Is A Rigged System

Patrick Marren
Principal

If you're like me, the past few months when you get on line in the morning, one of your first stops is FiveThirtyEight.com. Specifically, its three presidential prognostication models, "Nowcast," "Polls Only," and "Polls Plus.

FiveThirtyEight thoughtfully allows the bleary morning pre-coffee reader to choose exactly the level of panic or elation she or he wants to experience. If you are of a steady temperament and don't like to get too high or too low, you can choose "Polls Plus," which tempers current poll numbers with adjustments for the economy and other things that are expected to exert influence by November 8. If on the other hand you can't wait for the coffee to finish brewing to get your morning jolt, you can choose "Nowcast," which is the most immediate and volatile of the measures. It attempts to estimate what would happen if the election were held now, before your coffee. It tends to (over)react semi-instantly to the latest news, causing the reader to experience the maximum level of emotional whipsaw. And then in between is "Polls Only," which attempts to estimate the November 8 outcome without the fooferaw of additional tweaks, merely based on the pile of polls FiveThirtyEight samples and weights according to past performance. 

Anyway, in 2016, FiveThirtyEight has been more volatile than it was in 2012. In 2012, Mitt Romney hit his highest percentage chance of victory (38.9%) on October 12, after Obama's prospects had slid after his poor performance in the first debate. Obama rebounded with a strong second debate performance, and his percentage chance of winning steadily increased until it hit a peak of 91.4% on November 5, one day prior to the election. (FiveThirtyEight, then a feature of the New York Times, had just one estimate of presidential chances back then, so there was no option for the bleary reader to choose her or his level of emotional turmoil. Progress.) 

FiveThirtyEight followed up its 2008 premiere, in which it successfully picked 49 out of 50 states' presidential preferences, by running the table in 2012: they predicted 50 out of 50 second time around. Nate Silver was declared God of Nerddom, and shortly thereafter upped sticks and relocated to ESPN, which is less of a departure than it might appear, since Nate began his prognostication by contributing mightily to the "Moneyball Revolution" in baseball. His PECOTA model has become the industry standard for predicting future baseball player performance. (PECOTA is supposed to stand for Player Empirical Comparison and Optimization Test Algorithm, but real baseball fans know it was named for Bill Pecota, a journeyman .249 lifetime hitter who played every position during his major league career with the Royals, Mets and Braves.)

This cycle, things have been a bit more volatile, and also a bit hairier for Nate and FiveThirtyEight. First, he wrote a 2015 column in which he laid out the "Six Stages of Doom" a Donald Trump candidacy would have to get past before he could win the nomination. Whoops. Trump blasted through every stage without problem. Not only did he cruise to the nomination, he even surged into the lead against Clinton after the Republican convention, with a 55.4% chance of winning if an election were held on July 27. Even the staid Polls Plus, prior to the first debate on September 26, showed Trump as having 45% chance of winning the election at that late point. 

As in 2012, a debate has appeared to act as a turning point (though FiveThirtyEight staff disputed the idea that the debates would change much of anything in an on-line dialogue prior to the first one). Clinton's numbers began an Obama-2012-style rebound after the September 26 debate. Revelations of Trump misbehavior have since appeared to add momentum to Clinton. And now, a mere three weeks since FiveThirtyEight told us the race was more or less a coinflip, even the conservative Polls Plus has Clinton at an 84.2% chance of winning the presidency. 

So what does this tell us? 

Well, let's put to the side for the moment the issue of whether FiveThirtyEight or Nate Silver are good at their jobs of Bayesian analysis. Let's assume they are very good at that. Ninety-nine out of 100 states over two elections earns them some solid street cred. Let us give them their props. (Neither let us refrain from fruitful alternative approaches, such as that of the estimable Mr. Gerard Smith, of an upstart outfit calling itself the Futures Strategy Group, right here. He focuses on undecideds and comes to the conclusion that there is less volatility than may appear, and that Clinton has been on track for a solid victory for a while.)

But let's also back away a few steps and ask ourselves: What are we doing here? 

We're voting for president. 

What are the stakes here, for our lives, businesses, families, etc.? 

Maybe trivial, but possibly very great. Potentially huge.

Also very uncertain. No matter which of these candidates wins, we could find ourselves in economic crisis - or economic boom - for reasons only tangentially related to anything they have done. We could find ourselves at war with North Korea pretty much at any time, based on their actions over the past three decades. We ARE at war in the Middle East. Not just in Iraq, but also, as of this week, in Yemen. (If you didn't know about this, maybe you want to read about it here.) 

Of course, specific actions of the next president could also have huge ramifications for us. The president can declare war; the president can sway markets with statements, much less actions. The president, in short, matters.

So let's say Nate Silver and FiveThirtyEight know what they are doing, and Hillary now has an 80%+ chance of getting elected.

That means Donald Trump, now at his lowest point, has almost a 20% chance of winning. 

As Nate points out repeatedly in his book The Signal and the Noise, when a meteorologist says there is a 20% chance of showers and it rains on your parade, that does not make her a bad meteorologist. It makes you dumb for assuming that "20%" actually meant "No way." 

So, assuming that Nate and his minions know whereof they speak, unless they are saying everything is a 0% or 100% shot, they need to have a 20% outcome take place now and again, or their model is not correctly calibrated.

That means one out of five times, we should get President Trump. 

But this is not a Monte Carlo simulation. This is one time for all the marbles. Either Hillary wins, or Trump wins, and we need to be prepared in depth for either outcome. 

This is my problem with applying PECOTA to presidential elections. PECOTA works because every team has 100 or more prospects and players, and if PECOTA predicts one guy will hit .350 and he hits .180, well, there are 99 other guys to pick from, and one is bound to exceed his projection. It also works because baseball is kind of a closed system, in which change is very slow and past performance truly does help to predict future outcomes. Sure, Jake Arrieta had an ERA of 7.23 for the Orioles, and he's thrown two no-hitters for the Cubs since the O's traded him; but the Orioles have had other good pitchers come up to take his place. (Not AS good, but still.)

But this election is not like that. This election is more like Jake Arrieta having a 7.23 ERA for the Orioles, getting traded to the Cubs, and, let's say, going 30-0 twice by pitching from third base, or eliciting epileptic seizures in batters for two seasons.

This election has not been about the algorithm needing tweaking. It's about the algorithm being entirely irrelevant due to an out-of-left-field (sorry to re-mix metaphors) event. 

Maybe Congressional elections, with hundreds of candidates, are more amenable to Nate's approach. But presidential elections aren't like baseball. They are big, one-time events. Events that will have consequences that cannot be predicted with precision, but which need to be analyzed in advance in depth, so we'll be ready for them when the chaos hits and we have to make quick decisions.

The only approach I know of that can handle a situation where the outcome is unknowable, but the consequences serious, is scenario planning. You have to imagine multiple outcomes (E.g.: Hillary wins big; Hillary wins in a close election; Trump wins big; Trump wins in a close election; someone else unexpectedly wins.) It is only by ASSUMING a certain outcome that you can take that outcome seriously enough to plan against it, to work out how it MIGHT have happened, to work BACKWARDS to speculate about possible causes and effects.

And probabilities have NOTHING to do with these scenarios. Why not? Because you have to take EACH ONE OF THEM seriously. If you assign a probability of 45% to one scenario, and a 2% chance to another, which one do you think you will think harder about? Yet 2% probabilities come through every day. On October 11, the Cubs were down to a 1.7% chance to beat the San Francisco Giants in the 8th inning of their game, according to Fangraphs. They exploded in the top of the ninth and won. That the Cubs would slice through 5 Giant relief pitchers was a scenario that Giants fans had not entertained entering the 9th inning. (Of course, it was also a scenario that no Cub fan had ever even thought about in 108 years or so. Certainly I never had.)

So scenarios are NOT about probability. They are about, first, deciding what you truly care about - what affects your life, your business, your family, your world; then deciding what dimensions might define the future state of those things; then, systematically varying those defining dimensions, irrespective of probabilities, to try to IMAGINE things you are not already thinking about. 

That's where Nate, great prognosticator though he is - and he is, I am NOT being sarcastic here - could not anticipate Trump. It's because he did not work backwards FROM a Trump victory and ask himself, "Okay, this happened - HOW did it happen?" He needed to change his analytical framework; instead he stayed within it, and got Six Stages of Doom. Scenarios would have turned this around, and had him speculating about the Six (or more, or less) Things That Made Trump Win - before he had won. 

Aside from all this, analyzing a presidential election as if it were baseball might not make us great citizens either. If we start every day looking at whether a percentage has gone up or down, we start to think about politics as a pure spectator sport, one over which we can have no influence. It can make us fatally fatalistic, and the apparent volatility of the outlook can enervate the electorate. 

Mister Dooley, the character created by the journalist Finley Peter Dunne, famously said around 1900, "Politics ain't beanbag." Well, it turns out that presidential politics ain't baseball.

So if your world will be rocked by either of these two candidates being elected - or neither - you might want to invest in a little scenario planning.

Thoughts?