Stochastic Democracy

Polling: NC1, NC2, NC3, and Precinct-level Post-Strat

2011-11-05T16:28:00.002-04:00

Estimated Obama Approval by precinct in the North Carolina 3rd Congressional District

One thing we've been trying to do is push the limits of what can be done with a traditional poll of a district. This is a map of estimated Obama Approval by precinct, based on a multi-level Bayesian spatial model that took into account past election results, the 2008 Presidential Exit Poll, and the poll of 750 Registered Voters we did of the district. The precinct estimates should be accurate to about six points, which is pretty incredible considering that there are only a couple respondents per precinct.

Here's our first round of public polls:North Carolina's 1st Congressional District: [Obama approval, Congressperson Approval]. Click here for a detailed poll report with cross-tabs.

President Obama depended on strong support from the heavily African-American 1st Congressional District of North Carolina to eek out a win in the state in 2008. Since he retains the support of the black voters in the district, the President is still on a good track to repeat that showing in this district.

He leads Mitt Romney by 28 points, by a 60-32 margin, thanks to his pretty strong 56-41 approval rating. He also leads Rick Perry by a 60-33 margin, but remember all of these polls were taken in mid-October as test subjects for our new polling outfit, so considering Gov. Perry has since taken a nose-dive in his favorability numbers, he would probably do even worse today.

It is striking that even in this heavily Democratic district Governor Beverly Perdue has a negative approval rating at 37-44, but she still leads former Charlotte Mayor Pat McCrory by a 47-36 margin. The discrepancy stems from the fact that African-American voters approve of her by just 44% to 29% disapproval, but still vote for her over McCrory by a 66-16 margin.

Senator Kay Hagan does better than Governor Perdue, but worse than President Obama-- mostly because she's less-well known than the President and thus many supporters of President Obama are hesitant to back her-- she gets only of 70% of voters who approve of President Obama. But she still leads Congresswoman Renee Ellmers by a margin of 49-30.

Congressman G.K. Butterfield is moderately well-liked in the district, with an approval rating of 37% to 33% disapproval.

North Carolina's 2nd District:[Obama approval, Congressperson Approval]. Click here for a detailed poll report with cross-tabs.

North Carolina's old 2nd Congressional District is the classic example of a swing district with a slight Republican tilt: It was carried by former President George W. Bush by convincing margins twice, but bolted for President Obama by a narrow margin in the Presidential Election 2008, allowing him to win the state of North Carolina. Its incumbent Democratic Congressman Bob Etheridge was ousted by an even narrower margin by Republican Renee Ellmers after a personal scandal in which he attacked two teenagers tracking him with a video camera.

The probably most interesting finding in this poll: Fmr. Congressman Etheridge has regained some of his further popularity, leading Congresswoman Ellmers 46-40 in a hypothetical rematch. However, considering the district was made more strongly Republican in redistricting, this lead probably wouldn't exist under the new boundaries.

President Obama sports a negative 47-50 job approval rating, but leads Governor Romney by a 7-point margin-- 48-41. It is notable that here, as well, almost none of the undecided voters approve of President Obama's job performance, so once these voters come home, Romney should be in a narrow lead here.

Governor Perdue sports an atrocious 29-53 approval rating and trails likely Republican nominee Pat McCrory by an 11-point margin-- 36-47. With these numbers, reelection seems close to out of reach for her.

Senator Hagan leads Congresswoman Ellmers by one point in Ellmers' own district-- 42 to 41. Still, Ellmers is not terribly disliked by her district-- she has a 37 percent to 31 percent approval rating-- which is better than a lot of politicians these days.

47% of voters in this district want President Obama to become more conservative, while 18% want him to become more liberal, and 35% are comfortable with his current posturing.

North Carolina's 3rd District: [Obama approval, Congressperson Approval]. Click here for a detailed poll report with cross-tabs.

The third Congressional District of North Carolina is a classic Southern district: It is made up of some African-American voters, but mostly of ancestrally Democratic, conservative Whites. This is perfectly illustrated by the fact that Democrats have a 12-point Party-ID gap over Republicans here (38% of respondents identify themselves as Democrats, 37% as Independents, and just 26% as Republicans)-- but at the same time 50% of respondents say they're Conservatives, with just 16% identifying themselves as liberals.

President Obama is very unpopular in this district. He has an approval rating of 36 to 59 percent, trails Governor Romney by 12 points (37-49), and that's only going to get worse as people who still haven't formed an opinion on Romney, but disapprove of the President's job performance come home.

Governor Perdue has an approval rating of 28-54 here-- which is notable, since her home base of New Bern is located in this seat. She trails Pat McCrory by 11 points, 33-44. Winning this district in 2008 even while President Obama lost it heavily against Senator McCain was a cornerstone for Gov. Perdue's victory in 2008. She doesn't seem likely to be able to repeat that.

Senator Hagan trails Congresswoman Ellmers by three points here-- 37-40, but with 23% of voters undecided this race is very much in flux. Congressman Walter Jones is decently popular at a 39-28 approval rating, but primary polling we'll be releasing over the next days shows that he has at least something to worry about in the Republican primary, if a strong challenger emerges.

Our polls are of registered voters who've voted at least once in the last four years, and have been weighted by Age, Race, and Gender.

MRP, IRT, and Polling

2011-11-05T16:27:00.000-04:00

For the past three years, this blog has mainly focused on election forecasting and poll aggregation. Honestly speaking, we seem to be pretty good at that. But after having predicted US House district results with an average error of 2.5 points in 2010, there honestly isn't much room for improvement for finding out who is going to win the election.

Our final 2010 race forecasts graphed against election results.

That's why I'm really excited about newer statistical techniques that will allow us to answer more subtle and interesting questions about the electorate.

Estimated Obama Approval in the first half of 2011, derived from weekly PPP polling over six months(n~20,000). Blue is more Democratic, Red is Republican, Green indicates groups for which there wasn't enough data to reliably estimate opinion. Color scale is different for Black voters to attenuate Age and State differences (California is at 75% approval).

This map was created with a technique called MRP, which uses multi-level logistic regression in order to produce sub-group estimates that are much more accurate than traditional naive disaggregation of the data into various crosstabs.

Probability of supporting Jack Davis (I), Jane Corwin (R), or Kathy Hochul(D) as a function of IRT-derived ideology for the NY-26th Special Election.

To produce this graph, we used a Bayesian IRT model to estimate inherent respondent ideology based off of their survey answers, much like political scientists estimate legislators' ideological beliefs from their votes in congress.

Understandably, it's pretty hard to get pollsters to provide the raw individual level polling data that is needed for this kind of analysis. So we decided to build our own data by building our own pollster. We're aiming to bring the kind of sophisticated statistical techniques that previously were only seen in academic journals into the realm of timely analysis. Check back soon to see our first public polls.

Women Representatives are about as Liberal as their districts

2011-06-21T18:22:00.002-04:00

Yglesias writes

As measured by DW-NOMINATE scores, Republican men and Republican women are the same. But Democratic women are much more liberal than their male counterparts:

The next step that it would be interesting to see is whether this is just a coincidence in which no women Democrats represent the kind of districts that tend to elect moderate Democrats. You could compare everyone’s DW-NOMINATE score to their district’s PVI and see if there’s a gender gap in representation.

This is easy enough to check:

Estimated ideology among Representatives in the 111th Congress vs PVI(Average of Kerry and Obama vote). Idealogy estimated from Jackman's pscl package in R, which has some nerdy advantages over DWNominate.

Now for a regression.

Once you adjust for district partisanship, it doesn't seem that Women representatives are any more liberal than would be expected.

Generic Ballot Polling Review

2011-05-17T06:50:00.001-04:00

With renewed media focus on House Generic Ballot polling, it's a good idea to see how things have shifted since the election:

The past two months of Generic Ballot Polling. For a full table, click here.

In a familiar pattern, Rasmussen polling strongly diverges from everyone else. And the relative frequency of Rasmussen vs non-Rasmussen polling at any given point swamps any actual change of opinion in any naive poll filtering method.

Large Statistically significant house-effects

In order to get an idea about how many seats the Democratic party can expect to win from a generic ballot result, we do a simple univariate regression relating overall Democratic vote with the percentage of seats won by Democrats in presidential elections since 1948. This is somewhat crude, but because we don't know what the districts will look like in 2012, more sophisticated analysis isn't really possible.

In order to tease out differences in pollster differences and the LV-RV disparity, we've used our 2010 poll filter that adjusts for house-effects.

House-effect adjusted estimates of Democratic Vote-Share over time in the Generic Ballot among Registered Voters with one standard deviation confidence intervals shown.

That's a registered voter estimate. Republicans almost always perform better among likely voters than registered voters, and the extent of this difference is often called the "Enthusiasm Gap".

Discrepency between Likely and Registered polls in the last four national elections, estimated using Stochastic Democracy's Bayesian DLM model with House-Effects in a previous post.

Using this, we can construct estimates conditional on different turnout scenarios:

This is all consistent with a very close race for control of the House of Representatives in 2012. While this analysis shows a very slight edge for the Democrats, they face a disadvantage due to large Republican gains in state legislatures that will effect voting law and redistricting, and due to incumbency advantage working in the Republican's favor due to their strong performance in 2010. Intrade puts the Democrat's chances at 42%, which seems reasonable.

Preliminary Canada Post-Mortem

2011-05-03T09:41:00.005-04:00

With all ridings now fully reporting, there's now enough data to pick apart our model and see how it did.

How our model did in comparison to ThreeHundredEight.com, the primary Canadian Forecaster. We actually produced probability distributions instead of numbers, and the distributions for the Bloc and NDP were heavily skewed:

Pre-Election probability distributions for each party's seat count.

Our pre-election probability distribution for the number of Bloc Seats.

Our pre-election forecasts graphed against Election Results (Not including Saanich--Gulf Islands).

Table showing deviation of election results from polling by province.

Table showing average absolute error of our forecasts by party and by province. Our predictions were best in Quebec and worst in Atlantic Canada.

But our model did more than just produce forecasts. We put a lot of effort into making our model probabilistic so as to produce accurate confidence intervals.

And it seems to have paid off! 89.6% of election results fell within our 90% confidence intervals, 94% in our 95% confidence intervals, and 98.9 in our 99% confidence intervals. So when we were wrong, it seems we were exactly as wrong as we said we would be!

Mean, Median, Mode

2011-05-03T05:44:00.001-04:00

Summary statistics from the Bloc Québécois pre-election seat forecast distribution. Seems like a good Stats 101 example.

Results are in!

2011-05-02T23:43:00.002-04:00

The Conservatives have won a majority. Otherwise, our projections seem to be right on the money, and considerably better than the forecasts at Threehundredeight.com and Sexton at Fivethirtyeight at the New York Times.

The Conservatives did about 4-5 points better than polls showed, and while I need to look at the numbers in more detail, it does seem to be consistent with the idea roughly 15% of Liberal voters who told pollsters that their second choice was the conservative party...voted for conservatives.

http://www.cbc.ca/news/politics/canadavotes2011/#/284 is a good guide.

New Canada Forecasts

2011-05-02T20:13:00.001-04:00

Before I get to explaining the forecasts, which are quite enthusiastic about the chances of the NDP, it's important to stress how dependent they are on the strange situation in Quebec.

Graph showing polls in Quebec over time

Right now, polls have consistently shown the collapse the collapse of the Bloc, and based on regional polling we can expect them to receive about 23% of the vote. If that is indeed what will happen, then then the forecasts below are going to hold up very well.

The problem though, is that individual riding polls taken in the past week have shown individual Bloc candidates doing about 8 points better than you would expect based off of regional polls. This discrepancy is reminiscent of the 2010 election, where there were plenty of Southern Democrats who told local pollsters they would vote Democratic due to popular candidates and good campaigns and National pollsters that they would vote Republican based off of national issues. In 2010, it turned out that they were mainly telling the truth to national pollsters, but there have been elections where the opposite was true.

Based purely off of trends inferred from local polls, you'd expect the Bloc to win around 42 seats, entirely at the NDP's expense. Based off of regional polls, they'd get around 7 seats on average.

Outside of Quebec, things should hold up fine. Riding level estimates are available here and here.

Note the Seat distribution of the Bloc. Due to the wide uncertainty regarding Quebec, the Bloc could theoretically win almost every Riding in Quebec. They're just not particularly likely to pick up any particular seat. Uncertainty with regards to the Bloc comes at the expense of the NDP, which is why their distribution is so skewed.

Useless Graph of the Day

2011-04-28T01:57:00.002-04:00

Calculated from Simon Jackmon's excellent pscl package on R. Unfortunately, the 35th Parliament was the most recent one I could find with a machine readable roll call matrix (If anyone can send me something more recent...). The obvious outliers are due to by-elections and party defections.

Roughly speaking, these graphs are created by pscl by extracting latent dimensions that best describe variation of voting amongst legislators.

Voting seems to be two-dimensional, corresponding roughly to "Pro-government vs anti-government" and "Quebec Separatism". This is in contrast to the United States, where voting is strongly one dimensional on the basis of ideology. Ideology doesn't seem to be extractable from voting records in Canada, as the NDP appears between the Right and the Liberals, probably due to strong control over the agenda by the Liberals who were careful not to call votes that would differentiate leftist parties.

It's also odd that party unity is so much weaker amongst liberals than the opposition. It'd be interesting to see if that continued as time went on.

Canadian Federal Election Forecasts

2011-04-26T17:54:00.004-04:00

Forecasts for the May 2nd Canadian Federal Election. Click here for a larger version.

Probability Distributions for each Party's Seat Count

Opinion over time with 70% confidence intervals. Click here for a larger version.

Canadian Seat and Vote counts by region

These forecasts were made with Stochastic Democracy's Bayesian Canadian Election Model. It works by first estimating Public Opinion in each of Canada's provinces based off of regional polls, and then applying uniform swing on a region basis. The model takes into account all polls that were released before yesterday. Individual district-level vote predictions are available here, probability estimates are available here.

This is similar to what has been done by ThreeHundredEight, who has generously provided his extensive Canadian poll dataset. The advantage here is that this model is fully probabilistic, which allows rigorous answers to questions like "What's the probability that the Conservatives will win a majority of seats?" (30%), or "What's the probability that the NDP will end up with more seats than the Liberals?" (51%).

As a quick primer for American Readers, Canada has five major political parties: The Conservatives, The Liberals, The NDP (Can be loosely thought of as a Social Democratic party to the Liberals left), The Bloc Québécois (Quebec Separatists), and the Greens.

Because Canada elects it's Parliament based on Plurality voting in Geographical districts, Conservatives have been able to control government with less than 40% of the vote due to a fractured left-wing. Meanwhile, the Bloc Québécois have been able to obtain a disproportionate number of seats due to their concentrated support in...Quebec. The Greens tends to receive six to ten percent of the vote nationally, but has consistently failed to concentrate their vote to win and hold a single seat.

The campaign was mostly uneventful until about a week ago, when the NDP unexpectedly started to surge around the country. Our model now predicts that the NDP is now a very slight favorite to overtake the Liberals as the second largest party in Canada and to overtake the Bloc Québécois as the largest party in Quebec.

The big question mark is whether this surge will persist into the election next week, and to what extent left-wing voters will vote tactically, and if they do, who they will vote tactically for.

Miami Mayor Recall Election driven mostly by Republicans

2011-03-16T17:05:00.001-04:00

In a lop-sided result that shocked most advisors, 88% of Miami voters voted to recall Mayor Carlos Alvarez (R- Miami). Some Democrats suggested parallels with the likely upcoming recall elections of Republican State Senators in Wisconsin over controversial anti-Union legislation. Republicans were quick to point out that Alverez's recall campaign centered around the mayor's property tax increases and perceived concessions to county employee unions.

Alvarez recall vote by precinct. Alvarez did not win a single precinct with more then two voters.

Vote for Republican Senate Candidate Marco Rubio in 2010 vs Yes vote by precinct. Statistically significant but not particularly strong relationship.

The question is what extent this recall was driven by Democrats or Republican disapproval. Nothing passes with 88% of the vote without bipartisan approval, but a deeper look at the data reveals a disproportionately Republican electorate.

Turnout for the 2011 Mayoral Recall Election by precinct

2010 Senate results by precinct, falling mostly on ethnic lines. Note spatial similarity to the previous map.

A graph confirming the seeming relation between the last two maps. Precincts where Rubio performed strongly had much better turnout relative to 2010 then Democratic districts.

Election results under different year's precinct turnout levels.

These factors combined to produce an incredibly Republican electorate relative to 2010, a year that had by far the best Conservative turnout in a decade. Roughly 50% of votes in the special election came from precincts where Rubio received more than 55% of the vote, these same districts made up 37% of the votes in 2010. If precincts had turned out in 2008 at 2011 levels, Obama would have nearly lost the county, traditionally a Democratic stronghold.

This isn't bad news for Wisconsin Democrats, they seem to be doing fine. But if they prevail, it's safe to say that their electorate won't look anything like this one.

Let's Gerrymander!

2011-03-08T17:06:00.004-05:00

It's been 18 months since Stochastic Democracy wrote about Tobler's Law and its applications for Gerrymandering.

In short: Democrats tend to be clustered together, while Republicans are spread out. This creates a large and natural pro-Republican bias when States try to adopt seemingly 'fair' maps based on aesthetically pleasing, 'compact' districts.

But it seems to be intuitive that in a Democratic system a truly fair redistricting would be one that reflects the will of the voters- one in which the share of districts a party gets closely matches its share of votes.

StochasticDemocracy therefore determined the 'fair' share of Congressional districts Democrats should receive in any state by calculating the Weighted 2010 Congressional Averages by Voting Eligible Population- therefore getting rid of both the impact of illegal immigrants (who get counted in the Census but don't actually get to vote) and under-age residents.

We then apply a universal swing- so that the average Congressional Democrat received 50% of the vote. If the districts were fair, we would expect Democrats to win roughly the same percentage of seats in a state that they win in terms of votes.

We can use this to approximate the 'fair' amount of Democratic, Swing and Republican seats there should be in a state as follows:

The number of safe Democratic seats should be the percentage of votes that Democratic candidates would receive in a neutral national year, minus 5% , multiplied by the number of seats the state has, rounded to the nearest integer.

The number of safe Republican seats should be the percentage of votes that Republican candidates receive in a neutral year, minus 5%, multiplied by the number of seats in the state, rounded to the nearest integer.

The number of swing seats should be the total number of seats minus the sum of Democratic and Republican seats.

Finally, the following table also shows how many seats are expected to elect Democrats under the boundaries of the 112nd Congress, in a neutral year. Keep that in mind when comparing it to the first three columns- the first three columns use the number of seats a state will include in the next decade, the last column uses the number of seats a state was assigned for the last decade.

It turns out that it is possible in most states to create Gerrymanders that follow these rules. In most cases, they are much cleaner than the current lines.

Compare, for instance, the current Maryland map (6-2 Democratic) with our proposed 5-2-1 map.

Even of the some more daunting proposals are possible to draw.
Here, for example, is a wonderful map of New York State by Swing State Project user Johnny Longtorso that creates 8 GOP-leaning and 3 swing districts.

Or, just as interesting, a compact Alabama map by SSP user roguemapper with three districts that were won by Barack Obama in 2008- and that should be easily winnable for Southern Democrats. Of course, this map ignores the VRA requirement for an African-American majority seat. But wouldn't African-Americans in Alabama be better represented in Congress by three Democrats with a majority-Black primary electorate than by just one African-American Representative from a very heavily African-American district?

Granted, in some states it seems impossible to redistrict fairly- those are the states where votes are distributed very evenly throughout the state. It won't be possible, for instance, to carve out a GOP seat in Maine, where John McCain won just one tiny county, or three Republican seats in Massachusetts, where Pres. Obama's weakest county still gave him 53% of the vote. Fair representation being impossible in some states under our current system is, in fact, a troubling sign for our Democracy and should be a rallying cry for Election Reform.

But as long as we operate under the current rules, every effort should be made to ensure fair representation for every voter in the country, Democrat or Republican.

Preliminary 2012 Senate and Governor projections

2011-03-03T20:36:00.004-05:00

We at Stochastic Democracy are proud to announce preliminary forecasts for selected 2011 Gubernatorial, 2012 Gubernatorial and 2012 Senatorial contests- namely the 24 out of 48 races that have already been polled. We expect to expand our forecasts in this area within the next few weeks, but for now our predictions are little more than a smoothed average of available polling data.

Still, it's useful to have all of this data in one centralized place. All of the polls were done with Registered Voters, and so we have provided adjustments for different turnout scenarios in the same style as our Presidential forecasts.
2011 Gubernatorial Elections

The 2011 West Virginia Gubernatorial Race is fully open at this point. Well, that's maybe the wrong way to put it- West Virginians have clear preferences depending on if Congresswoman Shelley Moore-Capito (WV-02, R) gets in the race. If she does, she's a clear favorite, if not, then the leading Democratic candidates (SoS Tennant and Acting Governor Tomblin) are heavily favored.

2012 Gubernatorial Elections

If North Carolina Republican Pat McCrory decides to run for Governor, the race leans strongly in his favor. Incumbent Missouri Democrat Jay Nixon is favored to win his race against probable Republican nominee, Lt.Gov. Peter Kinder at this point.

2012 Senate Elections

This is a lot of data to digest and we won't attempt to interpret it right now. But depending on 2012 voter turnout and who wins the primaries, Republicans can either narrowly miss picking up the Senate or narrowly take it over. Informative!

Registered Voter Polls, the Enthusiasm Gap, and the 2012 Presidential Election

2011-03-02T18:14:00.010-05:00

Even though the election is more then a year and a half away, we already have 123 presidential polling match-ups in 27 states. Almost all of them are from Public Policy Polling - a well respected North Carolina polling firm that did very well in 2010. This is easily an order of magnitude more polls than we had at this point in the last election. The problem is that all of these polls are done on a sample of registered voters instead of likely voters due to the inability to forecast who is going to turn out to vote next November.

Discrepency between Likely and Registered polls in the last four national elections, estimated using Stochastic Democracy's Bayesian DLM model with House-Effects.

This can be a big problem. Because Republican support is concentrated amongst richer high-turnout groups, Republicans almost always do better among Likely voters than Registered Voters. This gap varies from election to election, almost disappearing at at half a point in 2008, but increasing five-fold to 2.6% in 2010. Because of this, it can be helpful breaking down results into different turnout scenarios in order interpret current Registered Voter polls.

Presidential vote estimates by candidate, Turnout, and state. Color scale runs from red to white to blue. Click for larger version.

This is a big map to digest. We've compiled all available state-by-state presidential maps and aggregated them by state, using Obama Approval and demographic information to fill in data for the states without polls via a 3 variable linear regression model (R^2~.96).

Obama needs 270 electoral votes to win the election

The take-away from this is that if the election was today, Obama would win under most turnout candidate combinations. Even under the most favorable Republican electorate in recent history, Obama would still defeat Gingrich and Palin in a land-slide. Romney and Huckabee can win, but only if they can keep the Enthusiasm gap near 2010 levels.

But this is only if the election was today. If the economy heads south due to an oil shock or a government shutdown, then Obama's baseline approval could decrease. I'd also expect a lot of Republicans who don't want to commit to Palin and Gingrich to come home. But as of now, Obama looks to be a favorite heading into the electoral cycle.

Candidate Quality in the 2010 Congressional Elections

2011-02-23T19:44:00.002-05:00

There's been a lot of talk lately about Tom Perriello, the former U.S. Representative for Virginia's 5th Congressional District and a potential US Senate Candidate in 2012.

The general question is- did Perriello, who lost by 5 points in a quite Republican district last year, overperform compared to how most Democrats did? And if he did, what could that be attributed to, given the fact that Perriello was quite liberal and didn't run away from the Democratic agenda?

This is a question that can relatively easily be answered.
A simple Linear Regression model with just three variables (two dummy variables concerning Incumbency and the Cook PVI of a given Congressional district) does a pretty good job in explaining the variance within the performance of Democratic Congressional Candidates in 2010, as you can see here:

Now, if we compare the output of this regression model- in other words, how a given Democrat *should* have done in 2010 to the real election results, which Democrats stand out as either particularly good or bad candidates? And why did they stand out- due to particularly good or bad opposition, due to their ideology or just because of weird quirks in their districts?

Let's look at the 20 Democrats who most substantially overperformed first.

The ideology scores are produced by StochasticDemocracy based on Roll Call Votes of the 111st Congress.

The Democrats who did very well can generally be clustered into three groups:
1) Very conservative Blue Dog Democrats in conservative districts, such as Dan Boren, Matheson, Taylor, Bright, Minnick, Peterson and Oliviero. Sure, some of these lost, but they didn't get the beat-down that a more liberal Democrat would have suffered in their districts. This is also true for moderate Democrats running in liberal districts, such as Dan Lipinski in Chicago.
2) Democrats who ran against tainted Republican incumbents, such as Joe Wilson, Michele Bachmann, Jean Schmidt, Virginia Foxx, Jason Chaffetz (conservative even by Utah standards!) and Roscoe Bartlett (who isn't ideologically controversial but at age 86 due for retirement).
3) Democrats running against California Republicans, who had a terrible year considering that their party was successful everywhere else.

Now what about the bad eggs, the Democrats who could have won but didn't? This is a somewhat more diverse group:

It includes, of course, the corruption-laden Kanjorski, a couple of very liberal candidates (Frank, Grayson, Mary Jo Kilroy and arguably David Cicciline who isn't out of the mainstream policy-wise, but openly gay), a couple of Democrats who just seemed to have underestimated the wave and neglected their campaigns (Kildee, Cleaver, Carnahan, Hare, Kosmas), some who had very charismatic or well-funded opponents (Halvorson vs. Kinzinger, Murphy vs. Fitzpatrick, Klein vs. West, Loebsack vs. Miller-Meeks) and, lastly, some who just had the bad luck of underperforming as Congressional challengers in conservative Districts even more than they should have (against King, Duncan, Luetkemeyer, LoBiondo, Huizenga, Miller).

What do we take from this? Running to the right seems to pay off for Democrats in conservative Districts- the list of overperforming Democrats reads like the who-is-who of Conservadems, while there are none to be found on the list of weak Democrats.

Second, running to the middle is paying off for Republicans- while several of their moderate (or at least mainstream conservative) members did very well- like LoBiondo or Candice Miller- their tea-party firebrands like Bachmann, Schmidt and Wilson tended to underperform. Virginia Foxx or Bachmann might well have lost in LoBiondo's Atlantic City-based district, while LoBiondo easily cruised to a better-than-2:1 victory.

The trade-off for both parties is obviously that middle-of-the-road Congressmen are less helpful for them in getting their agenda passed.

As we're looking forward to the 2012 Congressional Elections, it will be interesting how the Party's recruiters handle this delicate balance between legislative and political success.

Oh, and as for Perriello? Our analysis finds that- despite of the conventional wisdom- he didn't perform that much better than could have been expected from him. Ranking the Democrats from their best to worst performances, Perriello is 178th- slightly above-average, but not remarkably good.

Intrade and the Thune dropout

2011-02-22T23:53:00.010-05:00

Today Senator Thune of South Dakota announced that he wouldn't seek the GOP Presidential nomination in 2012. While Sen. Thune's support topped out at 3% in early GOP primary opinion polling, he'd been widely considered an insider favorite. This has reflected in the betting markets, where his odds to win the GOP nomination have traded as high as 25%. Even yesterday, Sen. Thune had been the fourth-ranking candidate on Intrade, placing behind only former Governor Romney of Massachusetts, former Vice Presidential Candidate and Governor Sarah Palin of Alaska and Governor Daniels of Indiana.

But in the aftermath of his announcement not to run, Thune's odds have collapsed from 9 percent to .4%.

Elections are zero-sum, so it's interesting to see who this has benefited:

The two main beneficiaries of Thune's dropout seem to have been former Governor Pawlenty of Minnesota and Governor Barbour of Mississippi. These two candidates have one thing in common: Like Thune, they are currently second-tier candidates who replicate the appeal of better established candidates who will need to drop out or become scandal prone in order for themselves to gain traction.

The problem with this theory is that former Governor Huntsman of Utah and Governor Daniels of Indiana have not gained any ground on the betting markets. A possible explanation is that Thune's announcement was cancelled out by concurrent events: A poll showing that Huntsman 50 points behind Romney in Utah, and Daniels' decision to compromise with Indiana Democrats and drop a controversial right-to-work bill that could hurt his primary appeal.

It'll be interesting to see if the markets stabilize or if prices converge to match the reality reflected in polling.

Egypt on Intrade

2011-02-01T07:40:00.000-05:00

Political Consequences of the Tax-Cut Comprimise

2010-12-11T18:14:00.005-05:00

In many ways, It's obviously premature to talk about 2012 Presidential Election. We're a year away from even the primaries, with no Republican candidates having announced, and we don't know what the agenda will be for either President Obama or the new Republican congress.

Still, academic research suggests that none of this matters: election results closely correlate with economic performance. A good example of this school of thought is the "Bread and Peace" model by Economics professor Douglas Hibbs. His model looks at just two variables, Weighted Real disposable income growth over the term and military fatalities, but still manages to explain 88% of variation in post-war election outcomes.

These models are not perfect, but provide a good tool to compare the electoral effects of competing policies. Macroadvisors, a widely regarded private-sector economic forecasting firm, has provided estimates of disposable personal income with and without the tax-deal:

While the Tax-Cut deal boosts overall income, it front-loads growth in 2011 and slows it considerably in 2012 due the the expiry of the pay-roll tax cuts and other Democratic provisions. Because voters make tend to place greater weight on growth right before the election, it's easy to see why this isn't electorally optimal for the Democrats.

Plugging these forecasts into the Hibbs model* bears this out: If the tax cuts don't get through Congress, President Obama is projected to be a slight favorite for reelection, netting about 50.7% of the two-way vote. If the Tax-Cut deal is implemented in it's current form, Obama can be expected to receive 49.95%. This difference seems small, but because the two forecasts are not independent, it is statistically significant.

It's important to stress that these results are completely dependent on Macroadvisor's economic forecasts. Still, using the knowledge currently available, it seems that the Tax-Cut compromise as currently negotiated is a net-negative for the Democrats.

The reverse would probably be true if the Democratic provisions are extended again in 2012, a situation that many people find likely. On the other hand, if the new Republican Majority implements further spending cuts, this could make the deal worse.

*This takes a bit of work, because the Macroadvisors forecasts are not per capita and inflation adjustment is done slightly differently in the model. Feel free to contact us for details.

The other issue is that the baseline involved is slightly murky, as Ezra notes. But it closely corresponds pretty closely to the "current" baseline if you go through the weeds of their public blog.

Preliminary House Post-Mortem 2010

2010-11-04T06:33:00.015-04:00

While there are still a couple of uncalled districts out there, enough returns have come in to start looking at the results and how the model did:

Vote Map

Behold our new Congress, based on provisional returns, colored according to Democratic Vote-Share. Compare with map on side-bar that shows our projections.

Swing Map

Swing in vote from 2008 to 2010. Red indicates districts where Republicans did better then last year, Blue indicates districts where Democratic standing improved, and White indicates no change.

Forecasts:

From the Monkey Cage

Stochastic Democracy predicted that Democrats would obtain 197 seats in the House of Representatives, 5 more than the currently projected outcome. This was the third most accurate forecast, 5 seats more accurate than FiveThirtyEight, and the most accurate one that also provided district-by-district vote estimates.

Missed Calls Map

Red indicates the 10 districts that Democrats were predicted to win but did not, Blue indicates 5 districts Republicans were predicted to win but did not. White indicates the 418 correctly called districts, with 2 districts still uncalled (NY-25 (Maffei) and CA-20 (Costa).

While this method of grading makes our site look good, it's not a very informative due to the record number of competitive districts this year. A much better way to evaluate forecasters is to look at how well they managed to predict vote-share district by district.

District by district, the model seems to have over-estimated Republican vote-share by about half a point. This is because the regression model appears to have inappropriately applied a uniform national swing to heavily Democratic urban districts. If we restrict ourselves to the 357 districts where the result was between 25 to 75%, the bias effectively disappears (4 hundredths of a percent toward the Republicans).

Mean absolute error was 3.2 points, while median absolute error was 2.6 points. This goes down to 2.8 and 2.5 respectively for the 25-75% districts.

But we provided full probability distributions as well as vote estimates for each candidate. It seems that our stated standard errors were quite accurate: 94.7% of results fell within our 95% confidence intervals. More generally:

5.2% of results came within our 5% confidence interval, 10.6% in our 10% interval, etc

In other words, our standard errors were well calibrated. Not only were we not wrong very often, but we were able to predict precisely how wrong we would be.

I don't have easily assessible data yet for the forecasts of FiveThirtyEight or Electoral-Vote, the only other forecasters I'm aware of who provided district level forecasts, and so I can't yet replicate this analysis to see how our performance compares. On the other hand, Gelman has looked at FiveThirtyEight's forecasts and found that his confidence intervals were too wide:

So, yes, Nate's forecasts do seem unconfident! There were 37 races where he gave the Republican candidate between 60% and 90% chance of winning, and the Republicans snagged all 37. Apparently he could've tightened up his predictions a lot. Wang appears to be correct.

I don't think that looking at win-percentages solely is a fair measure, since most races are not close at all and race outcomes are not necessarily independent. Gelman agrees:

But . . . before jumping on Nate here, I'd suggest some caution. As we all know, district-by-district outcomes are highly correlated. And if you suppose that the national swing as forecast was off by 3 percentage points in either direction, you get something close to calibration.

Some other notes of interest:

Stochastic Democracy's House forecasts were really a blend of two almost independent models: 1) A fundamentals-based regression model that took into account Incumbency, past election results, Income, Cook Ratings, and the Generic ballot adjusted for House Effects. And 2) A Bayesian state-space model that filtered polls, taking house effects, design effects, and potential industry bias into account.

The poll model that Democrats would have gotten 205 seats, while the regression model forecasted 186 seats. Paradoxically however, the poll model outperformed the Regression model in 55% of the individual races where polling was available. While more analysis is needed, it seems that while polling was somewhat more accurate than fundamentals most of the time. Yet for races where polling and the fundamentals differed considerably, ID-1 (Minnick), AL-2 (Bright), MS-4 (Taylor), VA-9 (Boucher etc (And CT-5 (Murphy) on the Dem side), the regression was considerably more accurate. There are a couple of theories as to why that could be, and it's definitely something we'll be looking at in the next couple of days.

Absolute Error of our forecast in a district vs the Number of Polls in the District

Update 2:23pm: One potential concern is if there is sufficient race-to-race correlation, then well-calibrated confidence intervals could in fact mean badly calibrated confidence intervals. See Gelman. More on this later.

Update 2:52pm: Another graph:

Our final race forecasts graphed against election results. For comparison, FiveThirtyEight and Pollster's R^2 seemed to have been .78 and .55 respectively, but consider those numbers preliminary.

Election liveblog

2010-11-02T19:18:00.004-04:00

Just to get this started-
the first couple of impressions:
IN-08: Ellsworth's open seat is gone, just like everyone expected.
KY-06: Congressman Chandler is doing not too badly here, leading by 7 here- he's projected to win by 4 by us. 50% of the district is already in, so this is relatively solid.
Indiana: Don't worry TOO much about Hill and Donnelly just yet. Yes, they're down by double digits, but their areas of strength haven't started reporting yet- for example Monroe for Hill, St. Joseph and LaPorte for Donnelly. When these come in and they're still down, Dems have a problem.
Virginia: With Boucher and Perriello both down by close to 10% with substantial parts of their districts reporting, could there be trouble brewing for all Southern Democrats? Boucher wasn't supposed to go down so easily.

When the Whole is more then the Sum of it's Parts

2010-11-02T10:33:00.000-04:00

*Note, check out our final forecast table complete with trend graphs for each race in the forecast tab!!*

Looking solely at the Generic Ballot Poll, our model sees the Republicans 6 points ahead of the Democrats (R+6) after filtering for sampling error and house effects.

But if we restrict our focus to district polling in races that have been polled the situation looks quite different: Democratic candidates are receiving 8% less then they did in 2008. Since Democratic vote share was roughly 56% in 2008, this corresponds to Democrats receiving 48% of the vote, R+4. There's good reason to think incumbency advantage plays a big role in this, so let's break it down:

The average swing in non-open seats is 6.5% (~R+1) and in open seats it's 8.7% (~R+3). The baseline from 2008 is around 56%. To break it down further:

Formerly D Open Seat (n=13) - 20.3% swing from 2008. One would expect this to be large because of the loss of incumbancy, but this is still really large. Could be a selection issue, that Democrats who saw tough re-election battles decided to retire.

Formerly R Open Seat (n=14) - 6.3% swing *toward the Democrats*. In order for there to be a swing toward the Democrats, either the overall swing must not be too large, or incumbancy advantage must be huge for these candidates. (High quality Republican candidates who were being pressured by leadership to forestall retirement until a good cycle and/or run for Senate and Gov?)

R Seats (n=23) - 5.2% swing. This is pretty small swing, consistent with D+1, which is broadly inconsistent with the generic. (Campaign effects? Dems have not spent any money in Republican held turf...)

D Seats (n=122) - 9.4% swing. This is consistent with R+6 or R+7, which is the generic. Breaking this down further:

Freshman D Seats (n=31) - 4.7% swing. Incumbency advantage once again.

Non-Frosh D Seats (n=90) - 11.2% swing.

We can look at this as some sort of over-sampling and stratify to the house, applying the average swing for each respective category to unpolled races. This isn't strictly correct, since districts are not polled at random, but because we're looking at swings and not vote totals, it shouldn't be a big issue. Extrapolating this out, we get that Dems should be getting 48.3% of the vote(and 208 seats), an R+3 environment.

The point: We see disagreement between House Polls (Implied R+4, 208 seats) and Generic polls(R+7, 197 seats), and substantial disagreement with the gallup poll(R+15, 181 seats). Generic polls and the Gallup generic especially have had a very bad track record other then 2008, though they've usually been biased toward the Democrats. Somebody is wrong, we'll find out in 10 hours.

New Projections: Republicans favored to win House

2010-09-06T22:30:00.004-04:00

[9/8/2010 - 6:11 am, our tables currently are experiencing a glitch. Check back around 4-5pm today]

Histogram showing distribution of possible outcomes in the House of Representatives in November

After a month hiatus, our forecasting model is back! The new projections are pretty unambiguous: 2 Months before the election, the Republicans now are projected to have a 75% chance of capturing the House. There is still time before election day, but the Democrats are running out of time to catch up. In past mid-term elections, the incumbent party has almost always continued to lose ground after labor day. It remains to be seen if this trend will continue.

In early July, we predicted that Democrats were set to barely keep the house. Since then, there has been a 4 point drop against the Democrats in the Generic Ballot, and that has manifested itself in individual district polls.

While the House has been volatile, the Senate candidates on both sides of the aisle who were ahead a month ago have steadily kept or expanded their leads. This slowly helps them as their opponents run out of time to compete.

The one exception to this rule has been the Florida Senate Race, where Republican Mark Rubio is facing Republican-turned-Independent Charlie Crist and Democrat Kendrick Meek. Crist came into the summer with a large lead over both of his opponents. But Crist has suffered after a post-primary surge by Meek in Mid-August. With Meek and Crist splitting the Democrats, Rubio is now the favorite in the race. Still, three-candidate elections tend to be extremely volatile due to factors like strategic voting. Because of this, our model still considers the contest a toss-up.

The other large surprise of the summer has been the unexpected primary loss of Lisa Murkowski, a popular incumbent senator from Alaska, to tea-party favorite Joe Miller. Joe Miller has had a weaker showing in the polls then Murkowski had, but still is ahead of Democratic Challenger McAdams. Our model currently assigns Miller a 94% of winning the race.

Changes

We've incorporated Governor forecasts into our model. These work in a very similar fashion to our Senate Forecasts. I don't have much of a narrative to push here, I think the tables speak for themselves. See top races on the side of the site or in depth forecasts of all races at our governor page.

We've also found that a pollster's House Effect in Senate and Governor races can be quite a bit different then in Generic and House races. This is could be due to different likely screens trying to measure the different sets of voters. We've incorporated two separate sets of house effects for each pollster for account to this. This hasn't had a very large effect, namely because we had already been doing this for large pollsters before.

Funky KY-3 Poll

2010-07-23T17:00:00.000-04:00

In general, I do my best to avoid critiquing individual polls. But in this particular case, the internal poll by John Yarmuth's Republican opponent seems particularly suspicious.

Rivercity’s latest poll showed support for Lally at 43%, Yarmuth at 44% and Undecided or Other at 13%. The automated poll was conducted from June 20-29, and surveyed 300 likely voters with a margin of error at + 3%.

1) A poll with 300 voters using perfect random sampling would have a margin of error of roughly 6%, not 3.

2) It's extremely unusual for an automated poll to be taken over 9 days.

3) Nobody has ever heard of this pollster

4) Yarmuth has released his own internal poll that shows him up by 26 points.

Because of these factors, unless we get additional information, we're not including the poll in our forecasts, though it will still be available on our publicly available polling database for reference purposes.

Testing...one..two...

2010-07-12T02:34:00.004-04:00

Welcome to Stochastic Democracy's new Election Projection System, in collaboration with Professor Wang of the Princeton Election Consortium and a soon to be named media company. Read about our methodology, look at in-depth national, state, and district level forecasts, or learn about pollster bias.

Histogram of number of Democratic House Seats if the election were today and in November

This is preliminary. Aspects of the model and formatting are likely to change in the coming weeks, so feel free to contact us if you have any questions, concerns, or ideas.

~~Update-David: There seems to be some quirks with the table generating code. As the team is either Asleep/In Poland/Stranded on a Chinatown Bus, this should be fixed by noon.~~ - Fixed, 10:30 am

Not All House Effects Are Created Equal....

2010-07-09T19:46:00.008-04:00

Estimated Rasmussen House Effects (Bias) by Race, USA indicates Generic Ballot Polling

I wouldn't read into IA and PA very much, outliers are to be expected when sampling 40 races.

Preliminary Graphs for House/Senate/National Forecasting model

2010-07-05T18:40:00.002-04:00

Stochastic Democracy is launching an a major Election Forecasting System tomorrow, featuring district level House, Senate, and National level estimates via a Bayesian Multi-level model that takes into account Pollster Bias (House Effects) and Pollster inaccuracy (PIE).

More detailed race-specific breakdowns, explanations of methodology, and polling data will be released tomorrow. In the meantime, feel free to ask questions in the comments section.

Top-Line Results:

Generic Ballot Two-Way Vote* - Democrats - 49.9% Republicans - 50.1%

Probability that Republican lead in Generic Ballot is real - 52%

Senate:

Expected Number of Seats: 55.7

House:

Expected Number of Seats: 222.6

Probability of Democrats keeping the House: 65%

Probability of Democrats keeping House if Election was today: 86%

*Two-way vote is defined as Democratic_Vote/(Republican_Vote+Democratic_Vote), which strips out undecided voters.

Tell me what you like, what you don't like, what isn't intuitive, and suggestions for alternative visualizations. Also, feel free to note if something looks funky:

Histogram of Democratic Seats in the 2010 Midterm election in November

Histogram of Democratic Seats in the 2010 Midterm election if Election was today

Estimated Two-Way Generic Ballot estimate over time (Question: "Who do you intend to vote for in your district")

The same as before, but this time with scary 95% confidence intervals

Distribution of Senate Seats if the election was today.

Color indicates expected percent of State delegation that will be Democratic if the Election was today

Senate map, colored by Democratic vote-share. Green indicates third-party candidates, Black indicates states without senate races.

Congressional district level forecast - Colored by Probability of Democratic Victory if Election was today

CD map, colored by estimated Democratic Vote-Share (Quadratic Color Scale)

For reference, the 2008 results under the same color scale

Pollster Bias relative to Pollster Consensus. Lower Numbers indicate Bias toward Democrats. (Click to expand)

How to Detect a Bad Pollster

2010-06-20T20:26:00.004-04:00

Nate Silver's new Pollster Rankings released last week have proven influential, leading to the dismissal of R2K to DailyKos, and prompting a mini-feud between Pollster.com and FiveThirtyEight .

Nate's methodology is fairly complex, but at heart, he judges pollsters by looking at how close their last poll was to an election-result. This seems intuitively fair, but mathematically, this poses problems for what he is trying to do.

Arithmetically, a poll before election day can be expressed as Poll=Opinion+Bias+Sampling Eror, so absolute error from an election result can be expressed as |Opinion + Bias + Sampling Error - Result| = |Last Minute Movement + Bias + Sampling Error|[1].

The problem arises from trying to study Sampling Error from a statistic whose effects are mainly dominated by last minute movement and Pollster Bias. Because last minute movement and Pollster Bias vary considerably between elections, adjustment would be very difficult. Even worse, this method throws away the data that comes from the vast majority of polls that come earlier in the cycle. There exists a better way!

In order to understand how to proceed, it's important to stress the difference between House Effects(Bias), Design Effects(PIE), and their implications for forecasting. It comess down to the difference between accuracy and precision.

Consider a hypothetical pollster with an absurdly large Design Effect but a negligible House Effect. Let's call it Zogby Interactive[2] .

Zogby performs just as well on average as an ideal pollster, but it's sampling error is much higher then expected. By comparing the variance of the poll series with that of public opinion, we can estimate excess sampling error. Note that Zogby Interactive Polls provide very little information about a race

Now consider another hypothetical pollster with a pretty small design effect but a large Pro-Republican Bias named Rasmussen.

Rasmussen polls follow Public Opinion almost perfectly, but with a pro-Republic Bias. By measuring difference from public opinion, we can estimate House Effects. Note that once Bias is estimated and accounted for, Rasmussen polls provide quite a bit of information about a race.

Nate Silver has noted both effects, and has tried to estimate House Effects separately in the past. But Opinion, House Effects, and Design Effects all require each other to estimate, and so any accurate model should consider them jointly.

In order to do so, we've set up a Bayesian Dynamic Linear Model and estimated it with MCMC. [5] The model is pretty simple, and is a slight generalization of the site's previous methodology that has performed pretty well [3]:

We assume that public opinion follows a simple random walk[4]. But public opinion isn't directly available. All we have are noisy and biased polls.

A Dynamic Linear Model, where x(t) represents public opinion, and y(t) represent observed polls, arrows indicate conditional probabilities. See footnote [5] for details

Actual Estimates:

One of our system's best feature is that it's a fully Bayesian probability model, allowing us to derive joint posterior distributions of all of our parameters.

Because of this, we can judge the primary debate between Mark at Pollster.com and Nate at FiveThirtyEight. Do Design Effects(PIE) vary significantly among Pollsters?

Short Answer: No

In order to check this, we ran our model on this cycle's Generic Ballot Polling, setting very weakly informative priors for each Pollster's PIE, without assuming any relationship between them.

Estimated Pollster PIE vs Number of polls in the Generic Ballot database.

The issue here is that PIE is a ratio strictly greater then one, and so greater uncertainty will drive the mean estimate in only one direction. On Andrew Gelman's suggestion, I imposed a Hierarchical model on the priors in order to account for this, leading to the following estimates:

Corrected PIE vs number of Polls in Generic Ballot

There's little reason to believe from this graph that Pollster Introduced Error in the Generic Ballot varies at all. The difference between even the best and worst pollster is not statistically significant from zero.

It seems that pollsters perform about 25% worse then would be predicted by ideal random sampling, which would add less then point of error. This is in agreement with previous research, but directly contradicts Nate's findings that polls have an added error of 3-4 points.

The discrepancy occurs because Nate's estimating method conflates House Effects with PIE. This inflates the estimates to be bigger then they actually are.

Preliminary House Effects by Pollster with Standard errors. Pollsters in Bold have effects statistically significant then zero. [6]

The Exception that Proves the Rule:

While the graphs were from the Generic Ballot Polling of this cycle, the results hold for other national and state level races from 2008, 2009, and 2010. It seems that for election horse-racing, accuracy does not vary significantly among Pollsters [7]. But polling for Obama Approval seems to have much higher Design Effects and little more differentiation between Pollsters.

Estimated Pollster Introduced Error vs Number of Polls in Obama Approval Database

It's not clear why Polling for Obama Approval is more unreliable then polling for elections. It could be that because there is no accountability in the form of an election result, discipline and standards are laxer then otherwise.

What Comes Next:

Preliminary Graph of Obama Job Approval with 95% confidence intervals, adjusted for House Affects and PIE.

Over the next couple of days, we plan to release an election forecasting model for House, Senate and Governor races, as well as introducing a house-effect and PIE adjusted tracker for the Generic Ballot and Obama Job Approval. This is to be done in collaboration with the Professor Wang at the Princeton Election Consortium and a soon-to-be-named major polling website. Expect more in the coming days.

Footnotes

[1] - 538 looks at all polls from 21 days before the election to try to correct for last minute mean reversion. Last minute movement should increase as we move further from election day, potentially creating another confounding factor.

[2] - This gives Zogby Interactive too much credit, they have a very large house effect as well as a design effect.

[3] - This is based on the previous work of Simon Jackman of Stanford and Mark Pickup at Oxford and the UK PoliticsHome Poll team, who both had generously made code available.

[4]This is a technical footnote, but there is a good deal of evidence that public opinion of major parties in Western Democracies does indeed follow a simple random walk, as opposed to one subject to "momentum" and "mean-reversion".

[5] That is to say,

Opinion(t) = Opinion(t-1) + e(t), where e(t) is a random variable with mean 0.

Meanwhile, a poll with n respondents on day t is a normally distributed random variable with mean Opinion(t)+Bias and variance [Opinion(t)*(100-Opinion(t))/SampleSize]*PIE^2, where PIE denotes Pollster Introduced Error or a Design Effect.

Under this specification, PIE can be interpreted as the Ratio of actual sample variance to variance predicted by perfect random sampling.

[6] House-Effects seem to vary a bit from race to race. We're planning to produce more precise House Effect Estimates by pooling parameters from different races using a further multi-level model.

[7] Except Zogby Interactive. Zogby Interactive did not show up in the graphs because they have not fielded any polls in the Generic Ballot. But their PIE in other races has been flagged as an outlier, having a much higher Design Effect then any other Pollster.

What do Tuesday's primaries mean for the General Election?

2010-06-10T09:43:00.001-04:00

This is actually my first post on this blog, even though I've been working with David closely for the last two years. I'm Rasmus, a 17-year old German high-school kid, who's somehow been drawn to US politics. Besides statistics, I'm also an intern and have worked with several local and statewide campaigns, especially in Montana.

Today I’m going to summarize the implications of the most important of yesterday’s primaries for the November 2 General Election.

AR-Sen

It probably doesn’t make a huge difference who the Democrats nominated to run against current U.S. Rep. John Boozman. The Republican led Lt. Gov. Halter by about 20 points, Sen. Lincoln’s deficit is even higher at roughly 25 points. Lincoln’s somewhat unexpected victory in the run-off election on Tuesday won’t change the fact that either Democrat would have lost this seat come November.

CA-Sen

The conventional wisdom is that Carly Fiorina is a weaker General Election candidate than former U.S. Congressman Campbell. And indeed, in pre-election polls Campbell was running several points ahead of Fiorina’s performance. But Campbell ran a weak primary campaign and was heavily outspent by Fiorina, so that the Republicans might actually have nominated the better campaigner, even though she starts off from a weak position and about 7 points behind incumbent Senator Barbara Boxer.

CA-Gov

The widely forecasted victory of billionaire candidate Meg Whitman over State Insurance Commissioner Steve Poizner is probably a good thing for Republicans. For once, they won’t need to spend money on the race- Whitman is most likely going to finance the campaign on her own. Also, Whitman has been a bit stronger than Poizner in the polls against Democratic nominee Jerry Brown, the current Attorney General and former Governor.

CA-Prop 14

California also voted to establish a so-called jungle primary system today, where all candidates for office run on the same primary ballot, with the top two vote-getters advancing to a run-off election. The proposition was strongly opposed by minor parties, who will lose their General Election ballot lines, unless they manage to place ahead of the top Democrat or Republican in a race.

The new voting system is expected to strengthen the influence of Moderates and Independents and will probably lead to increased centrism in both parties, as candidates struggle to position themselves close to the median voter of the whole primary electorate.

IA-Gov

This is a race where the primary made a real difference. The former Governor of 16 years, Terry Branstad, won his primary against Conservative insurgency candidate Bob Vander Plaats. Branstad enjoyed wide support in the establishment and had the endorsement of Sarah Palin, nevertheless the primary turned out to be a lot closer than expected- Branstad won by a 50-41 margin. Branstad will go into the General Election against incumbent Gov. Chet Culver as a heavy favorite, leading in the polls by a double digit margin, while a Culver-Vander Plaats matchup would have been an instant toss-up.

ME-Gov

The General Election has been impossible to poll yet, as the Republican primary field had seven competitive candidates to the Democrats’ four candidates. The Democrats had a four-point lead on a Generic Gubernatorial Ballot in October 2009, according to PPP.

The winners of the primaries are Democratic State Senate President Libby Mitchell, who was rated as a slight favorite pre-primary, and Republican small-town mayor , tea-party favorite and businessman Paul LePage. This is a minor surprise, as LePage had ranked 2^nd and 4^th respectively in the two public polls of the seven-candidate primary. Yet, he more than doubled his closest trailing opponent and took 38% of the vote.

NV-Gov

As expected, former Attorney General Brian Sandoval easily disposed of scandal-tainted incumbent Jim Gibbons and will face Clark County Commissioner Rory Reid (Clark Cnty has more than 75% of Nevada’s population). Rory Reid is the son of U.S. Senate Majority Leader Harry Reid. Sandoval will be a strong favorite in the General Election, leading Reid by double digits in every public poll that has been conducted so far. Reid was leading Gov. Gibbons in the polls, whose approval rating got as low as 10%.

NV-Sen

Tea Party favorite, former State Assembly Member and unsuccessful Congressional candidate Sharron Angle won a three-way Senate primary against former State Sen. and Party favorite Sue Lowden and businessman Danny Tarkanian. This means that she’s going to go up against Sen. Majority Leader Harry Reid in the fall campaign. Reid, whose approval ratings are in the low fourties, had been at a huge deficit against all three candidates earlier in the campaign, but he has rebounded as Lowden and Angle stumbled during the final stage of the primary- Lowden over controversial remarks that people could barter for health care coverage with things like chicken, Angle over the unpopular position that Yucca Mountain should be used as a nuclear waste repository, strongly opposed by Sen. Reid.

The Angle-Reid general election is a toss-up, as would have been the Lowden-Reid matchup. The only candidate who would have had a leg up on Reid was Danny Tarkanian, who ran an unremarkable campaign and was therefore not at the center of media coverage (and scrutiny). Tarkanian would have led Reid by about seven points, but placed third in the primary election.

ND-Sen

Universally popular Governor John Hoeven won the nomination of his party unopposed and will go on to win the General Election against Democratic State Senator Tracy Potter.

SC-Sen

The most stunning upset of yesterday’s primaries happened here, as unemployed veteran Alvin Greene won the Democratic Senate nomination to run against Sen. Jim DeMint, a leader of the national conservative movement. It had been widely anticipated that DeMint would run against Vic Rawl, a former State Rep. and current Charleston County Councilman. Rawl had raised $180,000 for the race and had the support of the SC Democratic Party, but lost the primary to Alvin Greene. Greene did not campaign for the nomination, didn’t raise or spend money, neither Rawl nor the SC Democratic Party Chair have ever met him or talked to him, and it is unclear how Greene managed to pay the $10,400 filing fee. Greene has failed to file necessary legal papers with the SC Secretary of State and the FEC, and it is unclear how the Democratic Party of SC will react to the primary result.

Jim DeMint, who is sitting on a huge warchest, won the primary nomination of the SC Republicans with more than 80% of the vote.

SC-04

Incumbent Congressman Scott Inglis has been forced to a run-off election here… finishing second at 27% in a five-candidate race where all other four candidates were running against him. Inglis voted for TARP, that’s what did him in. His run-off opponent and the next SC-04 congressman (it’s a R+15 district) is prosecutor Trey Gowdy, who won 39% in the first round. The support of the other three candidates is expected to break strongly towards Gowdy, as all four challengers ran on an anti-incumbent message.

SC-Gov

Despite a late-breaking scandal that she might have had an affair with a blogger (of all people!), Sarah-Palin endorsed Tea Party candidate State Rep. Nikki Haley has placed far ahead of the field in South Carolina’s gubernatorial primary. She just narrowly missed the 50% threshold that would have allowed her to avoid a run-off election. Now she’s going to run against second-place finisher and U.S. Congressman Gresham Barrett. That is, if Barrett wants a run-off. He would almost certainly lose, as Haley already got 49% of the vote in the first round, Barrett languished at 22%.

The losers of the GOP primary are Attorney General Steve McMaster and controversial Lt. Gov. André Bauer, who finished in the teens.

The Democratic primary was won by State Sen. Vincent Sheheen in an unexpectedly large landslide against Superintendent of Public Education, Jim Rex, and fellow State Sen. Robert Ford. Sheheen will go into the General Election with a double-digit deficit against either Haley or Barrett.

SD-AL

In the GOP fight for the right to take on Democratic U.S. at-Large Representative Stephanie Herseth-Sandlin, tea party candidate Kristi Noem narrowly beat the establishment favorite, Secretary of State Chris Nelson. According to a pre-primary Rasmussen Reports poll, the Blue Dog Herseth-Sandlin leads Noem narrowly, and the same would have been true if Secretary Nelson had been selected to be her opponent.

SD-Gov

In the Republican primary for the gubernatorial race, the popular Lt.Gov Dennis Daugaard, second-in-line to the just as popular, but term-limited Gov. Mike Rounds, handily beat State Senate Majority Leader Dave Knudson. He’ll be a heavy favorite in the General Election against Democratic State Senate Minority Leader Scott Heidepriem, while the battle of the two Senate Caucus leaders would have been a formidable and close contest, according to Rasmussen Reports.

VA-05

The incumbent Congressman in this Republican-leaning district is unabashedly progressive Rep. Tom Periello. No big surprise that the national GOP has this race listed as one of its prime targets, and establishment favorite, relatively moderate State Sen. Robert Hurt won 48% of the vote to 30% for the Tea Party-backed candidate, Jim McKelvey. Several prominent Conservatives have threatened before the primary to undertake or at least back an independent Tea Party candidacy, should Hurt win. A February PPP poll found that in such a scenario, Periello would be at 44% support, with Hurt at 27% and a Generic Tea Party candidate at 19%.

This is probably a seat where a Conservative candidate such as former U.S. Rep. Virgil Goode, who had been ousted by Periello in 2008, would have been better in such a district in an election cycle that looks good for Republican prospects so far- at least much better than 2008.

UK Forecasting Round-up

2010-05-07T12:31:00.004-04:00

Pollster Accuracy:

Winner: ICM

Forecaster Accuracy (Seats):

Winner: PoliticsHome

Forecaster Accuracy (Vote):

Winner: Electoral Calculus and ForecastUK

Coalition Combinatorics

2010-05-07T05:01:00.008-04:00

If the forecasts on The Sun hold up, then the seat distribution in the British House of Commons should be as follows:

Conservatives: 307

Labour: 258

Liberal Democrats: 57

Scottish National Party: 6

The Greens: 1

Right-leaning Northern Irish Parties : 8

Left-Leaning Northern Irish Parties: 4

Sinn Fein: 5

Plaid Cymru: 3

Normally, it takes 326 seats out of 650 in order to obtain a majority. However, to quote Jim Miller:

"the Speaker ordinarily does not vote and five members of the Sinn Fein party are not allowed to vote because they refuse to take a loyalty oath".

The Speaker is currently a Conservative, and Sinn Fein has five votes, so 323 votes are required for a majority.

There still is hope of keeping the Tories out of power. A Labour+LibDem+Green+SNP+Left Irish+Plaid Cymru coalition would have 329 seats, six more then needed for a majority. But the British seem to have an aversion to large coalitions and there is a very small margin of error in the remaining undeclared seats for this to be a possibility.

The plausible Conservative-Unionist coalition nets 315 votes, which is 8 seats short of a majority.

A lot of talking heads are calling the potential left-wing coalition unstable. But in such a coalition, only the Liberal Democrats would have much bargaining power. The other four parties in the coalition do not have enough seats to unilaterally bring down the government by defecting in a no-confidence vote. But in a Conservative-led Coalition or Minority Government, the Liberal Democrats would have the exact same power.

So if the Liberal Democrats want it, there exists a viable left-of-center coalition. It remains to be seen if that's what they want.

Update 7:49 Moscow Time: The results from every district are now in, so the numbers on this page are final.

Forecasting and Multi-Party Dynamics

2010-05-07T04:12:00.003-04:00

First, it seems like every major forecaster was fairly off last night in Britain, only because it seems that a huge proportion of LibDem voters switched to Labor at the last second.

Determining bragging rights will have to wait until the full results come in, but it seems that both FiveThirtyEight and PoliticsHome were both in the right ballpark when it came to estimating both Conservative Vote-Share and Seats.

It seems that Forecasters were quite good at tracking movement between the left-of-center coalition (Labour and the Liberal Democrats), but were much worse at determining the much more unstable inter-coalition dynamics.

This seems broadly similar to my experience modeling the Israeli elections last year, when the seats and vote share of the left-wing coalition (Labor+Kadima+Meretz+the Arab Parties) was easily predictable, while the actual breakdown within the coalition was not, with Kadima getting more seats then anyone had predicted. (This was probably due to tactical voting, because there was a hope that if Kadima was the largest party, they would be able to form a government)

Mathematically, this seems to be because movement between broad coalitions can be modeled pretty well as a simple random walk, while the structure of intra-coalition movement is much more intricate and unstable, subject to complexities like personality and tactical voting that are difficult to account for. This has been a surprisingly lightly-researched subject, and it's something that should be looked at moving forward.

From an actual policy perspective though, this might be irrelevant. The relevant metric for which party would preserve power was the number of seats that Conservatives would capture, and both the pollsters and the aggregation seemed to get this broadly right.

UK Round-Up

2010-05-05T11:07:00.005-04:00

Modeling the UK election can be fundamentally divided into two problems: 1) Estimating public opinion from polling, and 2) Determining how public opinion translates into seats in the House of Commons.

Poll Aggregation:

Charles Berry, Electoral Calculus, and most news organisations use a naive "average all the polls" approach.

PoliticsHome and the LSE's Hix-Vivyan model both use a sophisticated recursive Bayesian filter, inspired by Simon Jackman and similar to the model we used with success in 2008. It's primary advantage is that it accurately estimates and accounts for House Effects, pollster introduced error, and sampling error. Under relatively realistic assumptions, such models can be proven to asymptoticly produce the most accurate estimates possible given the data.

FiveThirtyEight averages the two approaches. [Update: 5/6/2010, in their latest forecast, FiveThirtyEight has joined the Naive Poll Average camp]

Translating votes into seats

Suppose that the Labor party drops by 10% nationally in the polls relative to their performance in the last election. How does that translate into district support?

The natural first approach is to subtract 10% from the 2005 election results in each district. But there are some districts where Labour got less then 10% of the vote. In these districts, our estimates of Labour vote share would be negative! This is called the Unified National Swing (UNS) model.

If Labor's vote share in the polls is 26%, and their share in the 2005 elections was 36%, then this implies that there are 27% less labor votes then there were before. So another approach would be to look at everything proportionally and redistribute 27% of the labor voters to other parties somehow. This ensures that there will never be any negative votes. This is called a proportional swing model.

From FiveThirtyEight, an example of a proportional swing model.

On the other hand, suppose we have two districts: One district full of die-hard Labour supports and the other with "swing" voters who happened to vote Labour in the last election. If a scandal causes a drop in the polls for Labour, the swing voters in the second district are far more likely to change parties then the die hard supporters from the first, while the proportional swing model predicts the drop will be equal. Worse, the transition matrix that determines how to distribute voters from one party to another is fundamentally impossible to estimate from polling data, adding an element of arbitrariness to forecasting.

Theoretically, both of the models have problems. Which is right empirically?

Showing little variation in swing across regions in the 2005 election, consistent with uniform swing

Election results from 1997, showing results more consistent with proportional swing.

Neither of them! Unfortunately, which model is more accurate has changed from cycle to cycle, as the graphs above show. FiveThirtyEight and PoliticsHome are the most prominent respective users of Proportional and Uniform Swing. They've had an interesting debate on the relative merits of their models that can be read in order at here, here, here, and here.

" In all the following charts, the red curve models Labour, blue models Conservative and yellow models the Lib Dems. The horizontal axis is the national vote share for that party, and the vertical axis is the constituency vote share."

A third approach has been proposed by Charles Barry. Because Uniform National Swing and Proportional Swing both work sometimes, why not fit a more general curve that has both as a special case? It's a really interesting insight, but unfortunately, the details of the model with regards to parameter estimation and theory require a bit more work. Despite this, it will be interesting to see how accurate this model will be. Moving forward, this seems to be the preferred approach in future cycles.

So What Will Happen!?

Not a clue! Most models which use Uniform Swing show that Conservatives will fall far short of a majority. Nate's model and Charles' model show more downside potential for Labour. Meanwhile, there are significant house effects in polling. If some of the more Conservative-Affiliated pollster's are right, then the conservatives might be able to squeak a majority.

My personal suspicion is that tactical voting is going to play a bigger role then is commonly assumed, and that this will likely benefit Labour.

But that's only a hunch, so I'll end with some graphs from the Hix-Vivyan model.

The estimated marginal probability distributions of the number of seats that each party will get.

Smoothed vote over time

Update: 5/6/2010: Charles Barry has put up a histogram on his site:

Also, check out Jeewa's write up of actual forecasts at Random Variable

Moscow

2010-03-29T09:31:00.006-04:00

Some of our readers may know that I've been living in Moscow for the last year studying abroad. In light of the Metro Bombings today, I want to let everyone know that I'm alright. From what I can tell, it seems that none of my friends or professors have been effected either. I have a quick informal write-up at DailyKos.

My prayers go out for the injured, sick, and their families.

Edit: My roommate, Parker Glynn-Adey, has a write up at his Moscow blog. To quote:

It seems natural and inevitable that these things would happen in Moscow. The city is not shaken to its core, there is no talk of retaliation. The only side effects are an increased police presence around metros and an eerie quietness in the subway. No one was selling trinkets to captive audiences aboard the metro today, no one was shouting into a cell phone, they were just going where they needed to go. Outside the my station the guards who usually drink tea and smoke were holding guns and batons while they drank tea and smoked. They didn’t seem too concerned.

Distributions of Ideology by Industry

2010-03-18T17:12:00.005-04:00

From Ideological Cartography,

Estimated distributions of ideology on a left-right scale of political contributers within individual industries.

The author takes advantage of the requirement that donors who contribute more then $250 to political campaigns state their occupation. He then proceeds to estimate the ideology of individual contributors by looking at the candidates that they contribute to. Somebody who donates to liberal candidates is probably more liberal then someone who donates to republicans.

This is a sound idea, but how can we determine the ideology of candidates? For the candidate already has held office, then the candidate's voting record can be scored via DW-NOMINATE. But what if the person has never served in legislature?

The site's way out of this, is to estimate the ideology of politicians by considering the ideology of their donors. But the ideology of donors is determined by the ideology of the politicians they contribute to! This logical problem is fixed via iterative magic.

This also allows us to estimate other interesting things, like changes in ideology of candidates over time:

Ideological Progression of Candidates during the 2008 Presidential Election: The presidential candidate trends are estimated in a two-step process. This first step recovered ideological estimates from an IMWA scaling that included all contributions from the 3125 PACs and 131,000 individuals that gave to two or more federal campaigns during the 2007-2008 Election Cycle. Holding the contributors estimates static, the presidential candidate trends are then smoothed over time using locally-weighted regression (LOESS) with a span of .3.

This method isn't perfect, but it's a good start! This line of research looks promising!

Rasmussen Polling Irregularities - A first post

2010-02-24T08:20:00.011-05:00

***Cross-posted at DailyKos***

Rasmussen polls are consistently to the right of other polls, and this is often explained in terms of legitimate differences in methodological minutiae.

Deviation from average of Obama Disapproval from Pollster

But there seems to be evidence that Rasmussen's house effect is much larger when Republicans are behind, and that it appears and disappears quickly at different points in the election cycle.

In order to estimate "instantaneous" house effect, we looked at all the Rasmussen 2008 polls(Senate, House, Governor, etc), and compared them with the filtered best estimate for public opinion on that day using pollsters excluding Rasmussen.

I'll hold off commentary for now, because these findings are preliminary, other then to say that House Effects seem to be more subtle then previously assumed, and that it warrants more study.

Note: All graphs are in terms of Republican Two-Way vote, defined as RepublicanVote/(Democratic Vote+RepublicanVote), as is standard in political science. "Margins", or RepublicanVote-DemocraticVote, are about twice TwoWay.

Estimated Rasmussen House-Effect vs "Closeness of race", defined as Abs(50-TwoPartyvote), estimated from all available pollsters excluding Rasmussen. This tests the idea that Rasmussen polls are reliable for close races, but that there is more ambiguity in non-competitive races. There seems to be a trend here, but the absolute value masks something...

Estimated Rasmussen House-Effect vs Estimated Republican vote from all non-Rasmussen Pollsters. The graph seems to show that Rasmussen House effects are much larger in races where democrats show large leads, making democratic-leading races seem closer then they really are. The effect is not symmetrical.

Estimated House Effect vs Days before the Election.

P-Value for "Rasmussen House Effect is greater than 0" over time. The graph seems to show Rasmussen polls have a Statistically significant Pro-Republican house-effect that appears during primary season in the beginning of the year, disappears during the summer, and then very rapidly appears right before the Republican National Convention

The findings are preliminary, my next post will expand the dataset to other elections and run the same procedure on Dailykos/R2K and SUSA polls as a control.

In the meantime, I'm looking for explanations or mechanisms that could explain the patterns above, because I don't see how they can be explained in terms of different voter screens...

Update 3/2 : Some people have noted that the three leftmost points on the second graph obscure the general effect. So I removed the "outliers" and reran the local regression. Here are some graphs:

Estimated Rasmussen House-Effect vs Estimated Republican vote from all non-Rasmussen Pollsters with outliers removed

P-value for "House Effect is significantly different from zero" for each point on the previous graph.

An Open-Source Pollster?

2010-01-31T04:38:00.002-05:00

A very interesting proposal:

...This instantly gave me the thought that the DailyKos community could be a... pollster. Yes, I know. DailyKos.com commissions polls with Research2000, which is great and all, but it isn't as good as I'd like it to be, for a couple of reasons:
...
We could be the most open, research-friendly pollster ever. We could release ~~anonymized~~ raw data to scientific researchers and Kossacks who want to play around with it. We could look at geeky polling things-- like, try to poll the whole population of a precinct and look at differences between our results and real (Census/Election) data. We could effectively make polling a lot better by giving statisticians and econometric students data to play with and do research on

Basically, the thesis is that the extremely low prices of new IVR(Interactive Voice Responce) polling firms like Precision Polling provide an opertunity for a community-run polling firm that could crowd-source statistical work in a transparent manner.

There are a ton of very knowlegable people in Statistics who have a lot of free time and a passion for politics, so I think the idea has a lot of potential. I really hope the idea gets off the ground.

A quick trick for approximating Median income

2010-01-16T13:33:00.005-05:00

Matthew Yglesias notes

— The internet should make it easier to do an international comparison of median income. I can’t find one.

There really is a paucity of data on median income for countries other then the US.

Luckily, it's pretty easy to find Gini Coefficients for a number of countries, as well as GDP per Capita(which roughly approximates average income). Fisk distributions* are commonly used by economists to model income inequality and have been shown to do a satisfactory job. Assuming that income is distributed under a fisk distribution, then the mean and gini coefficient provide enough data to estimate the median.

Examples of the Fisk Distribution for different parameters[Wikipedia]

Conveniently, this method also allows us to estimate 30th percentiles, shares of income going to the top 10%, as well as a bunch of other measures of inequality.

Testing this method against the countries where median and average income data was available, it's generally accurate to around 5%. This isn't perfect, but it's useful for purposes of international comparison.

For example, very poor countries are often the ones with the highest income inequality, and so looking at mean income creates an unintuitively large bias. Take a look at Afghanistan:

Estimated income distribution of Afghanistan

It also serves to illustrate the point that due to America's high income inequality, median incomes are not as high as one would expect from our very GDP per capita. Compare America's estimated income distribution with that of The Netherlands:

Estimated income distribution of The Netherlands and the US. Rescaled for visibility.

By Median, the US is 6th, just behind The Netherlands. For full rankings, I've made a spreadsheet available that shows the mean, median, and mode for 157 countries, as well as an applet that shows calculates arbitrary income percentiles and distributions. It can be downloaded here.

Screenshot from spreadsheet applet available for download.

*There are empirical reasons to believe the Singh-Maddala distribution would do a slightly better job, but the math involved in estimation becomes a lot trickier. The larger problem is the discrepancy between GDP per capita and mean income. Does anyone have a good source for average income or consumption? It's a component of GDP(Consumption+Investment+Exports-Imports), so the data has to be out there somewhere...

Update: Thanks to Regav for providing a link to OECD data for income distributions. This closes the discrepancy between GDP per capita and income that I was worried about, and has made the spreadsheet a lot more useful. Graphs and numbers updated. A spreadsheet showing median income for the OECD block of developed countries can be found here.

Gerrymandering, Compactness, and Tobler's law

2009-11-15T08:37:00.004-05:00

Via Professor Andrew Gelman, a fascinating paper on Legislature Districting (Not a phrase often said...) To quote the abstract:

When one of the major parties in the United States wins a substantially larger share of the seats than its vote share would seem to warrant, the conventional explanation lies in manipulation of maps by the party that controls the redistricting process. Yet this paper uses a unique data set from Florida to demonstrate a common mechanism through which substantial partisan bias can emerge purely from residential patterns. When partisan preferences are spatially dependent and partisanship is highly correlated with population density, any districting scheme that generates relatively compact, contiguous districts will tend to produce bias against the urban party. In order to demonstrate this empirically, we apply automated districting algorithms driven solely by compactness and contiguity parameters, building winner-take-all districts out of the precinct-level results of the tied Florida presidential election of 2000. The simulation results demonstrate that with 50 percent of the votes statewide, the Republicans can expect to win around 59 percent of the seats without any "intentional" gerrymandering. This is because urban districts tend to be homogeneous and Democratic while suburban and rural districts tend to be moderately Republican. Thus in Florida and other states where Democrats are highly concentrated in cities, the seemingly apolitical practice of requiring compact, contiguous districts will produce systematic pro-Republican electoral bias.

The fear with district design is that the designers could create twisted unnatural districts that ensures a party can win a majority of districts, even if they receive less votes then the opposition.

U.S. congressional districts covering Travis County, Texas (outlined in red) in 2002, left, and 2004, right. In 2003, the majority of Republicans in the Texas legislature redistricted the state, diluting the voting power of the heavily Democratic county by parceling its residents out to more Republican districts.[Wikipedia]

A standard proposed solution to this problem is to take the subjectivity out of drawing districts by requiring districts to be compact(Round). But the authors find that this seemingly fair solution creates a large anti-urban and correspondingly pro-Republican bias unless there are either an absurdly high or absurdly low number of districts.

Republican share of seats after algorithmically creating n compact districts.

The crux of their result lies in a vote-theoretic version of Tobler's law
, that the probability of two voters voting in tandem is roughly inversely proportional to their distance from each other. The paper find empirical evidence of the law in the five elections they looked at.

This is more than just an urban-rural divide because of it's voter-invariance. As can be seen from the map below, there are rural blue counties, and they tend to cluster near other rural blue counties.

Assortative mixing and conformity are the main causes of a Tobler-like relationship that comes to mind. That is to say, either Democrats tend to move away from areas with large numbers of Republicans, or people who live near lots of Democrats tend to conform to local consensus, or a combination of both. Of course, more research is necessary to say which.

But to return to districting, the authors seem to show that a non-trivial consequence of Tobler's law is that placing a compactness restraint on districts produces biased and unrepresentative legislatures.

Then how do we ensure fair and representative districting? Gelman suggests multimember districts as a way around the problem, an idea that Matthew Yglesias has mentioned before. But why not a restraint of the form "If voters from the last election voted under the the proposed districting, the proportion of Democratic seats to Republican seats must be as close as possible to the proportion of Democratic voters to Republican voters"? Of course using Proportional Representation makes this whole issue irrelevant...

Update: Based on questions, I think a summary would be useful:

f we get rid of "funny-looking" districts and make all districts roughly round with equal population, there will be a strong Republican bias. That is to say, some form of gerrymandering(in the form of funny looking districts) is necessary in order to have a representative legislature.

Update 2: In the comments, I wrote the following theoretical explanation for the relationship between Tobler's law and skewed legislatures:

Here's another way to look at it. Suppose that Tobler's law holds (That your probability of voting for the same guy as someone else is inversely proportional with distance) and that your districts have to be roughly circular with equal population(say, 100,000).

Let's consider a district centred in an urban district. Since population density is so high, the circle is going to be pretty small if it's going to have 100,000 people. Because the area is so small, Tobler's law predicts that the variation inside your district is going to be pretty small.

Meanwhile, a suburban district is going to be pretty big, so Tobler's law predicts higher variation in votes then in the smaller district.

The end result is that your urban districts will be really lop-sided, so that the urban party will "waste" most of their votes, leaving the rural party with more seats

The main way around it, is that we have to make our districts funny looking and twisted...which is roughly the status quo, actually. Multimember districts or PR would do it too...

h/t Visualisation of the House Health-Care Vote

2009-11-08T08:46:00.002-05:00

Professor Simon Jackman has a great visualisation of the Health Care vote over here. His post isn't very long, so I'll post the whole thing:

Here is a quick look at how Democrats split on the House vote on the Affordable Health Care for America Act, as a (logistic) function of Obama vote in their district.

Davis (AL-7) and Kucinch (OH-10) are the big “errors” among the “Noe” votes; Kucinch had been telegraphing his opposition to a too meek reform bill for some time. Davis is the same boat (“is this the best we can do?“).

Marion Berry (AR-1) is the biggest “error” among the “Aye” votes; he voted yes while representing an Arkansas district where McCain got 59% of the vote and Obama just 38% (but, perhaps reflecting much about that part of Arkansas, he was unopposed in the 2008 Congressional elections) and he seems to have long history of being in the forefront of Democratic reform efforts on health care.

To add another bit of information, the only Republican crossover was Joseph Cao of New Orleans, whose seat will probably belong to a democrat soon...

Preliminary Judgement

2009-11-04T07:34:00.009-05:00

While it's very important to wait for certified returns, it's now possible to get an idea of how our forecasts performed yesterday.

I'll write out more commentary and thoughts later, but I've got some errands to run and just wanted to get the raw data out there.

First, a summary:

We did better, but the difference between Pollster's performance and ours isn't large enough to be conclusive with so few races.

FiveThirtyEight didn't post margin estimates this year, only putting up probabilities derived from what could be described as a decision rule procedure. We didn't forecast the same races, but looking at the ones we both covered and comparing likelihood:

With only 5 4 races, this doesn't convey much information either. Still, I reserve bragging rights...

More soon!

Final 2009 Election Forecast

2009-11-03T14:04:00.009-05:00

****Cross-posted at DailyKos***

For those unable to wait a couple hours, our final pre-race forecasts are below.

[7:02 pm EST] Also, fellow Stochastic Democracy Contributor Rasmus is running a live-blog over at DailyKos covering both the big-ticket races and some important local races. It's already 3am in this time zone, but I'll try to post a little over there too as late as I can.

Summary:

New Jersey Governor race:
Corzine (D) 's estimated two-way vote: 49.62% (+/- 1.53)
Corzine (D) 's estimated three-way vote: 44.55% (+/- 2.54)
Corzine (D) 's estimated probability of victory: 33%
Most Likely Outcome : Chris Christie (R) victory (67% chance)

Virginia Governor race:
Deeds (D) estimated two-way vote: 42.69% (+/- 2.89)
Deeds (D) estimated probability of victory: Rounding error...
Most Likely Outcome: Bob McDonnell (R) victory

New York City Mayor Race:
Bloomberg (I) 's estimated two-way vote: 58.1% (+/- 1.47)
Bloomberg (I) 's estimated probability of victory: >99%
Most Likely Outcome: Micheal Bloomberg (I) victory

Maine Gay Marriage Proposition:

Prop 1's estimated support: 52.36% (+/- 4.67)
Prop 1's estimated probability of passage: 69.6%
Most Likely Outcome: Passage of Prop 1, leading to the repeal of Gay Marriage in Maine. (69.6% chance)

Special Election in House District NY-23:
Owens (D) ' estimated two-way vote: 46.75% (+/- 5.62)
Owens (D) ' estimated probability of victory: 15%
Most Likely Outcome : Doug Hoffman (C) victory (85% chance)
*Disclaimer: On the basis of a single Sienna Poll with 18% of respondents undecided.

Quick Definitions

Two-Way Vote - The two-way vote strips out third party support and only looks at how the leading candidate does compared to his strongest competitor. It has the useful property that a winning candidate will always win more then 50% of the two-way vote. Formula: 100*Candidate_1/(Candidate_1+Candidate_2)

Three-Way Vote - Like Two-way vote, but for three candidates. Formula: 100*Candidate_1/(Candidate_1+Candidate_2+Candidate_3)

New Jersey

Smoothed Corzine two-way vote and 95% confidence intervals. Smoothing done via Bayesian filtering

Democratic Candidate Jon Corzine is forecasted to win 49.62% (+/- 1.53) of the two-way vote and 44.55% (+/-2.54) of the three-way vote. This leaves him with a 33% chance of winning the election, making republican Chris Christie a 3-1 favourite. Christie and Dagget are expected to receive 45.18% (+/-4) and 8.58% (+/- 6.6) of the vote respectively.*

Note how much smaller the margin of error is for Corzine's share of the vote in comparison to Christie and Dagget. This is because most of the variation in the race has been voters jumping between Christie and Dagget, while Corzine voters have mainly kept put.**

This is a close race and there have been a lot of polls showing both Corzine and Christie ahead. But since yesterday, PPP and Quinnepac released pro-Christie polls with large sample sizes that pushed our estimate against Corzine. Keep in mind, these two agencies believe that conservative turnout today will be higher then other pollsters assume. If the other pollster's assumptions were right then Corzine will win, if PPP is right, then there is a much larger enthusiasm gap between Democrats and Republicans then is generally believed.

*Haven't had time to update 3-day estimates from yesterday, so they do not reflect today's polls

*Statistics Trivia: Why do the candidate's vote shares add up to 100? Because when jointly estimating 3 correlated random variables [u, v, w], the vector [Most likely value of u, most likely value of v, most likely value of w] need not be equal to [Most likely combination of u, v, and w]. Joint estimation of 3 or more random variables a little non-intuitive, this makes talking about multi-party races more complicated.

Virginia

Smoothed Deeds two-way vote and 95% confidence intervals. Smoothing done via Bayesian filtering

Democratic Gubernatorial Candidate Creigh Deeds is set to be defeated by Republican Bob McDonnell with 43.23% of the two-way vote.

New York

Smoothed Thompson two-way vote and 95% confidence intervals. Smoothing done via Bayesian filtering

Mayor Bloomberg will annihilate his Democratic challenger Bill Thompson with an estimated 58.1% (+/-1.47) of the two-way vote.

Maine

Smoothed vote and 95% confidence intervals. Smoothing done via Bayesian filtering

Proposition 1, to repeal the State Legislature's legalization of Gay Marriage, is set to pass with 52.36% (+/-4.67) of the vote. However, this outcome is not a sure thing. We estimate that the measure has approximately a 30.45% chance of failing.

This is a strong reversal from our last update, when the amendment had a 70% chance of failing. What happened?

PPP released a poll showing the Yes side 4 points ahead with a large sample size. In this particular case, the Maine election has been very lightly polled, so a single outlier poll with a large sample size like PPP's can push estimates a lot. This is especially true because the few other polls in this race have tiny sample sizes.

Like in New Jersey, PPP shows large Republican leads on the basis of high conservative turnout on one end, while nearly every other pollster forecasts relatively comfortable Democratic victories. It isn't obvious which side is right.

****Cross-posted at DailyKos***

Election 2009!

2009-11-01T22:20:00.005-05:00

Summary:

New Jersey Governor race:
Corzine (D) 's estimated two-way vote: 49.93% (+/- 1.78)
Corzine (D) 's estimated three-way vote: 44.55% (+/- 2.36)
Corzine (D) 's estimated probability of victory: 47.12

Virginia Governor race:
Deeds (D) estimated two-way vote: 43.23% (+/- 3.66)
Deeds (D) estimated probability of victory: Rounding error...

New York City Mayor Race:
Bloomberg (I) 's estimated two-way vote: 58.1% (+/- 1.47)
Bloomberg (I) 's estimated probability of victory: >99%

Maine Gay Marriage Proposition:
Prop 1's estimated support: 51.5% (+/- 3.87)
Prop 1's estimated probability of passage: 22.4%

Special Election in House District NY-23:
Not a clue...

Quick Definitions

Two-Way Vote - The two-way vote strips out third party support and only looks at how the leading candidate does compared to his strongest competitor. It has the useful property that a winning candidate will always win more then 50% of the two-way vote. Formula: 100*Candidate_1/(Candidate_1+Candidate_2)

Three-Way Vote - Like Two-way vote, but for three candidates. Formula: 100*Candidate_1/(Candidate_1+Candidate_2+Candidate_3)

New Jersey

Smoothed Corzine two-way vote and 95% confidence intervals. Smoothing done via Bayesian filtering

Smoothed Corzine three-way vote and 95% confidence intervals.

Democratic Candidate Jon Corzine is forecasted to win 49.93% (+/- 1.78) of the two-way vote and 44.55% (+/-2.36) of the three-way vote. This leaves him with a 47.1% chance of winning the election, making republican Chris Christie a (very) slight favourite.

Virginia

Smoothed Deeds two-way vote and 95% confidence intervals. Smoothing done via Bayesian filtering

Democratic Gubernatorial Candidate Creigh Deeds is set to be defeated by Republican Bob McDonnell with 43.23% of the two-way vote.

New York

Smoothed Thompson two-way vote and 95% confidence intervals. Smoothing done via Bayesian filtering

Mayor Bloomberg will annihilate his Democratic challenger Bill Thompson with an estimated 58.1% (+/-1.47) of the two-way vote.

Maine

Smoothed vote and 95% confidence intervals. Smoothing done via Bayesian filtering

Proposition 1, to repeal the State Legislature's legalization of Gay Marriage, is set to fail with 48.5% (+/-3.87) of the vote. However, this outcome is not a sure thing. We estimate that the measure has approximately a 22.4% chance of passing.

--
Expect an update tomorrow to incorporate last-minute polling.

And, as a quick note, an announcement to explain the paucity of recent posts: I've been in Russia! Specifically the Math in Moscow program. Oddly enough, people here don't find election forecasting to be terribly impressive...

More than 100% turnout?

2009-06-21T13:46:00.003-04:00

This isn't the most sophisticated way of looking at things, but I've been matching up census data with city level voting data.

The census gives a year-by-year age breakdown, so I've conservatively estimate turnout by counting the entire population over the age of 15(The voting age is 18, and the census is 3 years old), and assuming no mortality.

So far, I've only looked at the 10 subdivisions in the Province Yazd, and three provinces stand out with suspiciously high turnout. Saduq(صدوق ), Mehriz(مهريز) , and Taft(تفت) , with 97%, 104%, and 130% turnout respectively. Excluding those provinces, the average turnout was 77%.

Perhaps there is an innocent explanation (Immigration or Migratory workers?), but it warrents explanation. I'll expand the analysis to other provinces and see if the result holds.

Update: See Nate Silver

Iranian Statistics Round-up - Updated 11:35AM June 21st

2009-06-19T13:05:00.018-04:00

In the last couple of days, a dizzying number of different statistical arguments have been put forward regarding the legitimacy of the 2009 Iranian election.

In order to keep track of them all, I've made a summary of all the major quantitative arguments I've heard so far and my thoughts on them. Let me know by email or in the comments if I have missed any.

1) Professor Mebane's work

i) Second digit Benford Law test:

Previous research by the professor has found that testing the frequency of the second digit of vote returns is a more reliable measure of electoral fraud then the more traditional first digit test, which can often produce false positives.

He did not find any statistically significant discrepancies using a second digit Benford law test on city-level election returns.

However, his Second-Digit Benford Analysis was developed for precinct level data which has not been available. At such a high level of aggregation, sheer scale can overwhelm foul play. Because of this, conformity to Benford's law is not suggestive of authenticity in this case.

Update[4:35 pm]: Professor Mebane has obtained ballot-level data and indicated that serious discrepencies were found with Karroubi and Rezaee returns, will update when more information is available

Update[9:23 pm]: A report incorporating Ballot-Box returns can be seen here. To quote:

The initially released polling station data show evidence of signicant distortions
in the vote counts especially for Karroubi and Rezaei. No signicant distortions are apparentfor Mousavi's vote counts. There is very marginal evidence of distortions in Ahmadinejad's vote counts. A key to interpreting these results is understanding why the vote counts for Karroubi and Rezaei are typically so small.

Is it (a) inherently low levels of support, (b) voters strategically abandoning the candidates, or (c) fraudulent counts? If there is goodreason to believe either (a) or (b), then (c) is less likely.

ii) Election models:

Professor Mebane found that when conditioning 2009 data on first and second round 2005 data using simple models, he found behavior that violated "natural election processes". He came to the conclusion that it constituted "moderately strong evidence for fraud".

I don't feel qualified to comment on his assertion, but he is considered an expert in this field, and so I attach weight to the fact that he is convinced. Still, I would like to see his methods applied to other elections as a "control"

2) The work of Dr.Roukema

Update 4:50PM 5/21 :Updated to reflect comments from the author

His arguments are the following in order of "strength":

i)That reformist candidate Karroubi's vote returns have a large number of excess 7's assuming Benford's law or empircal variations on Benford's law

ii) That Ahmedinejad had an excess number of 2's and a deficit of 1's, according to Benford's law but not according to some Benford variants

iii) That none of the candidates have log-normal distributed returns except for Mousavi.

To me, i is interesting and merits further study, though even the author wouldn't characterize it as conclusive.

This is particularly interesting because the largest discrepancy between Iranian opinion polling and the actual results was the massive collapse in support of Karroubi and Rezaee. Every opinion poll showed their combined support in the double digits, when they ended up obtaining less than 3% of the vote between them.

The major criticism of ii is that there has been research showing that candidate vote returns often do not always conform to Benford's law.

Histogram showing the distribution of vote totals in voting areas is approximately log-normal, pulled from original paper.

As for iii: Statistically, I don't see why total returns on a district level being log-normally distributed would imply that every candidate would have log-normally distributed votes. On the contrary, Log-Normal distributions are not stable under addition, and so at least one of the candidate's returns would need to deviate from a log-normal distribution in order to maintain the observed total vote's log-normality. (At least if you ignore the fact that different candidate vote returns are not independent).

Nate Silver and Professor Andrew Gelman's commentary on all three points are invaluable. See here, here, and here.

Update 11:35 AM June 21st :

Dr.Roukema has updated his paper, which can be seen here. He finds that evidence that irregularities might have been concentrated in high population areas.

"Among the six biggest cities, why should those three cities that voted more for Ahmadinejad be exactly the same ones where there are 70xx votes for Karroubi?" - Dr.Roukema

To quote the abstract:

Three of the six most populous voting areas have
vote totals for K[Karroubi] that start with 7. All three of these have
greater proportions of votes for A[Ahmedinejad] than the other three voting
areas. Interpreting this as an overestimate of the true
vote assumed to be 50% to match other data, while retaining
constant total vote numbers and increasing votes for the
other three candidates in proportion to the average voting
percentages, would imply that the difference between A’s
and M’s[Mousavi] vote totals would drop by about one million votes.

3) Samuel Wang's look at Tehran Opinion polls

Professor Samuel Wang asserts that while the public opinion polls conducted for Iran at large were all over the place and of suspect quality (Polls ranged from Ahmedinejad +16 to Mousavi +32), polls for Tehran specifically might be more reliable.

He then looks at Tehran-specific polls and finds that the discrepancy between the Ahmedinejad's poll-forecasted winning-margin in Tehran and his official margin there was large enough to be statistically significant. He concludes "For now, my interpretation is that the official returns in Tehran are unbelievable."

Some thoughts:

i) I'm not sure what the polling area the polls referred to when they polled "Tehran", the translation wasn't clear. If they were polling "Tehran, Tehran", then the results would have been within the margin of error.

ii) Focusing on margins obscures the fact that in terms of actual candidate vote-share, the polls seem to have been massively off. Mousavi+Ahmedinejad was estimated to be around 60%, but ended up at 97%. This could have been due to an abnormal number of undecided voters or some other factor, but it should be explored.

4) Nate Silver's Analysis

Nate has had some interesting qualitative analysis and statistical commentary on other research, but his best piece so far has been this , where he shows that Ahmedinejad did not do very well in rural areas in the first round results of 2005, while doing much better in 2009. He posits that such a radical change in the rural-urban divide in so little time is suspicious.

Before: First Round Iranian Elections in 2005

After: First Round Iranian Elections in 2009

I have to check if this result holds on when I replace the Ahmedinejad variable with a "conservative" variable showing the combined support of the 4 conservative candidates in 2005. But so far, this has been one of the most convincing points in favor of fraud I've heard so far.

5) Miscellaneous criticisms

i) Mousavi lost his home region:

I don't find this suspicious. There were some polls conducted by Western Organizations that showed that Ahmedinejad had much higher support than Mousavi among Azeri's, Mousavi's ethnicity. This could be related to unconfirmed reports that Ahmedinejad was a popular administrator in Azerbaijan for 8earlier in his career.

ii) Ahmedinejad won Tehran, which should have gone for Mousavi:

Some polls showed that Ahmedinejad winning Tehran by around the margins that he did. Not only that, but he was formerly elected Mayor of Tehran. I don't find this necessarily suspicious by itself either.

iii) Counting was done implausibly quickly:

Paper ballots can be counted very quickly. It doesn't take very long to call a 63% lead.

iv) There are numerical discrepancies in the voting data.

To summarize the arguments I've previously made here:

There are about 100,000 missing votes, because Valid votes+Invalid votes is 100,000 less than "Total votes".

Also, the percentage of spoiled ballots in a district is highly correlated with the district's reformist candidate vote share, while being negatively correlated with Ahmedinejad vote-share.

Percent of ballots declared invalid vs Candidate vote-share

One simple explanation, would be that new voters are more likely to make mistakes and produce spoiled ballots. But this would imply that the surge in turnout mainly went to reformist candidates. If this was the case, I don't see how Ahmedinejad could have won.

v) The idea that Ahmedinejad's share of the vote stayed too constant while results were being announced to be real. Shown via following popular graph:

This has been thoroughly debunked by multiple sources, see here and here.

Yet Another Iran Stat Analysis

2009-06-19T01:24:00.005-04:00

Professor Sam Wang of Princeton, of Princeton Election Consortium fame, just posted an analysis of the 2009 Iranian election here.

Basically, he asserts that while the public opinion polls conducted for Iran at large were all over the place and of suspect quality (Polls ranged from Ahmedinejad +16 to Mousavi +32), polls for Tehran specifically might be more reliable.

Looking at a six-poll aggregate (which includes polls from before the June 3rd presidential debate), the polling median is Mousavi +4 +/- 4 (Median +/- MAD-based SEM), while the 3-poll aggregate (only includes polls from after the June 3rd debate) has a polling median of Mousavi +4 +/- 11.

The announced official result was Ahmedinejad +12% (51.6% to 39.4%), a discrepancy of 16 points. This discrepancy does not reach statistical significance compared with the last 3 polls (p=0.07), but does compared with all 6 polls (p=0.003).

For now, my interpretation is that the official returns in Tehran are unbelievable.

Some points:

1) While the three poll-aggregate discrepancy from the result does not meet the standard threshold for statistical significance (p=0.05), it's sufficiently close so that I would count it as such anyway. The 0.05 statistical significance threshold was not given to us by God, see here.

2) There might be some confusion as to what "Tehran" is. Ahmedinejad supposedly won by 12 points in the "Oshtan" (Region) of Tehran. Yet official results show that Mousavi won Tehran (تهران ), Tehran with 52% of the vote and Mousavi + 8.66. This creates a discrepancy from the polling aggregate of only 4.66 points, a result fully within the margins of error of both the three and six poll aggregate.

It is possible that pollsters conducting estimates of "Tehran" were only polling people in "Tehran, Tehran", much like polls of New York City wouldn't include anyone from Albany in their sample. People with local expertise should speak up.

3) I can't find the Tehran poll internals in English, but based on a relatively incomprehensible translation courtesy of Google, it seems that 34.4% of respondents in a pre-election poll indicated that they would not vote for either Mousavi or Ahmedinejad.

According to the official results, only 3% of voters in the Tehran province voted for somebody other than Mousavi or Ahmedinejad.(1% for reformist candidate Karroubi and 2% for conservative Rezaee).

The 34.4% might include Undecided/Don't know/Refused (knowledge of Farsi would be useful), but even then, this is a suspicious discrepancy that warrants further study.

Breaking - "I think the results give moderately strong support for a diagnosis that the 2009 election was affected by significant fraud."

2009-06-18T07:25:00.003-04:00

Professor Mebane, whose analysis we have linked to earlier, has just updated his report.

More in depth analysis on it later, but after looking at first-round data from 2005, he finds evidence of significant discrepancies.

See it here.

Money Quote:

"The results give moderately strong support for a diagnosis that the 2009 election was affected by significant fraud."

New Benford Law Analysis [Update:4:01 pm 6/18]

2009-06-17T09:17:00.008-04:00

Hot off the arXiv, Dr.Roukema has published an analysis of the 2009 regional level data, finding evidence of serious anomolies.

Some highlights:

Benford's Law says that numbers starting with the digit 7 should occur about log10(1-1/7) = 5.7% of the time. So for 366 voting areas, the first digit should be 7 for about 21 voting areas, plus or minus (one standard deviation) roughly sqrt(21) = 4.6. But the actual table published by the MOI gives 41 vote counts for Karroubi starting with 7. This is about four standard deviations too high.

They also found deviations from Benford's law that they deem highly significant(p=0.0069) when analysing Karoubbi returns. Karoubbi is the canidate who recieved less than 1% of the vote after recieving 17% of the vote in 2005, an outcome that many commentators have deemed suspicious.

After examining Ahmedinejad's vote returns, they find that if the standard version of Benford's law applies, then a larger number of voting areas have vote numbers that start with the digit 2, and find fewer than expected digits that start with the digit 1. The authors speculate:

If we were to consider the alternative hypothesis that someone interfered with the data in order to increase A’s votes, replacement of 1’s by 2’s in a few dozen voting areas would be one method of achieving this without leading to numbers that are “obviously” artificial.

Lastly, to quote the abstract regarding this alternative hypothesis

A less significant anomaly suggested by Benford’s Law could be interpreted
as an overestimate of candidate A’s total vote count by
several million votes.

Update:

I'm not sure how much weight to actually put toward this analysis. It certainly is interesting and warrants further research, but it is not itself conclusive(As the author would likely agree).

See Andrew Gelman (here and here) and Nate Silver. (Gelman has some good notes elsewhere on his blog as well) And to quote an Professor Mebane when asked to comment.

In general my research has not focused on first digits, because in the precinct-level data I've emphasized first digits are often arbitrarily distributed (far from Benford) while second digits match the distribution. Here of course the data are much more highly aggregated than precincts. My discussion of the second digits has so far also turned up a few oddities (e.g., Table 4), but I don't know what to make of them. I have some ideas of further things to look at. Overall I don't know that I expected the first digits to satisfy Benford's Law, so I'm not surprised that they are slightly off. But I don't dismiss the finding, either.

Update: changed to reflect the input of the author, Graph courtesy of author.

Iran Elections - Final Update for now [Update: 6/16 1:31 PM]

2009-06-15T01:11:00.012-04:00

Professor Mebane of University of Michigan, someone with quite a bit of expertise in election diagnostics, has posted a brief report here based on the city-level election data discussed earlier.

To summarize, the data conforms to Benford's law, but the scale is so large that conformity doesn't prove much. There are a couple of odd findings, but, given the limited data available, it's hard to say anything.

To quote Professor Mebane from private correspondance,

At this point I have no confidence in the election results but also not much concrete evidence to demonstrate they are invalid. If anyone has polling station level data, I'd love to see them.

Update: Professor Mebane has updated his analysis to incorporate 2005 second round district-level data.

In 2005 some opposition politicians called for a boycott of the election. The surge in turnout in 2009 is widely interpreted as meaning that many who boycotted in 2005 decided to vote in 2009. Hence towns that have high ratios should have lower proportions of the vote for Ahmadinejad (the coecient should be negative).

He then tested this hypothisis using a overdispersed binomial model, finding that it worked well for most districts.

In particular, the following districts stood out:

Tabriz, Azerbaijan Sharghi

Terhan, Tehran

Shemiranat, Tehran

Karaj, Tehran

Chabahar, Sistan va Balauchistan

Khash, Sistan va Balauchistan

Zahedan, Sistan va Balauchistan

Saravan, Sistan va Balauchistan

Yazd, Yazd

Anyone with any local knowledge of these regions should leave a comment.

In general, combining the 2005 and 2009 data conveys the impression that a substantial core of the 2009 results reflected natural political processes. In 2009 Ahmadinejad tended to do best in towns where his support in 2005 was highest, and he tended to do worst in towns where turnout surged the most. These natural aspects of the election results stand in contrast to the unusual pattern in which all of the notable discrepancies between the support Ahmadinejad actually received and the support the model predicts are always negative. This pattern needs to be explained before one can have condence that natural election processes were not supplemented with articial manipulations

The updated report can be seen here, while associated data and R-code can be seen here.

Preliminary Iranian Data Analysis

2009-06-14T21:13:00.008-04:00

After looking a bit at the data from the last post:

1) The data seems to confrom to Benford's law, which suggests that if it was falsified, it might have been falsified in an "organic" way(For example, "every 3rd vote for the opponent goes to the incumbant).

With data:

Bin Frequency Freq Theoretical Freq

1 95 0.259 0.301
2 85 0.232 0.176
3 54 0.147 0.125
4 30 0.081 0.097
5 32 0.087 0.079
6 23 0.062 0.067
7 17 0.046 0.058
8 20 0.054 0.051
9 10 0.027 0.046

2) The data isn't entirely clean, the "total" number of votes is larger than the "valid" votes + the "invalid" votes. This leaves about 100,000 votes unaccounted for. This isn't extremely unusual, though on the high side.

3) The number of invalid votes seems high in certain districts. In تركمن in Golestan for example, about 10% of ballots were declared invalid(This was a place where Mousavi did very well).

Furthermore, the percentage of spoiled ballots seems to be higher in regions where the opposition did better.

Statistically, the correlations were statisticly significant (p<.001) for every canidate except Rezaee. This could just mean that Opposition voters are younger and more prone to make mistakes, or it could imply voter suppression and tally bias. But still, invalid votes only made up 1% of ballots, and so even if this was the result of foul play, it wouldn't be enough to change the results.

All and all, I've yet to find convincing statistical evidence for fraud in the data. But I'll keep looking and post findings here. If anyone has any ideas or observations, please let me know in the comments or email me at dms at the-beach.net.

Breaking Update: Iranian Data Found

2009-06-14T17:45:00.002-04:00

I've found a lot of Iranian regional data here, from a source here.

I've just started the stat analysis now, we'll see if anything turns up. (What better way to spend a vacation in Paris?). I invite others to do the same.

Iran Update

2009-06-14T15:12:00.005-04:00

It seems that Iran doesn't post election returns publicly as a policy. All of the figures that have been touted (Ahmadinejad ahead in Tehran! Mousavi loses in native Azeri areas!, etc), have been figures reported by the state owned media. I'll keep an eye out, but I suppose we shouldn't be suprised that Iran isn't particularly transparent.

Regarding the graph from yesterday thrown around as proof of fraud,

Found at TehranBureau.org

More evidence has come out showing that the graph's high R^2 is actually typical of all election returns.

Nate Silver has posted an analysis based on a similated election night from 2008 US data, similar to the analysis that HudsonBayMark did yesterday with data from Latin America, both showing high R^2 values for election returns and concluding that the Iranian graph was not usual.

From Nate Silver's FiveThirtyEight

Most convincingly, VoteForAmerica managed to have real time election return data from the 2008 election laying around.

The empirical evidence cannot be argued; I could go through and create similar graphs for every state and the linear relationship with a very high R^2 would hold. For the record, I did analyze data from California, Minnesota, Vermont and West Virginia to verify this result.

This shows conclusively that the R^2 critique pushed by the Tehran Bureau has no basis. Of course, this doesn't actually prove the election was clean(I personally suspect it was stolen). To quote VoteForAmerica:

The direct result of this research seems to support the idea that the election wasclean, but that in and of itself is peculiar; the election outcome was almost too consistent for an 85% turnout. Either the election was rigged very carefully, or the riggers got lucky; my money's on the former. If a conscious decision had been made to alter the result of this election, it would seem illogical to ignore statistics. The people of power in Iran definitely had the means to ensure that the election appeared clean from a statistical point of view. Going forward, I plainly expect other anomalies to appear, but I highly doubt the smoking gun will come in the form of mathematical/statistical analysis.

Electoral Fraud in Iran?

2009-06-13T16:13:00.004-04:00

**Crossposted at DailyKos**

There has been a lot of discussion about a graph showing partial vote returns that Andrew Sullivan posted on his blog.

Yes, this obviously was a "divine assessment". They didn't even attempt to disguise the fraud. Which, to me, tells me they panicked. This graph is a red flag to Iran and the world.

Is this so unusual? I thought that looking at raw vote totals might make things look artificially fishy.

After some searching, the original data source turned out to be from here. From there, a graph of vote-share can be created:

Considering that 20% of returns had already come in by the first data point, this doesn't seem unusual at all.

I don't have any raw data lying around, but HudsonValleyMark ran a "simulation" of vote returns from an election dataset, finding similar amounts of variation.

Real data would be preferable, and if anyone has any data of real-time election returns out there, please share them in the comments. (I actually have some at home, because I worked on real-time election projection for my blog. But, I'm in Europe at the moment, so that it isn't too useful...)

This doesn't rule out election fraud of course, it only suggests that other avenues should be explored. I'm trying to find regional vote data, to see if I could use something some more traditional data-verification tools (Benford's law, etc). My knowledge of Iranian data sources is limited, so any help in this direction would be greatly appreciated.

**Crossposted at DailyKos**

Head's Up - European Parlimentary Elections

2009-05-13T14:18:00.002-04:00

The European Parliamentary Elections are coming up, and this site does an excellent job at forecasting.

The site doesn't seem to make any rigorous attempt at forecasting "temporal error", that is, take into account how public opinion could change between now and the election, though the regression component of their model might implicitly do so. This is forgivable however, because any attempt at such would involve restrictive assumptions due to the large number of parties and states.

Instead, their model results and distributions can be roughly translated as "What will happen if the election happens tomorrow".

It'd be fascinating to see if the non-normal distributions observed are an artifact of the small sample-size (n=1000) for his Monte-Carlo simulation or something deeper. Hopefully the authors write about this in the future.

I've requested the data to see if the model I developed for the Israeli elections might be relevant here, but in the meantime, this is probably one of the best election-forecasting websites created so far.

Unintended Side-Effects from Cap'n'Trade?

2009-05-08T22:04:00.006-04:00

MasterResource looks at the effects on global temperature of the proposed Waxman Climate-Change Bill. (OECD90 refers to getting the OECD-30 developed countries to agree to return 1990 emission levels, with other classifications listed here)

As the graph shows, not much is accomplished if the US acts unilaterally.

What concerns me though, is that the effect might be worse, as US cuts in demand cause a large drop in international oil prices due to the high price elasticity of fossil fuels. Depending on the energy efficiency of non-participating countries, this may actually cause an increase in total emissions! Of course, it will spur research and development that would spill over to the rest of the world. How this would all balance out isn't immediately clear. Atleast for coal, which isn't as price sensitive to demand, I'd suspect the latter effect would outweigh the former.

But I'll look around the literature to see if I can model this more rigorously...

Anti-Mean Reversion

2009-04-04T17:27:00.003-04:00

Andrew Gelman at FiveThirtyEight writes

Beyond that, there's a systematic pattern that Obama did better than the polls in Deomocratic states and worse than the polls in Republican states. Does this represent a real pattern of voters--perhaps people reverting to their more predictable positions at the last minute, with Vermonters moving to the Democrats and residents of Wyoming going the other way? Or maybe it's an artifact of the poll aggregation, with the predictions being pushed too close to 50%, on average?

This was a point brought up by Sam Wang at the Princeton Election Consortum back in November.

But since it was November, he dealt with election-day precertified data. Since we have it now, it's better to work with the certified data from December. This distinction makes It makes a large difference. On average, about 3% of the vote is left uncounted in the days immediately after election-day(With Alaska it was 43%!), and so post-election analysis might say more about the characteristics of votes that take a long time to count than anything else.

I quickly revisited the finding about likely voter models, and checked to see if it was true with the post-certified data-set. Checking, it seems Wang's results are very robust . I used my predicted margin because it did not incorporate any regression, making it a better representation of polling consensus.

Under a weighted-least square regression with robust standard errors (Weighted to account for differing sample sizes across states), the Actual margin was 1.64(+/-.6) points higher than the predicted margin on average (p=.0085).

Under a Least Absolute Deviation without any weights, the actual margin was 1.34(+/-.58) larger than the predicted margin (p=0.03) .

The dataset, containing FiveThirtyEight and Stochastic Democracy forecasts, along with certified vote totals, is available here. Feel free to download it and look for other patterns.

Gay Marriage Support Over Time

2009-04-04T13:52:00.002-04:00

Nate Silver has an interesting write-up where he model's support for Gay Marriage bans accross states. He then proceeds to predict at what point each state would reject a gay-marriage ban ballot initiative.

As he says,

It turns out that you can build a very effective model by including just three variables:

1. The year in which the amendment was voted upon;
2. The percentage of adults in 2008 Gallup tracking surveys who said that religion was an important part of their daily lives;
3. The percentage of white evangelicals in the state.

This is interesting, and confirms what most people would initialy suspect: Support for gay marriage is primarily determined by religion and decreases over time.

I wouldn't put too much stock the forecast numbers though. The R^2 for his model is only about .66, which is fine for confirming the overlying trends noted above, but really poor for forecasting purposes. It is however, a good initial model.

People don't change their minds about gay marriage very often. So fundementaly, the structural shift in opinions on gay marriage can be explained solely by looking at how quickly the young replace the old.

Estimating the support among the various age groups could be done individualy for every state by looking at raw survey or exit poll data. This would cut out the uncertainty behind indirectly estimating support via regression. At that point, the support is matched with demographic projections from the senate, and it would be straightforward to predict changes in public opinion over time.

This would automaticaly account for the different demographic structures of each state. For example, a state where young people flee would see it's support for gay marriage evolve differently than one where young people flock. Moreover, it would automaticaly incorporate the idiosyncracies behind a state like Utah.

Hopefully someone else does it. But if there's a demand, Rasmus and I might take a look into this when we can find time...

Partner Reduction and HIV

2009-04-01T10:22:00.011-04:00

Ross Douthat, defending the Catholic Church's actions regarding HIV in Africa, says

But it's my impression - created, in large part, by reading Helen Epstein's The Invisible Cure (and if there's a devastating rebuttal to her arguments, please send it my way) - that an awful lot of the money poured into condom-promotion over the years would have much been better spent promoting "partner reduction" in cultures inclined to promiscuity and de facto polygamy instead.

Leaving aside arguments of the practicality behind partner reduction, it's desirability relative to condom promotion is more complicated than he implies.

Consider two sexual networks A and B

Network A represents a case where a prostitute has three clients and a husband, while each of her clients have a wife(A relatively common situation in certain countries).

Network B is more egalitarian , every man and women has two sexual partners.

While the average number of sexual partners is higher in Network B than Network A, a disease would spread far easier in Network A.(This is because the prostitute is "close" to everybody, so she will quickly contract the disease and give it to everyone else).

Taken from Liljeros(2003)

While most people report only a couple sexual partners during their lifetimes, some report several hundred or more. The promiscuity of the average person isn't terribly relevant, it's the small number of "hubs", people who have sex with large numbers of people, who are most important to disease transmission.

In Epidemiology, this is represented by the formula for a disease's basic reproduction number, which tracks how far a disease will spread in a network.

p0 is the average number of infections produced by an
infected person in an uninfected population, σ^2 is the variance of the number of partners, and μ is the mean number of partners in the population.

My concern with Partner reduction is that it's only realistic means of implementation, social stigmatization of sex, might decrease μ a bit, but drastically shoot up σ.

To see how this could happen, imagine if we shifted to a culture where guys usually do not have sex with their pre-marital girlfriends, since that's not something that "good girls do". Instead, some of the guys get their fix by sharing prostitutes.

In this new world, all of the "good girls" have at most 1 partner, and the guys only have one or two. So there has been a lot of partner-reduction. However, because the variance of sexual partners has shot up due to the prostitutes, society has become more vulnerable to STD's.

Epidemiologist Elizabeth Pisani, author of a book I strongly recommend, explained this better than I here:

Because of these complications, I'd shy away from benefit-reduction for the purposes of public-health. After all, it's a rather straightforward case of social engineering, and conservatives like Douthat are usually against that.

Instead, the most effective means of targeting HIV is to focus on making sure the groups that do most of the transmission work(Intravenous drug users, sex workers, and people who have an enormous amount of sex) are using clean needles, condoms, and have access to antiretrovirals. Unfortunately, most conservatives don't seem to favor this approach...

***Cross-Posted at Daily Kos ***

Quick Special Election Note

2009-03-31T16:46:00.006-04:00

As most political junkies know, there is a special election today in House District NY-20 to determine the successor to now NY-Senator Kirsten Gillibrand.

Unfortunately, there don't seem to be enough non-internal polls for any conclusive predictions, but running what's available through a quick Kalman filter...

Date	Observed	Filtered	MOE
2/18/2009	42.5	43.12	3.92
3/9/2009	47.67442	47.7	3.2536
3/25/2009	52.22222	51.71	3.4496

With an observed volatility of 1.33(+/- .3) (With a sample size of three, keep in mind), The Democrat's probability of winning the race is roughly 68%.

Usual caveats about low special election turnout apply

***Cross-Posted at Daily Kos ***

Update: Democrat Scott Murphy indeed won the election, squeaking through with 50.23% of the vote.

International Fiscal Stimulus

2009-03-04T18:49:00.007-05:00

Justin Fox writes about the stimulus packages in other countries. At first glance, it seems that the US, China, and Spain are trying hard, while the rest of the world is shirking.

The concern is that if we in the U.S. do lots of stimulating and other economies don't, much of the money will just leak out overseas as we spend on imports but others don't buy our exports.

These concerns have echoed across the blogosphere. Germany in particular attracted a lot of ire for only spending 0.9% of GDP.

But money does not need to be in a specially branded stimulus package in order to be stimulative. Any deficit spending can do the trick.

I pulled up each of the countries mentioned in the Brookings report, and looked up their deficit numbers from the Economist's Intelligence unit.

It's unclear the extent to which these numbers include the stimulus packages mentioned in the Brookings report, but even if we solely look at deficit spending, Germany looks a lot less frugal(at 3.7% of GDP), while China doesn't seem as generous(at 2.7%).

Country	Sum spent (in billions of $)	Stimulus % of GDP	Deficit % of GDP
Saudi-Arabia	17.3	3.3	11.8
US	787	5.5	11.5
UK	36.35	0.9	11.3
Vietnam	1	1.1	7.3
Spain	113.37	8.1	7.2
Malaysia	1.88	1	6.6
New Zealand	5	3.7	6.5
India	6.3	0.3	6.1
Japan	110	2.3	5.4
France	33	1.3	5.4
Portugal	2.77	1.3	4.5
Italy	5	0.3	4.3
Thailand	300	3.3	4.2
Germany	103.3	1.6	3.7
Russia	20	1.1	3.1
Australia	10.15	0.9	3.1
Chile	4	2.2	3
China	586	6.9	2.7
Philippines	6.12	4	2.7
Hungary	6.5	4	2.6
Belgium	2.52	0.6	2.5
Korea	10.8	1.1	2.2
South Africa	3.76	1.2	1.7
Switzerland	1.34	0.3	1.7
Indonesia	4.5	0.9	1.5
Brazil	3.6	0.2	1.4
Netherlands	7.56	1	1.3
Mexico	54	4.7	1
Argentina	13.2	3.9	0.8
Norway	2.9	0.6	-9.8

County-Level Foreclosure Map

2009-03-04T15:08:00.005-05:00

Ezra Klein looks at this graph and writes

“What we had is less a foreclosure problem than a foreclosures in California, Nevada, Arizona, and Florida problem. The way you get 42 states with foreclosure rates beneath the national average is that those last eight states are post-crash dystopias inhabited mainly by squatters and feral dogs. And the way eight states bring down the economy is that the foreclosed assets were heavily leveraged: The whole country might as well have been the Golden State given that Citibank would bet $56 dollars on every buck of California mortgages.”

Matt Yglesias responds

But what I really want is some more fine-grained data. When I was at the Atlantic, we were able to put together a county-level map showing foreclosure rates (no reference to national averages), for my prescient charticle “There Goes the Neighborhood” in our January/February 2008 issue:

Unfortunately, the charticle was so prescient that the data is now hopelessly outdated. And I don’t really know how to get more updated data or make such a good-looking map

Luckily, Hotpads provides rather up-to-date Foreclosure heat maps. While they don't let you see the entire country at once at a county-level, a combination of screen-captures and photo-stitching allows the creation of this:

Not terribly pretty, but conveys the general point: Foreclosures are spreading, as the the poor economy, at first driven south by unexpected foreclosures, causes further foreclosures.

Unfortunately, not that much data analysis can be done, since the owner has refused to make the raw data available on a county-level. Though, someone might be interested with the CD-level data.

The Pirate Bay

2009-02-21T15:11:00.007-05:00

The Pirate Bay, a very heavily influential BitTorrent tracker and founder of a minor political party in Sweden, is in the fifth day of their trial for copyright infringement. My views on copyright are outside the scope of this blog, but I did notice one bizarre quote from the trial:

When Altin asked about the amount of copyright material tracked by TPB, Peter explained that he carried out a survey of a random 1000 torrents from the tracker and 80% of the content linked by the site was not copyrighted, noting that there is much more illegal material on YouTube.

As someone rather familiar with the File Sharing Scene, this initially seemed a little far-fetched to me. But Torrents exhibit Pareto phenomena, where most visitors are concentrated to the most popular downloads. Perhaps a lot of legitimate content was hiding in the long tail.

To obtain a "random" sample of Pirate Bay Torrents, I looked at the 30 most recent torrents, provided here, and checked each one for copyright infringement(The Sample size is rather small, but I can't think of an automated way to check for copyright infringement). I then ran a similar procedure for Youtube.

Bar Graph of results, with red lines showing 95% confidence intervals

I found that about 70% of the torrents on Pirate Bay looked at likely violate US copyright laws, while roughly 30% of the content looked at on Youtube likely violate copyright law. The margin of error, given the sample size, is +/- 16% with 95% confidence.

However, a good deal of the illegal content on The Pirate Bay was very old and obscure(Think, 1960's era Hungarian Jazz), and most of the illegal content on Youtube involved videos that used copyrighted music as a background.

So it's possible, depending on your interpretation of fair use, that Peter's comment is true under Swedish copyright law, but it still seems doubtful that his website is more "clean" than Youtube.

***Cross-Posted at Daily Kos ***

Israel Update

2009-02-10T23:23:00.011-05:00

It seems that I predicted the number of seats in the (Arab+Kadima+Labor+Meretz) coalition perfectly at 55 seats. I also correctly predicted that Gil and the Greens would miss the electoral threshold necessary to receive a seat.

I can't brag too much though, since I never released any party specific breakdowns, mainly since I couldn't get a good handle on individual party volatility. (As shown by the surprise shift from Labor to Kadima).

So far, with 99% of the votes counted, the break-down(in order of left to right) is:

Arab Parties: 11
Meretz: 3
Labor: 13
Kadima:28

Likud:27
Yisrael Bitenu:15
Shas:11
Haredi Parties: 12

Everyone is talking about Yisrael Bitenu's role as a "King-Maker", but I'd give that prize to Shas. Yisrael Bitenu can't fit in any majority grouping outside of the right.

Yisrael Bitenu is not going to be in a coalition with the Arab parties, for obvious reasons, without the Arab parties, Meretz+Labor+Kadima + Yisrael Bitenu = 59 seats=Not a majority.

And as for their place on in a right-wing coalition, Shas hates them. And if Shas defects to the traditional left-block, then it would push the left to a majority.

So, if anything, I'd say Shas is the king-maker. Shas could easily fit with Kadima or Likud, since it's demands are mainly unrelated to the peace-process(More religious funding, less pork shops, etc). The question is whether they would coalition with the arab parties.

Obviously, Netanyahu is probably the next prime minister. But, we'll see, coalition politics is weird...

More Israel

2009-02-03T21:04:00.008-05:00

I'm aware that I've promised updates for tonight, but a power-outage and my tendency not to save my work has forced me to set that back by a day or so. In the meantime, I'll go over some of the challenges with generalizing our model to the Israeli elections.

1) Israeli Pollsters usually do not publicize the raw percentages that each party receive in their polls. Instead, they run D'Hont parliamentary seat allocation algorithm used to allocate seats in Israel on their poll, and report the seat outcomes. This creates something best described as rounding error on steroids, entirely negating the advantages of the unusualy large sample sizes found in Israeli polls. Since the D'Hont method is intractable in terms of random variable algebra, this requires a good deal of Monte-Carlo simulation to determine the extent of the possible "rounding error", and incorporate it into the observation error distributions of our Bayesian filter.

2) In the Israeli parlimentary system, parties that recieve less than 2% of the vote have their votes thrown away. Since there are a couple of parties that hover near the electoral threshold, this creates problems for modeling.

The first problem, is that since pollsters only report the number of seats instead of percentages, a poll that shows a party recieving no seats could actualy have had 1.9% of the vote in the sample. Luckily, this Censoring problem is remarkably easy to implement in our Hidden Markov Framework.

The second problem, more fundemental to parlimentary democracy, is that whether or not these small parties reach the threshold has huge effect on the dynamics of potential coalitions. I use Monte-Carlo to do the best I can, but due to the non-tractability of the D'Hont Algorithm, sensitivity analysis is a bit difficult.

3) Moving from two parties to 14 is a huge jump, both analyticaly and computationaly. In order to keep things computationaly feasible(pending a new CPU), I've had to assume that day to day opinion changes are Gaussian. This is not true, as I've gone over before, but the approximation is still pretty good thanks to the Central Limit Theorem.

4) It's really hard to find information on Israeli Pollster Methodology. For example, I have no idea what Israeli pollsters do with undecided voters, or their sampling method. These are low concerns, since the election isn't close at all, but any information on this front from people who speak hebrew would be great.

Finaly, a quote from our Kos page comment forum that I especily enjoyed.

Expecting rationality from humans is a mistake. Expecting it from politicians is a bigger mistake. Expecting it from Israeli politicians ... well .... - plf515

Pretty graphs soon...

Stochastic Democracy: Israel

2009-02-02T22:40:00.011-05:00

Left- Smoothed estimate of support for the Left-wing Coalition.
CI- The 95% confidence intervals for what public opinion is today, based on the sampling error in the poll aggregate
FI- The 95% prediction intervals for what the outcome will be on election day, based on estimation of variance when treating polls as observations of a random walk(under a hidden Markov Bayesian framework).

More details tomorrow, but I've modified the Bayesian filter a little to accommodate the upcoming Israeli election(It's a fairly hefty generalization of what we did for the Georgia Senate Race). I'm still working out some of the kinks, but it seems that conventional wisdom is correct. We estimate that Benjamin Netanyahu has an 84% chance of becoming Israel's next prime minister.

As of now, we predict that the likely right-wing block (Likud, Shas, Yisrael Bitenu, Jewish Home, National Union, and United Torah) will win roughly 65 seats in the Israeli Knesset, beating out likely left-wing block (Kadima, Labor, Meretz, and the Arab parties).

The two "swing" parties which do not easily fit on the hawk-dove spectrum, Gil(Senior Citizens Party) and the Greens(Environmentalists), are currently right below the 2% threshold necessary to obtain seats, and the most likely outcome is that they will not obtain any representation.

Note, religious parties have often been in coalition with Labor and (recently) Kadima. Meanwhile, Meretz(and Shinui before it), have coalitioned with Likud.

But for predicting the likely government, we only need to consider 1st choice coalitions. For example, given the choice, Shas would prefer Likud to Labor]

In the past, there have been coalitions of national unity between Likud and Labor, but that is rather unlikely this time around.

Things to Watch For:

There are several parties that are right at the threshold. On the right, Jewish Home, a "Religious Zionist" party, is estimated to have roughly 2.5% support. On the left, Balad, a Arab party, is estimated to have 2.2% support.

The Left's main hope is that Jewish Home fails to make the threshold, while Balad, Gil, and the Greens each manage to get over 2%. If that happens, then it may be possible to lure the neutral parties into a coalition to barely keep Likud out of power.

Strategic Voting Recommendations:

Kadima, Labor, and Meretz voters should vote for Balad(Though they won't), Gil, or the Greens

Supporters ofRight Wing Parties healthily above the threshold should vote for The Jewish Home Party(But they most likely won't, since the Haredi parties are locked in some fued I don't really understand)

Gaza

2008-12-29T18:30:00.004-05:00

Much of the talk about the latest attack in Gaza has focused on the morality of the actors involved. It's easy to get sucked into these disputes, but they don't accomplish very much.

Instead, everyone can agree that the current status-quo of endless tit-for-tat violence is bad, both for Israelis and Palestinians.

But in order to move to an equilibrium that benefits both sides of the conflict, we need to better understand the dynamics of the parties involved.

With that in mind, let's analyze the "game" of the Gaza conflict:

The Game

First, assume that the Palestinian factions mainly act rationally to maximize money and power.

There are plenty of angry Palestinians who love to kill Israeli's, but they are for the most part exploited by smarter and more level-headed people for the purposes of power and money.

The players: Israel, and roughly two dozen Palestinian factions of varying power in Gaza.

The game goes as follows: Palestinian factions each individually decide how many rockets to send into Israel, and Israel responds with retaliation against Gaza. Israel's attacks reduce rocket capacity, but also inevitably kill civilians, creating "anger".

This anger fuels recruitment and foreign funding, and Palestinian factions use the manpower from recruitment in order to exploit the economic rents prevalent in a lawless third world country(Road checkpoints, protection money, ect). Palestinian factions send rockets into Israel as a means of capturing this anger driven recruitment and foreign funding.

But this anger creates danger. Radicalized youth are hard to control, and tend to get into costly turf wars. Worse, "insufficient action" against Israel can trigger assassinations(See Sadat, Anwar). And of course, war with Israel can be inconvenient(See Yassin, Ahmed) Most of the players in the game would prefer peace.

Where does this lead? Let's look at the payoff matrix of two equally matched Palestinian Factions.

This is a classic prisoner's dilemma problem. Even though both factions are better off if they refrain from violence, they are only better off if the other party refrains from violence as well. The only way that the Gazans can stop Israeli retaliation is if they all cooperate.

But unfortunately, two dozen factions are never going to unanimously cooperate on anything(And even if they do, factions don't have much control over their members), and individual factions, knowing this, proceed to attack Israel.

This is a fairly simple model to translate into math(Just assume that new recruits are allocated to each faction in proportion to rockets sent, and that Israeli retaliation effects each faction in proportion to their size.), and for most parameter values, an Israeli strategy of "proportional" attacks leads to steady streams of rocket attacks and retaliation, with occasional spikes in violence. This equilibrium is fairly robust to changes in assumptions.

Obviously, Israel is justified to want to "totally change the rules of the game", as Ehud Barrak said.

The question is, how?

More on that tomorrow, in the meantime, I'll step out of my math-major skin and say...

Don't Invade Gaza!

Gaza has been preparing for a potential Israeli invasion for over 3 years. If Israel invades, the number of dead Israeli soldiers would likely far exceed the number of Israelis that could realistically be hurt by Palestinian rocket attacks.

While this might be justified if there was a chance of lasting peace afterwords, there doesn't seem to be a plausible path in that direction. Most likely, there would be a temporary lull in violence, and soon enough, we would end up exactly where we are now.

Except of course, with more dead Israelis and Palestinians.

But as I said, more tomorrow...

Note: The follow-up post was published at our DailyKos site.

Post-Certification Post-Mortem

2008-12-26T14:19:00.009-05:00

Now that it seems clear that Democrat Al Franken is going to win the Minnesota Senate race, and with every state having certified their final vote totals, we can look at how accurate Stochastic Democracy fared this election cycle:

Electoral College:

Having some fun...

Our model correctly predicted the winner of the presidential race in every state except Indiana(Where we predicted Obama had a 48% chance of victory).

In terms of electoral votes, our model predicted that Obama would win 364 electoral votes, when he instead won 365.

Altogether, we called 50 out of 51 states/districts correctly, and our electoral forecast was off by 1 vote.

National Vote:

We predicted that Obama would receive 53.76% of the two-way vote. He went on to win 53.68% of the vote.

Senate and Governor:

We correctly predicted the winner of every senate race, as well as predicting that the Georgia Senate race would go to a run-off.

We also correctly predicted the outcome of every Governor race.

Comparison with FiveThirtyEight:

In terms of predicting winners, FiveThirtyEight and our site produced nearly the same predictions. The only differences were:

Due to the lack of polls, we did not provide a forecast for the Omaha district of Nebraska. FiveThirtyEight did provide one, though it was incorrect.

FiveThirtyEight predicted that Republicans would win the Georgia senate seat on Election Day, while we correctly predicted that no canidate would recieve a majority, sending the race to a run-off.

From our Pre-Election final Georgia Projection.

We predicted that Obama would win 364 electoral votes, while FiveThirtyEight predicted that Obama would win 353 electoral votes. Obama actually won 365 electoral votes.

But because of the winner-take-all system, this is not a good way to gauge accuracy. Instead, it is a better idea to look at how we did at predicting the margins in individual states:

Presidential:

In terms of mean absolute error, my model had a slightly lower mean absolute error than his. But the Kurtosis of his prediction residuals is much higher than mine.

That means that most of the time, his predictions were about as accurate as ours. But when his predictions were off, they were really really off. In his defense though, his model was mostly off in states that were not important(Washington DC, Wyoming, etc.)

But, everyone has their own metrics for this sort of thing, so for those who want to see the raw data, click here. As a final word though, I'd note that our model was much simpler then Nate's, while being either as or slightly more accurate.

Senate:

As can be seen, FiveThirtyEight outperformed Stochastic Democracy in the Senate. This suggests that House Effects and Pollster-Introduced errors might be more important in the Senate, and that future models should try to incorporate the effects.

Popular Vote:

Stochastic Democracy did quite a bit better than FiveThirtyEight, while Pollster falls in the middle. Of course, this isn't very statistically meaningful.

Also, since I don't want anyone to go through the tedious process of data collection, raw certified election results by state(Presidential, Senate, Governor), are available on a spreadsheet here.

Econometric Models

2008-12-04T10:34:00.002-05:00

Gelman has a nice write-up on Econometric models for election forecasting(Basicaly, models of the form "Plug in GDP Growth, multiply by inflation...").

The subtext behind these models is alluring, and really, most likely true: Campaigns don't really matter much, and elections are mostly determined by external events. But unfortunately, these models have theoretical issues, and they are not particularly useful when it comes to forecasting(Though they might be interesting for other reasons).

To see what I mean, take a look directly at Hibb's model, referenced by Gelman, here(The meaty details are on page 4). While the theoretical model seems plausible(Namely, that incumbant party vote share is determined by weighted real-income growth over the term and war casualties), the author still is fitting a non-linear model on 14 datapoints.

Temporarily ignoring how little data we have, these 14 datapoints were observed over 50 years. It's entirely plausible that the parameters involved have changed over the years(For example, voter expectations for real income growth were certainly different in 1952 than they were in 2004). And Crime certainly has some effect on elections, as does distributional effects of real income growth and demographic change. Unfortunately, there is far too little data to incorporate these real world complications.

Even if we assume the model is perfectly specified, it's off by 2.5 points on average on the data it's been fitted to. This isn't particularly impressive, considering that it uses a data input(Q3 growth reports) that isn't available until a week before the election. This doesn't perform very well compared to poll aggregates available from the same period.

Election outcomes might be a function of external events. But it is likely a very complicated function. Instead of estimating such a complicated function with limited data on shaky theoretical ground, there is a much simpler way of determining who people are going to vote for: Asking them directly.

Georgia Run-Off

2008-12-02T21:13:00.005-05:00

CNN has now called the Georgia Senate Run-Off to Saxby Chambless.

This isn't particularly surprising, considering the early vote totals.

Nate Silver said on his blog:

" A disappointing night for Democrats. On November 4th, Democrats became the de facto ruling party ... circa 2006 or so, the Dems got very good at figuring out what sort of messaging works when you're in the minority, but that's very different from the sort of messaging you have to do when you're in the majority. There's going to be a temptation in some circles to write this one off to poor African-American turnout or whatever, and that certainly is a large portion of the story. But I think the Democrats need to think carefully about what went wrong here as they begin to gear up for 2010."

I don't see why we need to assess "What went wrong here" too closely when the Democrats control 58(and maybe 59) seats in the senate.

But if we are going to start accessing blame, I'd say there are two main culprits here:

1) Georgia is a very red state

There might be a large number of black voters, but they already turnout well in proportion to their population, and no turnout operation can pull out a victory when Whites vote Republican 70% of the time(From the November Exit Poll).

Instead, Democrat's only path to victory in states like this is to convince large numbers of previously republican voters to vote for them(Like Obama did in Virginia, North Carolina, and Indiana).

I've heard a lot about GOTV efforts in Georgia. And getting out the vote is important. But it's not enough.

I never heard anybody, or more importantly, any ad money being spent, talking about why Jim Martin should be elected. The Republicans had a clear message "Stop the liberal takeover", and they hit it very hard. But I never heard anything resembling a coherent narrative on the Democratic side.

For some reason, we were able to form such a narrative in Indiana, but completely unable to do so in Georgia. I have no idea why, and would love to hear some commentary there.

2) Funding

This is more mundane. During the actual election cycle, Republicans had to distribute their meager funds and resources thinly across the country. After the election, they were able to send all their money to Georgia.

To some extent, this was true with the Democrats as well.

But campaign spending has rapidly diminishing returns(Approximately Square Root, if the Market Mix literature is to be believed). The first million dollars gets you volunteers and a GOTV program. The second million dollars gets you tastier food and nicer cocktails for your donors.

The Democrats were already spending lots of money in Georgia. Sending more organizers after such an extensive effort wasn't going to do much.

The Republicans on the other hand, barely had much of an operation at all. They had a lot to gain by sending in Republican celebrities and pouring money into the state.

And if on November 4th, the Democrat's organizational advantage couldn't eke out a win, it isn't much of a surprise that they couldn't do it after the Republican's redoubled.

Minnesota Links

2008-11-23T20:22:00.006-05:00

This seems like a good time to point out that Vote For America is doing some work on the Minnesota recount.

Update: FiveThirtyEight has it's own projections up. At first glance, Nate's methodology seems quite sound. But, as he says:

The error bars on this regression analysis are fairly high, and so even if you buy my analysis, you should not regard Franken as more than a very slight favorite. Nevertheless, there is good reason to believe that the high rate of ballot challenges is in fact hurting Franken disproportionately, and that once such challenges are resolved, Franken stands to gain ground, perhaps enough to let him overtake Coleman.

I quickly tried to replicate his regression, and a back-of-the envelope MOE is around +/- 600 votes. So it's a coin-toss.

I'm flying to France on Tuesday, so posting is going to be non-existent.

Assignment Desk: Black Turnout and Prop 8

2008-11-23T12:08:00.015-05:00

A asks

I'd be interested to see some rigor put to the thesis that high black turnout in CA pushed prop 8 over the top. I know some other people have run back of the envelope numbers, but I'd like to see some serious stats thrown at the problem.

Before we start, some numbers to remember:

Racial breakdowns of the 2008 and 2004 electorate, as well as a breakdown of the population.

Let's recall a useful arithmetic identity that holds in every county in California:

Prop8Support= (Number of White People)*(White turnout)*(White Support)+(Number of Black People)*(Black Turnout)*(Black Support) + ... [ continue this sum for Latinos and Asians]

The racial breakdowns of these counties are available from the Census Bureau, and racial turnout estimates are available from the California exit poll.

But Racial support is not monolithic. White people in San Francisco do not necessarily vote like White people in Orange County. In order to account for that, we perform a county-level regression on the election results to estimate racial support for each county.

We then revert the 2008 racial turnout proportions to what they were in 2004. This gives us an idea of what the vote would have looked like under a "traditional" turnout scenario.

If Proposition 8 fails under such a scenario, it gives credence to the idea that the surge in African American turnout for Obama was responsible for the Proposition's passage.

And that's precisely what happens. Under the "traditional" scenario , without the surge in African-American turnout, my regression model predicts that Proposition 8 would have failed with 49.3% of the vote(+/- ~ .6%) . While the election would have been close, the finding is statistically significant(Though right outside the margin of error).

So to answer A's question: Yes, it seems that the upswing in African-American turnout due to Obama was "responsible" for the passage of proposition 8.

I'm not going to make any commentary here, but I want to point out that even if this was the case, the election was really close. One could "blame" the lackadaisical No-Campaign, or the Mormon Church just as easily as one could African-Americans. So I'm not sure what the point of these kind of questions are.

But to return to my comfort zone, I want to point out that NLS regressions are a bit tricky. Because of this, I've made the regression output available here, and the California County Data(with election results and some regression variables) available here.

Update: Nate Silver at FiveThirtyEight seems to disagree with me.

Nate's "mistake" seems to be at:

Furthermore, it would be premature to say that new Latino and black voters were responsible for Prop 8's passage. Latinos aged 18-29 (not strictly the same as 'new' voters, but the closest available proxy) voted against Prop 8 by a 59-41 margin. These figures are not available for young black voters, but it would surprise me if their votes weren't fairly close to the 50-50 mark.

African American turnout in California increased from 30% of African Americans voting in 2004, to 50% of African Americans in 2008. This increase is too large to attribute to young black voters, which was the only group that Nate considered.

There seem to have been plenty of older voters who usually didn't vote, who wanted to vote for the first black president, yet were culturally conservative.

Update 2: OrganisedCrime asks

"White people in San Francisco do not necessarily vote like white people in Orange county"? Did/do black people vote the same throughout California, when voting on proposition 8? Where in California were the exit polls taken of black people and who did the polling? CNN? Did you break it down or take this into consideration in the black areas, as you did with white people?

I responded

That information was available on the regression internals on my website, but yes.

My model accounted for county-to-county variation for Blacks, Whites, Latinos, and Asians.

Other than the simple exit poll breakdown of turnout by race, no other polls were necessary, since support could be inferred using a county-level regression.

Assignment Desk

2008-11-21T18:40:00.000-05:00

Every week, tons of narratives are spun, most of questionable validity.

Have a favorite pet theory that you'd like to see evidence for? Suspect a pundit was lying but don't have the time, data, or expertise to prove it?

Today's your lucky day!

Post your question or suggested topic on the comment thread below, and Stochastic Democracy's math guy will address the best questions in a new post on the weekend.

Ask Away...

Georgia

2008-11-20T09:19:00.002-05:00

Georgia started early voting. Due to the Civil Rights Act, they have to keep racial statistics on turnout.

See here for data.

Black turnout is at 24%, compared to 30% in November. I don't see Martin winning with those numbers, but we'll have another day of data soon.

Also, exactly two Native American voters, one male and one female. Anyone want to bet it was a couple?

Race and Obama

2008-11-17T20:42:00.002-05:00

Referencing Charles Franklin's well known piece on racial polarization, Nate Silver has an alternative idea:

"The driving factor in determining how Obama performed vis-à-vis John Kerry, however, appears as though it might not be race, but rather how much Obama camaigned in a given state. According to the New York Times candidate tracker, Obama campaigned extensively -- by which I mean, he actually went out and spent a lot of time on the ground -- in 6 of the 15 Southern states. These include Virginia, North Carolina, Florida, and Missouri (where Obama campaigned extensively in the general election cycle), as well as South Carolina and Texas (where Obama campaigned extensively in the primaries). The other nine Southern states, Obama did not have more than a couple of apperances in, and several he did not visit at all."

The first and obvious objection, that Obama only campaigned in states where he thought whites would be favorable to him, was addressed by Nate.

I think the most telling example might be South Carolina, which Obama did not campaign in because of any particular demographic strengths, but merely because it happened to enjoy an early position on the primary calendar. In that state, Obama did 4 points better than John Kerry among white voters, even though he didn't really visit the state after January.

So does this story check out?

As much as I'd like to believe so, unfortunately not.

While I haven't yet looked at the 2008 election data(Expect that in a day or two, complete with pretty graphs), I've given a rough look at Racial Polarization in the 2004 elections before on a county-level.

On a county level, the relationship that Dr.Franklin was originally talking about, that Democrats do badly among whites where-ever black people live, seems robust, and has existed for several election cycles.

This isn't to say that Democrats shouldn't try to compete, that's a separate question. But it doesn't seem like the racial polarization in the South can be explained by differences in campaign effort.

Georgia

2008-11-17T09:21:00.002-05:00

Chambliss, the Republican Candidate for Senate for the run-off in December, seems to be involved with an ethics scandal.

Intrade doesn't think it will very significant, and we'll have to wait a bit for polls...

Minnesota Recount Prediction

2008-11-14T16:04:00.003-05:00

Via Andrew Gelman,

Check out this paper that shows most "undervotes" are going to be supportive of Franken.

Ou pas

2008-11-12T23:12:00.002-05:00

Alright, Alaskan district breakdowns still are not in. Without that information, the uncertainty bars on my model are too high for me to say anything but this: The Democrats just picked up seat 58.

It might come in later tonight, but I've got an early class tomorrow. So I'll be asleep.

Luckily, Nate Silver is an insomniac, so check FiveThirtyEight for Alaska updates until tomorrow morning.

Holy....

2008-11-12T19:46:00.001-05:00

Begich is ahead.

As of writing, it's a 3 vote(Yes, 3 votes) lead...

Still, my regression says the lead should hold....

Ted Stevens Update

2008-11-12T16:39:00.002-05:00

The Alaskan Board of Elections plans to count 50,000 out of the approximately 70,000 outstanding ballots today. Historically, they tend to update their website at about 17:30 AKST, which is 9:30 PM for anyone living under EST.

By then, we'll know enough about the situation to begin forecasting the winner. In the meantime, now would be a great time to read Nate's article on this.

Mini-Post

2008-11-10T19:07:00.002-05:00

I'm too busy today to check this out(Part of that post-election pile of work I was talking about), but I'll mention this in hopes that someone does it before I get to it tomorrow:

Nate Silver did some good math here. But a table of probabilities, based on supposedly exogenous assumptions? That makes the Bayesian in me cringe. Why don't we develop some plausible prior distributions for the relevant parameters so that we can get an actual number here?

Stay tuned...

Exit Poll Madness

2008-11-06T14:57:00.004-05:00

There is a nice discussion here about Nate's finding that Obama "underperformed" among Gay and Lesbian voters( Exit polls show Obama getting 70% vs Kerry's 77%).

Before I go on, I want to stress that exit polls have much larger margins of error than their sample size implies, because they are obtained from a very complicated stratified sampling technique. And since the company involved has been quite quiet about it's stratification assumptions, I can't even vouch for unbiasedness, let alone talk about variance.

But getting past that and assuming perfect random polls, then the margin of error of the observed change in democratic support among gays is approximately 2.5%, while the observed decrease was 7 points. (The margin of error is so small because the vote is so lopsided. Margins of error are much smaller in an 70/30 race then a 50/50 one)

So, it seems that the decrease might actualy be statisticaly significant and worth exploring.

Update: On Pandrew's requent, I looked at the 2000 data, showing Gore's 2000 support among gays to be at 70%. This is consistent with the idea that has appeared on the thread, that the 2004 election's focus on gay issues lead to an unnatural "bump" in support for democrats.

But the more I look at the data, the fishier this looks. I'm begining to think the whole thing is a fluke of their statification assumptions.

Pollster Inaccuracy

2008-11-05T16:16:00.013-05:00

While Pollsters were pretty accurate this season, they did make some mistakes, and it's important to learn from them.

So I ran a quick regression on the gap between Obama's performance in the polls and Obama's filtered poll average. (Don't worry, I took sampling error in the observations into account in the regression)

Altogether, the only trend that really pops out is that Obama did worse then expected in states with lots of young voters.

Evidence that Youth voters didn't turn out as much as pollsters expected? Possibly, but correlation does not equal causation. It could also be that states with lots of young voters have lots of culturally conservative parents. More research is needed.

I'm making the regression output available here, anyone else have any interpretations?

update: 7:46 PM, Fixed the incorrect link

How did the Polls do this year?

2008-11-05T14:41:00.008-05:00

Pretty Well

A chart showing Obama's performance in the election relative to the filtered average of polls in the state before election night. The CI lines show the 95% confidence intervals of the poll average.

The only important outcomes that were outside the margin of error were in Nevada, North Dakota, and Iowa. And out of so many states, that's to be expected.

I'm going to run a regression on the residuals to see what I can find, but in the meantime, the interested can see the spreadsheet here.

n/t

2008-11-05T11:43:00.002-05:00

Mon Dieu, Look At All That Blue...

2008-11-05T10:22:00.005-05:00

A cartogram, where state's are sized proportionally to their electoral votes. Based on states that our crack team have already called.

Lesson Of The Day

2008-11-05T09:13:00.022-05:00

It looks like altogether, our electoral predictions turned out to be fairly accurate.

Electoral:
Indiana was our only mis-prediction, and we gave Obama a 48% chance there. Omaha is still too close to call, but without polls, we made no attempt to predict it.

So we only miscalled 1 state, and it looks like Obama obtained 364 electoral votes, which we correctly identified as tied with 366 for being the most likely EV total.

Altogether, we called 50 out of 51 states/districts correctly, and correctly predicted the exact electoral vote breakdown.

National:
Too early to say with certainty, but two-way vote-share is at 53% so far, while we predicted 53.7%. Not too bad, and well within our posted margin of error.

As far as raw totals, we predicted would get 52.68% . So far, actual numbers seem to put Obama at 52.3.

Off by less than 4 tenths of a point! Not too shabby.

Senate:

Too close to call yet, but it looks funky. Alaska seems to be the main outlier(Re-electing a senator who was just convicted of corruption? Who wants to give Alaska back to Russia?)

But liberals shouldn't despair. If Stevens wins, he'll still have to resign his seat due to the conviction. That will spark another senate race later.

Turnout:

According to the good folks at George Mason University, my turnout estimate seems to have been fairly on the mark.

But it's too early to say for sure. So far, my estimate was off by about 2%.

Livecalling:

After all our hard work, we manged to call the race for Barrack Obama two and a half minutes before the major networks. One must appreciate the little victories...

Seriously though, we managed to call Florida, Virginia, Indiana, and North Carolina before 11 PM(and as of 11:32 AM, all of the major networks still have not called North Carolina and Indiana )

Meanwhile, I'm truly thankful to have had Rasmus and Edged working with me. Despite hailing from Germany and Canada respectively, they easily know more about this country on a county level than anyone else in the world.

After sitting in on some heated arguments about the minutiae of turnout assumptions in obscure counties I had never heard of, I'm shocked that neither of them have been picked up by a campaign.

Expect some posts today on precisely how well our model fared as soon as more votes come in. (With Regression!)

3 AM new post

2008-11-05T03:01:00.030-05:00

6:06 - OK, just as I was all ready to go, Minnesota give more numbers. Barring an absentee/early vote/provisional save, Franken looks in serious trouble. From Itasca and most of St. Louis he lost around 700 votes, while in my model he needed at least a thousand.

6:01 - My final call before I prepare to hit the sack is for Yes on 8, unfortunately. No looked like it was going to get within 300K, but never did. most of the precincts out are heavy Yes.

To recap, the races still not technically called are MO-Pres, MN-Sen, AK-Sen, and GA-Sen. NC-Pres and OR-Sen haven't been called by the AP or CNN, but OR should be, and while NC is close, given the early turnout and typically Dem lean of provisional ballots, I think that it will stay blue. Thanks for following along.

5:51 - Using 2004 turnout for MN, there are as many as 15000 outstanding votes. If these break like the rest of their counties, which is not guaranteed, there would be +2300 Franken in St. Louis, +400 in Itasca, +7 in Pine and -33 in Cass for +2674, plus a likely slightly Franken or tied at worst precinct in St. Paul. Right now, Franken is 2576 votes behind. If my assumptions are correct Franken could move ahead by 98 votes! They won't be, but that's the idea, the race will tighten again, but the question is if there's enough votes left in the state.

5:46 - A dKos poster noted in a thread there that there are 70000 early ballots submitted, favouring Democrats somewhat in their distribution. They started only Thursday, implying that the voters were aware of the conviction. Still an open race.

5:37 - I was going to call AK, but I will recant based on the fact that there has been no absentee votes counted to my knowledge, and that the extremely low turnout and very bad polling make me a little suspicious. Still, I think Stevens has all but got it, the absentee would likely skew more towards him, being from before his conviction.

5:30 - Begich is done, just a couple of precincts left and there's been no movement towards him. Alaska turnout looks really low, but it's the Democrats who stayed home, apparently.

5:22 - Getting more pessimistic on Alaska by the minute, even without any results coming in. A Georgia report I read, though not all that recent suggests that the amount of uncounted Georgia ballots may have been in the millions, or at least 1 million as of around midnight. I find that hard to believe, but you never know.

5:06 - Any thing you read about a "tight race in Oregon" is, frankly, complete and utter bullsh*t. There are another 100K votes of margin for Merkley just waiting to be counted in Portland.

4:53 - Ok, Coleman is now ahead by more than 2000, and it looks like St. Louis County may have some strong Republican sections, since I think the votes came from that blue area.

4:51 - Franken now 700 down, but this is the vote dripping in little by little from the last Republican County, while the Democratic counties may have shut down and gone home. Also, Obama is 600 votes short of carrying NE-02. Not an official result, though, so we can't call it.

4:47 - Haha, all my predictions are going down the drain, Coleman leads Franken by 500, but 1% in Hennepin and parts of St. Louis and Itasca are both out. Still think Franken will retake the lead, but less sure than I was 25 minutes ago.

4:39 - Alaska is down to the wire. of the 17 precincts out, 15 are in areas where Democrats lead or are unopposed in State House races. But there may not be enough votes to overcome Ted Stevens 3600 vote lead. Berkowitz is done, 16K back. My predictions of big Democratic movement look bad now.

4:37 - A couple of precincts in blue areas went red in MN, Franken's lead is back down to less than 400. No is closing the gap now, but even if it continues, it looks like too little, too late.

4:32 - LA is now nearly 50-50 on Prop 8, but with a million more votes left at most, it would need to win 2-1 and hope the rest of the state was even for a close race. I don't think it'll get there.

4:22 - Franken's lead is around 1100. Of the 5 counties out, 4 are Democratic, including St. Louis and Hennepin. Franken will win the night, but may very well face a recount.

4:19 - Looking at results in the Alaska house, it appears to me that there is more Democratic vote and Anchorage vote out than anything else. We shall see in Alaska.

4:05 - Alaska is now at 81% with Stevens and Young ahead, and Young ahead by a few points, but they are still at only around 2/3 of the vote. I think there will be Democratic vote coming. It will be close, though.

4:01 - Also about OR, they may have stopped counting in some places, leaving us stranded with an artificially tight race, where Merkley can probably expect at least a further 100K vote margin from the areas out. No are in trouble, they need 300K votes, but I don't see where they can possibly come from.

3:57 - Something strange in OR, where Multnomah County is reporting as 100% complete, but is well short of 2004 numbers. Probably just a glitch, but the race looks worse for Merkley than it deserves to be.

3:56 - Looks like NC will not be called until provisional ballots are opened, and that may not start until tomorrow. Good news is it should favour Democrats.

3:48 - Close House races: AK-AL has Young up, CA-04 is 2200 votes to the Republican of 200K total, 84%. ID-01 is a 4K vote lead for Minnick over Sali with 72% in. MD-01 is 1000 votes for the Democrat at 100% in. NJ-03 is 93% in and 1700 votes to the Republican. VA-05 is fully in and 1200 votes to the Democrat, but that may depend on provisionals. WA-08 is early but 7000 votes to Burner. Lots of counting in Washington still to be done, though.

3:44 - Coleman and Franken have exchanged the lead over the last couple of minutes. It's 2100 to Franken as I speak, with very little Coleman vote left. It may be a recount (not runoff as I briefly said), but I'm thinking Franken will finish ahead.

3:34 - Stevens looks ahead, but the current Alaska vote of McCain + Obama combined would lose to Bush 2004, many precincts are in but the vote is not.

3:32 - Franken's lead is now 300 votes. Nail-biting.

3:23 - It now looks like there definitely are still early votes in Georgia, as it has significantly less votes than a similarly sized state, North Carolina. How many depends on the differences in registration laws and overall turnout, but there are some left that could narrow the gap.

3:14 - Missouri now looks done, a 4500 vote lead for McCain and no more votes in blue areas. Looked like whatever St. Louis vote was left was in the Republican part of the county.

3:03 - LA is just 1/3 in, and the hope for No on 8 must be that the other 2/3 have basically the opposite feeling as the currently counted group.

3:02 - Perhaps sleep is making me a little less sharp, Coleman still has some strength in MN, this is a closer race than I assumed.

Sorry for the brief break...

2008-11-04T23:39:00.027-05:00

3:00 - In Oregon, Smith now leads, but our call holds, and a lot of vote is left out. Franken has the lead in Minnesota, I doubt he'll give it back for more than a few minutes. Nearly ready to call this one I think.

2:56 - Missouri is absurd. What I though was the final county gave McCain a 3K vote advantage, but I now see that St. Louis County still has 1% out. Still no call. Missouri is crazy.

2:37 - Discussion at Dkos suggests a huge number of early votes in Atlanta are uncounted. Obama is 240 K behind, but Martin is only 140K back, and Chambliss is nearly right on the 50-50 mark. If there's a lot of vote left, this is still an open race.

2:32 - Merkley is only 1400 up in Oregon, but as we said when we called it, the open areas are heavily Democratic.

2:30 - Franken is less than 200 votes behind in Minnesota. 13% are still out in Hennepin (St. Paul), and a lot of the Democratic North as well. He is favoured despite trailing.

2:23 - The votes coming in from Missouri have left the gap at less than 400 votes for McCain, but there is very little hope left for Obama. Still, between provisionals and a recount, I don't think a faithful call can be made.

2:19 - For Prop 8 Fans, all that really matters is whether LA reverses it's current 55-45 Yes lead.

2:15 - Chambliss has fallen to 50.32%, just 15-20K more for Martin will take Saxby to a run-off, and it looks like there is more in Atlanta, if not much.

2:07 - Obama has claimed a huge percentage of the remaining vote in St. Louis, and trails by less than 600 votes in Missouri, with only a tiny amount left. Some is in small Republican areas and a tiny amount is in Kansas City. However, this is still too close to call. Also Obama seems to be ahead by less than 600 votes in NE-02, though this is unofficial and may not be a final tally.

1:51 - Al Franken is now just 800 votes back in MN, but the outstanding vote looks to be bigger in blue areas than red, so he may be able to take it.

1:41 - A further John McCain call in Alaska. He's up to 162 against 364.

1:39 - We are calling the Oregon Senate race, where Smith seems to have no way to make up ground in the remaining areas.

1:36 - AK-Sen and House races look bad right now, but I suspect that this is not Anchorage and Juneau, Democratic strength areas.

1:26 - With John McCain seizing the lead in Montana, and Obama's Missoula base completly tapped for votes, we are calling Montana for John McCain. This boosts him to 159. Total: O 364, M 159 no calls on MO: 11 and NE-02: 1

1:06 - Missouri looks very close. McCain has a slight lead, but there is still vote out in both St. Louis County and Kansas City, and we have yet to see any result from Clay County. Depending on turnout in these areas, there may be enough for Obama to win late. We just can't say.

12:56 - Counting of early voting ballots in Georgia has cut Chambliss to just 51%, but there may not be enough for him to fall below the line.

12:49 - Montana is now even closer, but Obama's votes in Missoula are all counted. It looks like it will slide away from here.

12:37 - The Oregon Senate race looks close, but almost all the vote left out is coming from Portland and Lane counties, so it'll be very difficult for Smith to make up anymore ground.

12:31 - McCain has closed the lead in Montana for Obama to less than 500 votes, but there's still a lot of votes out.

12:28 - VA-02 and VA-05 are both very close with the Democrat ahead for a pickup, but are within a few thousand votes, and both worth watching.

12:16 - We hadn't called it yet, but NV will be an east Obama pickup, putting it at 364 for Obama and 156 for McCain (had some addition problems earlier).

12:08 - We are now comfortable calling North Carolina for Senator Obama, his small lead is enough, there is simply not enough Republican vote left to catch him. EV vote totals: Obama 359, McCain 156.

11:59 - Obama's speaking, so I'll take a short break. if I find anything breaking, I'll post it.

11:50 - There seems to be not much movement right now, we're mostly waiting for some small areas in some states like NC and GA. GA early votes have been counted, making the run-off a strong possibility.

Looks like we can call HI, WA and OR also for Obama. North Carolina looks nearly dead even, and we're not comfortable with a call there.

Congratulations, President Elect Obama!

2008-11-04T23:01:00.001-05:00

With the closing of California, and it's 55 electoral votes, we can officially put the junior senator from Illinois over the top, and declare him the next president of the United States.

10 PM closings

2008-11-04T22:02:00.010-05:00

10:58 - Based on the location of the remaining vote, Lake County and the Gary area in IN and the Farifax County area in VA, we are calling both states for Barack Obama, putting him over the top, and making Barack Obama the President elect of the United States!!!

10:53 - The remaining battlegrounds look like VA, NC, IN, MO, MT, NV and NE-02's 1 electoral vote. We expect easy calls on the presidential level for the other states yet to open.

10:47 - Obama is looking good in VA, but has fallen behind in NC by 16K votes, with Charlotte out. In Indiana, still nothing from Lake.

10:38 - A brief EV correction, we have it as 248-145 Obama.

10:32 - We're calling Arizona, John McCain's home state for him, putting him up to 143.

10:22 - North Carolina is closing hard, but mos of the Republican areas are in, so I expect the lead to widen.

10:14 - A delayed call for NM, gives 5 for Barack Obama, giving 239 for him. Obama looks very solid.

10:03 - We can make a further call for Barack Obama in the state of Florida, another 27 EVs that breaks John McCain's back. Obama 234, McCain 133 is the total to date.

We can now call Iowa, Idaho and Utah, the first for Obama and the second and third for McCain. 7 for Obama and 9 for McCain.