Problem retrieving data from Twitter

Cantor Loss shows Crowdsourcing, not Polling, is the Future of Prediction

Eric Cantor, the 2nd most powerful person in the House of Representatives, lost in the Republican Primary today to the relatively unknown Dave Brat. While others have focused on the historical nature of the loss, given Cantor’s position in his party, or on the political ramifications, I was most intrigued by the fact that polls conducted recently predicted Cantor would win by 34 points or 12 points.  In the end, Cantor lost by more than 10 points.

How did the polls get it so wrong?  In an age where people are used to blocking out web ads, fast forwarding through commercials, and screening their calls, using automated phone technology to ask people who they will vote for and assuming that you’ll get an unbiased sample (e.g. people who answer such polls don’t differ from those who do not answer automated polls) seems unwise.  The first banner ad got 44% clickthrough rates, but now banner ads are only clicked on by a small minority of internet users.  As response rates fall, bias is inevitable.

Pollsters may try to weight their polls and use new techniques to produce more perfect polls, but non-response bias will only get worse as consumers learn to block out more and more solicitations using technology.  On the other hand, a good crowdsourcing algorithm, such as the algorithm we use to produce Ranker lists, does not require the absence of bias.  Rather, such an algorithm will combine multiple sources of information, with the goal being to find sources of uncorrelated error.  In this case, polling data could have been combined with the GOP convention straw pollthe loss of one of his lieutenants in an earlier election, and the lack of support from Republican thought leaders, to form a better picture of the election possibilities as the non-response bias in regular polling is a different kind of bias than these other measurements likely have, and so aggregating these methods should produce a better answer.

This is easy to say in hindsight and it is doubtful that any crowdsourcing technique could have predicted Cantor’s loss, given the available data.  But more and more data is being produced and more and more bias is being introduced into traditional polling, such that this won’t always be the case, and I would predict that we will increasingly see less accurate polls and more accurate use of alternative methods to predict the future.  The arc of history is bending toward a world where intelligently combining lots of imperfect non-polling measurements are likely to yield a better answer about the future than one attempt to find the perfect poll.

- Ravi Iyer

The Moral Foundations of Environmentalists

I was recently asked about the Moral Foundations scores of those who are more concerned about the environment and so I analyzed the 15,522 individuals who took the Moral Foundations Scale on and also answered a question on the Schwartz Values Scale concerning how much of a guiding principle of their life it was to “Protect the Environment”.  I limited this analysis to those who placed themselves on the liberal-conservative spectrum, so that I could also control both for ideology and extremity of ideology, to some degree.  The results (beta weights controlling for other variables) of the regression analyses, predicting a desire to “Protect the Environment”, are below.

Moral Foundations of Environmentalism

My initial intuition was that ideology would be the greatest predictor, given how political the issue has become, but it appears that the Care/Harm foundation actually predicts as much unique variance as ideological identification.  From an intuitionist standpoint, this makes sense as the specific care you feel for Polar Bears may drive one’s values more than more abstract concerns about the ocean’s water level, similar to the way that charities appeal to emotions with specific cases of need as opposed to statistics.  Still a great deal of variance is indeed predicted by which ideological team you are on.

Also interesting to me was the significant, but small, negative relationship between ingroup loyalty and attitudes toward the environment.   The item I used from the Schwartz Values scale is part of a subscale designed to measure Universalism, which relates to Peter Singer’s idea that we should expand our moral circles.  While it is certainly possible to care both about one’s smaller circle/family and one’s larger circle/animals/trees, there is some tension there, especially in a world with limited resources where environmental choices that benefit the world at large, may negatively impact one’s local community.

There are certainly limitations to these results taken from a particular sample, so take them with a grain of salt.  And there remains a healthy debate about which moral concerns are more central, so there certainly are moral concerns that may predict environmental attitudes that are not measured here.  Still, these results converge well with what we see in the world.  Environmentalists tend to be liberals who are particularly concerned about the welfare of distant others, perhaps expanding their moral circle to include animals, oceans, and trees
- Ravi Iyer

On Mitt Romney and The X-Files

Those of you interested in political psychology and data science might enjoy my latest post on the Ranker Data Blog entitled Mitt Romney Should Have Advertised on the X-Files.  In it, I explore correlations between liking Mitt Romney and liking various TV Shows on lists on, replicating analyses which the Obama campaign purportedly conducted in the last campaign season, and finding that the X-Files and Mitt Romney have a surprising correlation.  From the post:

As you can see, the X-Files appears to be the highest correlated show, by a fair margin.  I don’t watch the X-Files, so I wasn’t sure why this correlation exists, but I did a bit of research, and found this article exploring how the X-Files supported a number of conservative themes, such as the persistence of evil, objective truth, and distrust of government (also see here).  The article points out that in one episode, right wing militiamen are depicted as being heroic, which never would happen in a more liberal leaning plot.  Perhaps if you are a conservative politician seeking to motivate your base, you should consider running ads on reruns of the X-Files, or if you run a television station that shows X-Files reruns, consider contacting your local conservative politicians leveraging this data.

- Ravi Iyer

The Gaza Conflict and Being Pro-Peace rather than Anti-War

One of my favorite Mother Theresa quotes is: “I was once asked why I don’t participate in anti-war demonstrations. I said that I will never do that, but as soon as you have a pro-peace rally, I’ll be there.

The current conflict in Gaza between Hamas and Israel requires the thoughtful liberal to navigate a few seemingly conflicting thoughts. On the one hand, liberals generally believe that peaceful means are more effective than military means at achieving long term success. This often manifests itself in opposition to military action, such as the Iraq war, Vietnam war, etc.. On the other hand, there is no country or government that would tolerate missiles being launched at their large civilian populations and the Israeli response to missiles being launched from Gaza is a response that every nation would take if in the shoes of the Israelis. Defending civilians against attack is just.

The point of this blog post is to point out that you don’t have to choose between being pro-peace and remaining anti-war, as these attitudes, while related, are not perfectly correlated. In a paper that is forthcoming in the journal Political Psychology, we found that you can find meaningful differences in what being pro-peace and being anti-war predict.  Being pro-peace relates to caring about others, while being anti-war is related to attitudes toward authority, for example.  The multi-dimensional nature of peace-war attitudes is reflected in the real world in the above quote by Mother Theresa, and by the assertions of soldiers and politicians everywhere that their ultimate goal is peace.  In the current conflict, it is perfectly reasonable to believe that the Israelis have every right to defend themselves against attack (therefore not being anti-war), while also faulting both Hamas and the current conservative Israeli leadership for not pursuing peace more vigorously (therefore being pro-peace).

Indeed, I’m writing this in part as a response to a beautifully written essay by Jessica Apple, a writer who lives in Tel-Aviv, which ends:

And as Israel pummels the Gaza Strip, there is no Israeli political leader saying, as Rabin did, “Enough of blood and tears.” [the leader of the opposition party] has, in fact, supported the government’s actions as just, without questioning whether they are wise…..I do agree that Israel has the right to protect its citizens. But I condemn Israel’s current leaders for failing to recognize that the best defense is peace.

The full essay is well worth reading. I pray for the welfare of all the innocent people caught between forces beyond their control in the region and hope to see peace prevail before it is too late for both sides’ welfare.

- Ravi Iyer

Bill O’Reilly, Sarah Palin and Paul Krugman need to get out of Maslow’s Basement.

Losing an election is tough and I have immense empathy for those who have a heartfelt vision for their country that was not fulfilled on election day.  Most people who care deeply about the election, Democrats and Republicans, do so out of a real desire for the country to do better and it’s unfortunate that the results have to disappoint so many well-meaning people.

That being said, there are some conservatives who have implied that those who vote for Obama simply want free stuff, while some liberals imply that billionaires who support Romney do so out of self-interest.  Consider this quote from Sarah Palin:

We’re not explaining to the rest of America, who thinks that they’re going to get a bunch of free stuff from Obama, that you have a choice. You either get free stuff or you get freedom. You cannot have both, and you need to make a choice.

Or consider this quote from Paul Krugman:

billionaires have always loved the doctrines in question, which offer a rationale for policies that serve their interests….And now the same people effectively own a whole political party.

And in reaction to the election results, Bill O’Reilly opined that the reason that Obama gets support is that

There are 50% of the voting public who want stuff.  They want things. And who is going to give them things? President Obama.

What these three quotes have in common is that they all make a common  mistake about how we view the motivations of others.  Chip and Dan Heath call this “getting out of Maslow’s basement”, which refers to Maslow’s hierarchy of needs depicted below.

Maslow’s idea was that motivation can be grouped from lower level needs such as wanting “stuff” to higher order needs like caring about others, fulfilling values, etc.   The implication of O’Reilly, Palin, Krugman, and many partisans, is that the other side is motivated by these lower level needs.  It is a common mistake, made in many domains to believe that others are motivated by lower level needs. Chip and Dan Heath have shown that we all assume that other people are motivated by lower level needs, but that we ourselves are motivated by higher order needs.  The truth is that most everyone is actually motivated by higher order needs. In the below video, they explain one of many studies showing this.

It is easy to let partisanship help you impugn the motives of others.  And there is no doubt that some amount of self-interest helps shape our values.  However, most people who care enough to vote do so out of higher order considerations.  Indeed, nobody stands in an 8 hour line to vote out of self-interest. They really do want to help the poor or promote economic growth and freedom.  And if we ever want to fulfill the bipartisanship we desire in the world, we would do well to understand the sincere motivations of others.

- Ravi Iyer

Early Voting is a Social Influence Tool, so tell everyone when you vote!

As a political junkie, I’ve been reading the spin on early voting with interest as each side talks about how they are using their ground game to get people to the polls.  Some have suggested that early voting doesn’t matter as it simply gets those people who would have voted anyway to vote earlier.  That may or may not be true, but I believe that such analysis is missing one of the most important aspects of early voting.  Specifically, it creates a longer period for social influence to influence voter turnout.

One of the more interesting practical findings in social psychology is the finding that injunctive norms (e.g. telling people that they should vote) often do not work as well descriptive norms (e.g. telling people that everyone else voted).  In work made famous by Robert Cialdini, psychologists have found that telling people not to litter works quite badly if they perceive that litter is common.  This paradigm has been put to practical use in hotels which often tell you about the towel recycling behavior of other guests, rather than just asking you to recycle your towel.  The below graph shows the rate of towel recycling given injunctive vs. descriptive norms.

Does this principle apply to voting?  Absolutely!  Indeed, in the largest study I’ve ever seen, involving 61 million Facebook users, the Facebook data team successfully increased voting, verified by actual public voting data, by giving users the opportunity to see which of their friends had voted.  However, this occurred over a single day, where users had to login to facebook, report their voting, and have their friends see it in time to affect their behavior.  Imagine if social influence could occur over a period of days or weeks.  This is the true power of early voting.  It creates an environment where voting becomes the norm and who wants to be the person left out?  So even if early voting turnout efforts are bringing in people who would have voted anyway, it’s likely that their behavior affects others.

Most polls show that the more people who vote, the more likely Obama will win this election, as polls of registered voters tend to show higher support for Obama than polls of likely voters, given that conservatives tend to vote more consistently.  Obama wants more people to vote.  As such, it makes sense that Obama’s campaign would promote early voting as they need to create an environment where the normative thing to do is to vote.  Everybody is doing it and who wants to be the one left out.  In contrast, Republicans do better in low turnout elections since their supporters tend to be more consistent voters.  It is no accident that Democrats consistently want polls to be open longer, while Republicans often resist.  As such, it is questionable whether Republican efforts to get conservatives to vote early are useful, especially if the voters are people who would already vote anyway.  The effect of an environment where everyone is voting is likely to stimulate voting amongst groups that traditionally vote less often and lean Democratic (younger voters and hispanics).

For people who want to affect the vote, the task is clear.   If you are a Republican, it might pay to be more subdued about your voting, and share it only with those who you know agree with you.  If you are a Democrat, tell everyone you can about your vote to broaden the electorate, and take advantage of early voting to tell people that you voted over a longer period of time.

- Ravi Iyer

ps. I voted two days ago.  It felt good. :)

Be the Bipartisanship you want to see in the World by Supporting our next President

The point of politics is to make people’s lives better.  Liberals and conservatives may disagree about how to do that, but despite the heated rhetoric, there are a number of broad goals that most anyone would agree upon.  Consider a survey we recently conducted concerning liberal and conservative preferences about the kind of place they would like to live.  While there are differences in terms of priorities, the top 5 desirable attributes are largely the same.  Everyone wants a strong economy, safety, clean air and water, and good medical care.

Top 5 Desirable attributes in a city by ideology:

Unfortunately, roughly half of the country is going to be disappointed by the results of the next presidential election.  Both history and psychology tell us that this disappointment will likely lead to some amount of demonization of whomever wins, reflexive opposition, and incivility.  This may lead to outcomes that nobody wants, such as what occurred during the debt ceiling negotiations.

Thoughtful liberals, conservatives, and both presidential candidates have talked about the need to transcend partisanship in order to attempt to create better policy and a better country.  The results of the next election are likely to disappoint some of these thoughtful people, yet it also represents an opportunity for them to be the change they wish to see in the world, by consciously resisting the impulse toward demonization and reflexive opposition.  It represents an opportunity to back up words of bipartisanship with action, at a powerful moment when everybody will expect the opposite.

Supporting our next president does not mean that you need to support their policies.  We can disagree without being disagreeable.  But supporting our next president does mean that we hope they succeed at goals that we all share such as creating a safer, cleaner, healthier, and more prosperous world.  It means hoping that the unemployment rate goes down, not up.  It means hoping that the poor receive the help they need, whether by charity or government, and that terrorism is stopped, whether by military or diplomatic means.  Whomever wins, let’s support them by truly hoping they succeed at our shared goals.

If this resonates with you, consider joining our facebook group and pass this message on to your friends.  Positive change always starts with small groups of people who believe in something.

How Coherence Defines Conservatism

One of the pitfalls in doing political psychology research is that it is tempting to define an ideology using the perspective of whatever you study.  Researchers necessarily (and I’m sure I do this too) talk about the novelty and uniqueness of their findings in order to convince editors of journals of the objective importance of their work.  In my technology career, we often think of connected variables as part of a “graph”, indicating that any individual finding is likely part of a larger pattern.  I believe that there are a number of psychology findings and news stories about conservatives that are actually part of a larger pattern, where each finding is actually an example of how conservatism can be defined by a desire for greater coherence.

What is coherence?  It is an idea that grows from the common psychological finding that cognitive dissonance is unpleasant, so people seek to create the absence of dissonant thoughts, beliefs, and emotions in their lives.  This absence of dissonance is what we can call coherence.  My graduate school advisor, Stephen Read, has studied it extensively in a variety of contexts, and, in a project led by my colleague Brian Monroe, modeled a variety of social psychological findings about attitudes.  My suggestion in this blog post is that, in a similar fashion, a large number of observations about conservatism can be explained by the idea that conservatives seek more coherence than liberals.  Below, I will list these observations and you can judge for yourself whether there is a broader pattern.

It bears noting that most of the above differences can be framed as positive or negative, depending on one’s ideological desires.  Coherence, by itself, is neither good nor bad, and can be both adaptive and maladaptive in different situations.  One of my colleagues once said that there is value in reviewing research from a particular perspective and pushing that review as far as one can go, even if one might be wrong.  There is certainly a ton of research I am unaware of and perhaps there is research showing contradictory evidence for my conclusion that conservativism is defined by coherence.  Or alternatively, perhaps readers are aware of more research on liberal-conservative differences that can be explained through the lens of coherence.  I would appreciate either type of information via comment or email.

- Ravi Iyer

Escaping Cycles of Extremism: What the Libyan Mob and Sam Bacile have in common.

The recent killing of Americans in Libya by a mob of protestors who were responding to an intentionally offensive youtube video, created by Sam Bacile and friends, illustrates a fundamental truth.  Extremism begets extremism.  Killing begets killing. Violence begets violence.

It is a truth that directly relates to the cycles of incivility that we see in American politics and a truth that social psychologists often study, because group level reactions to conflict, extremism, violence, and incivility/demonization are fairly predictable; they incite more of the same.  Indeed, there is clear evidence that Sam Bacile, Terry Jones, Osama Bin Laden, Charles Manson, and other extremists understand this implicitly and commit their extremist acts with the idea of inciting a wider war.  In this case, a desire for a wider conflict is what the Libyan Mob and Sam Bacile have in common.  Psychology research backs their methods.

Given the reliability with which extremists can create cycles of violence, it remains imperative that those of us who want reduced extremism, incivility, and violence realize the situational causes and consider how to frame things as a cooperative goal of moderates vs. extremists, instead of a Muslims vs. the West frame that extremists on both sides would prefer.  It’s an imperative that Martin Luther King put as follows:

Hate begets hate; violence begets violence; toughness begets a greater toughness. We must meet the forces of hate with the power of love… Our aim must never be to defeat or humiliate the white man, but to win his friendship and understanding.

The ultimate weakness of violence is that it is a descending spiral, begetting the very thing it seeks to destroy. Instead of diminishing evil, it multiplies it. Through violence you murder the hater, but you do not murder hate. … Returning violence for violence multiplies violence, adding deeper darkness to a night already devoid of stars. Darkness cannot drive out darkness; only light can do that. Hate cannot drive out hate; only love can do that.

- Ravi Iyer

ps.  crossposted from

Which undecided voters use the internet to figure out who they should vote for?

In 2008, I co-founded as a way to help people with the question “Who should I vote for?”  In 2008, it served over 500,000 people, but we didn’t get any demographic information at the time, so, while valuable, I couldn’t answer many of the questions I wanted to answer about the use of “candidate calculators”, which is a name sometimes given to sites that allow you to enter your political opinions into a website, which then attempts to match your opinions to those of political candidates.  In 2012, I added a few optional questions to the end of the quiz that asked the age, gender, political ideology, and planned candidate choice of quiz takers.

Right now, we rank #2 or #3 (I’ve seen both) for the search query “who should I vote for” on Google and according to Google, about 15,000 people have searched for that query over the past 30 days, with about 5000 clicking on  Some number of people do not fill out our surveys (25% bounce rate) and of those, only 30% or so fill out our optional demographics questions.  Browser referral information isn’t always sent, so I can only identify 470 who definitely typed in “who should I vote for” into Google to come to our site during the 2012 election, the bulk of which occurred in the last 30 days.  Still, I think it’s perhaps indicative of the kind of person who searches the internet for voter information.

Who is this person?  The average age was 30.3 years old (SD = 11.1), with people as young as 12 and as old as 87 taking the survey.  56% of quiz takers were male.  Judging by the below charts, the average person who asks Google who they should vote for really is likely to be undecided and moderate.

What do these voters care about?  In order to eliminate the effects of the liberals and conservatives, I looked just at the 201 people who said they were moderate or apolitical.  Here is the list of issues they care about in descending order of importance.

Issue Importance

And here are their stances on these issues, with questions they agreed to listed first, and questions they disagreed with listed last.  Note that these questions were asked on a 7 point scale with 1 = strongly disagree, 4 = in the middle, and 7 =  strongly agree.

Statement Agreement

Statement Agreement

What can we conclude from these analyses?  It seems like the kinds of people who are asking for help on the internet are people who might be classified as populists.  They appear to be mainly younger men, who want compromise in government, favor liberal policies like higher taxes on the wealthy, higher spending on education, and more corporate regulation, but also favor conservative policies like stricter immigration enforcement and stricter controls on government spending.  Of course, perhaps taking the average of these undecided voters obscures differences among these voters.  Also, these results are likely to generalize best to the types of individuals who are actively using the internet to figure out who to vote for, since our sample all typed in “Who should I vote for?” into Google and then took the VoteHelp quiz.  On one promising note for these analyses, these results do seem to converge with the media’s depiction of the voters who both campaigns appear to be trying to woo right now.

- Ravi Iyer