• Human beings have some degree of free will which confounds reproducible results. Some will dispute this, and maybe humans are more constrained than we might think, but that is a far cry from believing that human behavior is fully determined.  In contrast, a computer is actually incapable of producing a random number that is not fully predictable, though it can measure an ostensibly random event outside of itself and report that as a random number. If psychology or neuroscience ever produces a machine that can predict which number a person will pick from 1-100 with 100% accuracy in the same way that you can predict a machine's decisions (if you know the inputs), I will retract this paragraph, but I doubt that will ever happen.  As long as you allow for some degree of free will of the subject matter, the scientific paradigm of reproducibility breaks down.  Imagine if the gas in our car could decide not to combust or the electricity in our refrigerator could decide not to flow in the direction intended.  How reliable would our cars and appliances then be?  Physicists should be quite thankful that atoms are not so willful.
• We care about individual human beings, not the "average" human being. When we do studies on electricity and figure out a method to conduct electricity with 10% less loss of energy, without any side effects, we have undoubtedly made a worthy discovery.  It doesn't matter if the same intervention causes some energy that would otherwise be captured to be wasted, as long as the net effect is positive.  Energy is fungible.  In contrast, human beings are not.  If we conceive of some intervention that makes 10% of people more happy, undoubtedly it will make x% of people less happy.  All psychology studies are full of people who react in the opposite direction to the published results.  For example, people generally conform to social norms...but we all know that there are people who do the complete opposite and rebel.  This can be mitigated by measuring individual differences, but the problem is that human beings are both the consumers and the subjects of our research.  People care about what makes them themselves happier, not the "average" person.  Since psychologists cannot say with certainty what will happen to any individual person, the consumer of psychological knowledge has to filter anything said to them through the filter of their own individuality, in order to derive utility.  While our results may often be reproducible on groups (despite free will), they never encompass 100% of people and so there will always be some degree of subjectivity to a result.  For this reason, a memoir where a person can be more certain of the applicability of a phenomenon, due to the deep description of a situation, really can lead to more useful insight than a study that describes what scientifically happens to an "average" person, from the perspective of a knowledge consumer.
• Human beings are highly evolved social animals that can intuit complex things about other human beings. The observer article highlights the findings of this probability sample survey where it was found that people believe that psychology is a science, but also that everyday life provides training in psychology (see below chart).  In both the original study and the APS article, these statements were presented as somewhat contradictory pieces of evidence, but I don't think this is necessarily true.  I believe both are true.  People intuitively know a lot about how people work.  Their perceptions may be biased by experience, but then again psychologists are biased by sampling in the same way.  I firmly believe that anyone who believes that you can't learn something profound about the human condition from a good book hasn't read a good book (or is blinded by their own biases).  I cannot see how psychologists can be truly useful to the world as long as we believe that our methods are the only ones that speak to questions about human psychology and ignore outside wisdom, much of which is more advanced than what we study.

Personally, I agree with the survey respondents in the above study.  Psychology is a science and psychologists are among the best scientists in the world, if only because we have to deal with a subject that is willful, variable, cares about itself, and where the bar is so much higher for producing marginal knowledge above that of the average person.  Psychologists should be applauded for that.  Yet, I believe that trying to tell people that our image as a discipline rests upon the belief that what we do is very similar to what physicists and chemists do will always fail because the above three reasons are self-evident to most consumers of psychology.

Psychology is what it is.  Trying to sell it as more may lead psychologists be less trusted, as we seem more out for our own gain than for doing something useful.  It may also lead psychologists to adopt unhelpful procedures, designed to prove our scientific-hood, as opposed to being of use to others, and/or to ignore those methodologies seen as less "scientific" (e.g. qualitative studies or narratives), in order to be associated with methodologies seen as more scientific (e.g. neuroscience, as in the article), even when we could learn something from both.  There is truth in any view of the world, and the best truths are arrived at from a variety of angles, in the same way that the use of many measurement techniques reduce overall error.  According to the above graph, under 50% of people (in 1986) believed that psychology has an impact on their daily lives.  I absolutely believe that psychologists should continue to use the scientific method as best we can.  However, I also believe that psychologists may actually make more of a real impact if we realize the inherent limitations of our subject matter and stop worrying so much about whether we are grouped with the physicists or the sociologists.

- Ravi Iyer

Unfortunately, the world isn't necessarily organized in that fashion.  Few would believe there are such categorical distinctions.  Prospective dates have some degree of positive and negative qualities, rather than attributes being merely present or absent.  Are people either qualified or not for a job?  Most people instead belong along a continuum of professional ability, with some being very qualified (way above being merely adequately qualified) and some people being just below and just above the border of qualification.  Politicians aren't uniformly liberal or conservative and we routinely see partisans on both sides upset at those who aren't extreme enough and who toe the partisan line.

This may seem obvious, but the reason I bring it up now is that while most everyone would agree with this fact, when thought about more carefully, still many people continue to argue as if things are categorical.  There are two recent examples on the yourmorals blog.

First, the comment section of this post has become a debate (for many) over whether psychology is objective (science) or subjective (art).  Allow me to quote Gene, from this thread:

there is SOME objective knowledge that comes from psych research (anything that can be experimentally shown, is predictive, even if only statistically, it has value).

If you want to get really nitty gritty, even physics is not completely “objective”…it’s merely instrumental to understanding objectivity (see here: http://en.wikipedia.org/wiki/Instrumentalism)

Most things are not completely objective or completely subjective, especially where human affect, behavior, and cognition is concerned.  Yes, psychology is less objective than physics...but it's more objective than sculpture.  If I think that Paul McCartney sings better than I do, is that an objective or a subjective fact?  It's objective in so far as a survey of people would detect a very large statistically significant difference between perceptions of our singing.  But it's subjective in so far as it may not be true for a particular person (e.g. my wife and my mom).

What complicates things further is that many people who read psychology don't really care about what happens to most people, but rather how the research applies to them.  Consider this very useful overview of how changing our consumption patterns can make people happier.  One of the recommendations is something that I tell people often, that experiences lead to more happiness than material things, an opinion shared by 57% of a national sample (and shown to be true for most in experimental research).  Yet, 34% of those people disagree (and some don't benefit in experiments).  So is the statement that "buying experiences leads to more happiness than buying things" an objective or a subjective fact?  It's true for a majority of people, but not for a significant minority.  It's likely true for many groups, but certainly not all groups.  Yet many people still think we can definitively decide if psychology is objective or subjective, even though humans, unlike inanimate objects, don't react predictably to situations, except perhaps in aggregate (e.g. we have free will or at least the illusion of it).  I can find truths that apply to all rocks or all electrons, but not for all humans.  But I can find truths that apply to many humans or most humans, and that might give someone insight into themselves, which is a valuable thing.

A second instance of categorical thinking on the yourmorals blog of late is Pigliucci's critique of Haidt's recent SPSP speech.  Haidt pointed out that there is underrepresentation of conservatives in social psychology compared to the population and cites both self-selection and discrimination as issues to varying degrees.  Many people (understandably) focus on the sexier charge of discrimination, and Pigliucci answered that he "suspect(s)  the obvious reason for the “imbalance” of political views in academia is that the low pay, long time before one gets to tenure (if ever), frequent rejection rates from journals and funding agencies, and the necessity to constantly engage one’s critical thinking skills naturally select against conservatives."  But what if causality was continuous and not categorical.  Pigliucci may be entirely right about his obvious reason, yet there still could be some amount of discrimination.  Indeed, if there is one student somewhere whose ideas are supressed (and there was at least one in Haidt's talk), then there is at least some degree of both self-selection and discrimination, meaning that a debate over what statistically causes underrepresentation misses the point.  Bear in mind that these are not just data points, but actual human beings.  One human being discriminated against is one human being we could serve better, even if the vast majority of under-representation is due to self-selection.

I'm obviously biased in the above debate, but these thoughts are not a response to that debate, but rather a response to almost every debate and decision I see in psychology.  Some other things that are continuous, and not categorical:

Journal Publication - Editors have to make categorical decisions to accept or reject papers, yet many papers that are accepted never get cited, while other papers are published through sheer persistence down the chain of  journal prominence.

Statistical Significance - A 94.9% chance of being right is not that different than a 95.1% chance of being right, yet it is treated as a categorical distinction called "significance" because we need to be able to say whether something is true or not, when in reality, all we have is some evidence toward the truth, that varies to some degree.  Even the best paper does not definitively prove anything and even the worst paper is some evidence toward something.

Authorship - Many people work on papers (often undergraduate research assistants) and are not authors, while others do fairly little and receive authorship.  Sometimes the first author does 90% of the work and sometimes they do 51%.  Yet they still receive the categorical distinction of first author.

Psychological conditions - Few psychological clinical conditions are categorical.  In reality, people have some degree of anxiety, rather than having or not having an anxiety disorder.  Yet, for insurance reasons, people have to be diagnosed categorically as having a particular condition.

Psychological constructs - Is shame the same as guilt or different?  Is shame the same as sadness?  Is shame the same as happiness?  The truth is that shame is somewhat like some of these constructs and less like others of these constructs.  Categorical distinctions between such constructs are useful for publications, but don't really reflect the continuous nature of the real world.

I am sure that if I thought more, I could come up with many more examples of things that are continuous, but treated as categorical. In academia, perhaps we can eventually change our systems, leveraging technology, to acknowledge the continuous nature of things.  My real-world hope, as someone who believes that a world with less conflict is better than a world with more conflict, is that perhaps seeing things as continuous, rather than categorical, means that people will be less likely to make harsh judgments of others based on the idea that their beliefs are the categorical caricatures that we make them out to be.

- Ravi Iyer

As someone who studies political psychology, I have two main self-serving thoughts.  First, findings in political psychology would support the idea that most of this is due to self-selection.  We know that liberals score higher on measures like openness to experience, challenging the status quo, enjoying effortful thinking, having existential angst (searching for meaning) and placing a value on stimulation.  All of these findings are published and replicated in our YourMorals dataset.  These are all traits that can be framed as positive (enjoying new things, wanting to be an agent of change) and negative (disrespecting tradition, being narcissistic) in the 'real world', but are useful in academia.  Personally, I could be earning more money and likely doing something more objectively useful, but I like the stimulation of working in the world of ideas and it helps ease my existential angst.  This cluster of traits describes some part of most academics I know.

If you see the actual talk (video below), you'll notice that Haidt presumes a fair degree of self-selection and does not set representativeness (e.g. 40% conservatives in the US means we should have 40% in psychology) as a goal, perhaps for this reason.

Still, much of the talk is about discrimination (e.g. the analogy of the closeted homosexual) and so I see why many bloggers might have picked up on the discrimination angle.  I am not saying that there is not some peer pressure exacerbated by the assumption that everyone in the room is liberal...but my experience is that self-selection causes that environment more than the reverse.  That does not mean it isn't a problem.  It is and we should do something about it.

The main problem, from the perspective of someone who wants to understand political attitudes and ideology, is that it's really hard to study something you have no experience with.  Imagine what a collective of non-parents would think of parenting from a completely outside perspective.  Giving up sleep, friends, leisure, and money for an infant that cannot even smile might seem delusional, which is exactly the way that some psychologists see conservative ideology...as a product of some kind of mental fault.  It is only from the inside that sometimes things make more sense.

Those of us who study ideology often have nobody on the inside of conservative movements to help us make sense of them.  It is for that reason that I'd love to see more research conducted by conservatives.  Conservatives don't just have different perspectives on politics, but also in all sorts of other domains.  Until then, I'll have to settle for befriending them wherever I can and plying them with liquor to get their inner thoughts.  As a liberal who wants to persuade conservatives, such understanding is essential, unless I simply want to cheerlead amongst people who already agree with me.

In some ways, it's part of a larger problem in psychology where we ask relatively inexperienced (outside of academia) individuals to theorize about the nature of human experience.  Business school students are expected to have business experience to get into business school, yet social psychologists often have very limited experience with human social life before investigating it.  Given that, is it any wonder that many people feel that memoirs offer as much insight into the human condition as psychology journals?  Having a diverse set of experiences and perspectives within political psychology can only make our work that much more interesting.

- Ravi Iyer

ps. you can read Jon's official piece along with many reactions of other more prominent psychologists on Edge.

Besides dovetailing with my research, I think there is a practical value to be taken from this video. I often find myself concentrating on what I am doing, sometimes forgetting why I do things. In a world where we all have too many paths to choose from, we sometimes choose the path that has the most urgency (extrinsic motivation) rather than the path that is the most meaningful (intrinsic motivation). In business, that might mean doing whatever generates a profit now, rather than what satisfies the business' core mission. In academia, that may mean writing a paper for publication sake (extrinsic reasons) rather than exploring ideas that may not just get published, but also may serve some larger purpose. If you are inclined to explore these theories/ideas further, I might read more about self-determination theory, which talks about how intrinsic, rather than extrinsic motivation, leads to better human functioning, in addition to the benefits described in the above talk.

- Ravi Iyer

What interested me was not the data, but the critique of how psychologists attempt to illuminate the human condition.  A few quotes from the chapter summarize the points I'd like to emphasize.

Psychologists often study phenomena in isolated, artificial environments, which allows researchers to necessarily isolate variables of interest, but....

Aiming to isolate phenomena, scholars in this research enterprise are prone to devise somewhat peculiar and overly constrained assessments of moral functioning that are remote from everyday moral experience.

A gold nugget in Gilligan’s (1982) critique of moral psychology was her skepticism concerning such constrained dilemmas and her advocacy for assessing moral judgment more naturalistically, tapping moral problems from individuals’ own experience.

Another paradigmatic assumption to which we draw attention asserts that people are psychologically “cut from the same cloth,” uniformly operating by the same moral psychological

processes. This assumption is manifest in the frequent reliance on a single type of research participant (e.g., undergraduate students garnering course credit), a lack of consideration for

individual differences, and a homogenizing “people” label.

Our proposal contends that lab experimentation should be balanced with real-world observation of socially significant affairs and that morally relevant aspects of personality should

be tapped across all levels of personality description. Different methodologies should be mutually informative. Multiple lenses on the same phenomena contribute to a more comprehensive understanding, whereas divergent findings across methodologies hearken our attention.

So what can psychology tell us about moral reasoning that literature and the humanities, or simply reading the newspaper thoughtfully, cannot?  I would say not much, but rather that psychology can help buttress what can be learned by other methods and vice versa. They both get at the same questions. A colleague of mine once shared that he thinks of psychology studies as statistical parables, in the same way that stories of the real or fictional world provide us with different kinds of parables. Anyone who has read a really good novel might believe Ralph Waldo Emerson's quote that "Fiction reveals truth that reality obscures."

The authors I quote above want us to use multiple lenses to understand the human condition, referring to the lenses that psychologists might use (different samples, different methods). I would further extend that analogy to all fields that attempt to understand the human condition, such as literature and the humanities, but also just reading the news. This is not to say that there is not something powerful about quantitative analysis and methodologically rigorous psychological research. But as I step back from the research, I find that I'm only convinced by findings where there is a web of evidence, of the type that one researcher, paper, study, method, or discipline, could never produce...where the statistical parable has been replicated in other ways by other people and is echoed in situations I've faced and news stories I've read about. Fortunately, the internet and semantic web technologies promise to make it easier to discover such webs of evidence...but that's a subject for another post.

If you have the patience, it's worth reading the results of the conference in Herzilaya, but if not, perhaps I'll make a practice of summarizing some of the other chapters as I read them. Social psychology can be unfortunately unintelligible, in ways that literature is not.

- Ravi Iyer

Academic publishing is not easy. In psychology (though we'd be happy to publish outside of psychology), it's not enough just to have a valid results, but the results often have to be novel as well. Therefore, many replication studies may not be publishable or may only be publishable in lesser known journals or just on this blog. That doesn't necessarily make that endeavor unworthwhile, as replication, or the failure to replicate, is an essential part of the scientific method, but we want people to know what they are getting into. We're open to anyone who is motivated to publish in peer reviewed journals, and there is no inherent reason that limits this to academics. However, it's a labor intensive process with no monetary reward, so it's quite possible that only those with an eye toward building an academic CV might be interested.

Here is a running list of potentially publishable results which are in our publication queue, but there are many more possibilities. We are open to proposals on a variety of topics. Some of you might be interested in a specific topic and might find this list of measures useful in determining if we have data on that topic.  Data might potentially serve as the 1st study in a 3 study package where a community sample reinforces the results of a lab experiment, or as convergent evidence in something you already are working on. In rare cases, we may even be willing to collect new data using additional measures, even including experimental methods, if your ideas are compelling enough. However, there are only so many resources we have and the degree of effort required is definitely a consideration, balanced against the contribution which could be made. Also bear in mind that some number of papers are already in progress, and it may be possible that your idea is already being worked on.

If you are interested, please use this form to contact me as it has important questions to be answered. Beginning any publication process is a commitment and we would obviously like to work on projects that have successful conclusions. Thanks for your potential interest.

- Ravi Iyer

VoteHelp 2008 Election Visitors

VoteHelp served hundreds of thousands of visitors, so I'm guessing many millions took similar surveys when you combine traffic from all 2008 election calculators.  Traffic spiked noticeably during decision making periods (Jan-Feb primary and November election) with a low bounce rate, indicating that it served it's purpose of educating the electorate. There is clearly demand for such time saving services.

The ironic thing is that people know far more about presidential candidates compared to other elections. In 2010, how many people know much about local judges, state senators, or even our congressmen. People have better things to do, even political junkies like me, and it is understandable that people rely on partisanship rather than issue positions when making voting decisions. As much as votehelp was useful in 2008, it could be even more useful in 2010 if it could change the equation, such that becoming informed on individual issues was simpler.

However, the task of assembling data was difficult in 2008. We had some funding, but even for one election, the expense of the research was not small. Repeating those methods, even just for congressional races, would be prohibitively expensive. I was hopeful that the convergence of new data sharing technologies (APIs, XML, the semantic web) and databases (open government data sources) might facilitate this process. I subscribe to mailing lists about parsing political data, follow the Sunlight foundation on facebook, and am aware of few organizations like OnTheIssues and Project VoteSmart which track issues, some of which have APIs. Could I combine these projects into a mashup of data that would inform 2010 voters?

Unfortunately, a few days later, I have to admit defeat. There is tons of data out there. But it just isn't complete or meaningful enough. For example, VoteSmart has a wonderful service where they have interest group ratings for candidates.Theoretically, these interest groups could take some of the open government data on votes and create composite viewpoints, based on their issue perspective and reflected in their ratings. However, ratings only exist for prominent politicians like Barbara Boxer and not for challengers like Carly Fiorina (her likely opponent in the California Senate race) or Steve Poizner. Fiorina may not have much of a record as a businesswoman, but Poizner certainly should have some ratings from his other official offices. Further, below is a graph of the interest group ratings which exist for Boxer.

Interest Group Ratings for Barbara Boxer

The vast majority of ratings are either 100 or 0, which leaves little room for nuance. The increasing partisanship we see in washington is reflected in these ratings such that there is little predictive power beyond whether someone is a democrat or republican. Perhaps interest groups, which are necessarily partisan, aren't the best aggregators of knowledge as their views are necessarily extreme and therefore their opinions of legislators are equally extreme.

I don't think the world needs more open government data, at least for informing the electorate in voting decisions. Maybe that helps the press uncover corruption, but what seems more important are objective ways to aggregate data and create meaning out of the tidal wave of public data. Political scientists and psychologists can play a role in objectively extracting meaning from this data, along with web developers and data architects who make this data available. If anybody has ideas on how I might be able to do this for 2010, I'd love to hear them as I would love to work with smart, resourceful people on these issues. Please drop me an email or a comment. Until then, it looks like votehelp will have to wait til 2012.

- Ravi Iyer

A long time ago, I attended my favorite event in Los Angeles, the LA Times Festival of Books, and picked up the book Consilience, by E. O. Wilson. Consilience literally means the "jumping together" of knowledge and Wilson talks about how there is a potential orderliness or unity of knowledge that is possible across academic disciplines.  I was attracted to this book because it captures an overarching theme about how I have come to view the world. Everything I read these days jumps together into some grand puzzle, always incomplete, but not unsolvable, and social psychology, the field I am trained in, is a natural glue (not the only possible glue, but a useful one for me). Almost anything can be studied by social psychologists...culture, health, gender, marketing, politics, morality, sports, poverty, love, justice, religion and death are all prominent topics that social psychologists study. I dare say that list includes most any big question that people care about.

That being said, "if all you have is a hammer, everything seems to be a nail", and psychologists often fall prey to this saying in thinking that the use of statistical methods is the only valid way of examining the world.  A well designed experiment can tell us that something can happen (under often artificial circumstances), which is important knowledge, but a good book about someone's life can tell us unequivocably what actually has happened, at least to one person.  And a lot of times, we care as much about what happens to individual people as we care what happens to the "average" person, as we happen to be one of those individual people.

A Booth at the Book Fair on "Happy Science"

Consider an age old question like 'what makes people happy?'. To be sure, it's a vague question that scientific/quantitative methods can make more precise. But the people at the booth from the book fair in this picture (left) probably have a reasonably well thought out perspective on what makes people happy as well. Philosophers probably think the answer lies in contemplation. English majors might think the answer is revealed by great literature. Neuroscientists and biologists seek answers in brain chemistry. Religious scholars in religious texts. Psychologists randomly assign people to do things and see if it makes them happier. Who has the answer? I would say nobody...and everybody...Just as any psychological finding is made more robust by the convergence of findings using multiple methods by multiple researchers in various settings on diverse individuals...so too is any greater theory about the human condition more easily believed through the convergence of knowledge across disciplines...or consilience.  So I will often write book reviews on this blog linking what I've learned from the book with what I have learned in particular psychology studies.

For those who embrace this convergence, it's an exciting time. People are generating far more quantitative data as every facebook interaction, google search, credit card swipe, & GPS location can be mashed up into some application or graph that provides some evidence of the human condition. People are generating far more qualitative data as well, in the form of countless public blogs, forums, tweets, and facebook posts. Logic, statistics, & the scientific method can be used by people of any discipline to take this wealth of data and produce convergent knowledge.

I still plan to focus on posting graphs about quantitative findings that relate to psychological theories on this blog.  But one of the main purposes of this blog is for me to store my own thoughts. My thought processes about the psychology of anything would be incomplete if I didn't have a place to store experiences that didn't explicitly have any data component to them...the random news article, observation, book review or quote that provides external validity to anything psychologists study. It is one thing to see something in a psychological experiment.  But sometimes you only know it is real when you see the same thing exhibited in a character in a novel, in a quote from a politician, or in an essay by a philosopher.

Book Reviews:

• Tony Hsieh, liberals, and libertarians prefer buying experiences to materialism – A Review of Delivering Happiness
• The Definition of Moral Hazard and A Review of The Big Short
• Why is Warren Buffett liberal on the estate tax? A Review of The Snowball.
• The Present Hedonism Time Perspective of Motley Crue Members, Liberals, and Libertarians
• On Hyperpartisanship, Hypermoralism, and the Supernormal Stimuli of Modern Politics
• Appreciating American Libertarians – Insight from Ted Conover’s Book, Rolling Nowhere
• Psychological Causes of Violence in Sports Riots

Other Consilience posts:

• Tony Hsieh, liberals, and libertarians prefer buying experiences to materialism – A Review of Delivering Happiness
• The Definition of Moral Hazard and A Review of The Big Short
• Why is Warren Buffett liberal on the estate tax? A Review of The Snowball.
• The Present Hedonism Time Perspective of Motley Crue Members, Liberals, and Libertarians
• On Hyperpartisanship, Hypermoralism, and the Supernormal Stimuli of Modern Politics
• Intrinsic, not Extrinsic Motivation Leads to Greater Reward – 2 Theories
• Intrinsic, not Extrinsic Motivation Leads to Greater Reward – 2 Theories
• Appreciating American Libertarians – Insight from Ted Conover’s Book, Rolling Nowhere
• Appreciating American Libertarians – Insight from Ted Conover’s Book, Rolling Nowhere
• Psychological Causes of Violence in Sports Riots
• Psychological Causes of Violence in Sports Riots
• What can psychology tell us about moral reasoning that literature and the humanities cannot?
• Can open government data inform voters in the 2010 election?
• Book Reviews – Consilience between psychology and books I read.
• Book Reviews – Consilience between psychology and books I read.
• Methland by Nick Reding: Moral Maximizing and the Drug War
• What the positive psychology approach can learn from Barbara Ehrenreich’s Bright-Sided
• Gratitude Video from Conan O’Brien and Louis CK
• France to consider measures of gross national “bonheur” (happiness)
• Hyperpartisanship & Obama’s speech to kids in US schools
• Three polls which point to differing underlying fairness principles driven by differing goals
• Democracy Promotion vs. Dignity Promotion

At the same time, I've been working with colleagues on a paper about experiential vs. material purchasing styles, for which we have found convergent correlations all suggesting that experiential purchasers are dispositionally motivated towards seeking new, stimulating experiences to promote positive emotion, while material purchasers often seek to avoid negative emotions. This is supported by the fact that, in the YourMorals.org dataset, experiential purchasers report higher levels of openness to experience, lower levels of neuroticism (both measured by the Big Five Personality Inventory), and lower levels of disgust (as measured by the Disgust Scale). The disgust finding does not necessarily fit with the idea that experiential purchasing is related to seeking new experiences, unless one looks at the literature on disgust. In particular, this study theorized about such a relationship and confirmed it by reporting correlations between disgust and big five personality dimensions.

It occurred to me that I could contribute to the original studies' findings, by examining the same correlations in our dataset, using a more diverse and far larger sample, and perhaps even including some internal cross-validation.  The results are summarized in the table below.

Disgust Scale Correlations with Big Five Personality Traits

The main hypothesis of the original study actually dealt with the two robust relationships found in our dataset, specifically that disgust is negatively related to openness to experience and positively related to neuroticism. In all, these two relationships stand out as robust across groups and in both studies. Interestingly, the correlation between openness to experience and disgust is weaker in the two most 'rational' groups, edge.org and libertarians, which might be worth pursuing later. Given the smaller sample size and restricted diversity of the original study, I'd be inclined to say that conscientiousness and agreeableness are not robust correlates of disgust, though this could be an effect of the fact that yourmorals.org uses a different measures of Big Five personality traits from the original study.

Can I publish this finding? It's only correlational and says nothing about causality. It really doesn't say much that is new, but rather confirms the original study, more or less. Still, the 26 papers which cited the original study would be slightly more improved if they could cite this finding as well, since it's the same basic study with a different (larger and more diverse) sample. This is where the discussion of the peer review system converges with this analysis. According to this paper,  "many natural science fields operate on a norm that submissions should be accepted unless they are patently wrong." In contrast, psychology papers are often rejected, not because they are wrong, but because they are not interesting or novel enough.

The paper and the listserve discussion bring up many points related to this, but one relevant one to this finding is that it is hard to build a cumulative science when you don't reward replication, but instead reward novelty. The end result is that you end up with a series of slightly different perspectives on the same subjects, all named differently, where authors are constantly trying to come up with something new rather than building on something existing. This may help academics, but it makes it very difficult for these theories to be used in the real world. Any research on humans is likely flawed in some way. Can anybody do double-blind experiments on representative samples of people with behavioral measures? The public is wisely skeptical of any social science finding as are academics...but the solution might lie in publishing more replications rather than in restricting the publication process toward the mythical goal of the perfect, novel study. No single study proves anything when dealing with research on people. It's the convergence of lots of studies that might potentially be convincing enough to outsiders.

- Ravi Iyer

ps. if anyone wants to write this up and publish it traditionally, feel free to contact me

The timeline went something like this...

• March 2010 - de Rugy publishes a paper alleging that Democratic districts received more money than Republican districts from stimulus funds.
• April 1, 2010 @ 11am - Silver challenges her assumption in that she failed to take into account the fact that the districts receiving the most funds were state capitols, which ostensibly were supposed to send funds onwards.
• April 1, 2010 @ 4:42pm - de Rugy shares her data, concedes some points (including the need to check for capitols), while giving explanations for other points and maintaining her larger finding and taking some offense for being accused of bias.
• April 1, 2010 @ 7:35pm - Silver responds to her response, praising de Rugy for her openness, tempering his accusation of bias as the sort of unconscious bias that all social scientists have, and perhaps finding a middle ground in conceding that there may be some unconscious bias effects or particular project effects which account for her initial finding, which may or may not survive the inclusion of state capitol-hood as a controlling variable.

I imagine that both of them are right now crunching the numbers and figuring out some far more accurate interpretation than either of them would have come up with on their own. The best part is that if I wanted to, I could download the data myself and join in on the fun, perhaps merging in another data source if I so chose. Perhaps someone else is doing that right now too.

I found the exchange so intriguing that I took a break from working on a paper I'm writing about libertarian moral psychology (getting me to take a break actually isn't that hard, unfortunately). When I finish this paper, the timeline is likely to be something like the following:

• I submit the paper to a journal.
• 4 Months later - I receive 2-3 reviews of my paper. If they liked it (~30%), I can edit the paper to respond to reviews and move to the next step.  If not, I go back to step 1.
• 2 Months later - I resubmit the paper.
• 4 months later - If I'm lucky I may get the paper accepted (~30%), but more likely is that I have to do another round of edits which takes another few months or in rarer cases, the paper is rejected after this stage and I go back to step 1.
• 2 years later - maybe 50-100 people have read my paper, which now contains an outdated literature review and dated conclusions.  If someone wants to challenge my results, their paper may come out around this time. Few people outside of academia can read my paper due to the need to subscribe to the journal in question. I can't update my paper and have to have a whole new set of findings rather than being able to add a single study or clarification to a part of the existing paper.

Now the process that I described has it's merits. It produces more carefully thought out work, reviewed in depth by experts in the field. It's probably essential in some areas, but it's merits are dependent on the situation and I'm not so sure it's the best method for social science research that is supposed to be used by society in some timely fashion to have positive social benefit. Is that not the real goal of social scientists, rather than CV building?

As Nate Silver points out in his critique of de Rugy's piece, there is inherent unconscious bias that all social scientists encounter when they do any research. Peer reviewers don't reanalyze your data and they rely on your own description of methodology, so they really can't address many possible sources of bias, conscious or unconscious. All research is somewhere between a zero and one in terms of conclusiveness and it only moves close to a one after many people have replicated it, in my opinion, as research is inherently unreliable when you are dealing with people.

What if social scientists all self-published (maybe let's call it sharing rather than publication) on the internet? Overall quality would go down, no doubt. Sharing of replicated results, null findings, and perhaps most importantly, failures to replicate, would probably increase a lot though. Academia would lose a monopoly on research as anyone with a stats program could weigh in and data sharing would become the norm for controversial results. Also, separating the wheat from the chaff is a problem that computer scientists, Google, Digg, Slashdot, and countless others are continually solving. There is tons of research that gets published and then nobody every cites it, so the peer review couldn't have done that well at it's gatekeeping process. What if "getting published" was no longer the standard for acceptability, but rather the number of positive votes/comments of the people who read the article, and you could continually edit and revise your article to make it better, linking to people who replicate your study and updating your literature review and conclusions to keep current. I could envision a post-sharing review system that would actually improve quality by making the review process completely open and transparent, giving extra credit to those whose data has been re-analyzed independently, replicated by others, and read by experts.

There are a million considerations I'm probably leaving out right now, both positive and negative, but given the way that social science data is being generated and the pace the world is moving, it seems unlikely that the peer review process can resist these disruptive forces. Right now, the peer review process confounds sharing research with praising the research in question and maybe there are ways to separate the two goals so that they don't have to happen simultaneously.