Data Science & Psychology Data Science applied to Values, Morals, Politics, & things that matter.

Nate Silver and Veronique de Rugy demonstrate how a more modern peer review process could work.

As someone who was in the dot-com world for years before entering academia, I've always felt that the peer review process could be made far more efficient and while I'm not 100% sure what form that would take, it might look something like a recent exchange between Nate Silver, an Obama supporter who runs fivethirtyeight.com (which I read religiously during the 2008 election and which is the first site I turn to when I seek to interpret polling data), and Veronique de Rugy, an economist with a libertarian bent.

The timeline went something like this...

• March 2010 - de Rugy publishes a paper alleging that Democratic districts received more money than Republican districts from stimulus funds.
• April 1, 2010 @ 11am - Silver challenges her assumption in that she failed to take into account the fact that the districts receiving the most funds were state capitols, which ostensibly were supposed to send funds onwards.
• April 1, 2010 @ 4:42pm - de Rugy shares her data, concedes some points (including the need to check for capitols), while giving explanations for other points and maintaining her larger finding and taking some offense for being accused of bias.
• April 1, 2010 @ 7:35pm - Silver responds to her response, praising de Rugy for her openness, tempering his accusation of bias as the sort of unconscious bias that all social scientists have, and perhaps finding a middle ground in conceding that there may be some unconscious bias effects or particular project effects which account for her initial finding, which may or may not survive the inclusion of state capitol-hood as a controlling variable.

I imagine that both of them are right now crunching the numbers and figuring out some far more accurate interpretation than either of them would have come up with on their own. The best part is that if I wanted to, I could download the data myself and join in on the fun, perhaps merging in another data source if I so chose. Perhaps someone else is doing that right now too.

I found the exchange so intriguing that I took a break from working on a paper I'm writing about libertarian moral psychology (getting me to take a break actually isn't that hard, unfortunately). When I finish this paper, the timeline is likely to be something like the following:

• I submit the paper to a journal.
• 4 Months later - I receive 2-3 reviews of my paper. If they liked it (~30%), I can edit the paper to respond to reviews and move to the next step.  If not, I go back to step 1.
• 2 Months later - I resubmit the paper.
• 4 months later - If I'm lucky I may get the paper accepted (~30%), but more likely is that I have to do another round of edits which takes another few months or in rarer cases, the paper is rejected after this stage and I go back to step 1.
• 2 years later - maybe 50-100 people have read my paper, which now contains an outdated literature review and dated conclusions.  If someone wants to challenge my results, their paper may come out around this time. Few people outside of academia can read my paper due to the need to subscribe to the journal in question. I can't update my paper and have to have a whole new set of findings rather than being able to add a single study or clarification to a part of the existing paper.

Now the process that I described has it's merits. It produces more carefully thought out work, reviewed in depth by experts in the field. It's probably essential in some areas, but it's merits are dependent on the situation and I'm not so sure it's the best method for social science research that is supposed to be used by society in some timely fashion to have positive social benefit. Is that not the real goal of social scientists, rather than CV building?

As Nate Silver points out in his critique of de Rugy's piece, there is inherent unconscious bias that all social scientists encounter when they do any research. Peer reviewers don't reanalyze your data and they rely on your own description of methodology, so they really can't address many possible sources of bias, conscious or unconscious. All research is somewhere between a zero and one in terms of conclusiveness and it only moves close to a one after many people have replicated it, in my opinion, as research is inherently unreliable when you are dealing with people.

What if social scientists all self-published (maybe let's call it sharing rather than publication) on the internet? Overall quality would go down, no doubt. Sharing of replicated results, null findings, and perhaps most importantly, failures to replicate, would probably increase a lot though. Academia would lose a monopoly on research as anyone with a stats program could weigh in and data sharing would become the norm for controversial results. Also, separating the wheat from the chaff is a problem that computer scientists, Google, Digg, Slashdot, and countless others are continually solving. There is tons of research that gets published and then nobody every cites it, so the peer review couldn't have done that well at it's gatekeeping process. What if "getting published" was no longer the standard for acceptability, but rather the number of positive votes/comments of the people who read the article, and you could continually edit and revise your article to make it better, linking to people who replicate your study and updating your literature review and conclusions to keep current. I could envision a post-sharing review system that would actually improve quality by making the review process completely open and transparent, giving extra credit to those whose data has been re-analyzed independently, replicated by others, and read by experts.

There are a million considerations I'm probably leaving out right now, both positive and negative, but given the way that social science data is being generated and the pace the world is moving, it seems unlikely that the peer review process can resist these disruptive forces. Right now, the peer review process confounds sharing research with praising the research in question and maybe there are ways to separate the two goals so that they don't have to happen simultaneously.