Nate Silver and Veronique de Rugy demonstrate how a more modern peer review process could work.

1Apr/106

Nate Silver and Veronique de Rugy demonstrate how a more modern peer review process could work.

As someone who was in the dot-com world for years before entering academia, I've always felt that the peer review process could be made far more efficient and while I'm not 100% sure what form that would take, it might look something like a recent exchange between Nate Silver, an Obama supporter who runs fivethirtyeight.com (which I read religiously during the 2008 election and which is the first site I turn to when I seek to interpret polling data), and Veronique de Rugy, an economist with a libertarian bent.

The timeline went something like this...

I imagine that both of them are right now crunching the numbers and figuring out some far more accurate interpretation than either of them would have come up with on their own. The best part is that if I wanted to, I could download the data myself and join in on the fun, perhaps merging in another data source if I so chose. Perhaps someone else is doing that right now too.

I found the exchange so intriguing that I took a break from working on a paper I'm writing about libertarian moral psychology (getting me to take a break actually isn't that hard, unfortunately). When I finish this paper, the timeline is likely to be something like the following:

I submit the paper to a journal.
4 Months later - I receive 2-3 reviews of my paper. If they liked it (~30%), I can edit the paper to respond to reviews and move to the next step. If not, I go back to step 1.
2 Months later - I resubmit the paper.
4 months later - If I'm lucky I may get the paper accepted (~30%), but more likely is that I have to do another round of edits which takes another few months or in rarer cases, the paper is rejected after this stage and I go back to step 1.
2 years later - maybe 50-100 people have read my paper, which now contains an outdated literature review and dated conclusions. If someone wants to challenge my results, their paper may come out around this time. Few people outside of academia can read my paper due to the need to subscribe to the journal in question. I can't update my paper and have to have a whole new set of findings rather than being able to add a single study or clarification to a part of the existing paper.

Now the process that I described has it's merits. It produces more carefully thought out work, reviewed in depth by experts in the field. It's probably essential in some areas, but it's merits are dependent on the situation and I'm not so sure it's the best method for social science research that is supposed to be used by society in some timely fashion to have positive social benefit. Is that not the real goal of social scientists, rather than CV building?

As Nate Silver points out in his critique of de Rugy's piece, there is inherent unconscious bias that all social scientists encounter when they do any research. Peer reviewers don't reanalyze your data and they rely on your own description of methodology, so they really can't address many possible sources of bias, conscious or unconscious. All research is somewhere between a zero and one in terms of conclusiveness and it only moves close to a one after many people have replicated it, in my opinion, as research is inherently unreliable when you are dealing with people.

What if social scientists all self-published (maybe let's call it sharing rather than publication) on the internet? Overall quality would go down, no doubt. Sharing of replicated results, null findings, and perhaps most importantly, failures to replicate, would probably increase a lot though. Academia would lose a monopoly on research as anyone with a stats program could weigh in and data sharing would become the norm for controversial results. Also, separating the wheat from the chaff is a problem that computer scientists, Google, Digg, Slashdot, and countless others are continually solving. There is tons of research that gets published and then nobody every cites it, so the peer review couldn't have done that well at it's gatekeeping process. What if "getting published" was no longer the standard for acceptability, but rather the number of positive votes/comments of the people who read the article, and you could continually edit and revise your article to make it better, linking to people who replicate your study and updating your literature review and conclusions to keep current. I could envision a post-sharing review system that would actually improve quality by making the review process completely open and transparent, giving extra credit to those whose data has been re-analyzed independently, replicated by others, and read by experts.

There are a million considerations I'm probably leaving out right now, both positive and negative, but given the way that social science data is being generated and the pace the world is moving, it seems unlikely that the peer review process can resist these disruptive forces. Right now, the peer review process confounds sharing research with praising the research in question and maybe there are ways to separate the two goals so that they don't have to happen simultaneously.

Tagged as: academia, peer review, social scientists Leave a comment

Comments (6) Trackbacks (1) ( subscribe to comments on this post )

Jason
April 1st, 2010 - 23:38

Would this be substantively different from – essentially – blogging your own research?
Ravi
April 1st, 2010 - 23:54

I suppose not, though I don’t know if I have a concrete idea of what an ideal system would look like. What would you say the primary problems would be if people blogged their own research? I think someday all research groups will have some kind of dynamic internet presence. Just noticed that you have your own blog…do you blog about your own research ideas? Or only stuff that has already been published? I’m interested in getting other people’s ideas about these kinds of issues (hence my post).
Nance
April 2nd, 2010 - 06:00

I think we have to bow to the inevitable direction of publication now (blogs, research groups’ proprietary websites, etc.) and watch the new formats evolve toward new kinds of unimagined excellence. I read somewhere this week of math problems being solved by bloggers whose comments offered bits and pieces to produce a solution…much like a group of mathematicians clustered around a blackboard, but pulling in numbers of and diversity of contributors in unprecedented ways . The news moves too fast, the moment is too pregnant for the old processes. The concept of ownership of data and conclusion would have to change, I think, toward something more globally owned; the emphasis might become origination and contribution rather than ownership as we currently think of it. Cooperation would ultimately out-produce competition in academia. It sounds thrilling!
Johnnie Linn
April 9th, 2010 - 11:06

The feedback process can be speeded up if you present your paper at a conference first. The input you get there is likely to be similar from what you would get from referees later, especially if you are presenting your paper at the conference whose society’s journal is the one you want to publish in.

Conference proceedings are becoming increasingly available on line. That means your work can be more easily found before you have reached the publication stage.

I think that journals that are associated with major research universities will have less of a relative impact in the future in fields where research is not capital-intensive. Lesser journals will have an increasing relative importance since they can disseminate information worldwide at no greater cost than the older journals.
Jason
April 10th, 2010 - 13:50

As a general rule, I don’t blog about amorphous research ideas (ever), and if I blogged my own research it would not be until AFTER it had been published in whichever journal and I could link to the original material. AND I’d have to make it abundantly transparent that I was blogging my own research and inherently biased. And I’d make an effort to invite relevant other bloggers to discuss the research as well.

Then, there’s the issue of (at least for certain kinds of research), wanting protection from, say, animal rights extremists. If that became an issue, it would almost certainly stop me from blogging my own research.
Ravi
April 10th, 2010 - 22:54

Jason, I apologize in advance if I’m reading you wrong, but I can’t help but think that you have a quasi-moral opinion about “blogging your own research” that makes me feel judged. I think the inherent bias of any blogger is self-evident in the fact that it’s their blog…it’s someone’s opinion to be taken for what it’s worth, judging the source of the information as part of that process. It’s pretty obvious when something is a person’s own work as most people say so fairly explicitly. I do.

I definitely think peer-reviewed research is likely of higher quality and people should treat it as such…but it’s certainly not the only quantitative data that’s worth sharing. Gallup and FiveThirtyEight.com share worthwhile quantitative data all the time. Peer review isn’t usually about accuracy, but often about novelty or how much some piece of data advances the science. Reviewers don’t reanalyze your data or make sure you actually did the methods you said you did. Rather, they assess how much your results advance the discipline you are in and whether that contribution is worthy of their journal…you could have an entirely accurate result…and reviewers will decline it because it’s not novel enough or because it’s hard to interpret, not because it’s inaccurate. Unless reviewers start auditing labs and datasets, they can’t really assess accuracy. The peer review process ends up being more about CV building, rather than sharing information….If it were truly about sharing information, we would see a lot more null results, failures to replicate, and insignificant findings, all of which are accurate results, but not novel enough….we’d see auditing of accuracy rather than novelty.

…but I digress, my main point isn’t really about the peer review process, but rather the fact that blogging is an inherently opinionated and subjective exercise, which is part of what readers sign up for….so unless people actually deceive others, I don’t see anything strange about blogging about one’s own amorphous non-peer reviewed research ideas. If people don’t like it, they can stop reading or comment on how crappy the post is…:) Indeed, the interactive medium invites more review in some ways (see Nate Silver and De Rugy’s thread above), just as you are somewhat reviewing this post by your comment.