The Business of Psychology: Will the peer review journal article system be changed by technology?
In a sense, academics have been 'crowd sourcing' for years. The first documented case of peer review was in 1665 (according to wikipedia), though this only became a standard in the later part of the 20th century. Peer review refers to the process whereby other academics review the work of potential authors of new knowledge to insure that this work is of sufficient quality. Peer review spreads the work of editing a journal among a wide array of researchers and also allows for editors to forward papers anonymously, allowing the works of nobel prize winners and humble graduate students to stand on their own merits. It's a system with many virtues that has served academia well.
Still, technology has changed the way we communicate in almost every arena and the pace of that change seems to be accelerating. Will the peer review system survive? If not, what will take it's place? I don't know the answer to that question, but perhaps examining some of the areas where the current business of psychology and the current world don't match will lead us to some answers.
- It's slow for authors - Peer review is already derided as a slow process and given the pace of the modern world, it seems inevitable that change needs to occur in this area. There are too many researchers producing too much work for unpaid experts to keep up willingly. This will only get worse as online sites like Facebook produce mountains of data that should be analyzed.
- It's binary - What makes matters worse about the speed of the process is the fact that decisions are binary. You either are accepted (rarely in social psychology) or you are rejected. Yet papers exist on a continuum and some research gets cited hundreds of times while most research never gets cited at all...which is a chilling fact given how much effort and time went into it. All that work that reviewers do in their expert review of the research gets lost in the binary nature of their result and the fact that their comments are never revealed to the public.
- It's slow for readers - Invariably, the research that is most interesting is the research that is going on right now. How are discoveries expected to be made if cycles of research are delayed for years by inefficiencies in sharing information?
- It's static - Most papers in psychology contain a review of current literature and a discussion which talks about why the paper helps advance current knowledge. Unfortunately, that information becomes outdated soon after it is written and is even more outdated by the time it is published.
- It confounds praise with publicity - Having your article published serves two purposes. It helps you get a job in that it proves the worth of your work. It also allows other people to read your work and build upon it. However, these two things don't necessarily need to go together. 'Unsuccessful' research needs to be shared as 'failure' can be very instructive. Meta-analyses and other research aggregation techniques require that information.
- It is inefficient for authors - In most businesses, people specialize in certain tasks. In order to be a standout psychologist, you need to be able to be able to combine knowledge of psychology with writing skills and knowledge of statistics and increasingly, technical knowledge to collect data online. Few people can do all of these things well.
What kind of systems should stand in place of the current peer review journal article system? I don't have the answer to that, but I hope to talk about ideas for how technology may change the peer review system in successive posts. The problems of information overload facing academics is the same problem which everyone has these days. And people are continually improving systems which address this problem through innovations like crowd-sourcing (digg), leveraging social networks to filter information (facebook), collaborative writing (wikipedia), and sharing data across data sources (semantic web and freebase).
I'll write more in successive posts about specific solutions, but I think an ideal system would be one where all data is published, but the prestige that comes from a publication is awarded by crowd-sourced ratings and reviews, not by the act of being published. The publishing of a paper is the beginning of a conversation with the world, not the end. I say data, because I think people should be able to publish data or literature reviews or a combination...but that it should be possible to publish data for others to analyze, just like people publish theories for others to test. Analyses and literature reviews should be separate as analyses should remain relatively static while literature reviews and conclusions will inevitably change and should be updateable by the original authors (eg. see Psychwiki). Any review of a paper should include semantically tagged ratings of the research so that others can combine these ratings into meta-analyses. Indeed, eventually all data should be semantically tagged such that the aggregation of data points, not just studies, is possible.
I know that is a dense paragraph and I know I'll want to change it as soon as I hit publish....and the beauty of the internet is that I can. But rather than pretend I have the answer (I don't), I'll try and blog about innovations to the publishing process and the business of psychology in later posts, all in the business of psychology category of this blog.