Monday 10 March 2014

Does peer review give too much power to malcontents?

 

Peer review now has sacrosanct status. It is seen as equivalent to quality control in licenced medicine: a guarantee that the product will do you no harm, and that it may very probably do you a lot of good. It is sold as the gold standard, separating the precious metal from the dross, ensuring that everything which goes through the review process is of the highest standard.

This perspective is beloved of academic publishers. whose authors write for nothing (indeed, they are indentured labourers in academia) and whose reviewers review for nothing, and then the publications are sold for extortionate sums. $35 for one academic paper? You could buy a meal, a newspaper, a magazine, a romantic novel and still have change for several coffees.

Worse, anonymous reviewers can exercise power without responsibility: “the prerogative of the harlot throughout the ages”. They can bitch, spit, claw and slash, till the original work is in tatters. The supplicant, seeking promotion or mere survival, concedes all, and puts his name to a paper the reviewers have written for him, making him say things he does not believe, and commonly, cannot stomach. As the published papers accumulate he advances up the academic ladder, and looks forward to getting his revenge, either on his reviewers if he has found out who they are, or on his worst rivals, the bright young things snapping at his heels. The cycle of disparagement and suppression of contrary imaginations continues.

It is not all bad news. Some papers are rightly rejected; many are improved; some reviewers are kind-hearted, encouraging, helpful; it is even possible that some of the standard expressions of authorial gratitude to nameless reviewers are heartfelt. Anonymous review encourages honesty as well as spite. Sharp criticism may lead to great scholarly effort. It may also lead to some authors taking up farming, to the great benefit of academia, if not always to farming.

However, there is a quicker way to do all this. The authors could circulate their paper to friends, and incorporate some of their suggestions. They they could post it up on an open access website and invite reviews, thus getting several different public perspectives. It would be a more open and complete procedure. It would also be much faster. It would still be peer review, but with accountability and with far better metrics. The reviewers would be able to build up a profile: fair minded/usually fair minded/harsh/poisonous. Reviews could be counted, assessed for quality as above, and counted towards academic output in an open way. The way authors struggle to deal with criticisms could also be seen in an open way. Above all, no author could ever complain that one of their ideas was strangled at birth because of the psychopathy of a few anonymous critics.

This posting was not peer reviewed. Would you like to do so now?

4 comments:

  1. Peer review can also have unintended and unfortunate side effects. Every time I have come across some dubious piece of statistical analysis and queried the researcher about it, the response - thus far - has been "it wasn't in the original paper, but one of the reviewers made me put it in".

    ReplyDelete
  2. @ ASabisky: that's golden! peer reviewers who advise the editor "accept - with no revisions" don't stay reviewers long:) they have to justify their existence, so must find something to nitpick:) sometimes i knew who 1 of the helpful reviewers was, which i'm sure is common, b/c in such focused small fields, you may know who it is from the comments -- but in the publication game, when told "accepted with the following revisions..." one doesn't ask questions - one does the revisions (maybe maintaining some integrity by barely doing the most stupid of the requested revisions:)
    journal reviewers options:
    1) accept with following revisions (e.g., incorporate all of reviewer A's comments)
    2) major revision - resubmit (no promises!)
    3) reject outright (thanks, but no thanks:)
    when we sent articles back with #3, sometimes we'd get letters of outrage - "how dare you... i am thus & so... & i am highly regarded... i've had a stellar career, bla bla bla" -- it was cute! :)

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Davide Piffer and Emil Kirkegaard have created a new journal due to dissatisfaction with traditional peer review. It is similar to what you have suggested here.On their journal website, you can see how their peer review process works: http://openpsych.net/forum/showthread.php?tid=5

    ReplyDelete