Opiniomics

bioinformatics, genomes, biology etc. "I don't mean to sound angry and cynical, but I am, so that's how it comes across"

Why anonymous peer review is bad for science

I got into a really long, and often interesting, conversation on Twitter a few days ago about the merits, or not, of open peer review. 140 characters is a bit limiting, so I am putting my arguments here.

My regular readers know that I am a big supporter of open peer review, and I have signed grant and manuscript reviews for about 2 years now – crucially, I sign them whether they are positive or negative.  However, what I really want to do is change the way we see peer review – in my opinion, we should see it as a supportive and collaborative process by which a group of independent scientists assess the quality of a body of research, suggest ways in which it might be improved and decide whether it is ready for publication.  Some of this is encapsulated in my reviewer’s oath.   Peer review doesn’t have to be the often awful process that it often is, and it also doesn’t have to be easy on the authors.

Anyway, below are the arguments that I have read about why peer review should remain anonymous.  My response is below.

“Anonymous peer review protects reviewers”

I am immediately concerned by the language here.  Protect reviewers from what?!  Is peer review suddenly a dangerous activity?  The point is made even more forcefully in this quite terrifying piece, which states:

It might be especially difficult to find referees for authors who hold positions of power and influence, or for those who are considered quarrelsome or vindictive by their peers. In particular, younger, less-established scientists … would be reluctant to reveal themselves, for fear of retaliation from their more powerful colleagues.

OK, OK.  Hands up who in science is happy with the idea that in our field “powerful” and “vindictive” scientists might want to “retaliate” against someone who has reviewed their work?!!  Does anyone seriously think that that’s OK?  Why are we even discussing peer review if we work in a field where this might happen?  Stop the clock.  Time out.  Pause.  Retaliatory, revenge attacks, by anyone, should be considered serious scientific misconduct and the perpetrators should be identified and sacked.  It’s as bad, worse, than plagiarism or making up data.  This kind of activity should spell the end of careers.  Are we expected to sit back and accept that this kind of thing might happen?!

Of course, this argument in support of anonymous peer review is actually a very powerful argument against it.  Powerful or vindictive scientists are only able to take revenge attacks because they can hide behind a cloak of anonymity.  If their reviews were published, alongside their names, then the community would soon recognise if they were behaving badly, and action would soon follow.  The collective scientific community are actually very ethical and moral, with a strong sense of fair play.  Should a scientist identify retaliatory, revenge reviews, with evidence, we would all listen.  And act.

“Open peer reviews are ‘soft'”

I refer you to this paper, which states:

Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review

So in fact there is no evidence that open peer reviews are in any way “soft”. Hilariously, the Nature Neuroscience piece throws out this peer reviewed, scientific evidence in favour of “an informal poll of some of our referees”.  What?!  You can’t just throw away a paper’s conclusions based on a conversation you may or may not have had with a few carefully chosen referees!

There is no evidence that open peer reviews are soft, and I therefore declare this argument invalid.

“Open peer review promotes ‘quid pro quo’ favourable reviews

Yes, in fact I think open peer review probably does enable this kind of unethical, fraudulent behaviour;  my simple point is that anonymous peer review doesn’t stop it either!  They’re as bad as each other.  There is absolutely nothing to stop an anonymous reviewer emailing the authors of the study, praising their amazing research, pointing out they were reviewer 2 and inviting them to review their future papers.

Open peer review does not solve the “favourable reviewer ring” problem; but nor does anonymous peer review!

“Anonymity removes egos from the equation”

This one actually make me laugh out loud!  Sure, because as anyone who has ever been involved with anonymous peer review, egos are completely absent…..

“Open review will make it hard to find reviewers”

Yes, I think this is probably true and is backed up by the BMJ paper above.  I have one simple riposte to this.  When you look at a body of work, with the aim of reviewing it, the only thing you should be thinking about is the quality of the science (and perhaps also the impact, if the editor wants you to consider that).  If you would be scared to reveal your name to the authors, then you are already showing evidence that you’re unable to give an objective review.  You are already thinking, consciously or subconsciously, about who the authors are, what they represent to you, how they might affect you and your work.  You cannot be objective and you shouldn’t review the paper, anonymously or otherwise.  No review is better than a biased review.

“Anonymity corrects for unequal power relationships”

No, anonymity enables unequal power relationships, and open data, open information, open peer review equalizes those power relationships and ensures all interactions can be assessed independently for fair play.

One of the most telling quotes from the Nature Neuroscience piece is this one:

The journal also experienced an occasional breakdown of the peer-review process, in which authors and referees bypassed the editors completely in negotiating how a paper should be revised.

OH MY GOD!  Stop the planet!  Everyone just HOLD!  This is serious!  Imagine, authors and reviewers collaborating together to make a piece of work better, outside of the control of a journal!  THIS MUST BE STOPPED!

(seriously, you can’t make this up!)

In case you hadn’t noticed, the BMJ has practised open peer review since 1999, and they took this decision on the basis that it was simply more ethical, without cutting down on quality. Good on them.  They seem to be doing OK for themselves! If you don’t have time to read it, here are some choice quotes:

All editors have seen curt, abusive, destructive reviews and assumed that the reviewer would not have written in that way if he or she were identifiable

openness should eliminate some of the worst abuses of peer review, where reviewers—under the cloak of anonymity—steal ideas or procrastinate

Openness … links accountability with credit

Science is progressively moving away from anonymity.

we hope our small move will contribute to a broader culture change so that junior researchers cease to fear reprisals from senior ones

Inspirational stuff, I am sure you’ll agree.

What else can I add to this?  Just my own experiences.  Putting my name next to reviews has made me, without exception, give more thought to what I write, and that can only be a good thing.  I think more about the evidence for my criticism and whether those criticisms are valid.  More importantly, I empathise with the authors.  This is a good thing.  It doesn’t make me soft, it just helps me understand.  By far and away the biggest reaction I have seen is one, not of anger, but of respect.

Detractors will say that perhaps I have had a negative reaction, I just haven’t seen it; and perhaps they are right.  In fact, the only way I would see is if those reactions were open, and signed with the names of the reviewers responsible.  Which makes my point very nicely.

49 Comments

  1. All of your arguments assume that any negative consequences would be deliberate, conscious misconduct. In fact, all the evidence shows that we are very, very susceptible to unconscious biases. Have you ever been sent Christmas cards or stickers by a charity? That’s because they know you’ll feel a strong urge to reciprocate by sending a (much mroe valuable) donation, though of course you won’t be thinking of it as quid pro quo. Like it or not, you’re susceptible to the same impulse when you review something by someone who you know has given you a favourable review (and the reverse impulse – the inclination to punish – if they have given you a negative review). You won’t be thinking of it as “being nice to them because they were nice to you” or as “punishing them for criticisng you”, but you’ll go into the reviewing exercise more (or less) inclined to believe that the work is by a scientist with sound judgement, and less (or mroe) predisposed to look for hidden flaws.

    The same sort of unconscious bias kicks in if we read something – a paper or a review – by someone famous versus a newcomer, and often if we see a paper from the US versus one from China, or by a woman versus a man. It’s horrible to think that we are susceptible to such biases – but we all are. If you think you are immune, try an implicit association test: https://implicit.harvard.edu/implicit/ and read articles like this: http://blogs.scientificamerican.com/unofficial-prognosis/2012/09/23/study-shows-gender-bias-in-science-is-real-heres-why-it-matters/

  2. I think you misrepresented my comments about egos. It was implicit that egos and prejudice will exist under any format of peer review. The point was that I don’t think that it’s always constructive to be able to (a) base an opinion of a manuscript on the status of its authors (institution, seniority, gender, ethnicity, etc); or (b) be able to direct one’s wrath onto another professional after being perceived to have been unfairly judged.

  3. I also wanted to say that I have only once received an abusive review in all my years, and it was signed. The reviewer was a person who felt confident in his position of power within the scientific community and presumably knew that signing his review would give his (rather personal) abuse more impact.

  4. As a junior scientician sitting on the sidelines this whole exchange on Twitter has been well … delicious! 🙂 It was very lively, very animated, and very entertaining.

    I will have to respectfully disagree with Mick. Double blind, with all its flaws, is my preferred system. I have witnessed and continue to witness first-hand the negative effects of bias. As I may have already mentioned, reputation can be double edged sword in academia. Often times individuals, unconsciously or not, wiil make a priori assumptions about the quality of a scientists work just based off of his or her reputation. This is human nature and not too surprising but IMO completely unacceptable in academia / science.

  5. However, what folks on both sides should be doing is buttressing their arguments with hard data as Mick seems to be doing here. Your own beliefs, experiences and assumptions may lead to you concluding that a certain option is better but if lots of data show that one system is noticeably better than the other, then it should be set as the default option.

  6. I accept that there is “unconscious biases”, but surely the best way to discover and deal with these is by using open data? The current system of anonymous peer review, the reviewer is still subject to the same unconscious biases because the reviewer knows the authors of the paper. Those biases are then hidden first because the reviews are not published, and second because the reviewer’s name is never revealed.

    You may suggest that the answer is double blind, but of course, this cannot work. There are so many ways in which a reviewer can identify authors, e.g.:

    – the language used
    – references and citations to previous work
    – pre-prints
    – use of unique facilities or resources
    – access to unique data
    – access to unique cohorts
    etc etc

    Therefore double blind would not work because in a significant percentage of cases, the authors would be identifiable, and we’re back to the current system of anonymous peer review.

  7. Ah, my apologies, I did not mean to mis-represent you. I was working from hastily prepared notes. I agree with you, both (a) and (b) should not be allowed, and I believe open peer review would help remove (b). Unfortunately, both anonymous and open peer review are bad at (a). Double blind is not feasible (see comment above)

  8. Was the review also published? i.e. was it a matter of public record that this person had been abused? I’m sorry if I wasn’t clear, but when I say “open peer review” I mean the reviews are signed *and* published.

  9. Thanks for posting! Unfortunately, double blind cannot work. There are so many ways in which a reviewer can identify authors, e.g.:

    – the language used
    – references and citations to previous work
    – pre-prints
    – use of unique facilities or resources
    – access to unique data
    – access to unique cohorts
    etc etc

    Once a reviewer has identified the authors, we are back to square one.

  10. When you refer to “authors”, you presumably mean that the PI is easily identifiable? Double blind may, at least in some cases, conceal who the first and middle authors are. As such, it would be interesting to know if e.g. the influence of implicit racial or gender bias would be weakened by concealing the author list during review. Note that I am not arguing for the existing system here. I am just not convinced that double-blind review really does have the same outcomes (not sure if there is data on this).

  11. I think there is a middle ground to have accountability but maintain anonymity. I suggest the following experiment:

    1. Have a shared database of anonymous reviewers and their associated “Reviewer Rating”
    2. Reviewer Rating is established similar to a reputation score on Stack Overflow or eBay. After you peer review a manuscript, your Rating will be updated by:
    a. the authors’ rating of your review with a score of -1, 0, or 1
    b. the other reviewers’ rating of your review with a score of -1, 0, or 1
    c. the running total of your reviews

    This would quantitatively incentivize reviewers to be both thorough and fair, and prevent a given manuscript from being reviewed by a panel of 3 scientists with a cumulative Rating of -20. In addition, a researcher’s score could be solicited (not unlike a credit score) when a scientist is up for tenure, applying for a grant, etc as a measure of their community involvement.

  12. Quite clever! I like it! There’s still a problem of bias (i.e. reviewers with high scores misusing their position) but there seems to be sufficient checks and balances to ameliorate this.

  13. Commenting to cc7740 below:

    Unfortunately, the bias is actually *usually* not targeted at the 1st authors (PhD students or postdocs), but is instead directed at the PI as part of a 20 year grudge… So it’s actually the younger researchers who suffer for the (potential) ego-bashing activities of their PIs (who don’t care nearly as much about whether a paper is published this year or the next)…

  14. Very glad you like it! Implementation would depend on the presence of a neutral, central authority who would have the behind-the-scenes mapping of researcher anonymous ID -> Rating.

    Currently, the journals each have something like this, but it is not shared between journals, and the data is anecdotally/qualitatively stored in editors’ memories and impressions of reviewers.

  15. I agree that there can be problems with “guilt by association” (a reason to avoid PIs with bad reputations), but was specifically referring to implicit bias. There do appear to be some observations what happens when you change to double-blind: http://blogs.nature.com/peer-to-peer/2008/01/doubleblind_peer_review_reveal.html

  16. I do agree that in many situations open peer review has the potential to be better. For one thing, it represents a potentially terrific teaching and learning resource helping us to train students as better scientists. However, I don’t think a one-size-fits-all solution is ideal. I believe that authors should be able to decide what’s best for their paper. They should be offered a check-box during the submission that allows them to choose either full-open peer review, or double-blind peer review (including the editor). What we can agree on, I think, is that the current system of single-blind review sucks.

  17. This is an interesting discussion and although I think it is clear that the current system of single blind peer review is the worst for science and scientists it is less clear to me whether open or double-blind review would be the best mode to replace it.

    In my view the best arguments for open review are that it presents the strongest incentive for reviewers to write quality reviews, that reviewers would be able to get credit for their work and that they would be accountable for their reviews.

    However, I do worry that open reviews are more likely to be influenced by human bias, relationships and review history. We all invest heavily in our science and having it negatively evaluated will always be difficult to accept. It is only human that such an experience would influence the reviewer if on the next occasion author and reviewer roles were to be inverted. Moreover, scientists are heavily connected and professional relationships often become personal relationships. Will I really be able to openly negatively evaluate the work of my good friend? I am not so sure.

    Your central argument against such bias in open reviews is that reviewers are now accountable for their reviews. But that postulates that there is objectively the ‘right’ review for a paper and that it would be possible to objectively identify reviews as too positive or too negative. But reviews are to a large degree based on personal experience and apart from extreme cases it will not be possible to establish that a review was biased for personal reasons.

    So I think although open peer review is the ideal scenario, double-blind is more realistic given how humans work. And I don’t agree that it is not possible. It is true that there will be a good number of papers were it will be easy for the reviewer to guess the authors. But there will be a large number of cases were that is not the case and were review is truly double-blind. And even in the cases were it is obvious from which lab a study originates. Are you sure it is really from big famous PI Abc or is it maybe her postdoc who is now setting up his own lab? This sort of ambiguity would already make a difference.

    In any case I believe reviews should always be published next to the paper and although pre-publication review is valuable we need much greater emphasis on post-publication review.

  18. I’m with Mick; I’ve been doing open peer review (and frequently posting the summary bits of my review on my blog, after the paper is out). Ideologically, this is the right way to go; now let’s make it work!

  19. Are you really this delusional? Anonymous peer review protects junior untenured people in the field from retaliatory behavior by BSDs in their field, plain and simple.

    Your example is just wishful thinking:

    “Of course, this argument in support of anonymous peer review is actually a very powerful argument against it. Powerful or vindictive scientists are only able to take revenge attacks because they can hide behind a cloak of anonymity. If their reviews were published, alongside their names, then the community would soon recognise if they were behaving badly, and action would soon follow. The collective scientific community are actually very ethical and moral, with a strong sense of fair play. Should a scientist identify retaliatory, revenge reviews, with evidence, we would all listen. And act.”

    The concern isn’t so much when Professor MyLabIsWayBigger tanks my paper as an anonymous reviewer for reasons of personal interest. The concern is that when I am invited to review that full professor’s work I want to be sure I am free to call out any bad science that makes its way into the paper, without worrying that my reviews will factor into Professor MyLabIsWayBigger’s future review of my grant during study section. All it takes are the usual stock reviews in this climate to sink a proposal, not obvious vindictiveness.

    I agree that the scientific community should be held to the same ethical and moral standards as other human institutions — but my experience with the inner workings of the politics of academic institutions is a bit different, and I don’t think every dean or department chair looks at it the same way you do. Actually I’m fairly sure of it. Senior R01 funded faculty are protected and valued far far above junior kids just struggling for their first trainee grant. In the meantime it is reasonable to be a realist about how the world actually works. Anonymous peer review is like anonymous voting — both should be kept that way to prevent intimidation.

  20. The only reason that Professor MyLabIsWayBigger can tank your future grant is because they can hide behind a cloak of anonymity. Remove that cloak and you remove their power.

    I appreciate what you’re saying about the inner workings of academic institutions, but that’s exactly what I want to change. Powerful, vindictive old professors shoudn’t be allowed to be that way, and the only weapon we have is to remove their cloak of anonymity. Expose them. Here, World, in all his/her glory is Prof Vindictive, who tried to destroy a young scientists career because of a review.

    The voting analogy doesn’t work. Yes, anonymous voting is in place, but there is no anonymous intimidation. The intimidation is out in the open, because if someone tries to intimidate you, you know who they are and you report them. Anonymity in academic peer review protects the agressor as much as anyone. Why allow that?

  21. You say that open reviews are more likely to be influenced by bias, but I would say that anonymous reviews are influenced by bias too. You make the analogy of reviewing a friend’s work – well, as a reviewer, you know your friend’s name just the same if you’re anonymous or not. The difference with open peer review is someone can notice and say “Hey! Isn’t it a conflict of interest that X reviewed Y? They’re close friends and I see them holding hands in the park every Thursday night!”. With reviewer anonymity, there is no way we can do that. The only way we can detect bias is with open data. Anonymity encourages bias.

    Doube blind suffers from the same issue. It is not open to inspection. It is not open to analysis. WIth open data we can detect bias, we can detect reviewer rings, we can detect unconscious favourtism, we can detect nepotism, we can detect conflicts of interest.

    Double blind is also bad for people in small fields where everyone knows one another, and only holds up until people meet at a conference and start discussing the papers they have under review.

  22. Would you, Bogus Stackframe, have chosen your words more carefully (e.g. the delusional comment) if your identity had been posted alongside your review of Mick’s blog post? I’m sure you’ll say no, but I’m guessing the real answer is probably yes..

    Thank you for demonstrating one of the points Mick was trying to make about peer review.

  23. “All it takes are the usual stock reviews in this climate to sink a proposal, not obvious vindictiveness.”

    I do think this is a legitimate concern. There are powerful people who are quite adept at making subtle digs are people lower down in the pecking order, all under the guise of asking questions, or of feigned concern. And it isn’t necessarily detected by all.

    I suspect that in order for this not to cause a problem for lower-ranking people, we also need to move away from the culture of labelling each others papers as insufficiently “exciting” that appears to be creeping in.

  24. If someone is vindictive enough, they can sink those proposals anyway. Every single argument people are giving me against open peer review, also applies to closed peer review. Powerful people can kill papers and grants whether they, or the authors, are anonymous or not!

  25. They’d have to know who “wronged” them in order to retaliate, surely? Unless we’re not defining vindictive in the same way here.

  26. I suspect someone who is vindictive enough to kill grants/papers because they get a poor review is also vindictive enough to kill grants/papers that happen to be in the same area of science as them.

  27. That may be the case for some silverbacks but not others. Without evidence one way or another, I can see why people might be worried and less inclined to agree to review. I don’t see how dismissing concerns outright helps.

  28. I totally agree that single-blind review is massively subject to bias, which is one reason, why it is so bad. True double-blind review should do the job, but again I agree that it is impossible to make work for a significant number of papers. But I still think it would be a big step forward.
    You central argument that open review is safe is that misuse can be detected and reported. But how and to whom? Opinions about a paper are going to be diverse, so how to spot a biased review? Somebody disagrees with me so his/her review was probably biased? And when you detect misuse where do you take your concern?
    I just think the scientific community is good at identifying the problems and very bad in doing something about them. An example: It is probably pretty generally accepted that evaluating scientists according to the Impact Factor of the journals they publish in is inadequate. Many have now even signed a petition about it (DORA). But has something changed? Not as far as I can see. If you look around who is hired or even just invited for interview it is pretty obvious that still the defining factor is IF. So what to do about an institution where 9/10 interviewees have CNS papers? Report it? To whom? Obviously the hiring committee will say that they studied all 300 applications in great detail and these 9 were the best. Something similar will apply to misuse of reviews. People will see it, but little will be done.
    Also where to draw the line? X and Y hold hands in the park, so they should probably not review each others work, but how about A and B who hang out together at the bar at every meeting and obviously get along very well, does that also make them not suitable?
    I just think the problem with reviews is that they are done by a small number of people. If we establish a culture where it is the norm that you comment about the papers in your field (e.g. in Pubmed Commons) and therefore every paper gets many comments, than the odd one from Prof. MyLabIsWayBigger does not really matter. That is what we should be working towards.

  29. I don’t dismiss the views, what I said above is simply that if you are so scared of the authors that you are unwilling to give your name, then you are *already* incapabale of being objective, and shouldn’t review anyway, anonymous or not

  30. If professors and students alike were identifiable and accountable to their comments then it would constrain them to reasonable and defensible arguments in reviews. If someone showed a pattern of unfair behaviour they would very quickly be outed thanks to social media. Hiding behind anonymity (such as username codes) is soooo last century 😉

  31. Peer review is absolutely required in science and in particular in scholarly communication and publishing. Non anonymous peer review is a better way of quality assessment and public post-publication peer review (4PR) the best. This requires transparency about both the reviewers’ identities and their expertise in their specific field of research. If managed in that way readers and authors will benefit from this reviewing which could result then in a discourse rather than simply, an approval (or rejection) of the authors’ work as it is now in anonymous peer reviewing. If all participants in such an open 4PR process will acknowledge this principle the scientific community would benefit significantly in terms of a more useful discussion of new scientific results than it had happened before. So let us adopt that 4PR process rather than waiting to be bothered again week by week by journal editors to submit anonymously a closed review about a manuscript which we haven’t read before (or already twice because we already rejected it for another journal) and in which we may be not interested to read now because of lack of time.

  32. I agree completely with these ideas, and I think the nominal sense of objectivity afforded by “anonymous” peer review is fairly naive. Just receive my first signed review and gave my first signed review, and it was really nice!

    Another important factor that gets discussed less often is that in the current system, editors have more information than you as an author do. Sometimes you’ll get 2/3 good reviews and its fine. Sometimes not. Whether the editor is willing to override the reviewer can often depend on relative stature more than the content of the review–after all, the editor is playing the game as well, and probably doesn’t want to override Prof. PowerPlayer who gave the negative review. This definitely happens. The editor can have an agenda behind who they send reviews to and who they listen to. So no matter how much blinding is possible (even double blind doesn’t really seem plausible), as long as we have editors choosing reviewers and deciding who to listen to, there will be information asymmetry. Far better, in my mind, to have reviewer identities open–puts a bit of the spotlight on editors, also.

    One other consequence of signed reviews: if you get a bad one, you know who to exclude next time you submit… 🙂 I think that would mostly be a good thing, actually.

  33. I haven’t yet submitted a paper to a journal, nor have I published, so I am curious: what exactly is a bad review? Is it one that offers no constructive feedback, no feedback at all, unnecessarily harsh … etc.

  34. Your posts is a bit naive in dismissing very legitimate concerns people have when reviewing papers openly. Other than unconscious biases, there are some psychopaths in positions of tremendous power in academia, and they take negative feedback very personally. The assumption that “something” will be done about vindictive individuals who are valuable to the university for one reason or another is just plain wrong.

  35. A bad review is any review that votes to reject your paper… 🙂 Seriously, though, bad reviews are the ones that are just mean and don’t make much sense. These are typically some (proper) subset of negative reviews. All the best as you enter the world of peer review. It’s harsh and mostly a waste of time, but in the end, it’s undeniably nice to see your work in print!

  36. Look at your wording – you’re essentially telling me that vindictive psychopaths exist within Universities. Do you think we should do nothing about that? Anonymity enables that behaviour. Openness is the only way to stop it.

  37. A bad review is anything that isn’t a good review. A good review is one that provides constructive criticism, that identifies any faults in the research and suggests realistic ways in which those faults can be fixed. A good review is polite, but firm. A good review is well researched and referenced.

  38. Well I have just dealt with an important Italian collaborator of Prof MylbabIsbigger by the name of Stronzo Bestiale. Stronzo blocked my paper in PLoS One and by the time I had finished the appeals and put it in PeerJ he had published an overlapping set of results. So Stronzo is now going to be in my acknowledgements on every paper I ever write on phylogenetics as my ultimate inspiration. Without Stronzo I would never have cared less about viral evolution. I must also remember to add commas to my reviews of his work in Stronzo’s honour as this seems to be a criteria for rejection.

  39. You write:

    “Retaliatory, revenge attacks, by anyone, should be considered serious scientific misconduct and the perpetrators should be identified and sacked.”

    If only it were so easy. Personal experience on a search committee: one member of the search committee voices the *suspicion* (it’s anonymous review, right?) that one of the candidates was a reviewer on their rejected grant proposal. The candidate did not get invited for an interview. This is how quickly you get ruled out from a job, even though it was an anonymous review. Should the entire committee be fired, just because they don’t want someone at their university who disagrees with one of their colleagues? Anonymous reviews allow at least for the potential to voice that sort of criticism in a way that doesn’t automatically disqualify you from a job.

  40. Erm, but this happened when the reviews were anonymous, right? So anonymous doesn’t solve the problem!

  41. Do you have any ideas on what exactly we should do? Frankly speaking, I don’t. Psychopaths are everywhere and universities are not somehow immune to them. I would even argue that psychopathy predisposes people to power – psychopaths are both more motivated to get it and don’t have scruples that would hamper them on their way. Plus they can be extremely charming on the outside, which makes their psychopathy completely invisible to casual observers such as university authorities and their more distant colleagues and peers.

  42. To the long list of arguments in the text, I would like to add that anonymous peer-review is also a way to hide the incompetence of the reviewers.
    As an editor, you do not get the reviewers that you want and you can come to dire situations where nobody wants to review the manuscript. It is hard to avoid such situations in the current system, but a lesser evil would be to acknowledge it instead of pretending that everything went fine.
    I learned this in a strange way when I was a post-doc. I was offered to review a paper for Nature Biotechnologies. I accepted, under the condition that I would not be anonymous. The editor told me it was OK, but nevertheless edited out the disclosure from my review at the last moment, saying that I could disclose my identity to the authors directly if I wished. I was never sure why, and I wonder if it has anything to do with the fact that I was only a post-doc.

  43. As a politic scientist, I have to admit that my discipline is one of the most corrupted, closed circle of chosen ones, and it suffers from this condition a lot. My holistic approach to science suffers every time I try to publish, because I know there are many ego maniacs out there who do not accept they are being wrong…

    The best example is with the mindless usage of linear regression in almost all types of quantitative research. The scientific standards of the discipline are weak and does not correspond to a current knowledge. If we look at the political science from the point of another scientific discipline, like physics, we realize that its methodological standards are very weak. But the anonymous revievers constantly demand us to do mindless and senseless tasks, like always testing for null hypothesis, which have no to very small impact especially when the whole model is flawed from the very beginning. It is almost impossible to get to the better journals without accepting the current substandards.

    I believe that open peer review would help immensely because academic debate would be opened on a new front, and it is something we desperately need in quantitative social science.

  44. Reasonable and defensible comments can still be biased.

  45. Look at the evidence:

    Though 50-70% of reviewers believe that it is easy to identify authors, when asked to try, around 70% fail to do so: http://a-c-elitzur.co.il/uploads/articlesdocs/PeerReview2.pdf (page 296)

    Double-blind review does increase the representation of female authors in the pages of journals:
    http://www.ncbi.nlm.nih.gov/pubmed/17963996

    I think you are also forgetting the role of editors. Regardless of whether reviews are open or blind, a good editor will detect, disregard, and not pass on any comments that are obviously malicious. And it is only obviously malicious comments that open review has any hope of discouraging — reviewers will continue to get away with unhelpful or biased reviews either way.

  46. Prof MyLabIsWayBigger has that power whether anonymous or not. Only if he openly states (or otherwise makes very obvious) that he is vindictively tanking your grant application because you criticised his work is he likely to be criticised for this behaviour. Merely giving a “meh” review of your proposal is, as others have pointed out, enough to tank it.

  47. Reblogged this on The Writing Lifestyle and commented:
    As a Biology faithful and a pessimistic reviewer of all things written, I found this post interesting.

  48. This NYT piece on the handling of a sexual harassment case by Yale University is quite an interesting read in connection with this discussion (http://www.nytimes.com/2014/11/02/us/handling-of-sexual-harassment-case-poses-larger-questions-at-yale.html?_r=0). It highlights how little action was taken when very serious allegations were made against a famous faculty member. Particullarly interesting is the passage about the boyfriend of the victim whose carrer was stalled by the offender. The committee investigating the case “stopped short of saying negative actions like removing him from a grant had been retaliatory since so many factors, including subjective ones, come into play in a cardiology chief’s decisions.”

    Provided that the NYT report is accurate than this appreas to be a very clear cut case and it is shocking how the situation was and still is handled. So what are the chances that something will be done if there are complains about someone in power who appears to have negatively judged someones work in response to an earlier open review by that person? The response really writes itself: It is impossible to prove that the negative assesment was retaliatory since so many factors, including subjective ones, come into play in Prof. MyLabIsWayBigger decisions.”

  49. But you have a choice not to identify the authors and that decreases the *unconscious* bias.

Leave a Reply

© 2017 Opiniomics

Theme by Anders NorenUp ↑