Opiniomics

bioinformatics, genomes, biology etc. "I don't mean to sound angry and cynical, but I am, so that's how it comes across"

The reviewer’s oath

A quick one for Monday morning 🙂

I was unfortunate to receive this morning a negative review of one of the papers we have under review.  Now, this review is terrible.  It contains so many factual inaccuracies that the reviewer cannot possibly have read the paper, our software tutorial, the code nor the documentation.  It contains ridiculous, unsupported conjecture.  It is just a really terrible review, and I can only conclude the reviewer is trying to block the publication of our paper/software.

In a future blog post, I may dissect the review piece by piece, but I am giving the journal time to respond.  However, as an example, one of the criticisms is that our software may contain bugs.  They don’t actually identify any.  They simply state that our software may contain bugs, and it is therefore better for biologists to do it themselves as then they’d know exactly what they were doing.

I kid you not.

If “software may contain bugs” was a valid criticism, nothing would ever get published.

I am briefly reminded of a review I received for a paper when I was acting as editor for PLOS ONE.  The paper is published now: Evaluating de Bruijn Graph Assemblers on 454 Transcriptomic Data. The review, in its entirety, was:

It does not provide significant advance in the field

Again, I kid you not.

I really wish I could tell you the name of the reviewer, but I have ethics.  Needless to say I ignored them and asked other reviewers to help with the paper.  It is truly awful: the authors have spent months on this paper, they have written 9 pages of review and analysis, and that is the review they get?  Words fail me.  Don’t they deserve more?!

As any of my twitter followers know, I support open peer-review, and I sign reviews for both papers and grants.  I am also an editor for PLOS ONE and Frontiers in Bioinformatics and Computational Biology.  I am open, with a capital “O”.

Needless to say, the reviewer I mentioned above comments on our work from behind the wall of anonymity.  I really do not believe they would have the nerve to provide such an inaccurate, dishonest review if they had known that I would be able to see their name.

Frankly, reviews that are simply designed to block publication of papers border on scientific misconduct, and those who give them should be publicly identified.

Removing anonymity is important.  Open review is important.  It gives the reviewer a different perspective.  I know, because I practice it.  It makes you more constructive.  It makes you think “How can I help?” rather than “How can I criticise?”

So, I want to start to create “The reviewer’s oath”.  Here is a beginning:

I, the reviewer, promise:

  i) to not hide behind a screen of anonymity
 ii) to be open and honest with you (the authors) at all times
iii) to be constructive in my criticism
 iv) within the rules given to me by the journal, to assist you in every way 
     I ethically can to get your manuscript published,
     by providing criticism and praise that is valid and relevant

I am sure there are other things that we can come up with – what do you think?

And who wants to sign up to “The reviewer’s oath”?

Edit: 10:52 11th Feb 2013

Nick Loman had this to say on twitter.  I couldn’t agree more!

 

27 Comments

  1. The rationale behind anonymous reviewing is to avoid negative impact on the reviewer due to the very same type of petty vindictiveness you point out in the first case. Both problems that you identify in your post can be solved by good editorial practice, as indeed you show for the second case.

    A junior researcher’s career can be ruined by vindictiveness of a senior one. Signed reviewing will cause junior,and even senior, researchers to shy away from writing bad reviews. Even more, it may cause them to write dishonestly good ones to curry favor with whom they perceive to be important for their career. Indeed, open reviewing may quickly create “cliques” of favorable reviewing.

    Anonymity is no guarantee for honesty and fairness, as you have pointed out. But signed reviews would be even less so.

  2. I take your point, but I completely disagree. As I said, I am putting my money where my mouth is and signing reviews – even negative reviews. The fact is that, for me, it has had a very positive effect in that I am constructive, not destructive.

    Signed and open reviews would eliminate the truly vicious, vindictive reviews because I don’t believe some of these people would stand beside their review when it is scrutinised by a wider community of people.

    The problems you identify almost certainly occur anyway – the key to solving all of these problems is openness.

    Perhaps you don’t have the courage to stand by your reviews, Iddo? 🙂

    • I have to agree with Iddo, in that I know that vindictiveness goes both ways: from reviewer to author, and from author to reviewer. Scientists are human, and often more thin-skinned than they let on; why wouldn’t you be when the dividing line between you/your job is so thin (in terms of hours worked and your life’s meaning) in so many cases?

      An asymmetric arrangement in which the reviewer is anonymous but the author is not permits selective criticism – either unjustified vindictive comments, or an easy ride – on the basis of the author.

      The opposite arrangement (anonymous author, named reviewer) permits selective treatment of the reviewer by the author after the fact: vindictive or beneficial.

      Losing anonymity on both sides – your ‘open reviewing’ – puts a different pressure on, in exactly the way Iddo says: people fearful of the author’s power (and let’s not pretend that there isn’t an unequal distribution of power in terms of funding/promotions/grant co-applications/conference presentations etc.) may be tempted to give them an easy ride. Sure, open unjustified vindictiveness may disappear – except for sociopaths – but at what cost: sufficiently open and thorough criticism?

      Anonymity on both sides is, I think, better. The social loading of a power relationship (in either direction) disappears so long as anonymity is maintained. There should be no easy rides given unless the reviewer is lazy, and no vindictive reviews unless the reviewer is just misanthropic 😉 However, author anonymity is difficult, if not impossible, to maintain: inspecting the references is probably enough to identify people; also you may have seen the work previously at a conference, and talked about it with the author.

      So, what’s the second best option, given that this is impractical? I think that the odd vindictive criticism is acceptable, given that any non-substantive claims can probably be shown to be so (and have been, in my experience), in a way that giving ‘powerful’ authors a comparatively easy ride is not. None of the options is perfect, but I like the options that are most easily-correctable best, and I think that this is more easily-correctable.

      FWIW, whenever I’ve had a signed review (and some of the commenters here and on Twitter know who they are), I always assume that they have the best interests of the manuscript at heart, and I appreciate their openness. I just don’t want to force that openness on others.

    • >Perhaps you don’t have the courage to stand by your reviews, Iddo?

      You mean am I worried that someone who has never met me would make personal attacks on my character rather than addressing points of substance in my argument?

      I don’t know. Should I be?

      • Ha ha ha, not that I’m aware of, no 🙂

        I don’t really agree with your arguments, I genuinely think signed, open review will eliminate truly awful reviews and will force reviewers to be constructive. As I said, if a junior researcher wants to provide a good review to a senior researcher, in order to curry favour, they can do it under the current system and then let the senior researcher know by other means. Ultimately you cannot avoid this kind of academic fraud. However, with open peer review, I *do* think you can eliminate the kind of useless reviews I mention above.

        I think evidence from journals practicing open review backs me up, though N is small.

        Have you ever signed a review?

  3. More progress would be made unilaterally if the review mantra was, “How can I help?” vs. “How can I criticize?” I did my first open review recently, and yes, it was a bit scary, but in the end, I was happy with it, and it was a good experience.

  4. I’ve had the open vs closed review discussion a few times with advisers and heard the same suggestion that folks would be upset and hold a “negative” review against a reviewer in the future. I imagine there would be some of that initially, but if everything is open, I think those situations would pretty quickly be worked out. If the review is constructive, it shouldn’t be viewed as a negative.

    I also think there is already a lot retribution in the review process, just the reviewer can do it with almost no oversight. Most people I have talked to have had some anonymous review they felt offered little critique or suggestion and was just obstructive and ranting. I’ve had a reviewer suggest we withheld negative data. When these types of reviews come anonymously and are never seen by anyone but the author and and editor, there is no way to really rebut them.

    Short of full open review, I think a good first step is to publish the reviewers comments. It has been my experience that these reviews tend to be more thoughtful and constructive, even if the reviewers identity remains unknown.

  5. Frankly, reviews that are simply designed to block publication of papers border on scientific misconduct.

    No. Such reviews are scientific misconduct, no bordering about it.

    Those who give them should be publicly identified.

    Abolsutely yes! At present those who invest time and effort in good, detailed, helpful reviews get very little credit for it. And those who underperform, whether through laziness or malice, don’t get called on it. Whatever reasoning once motivated the tradition of reviewer anonymity and review secrecy, it’s long outdated.

  6. I think that transparency/open peer review is a good thing. However and unfortunately, if it was the rule and not the exception, I am not sure that many reviewers will still accept to review papers 🙂

  7. Nice post. There is a interesting work by one of SimplyStatistics blog authors on the benefit of complete open reviewing process published last year.
    Cooperation between Referees and Authors Increases Peer Review Accuracy. http://www.plosone.org/article/info:doi/10.1371/journal.pone.0026895. sure it will be interesting for you and the conversation on it.

  8. It seems to me that we cannot always trust authors for not going after negative reviews (even if these are fair and thorough). Perhaps making open reviews the default, but give the option to make is anonymous if the reviewer decides to do so, is enough of a nudge to vastly improve the quality of reviews. But systematic open reviews… this is tough.

  9. I am all for it. Where do I sign?

  10. The “open review as default, but with opt out if the reviewer really needs it” policy is the one we are currently using for GigaScience (see this recent editorial: http://www.gigasciencejournal.com/content/2/1/1/abstract). The bioinformatics community seems to have no problems working in an open-source environment when they are producing software, so why shouldn’t it be the same when they write up the same tools as papers? The medical community are far ahead in this respect, as the BMJ and BMC-series medical journals have been using open peer-review for well over a decade, and somehow they manage to get it to work. The BMC-series is an interesting example, as 50% of their journals use it and 50% don’t, and they don’t seem to have any problems getting sufficient referees for medical papers, even when they are in areas that overlap a lot with their biology journals (e.g. BMC Medical Genomics and BMC Medical Genetics).

  11. I like the idea of a reviewer’s pledge. With respect to reviewer anonymity, there’s another point worth adding. Usually when I review, I know less about the specific area than the authors, and have perhaps a day or two to not only get up to speed, but to provide accurate and useful critiques. This is a difficult task. Anonymity lets me risk being a bit sloppy. If my review makes a point which is wrong, the authors can rebut it to the editors, privately, with little harm is done to science or the reputations of anyone. Even an invalid concern might improve the manuscript, by the revisions to help other readers avoid making similar errors. If my reviews were made public, along with my name, I’d have to be much more careful — with the result that I’d hesitate to raise concerns that might be shown to be unfounded. Science would suffer, since potentially valid concerns would not be raised. I might also decline to review at all, or review more slowly. Privacy of reviews serves a number of functions, for authors, editors, and reviewers.

  12. As much as I agree with you, I think you should not disclose any review. The review “It does not provide significant advance in the field” is private, it belongs to the referee, the authors and the journal, and you shouldn’t *publish* it without their permission. The authors chose to publish it in a journal where reviews are not public (otherwise they could have chosen something like Biology Direct).

  13. I see both sides but lean on anonymity. It should be the editor’s role to ensure the review is constructive and in good faith. Just the way you pointed out to a one line review! The editor cannot be simply sending automated emails but be more involved in the process and read the reviews for sanity and tone.

    However, the pressure to disclose names, especially for junior faculty can be huge. Not everyone takes constructive criticism well 🙂

    • Well, if you’re a scientist and you don’t respond to criticism well then you’re in the wrong job – seriously.

      What I love about the open peer review model is that everything is out in the open. If you do encounter some prejudice because of a review you have submitted, then you can get that out in the open too – blog, and name and shame!

  14. As a junior researcher it takes guts to sign your reviews, but I’m all for it. By default, I’ve signed the past dozen or so journal reviews I’ve submitted, including negative reviews. Most recently I signed a highly critical review where I pointed out repeated plagiarism in a paper. I wonder what the implications will be, if any, for such an action…

  15. I have usually signed reviews. When I have not, it has been because I knew that it would just end-up in increasingly pointless correspondence with authors who cannot take even constructive criticism. It is a good principle, but takes guts for a relatively junior researcher.

    But the way, Nick Loman should learn how to spell principle! 😃

  16. I was a full-time editor for over 10 years and what got to me more than the sloppy or obstructive reviewers were reviewers who wrote one thing in the report to authors (usually on the positive side of lukewarm) and in the confidential letter to the editor would spell out all the defects of the paper with a recommendation categorically not to publish.
    This approach lacks honesty and integrity, and puts the editor in a very difficult situation. And happens more often than you think.
    As for terse, unhelpful reviews, these would go back to the reviewers for revision (seriously), or at the very least would be followed up by phone.

  17. How about:

    0) To actually read the paper under review and base my comments on factual analysis of what I read

    ?

  18. I’d happily sign up for such an oath, but who would want me as a reviewer?

  19. I make it a policy never to write private comments to the editor when reviewing. Everything I have to say about a paper goes to the authors. Of course, it takes me forever to do a review, because I have to stop and think every time I get to something I don’t quite understand or agree with, so people have gradually stopped asking me to do reviews—speed is more important than thoroughness these days.

  20. I certainly agree that removing the anonymity of reviewers will substantially increase the quality of the reviews. It does have many advantages and I can see it working very well with good papers that deserve to be published. Papers of influential professors in which a junior investigator finds significant problems represent a much bigger challenge to an open system.

    In my experience the bioRxiv allows for open reviews from some of leading experts and good visibility. To some extent this solves the problem since “bad” papers will not get reviews.

  21. inspired by Mick, I also wrote my reviewer oath if you are interested : http://aninfinityofhypotheses.wordpress.com/2014/07/01/my-reviewers-oath/

  22. Hi,

    Our journal paper submission is in the process of third review. One reviewer has been quite abusive all along. His/her comments have been “There is lack of novelty in this paper. The quality of English is very poor. There are already better techniques.” “I am not convinced at all that this paper has any new contribution. The Manuscript is also verry poorly organized. The work is not an advancedment of presently available layout compression and decompression techniques. The English continues to be of very low standard.” and “The organization and presentation of the paper has not improved at all. Many important implementation details are still missing. Results are questionable. There is no significant contribution or advancement. The quality of English has not improved at all. ”

    What have make us even angry are that the Associate Editor’s comments are “One of reviewer has strong concerns about the novelty of the idea and the quality of writing.” “One reviewer has some serious concerns, which must be addressed.” and “Please address the concerns of Reviewer # 3.” !!!!!

    We suspect that the nasty reviewer has written in the confidential letter to the associate editor about all the defects of the paper with a recommendation categorically not to publish.

    In the end, we have decided to file complaint to Editor-in-Chief. We really hope that the Editor-in-Chief would do something for us. Sometimes we wonder if we should just withdraw and submit to other journal.

    Thanks.

Leave a Reply

Your email address will not be published.

*

*

code

© 2017 Opiniomics

Theme by Anders NorenUp ↑