On non-anonymous reviewing

on

Some journals ask reviewers not to reveal themselves. A review process in which the reviewers are anonymous, unless they choose not to be, makes sense. But why shouldn’t reviewers be free to reveal themselves if they wish?

Twice, I have received non-anonymous reviews. In both cases, receiving the non-anonymous review was a thrill. Both reviewers were researchers I highly respected, and their positive opinion of my work meant a lot to me. In one case, the reviewer asked the journal editors to forward a signed review. In the other case, the reviewer sent me e-mail directly with the review attached. That review, while positive, had many excellent suggestions for revisions. Receiving the review more than a month prior to receiving the packet of reviews from the journal enabled us to get a head start on revising the paper, which was the reviewer’s stated reason for sending it to us directly.

I do not know why some journals prohibit reviewers from revealing their identities. I can think of a number of reasons, but none that outweigh the possible benefits, such as community building, that allowing reviewers to choose to review non-anonymously supports. Thoughts anyone?

On a related note, who owns the copyright to a review? This question came up recently when, according to the authors of a paper, the arXiv refused to allow uploading of a paper because it contained reviews for which the authors did not have the copyright. Is it true that a reviewer has sole ownership of the review, or does the receiver of the review also have the right to publish it? If not, how much of a review can an author post on a site like My Review Sucks? Also, if the copyright is retained exclusively by the reviewer, that creates many orphaned works, since the anonymity of most reviews means it is difficult to find the copyright owner to obtain permission to publish.

8 Comments

  1. I certainly think non-anonymity should be an option. I imagine that mostly positive reviewers would exercise this option, but I really can’t see the downside. A similar option would be to mark the review as creative-commons so that it could be freely republished.

    Certainly it behooves journal publishers to take the lead on improving the reviewing process if they want to maintain their relevance.

  2. One reason to discourage or prohibit reviewers from revealing their identities is corruption, especially log-rolling. The nefarious Dr. X reviews your paper and recommends it be accepted, while showing that he *might* have recommended rejection. In some other forum, X might anticipate that you will reciprocate in his or her favor. This is clearly a bad thing.

    A further concerns is that open reviewing can encourage disciplinary factionalism. Let us suppose that Contemporary Occult Studies are divided between the East Coast Traditionalists, centered on Miskatonic and Oxbridge, and West Coast Postmodernists at UC Sunnydale and Arkham West. If reviews aren’t anonymous, you might feel that an editor has been deeply unfair because all three reviewers of your best student’s CHI paper were Easterners. This might in fact be random, or it might be a Message to you from the Program Chair, or it might arise because almost all available reviewers who are proficient in the analytical technique you used and whom the Program Chair knows to be reliable were trained in the East. Whatever the cause, the cloak of anonymity can help preserve good nature where open reviewing will draw battle lines. (Compare the outcome of the old AI neats/scruffies battles with the current situation in macroeconomic theory or the Theory wars in 1990’s historiography).

    Finally, the cloak of anonymity helps draw a convenient shade over small mishaps in reviewing assignments. In the old days, conflicts were simple: you can’t review papers from your own department, your thesis advisor, your students, or your relatives. Program chairs and editors could reasonably be expected to anticipate all these, and the odd border case seldom caused much trouble. Nowawdays, on the other hand, there can be all sorts of social complexities which it seems hopeless to expect a program chair or editor to know. Jenkins and Jarndyce had a fling in graduate school, or Raskal’s daughter (from his first marriage) is doing her doctorate with Prof. Marmalade, or Profs. Orland and Johnson had a terrible fight last April at a conference in Goteborg. Realistically, you can’t know (and don’t want to know) everything when you assign papers, and you hope that (a) reviewers will tell you about the really egregious mis-assignments and (b) the rest will come out in the wash, or be revealed through careful reading of the reviews. (This is another reason, incidentally, why paper ratings are much less significant than commentary; Orland can reduce Johnson’s rating from a 9 to a 6 from spite and say “I am a tough grader!” but, in comments or at a PC meeting, Orland will have to make a scientific case and not just issue an opinion.)

  3. […] This post was mentioned on Twitter by Mark Bernstein, Gene Golovchinsky. Gene Golovchinsky said: Posted "On non-anonymous reviewing" by Eleanor Rieffel http://palblog.fxpal.com/?p=4325 […]

  4. Mark – Do you think these concerns outweigh the potential benefits of allowing reviewers to review non-anonymously if they so choose?

    I see many reasons against a completely open review system in which there is no anonymity. I am not arguing for that. I agree with Daniel’s guess that reviewers would choose to make only positive reviews non-anonymous, and would probably only make a select subset of their positive reviews non-anonymous. I can see a few downsides, but also significant upsides. For example, you point out that allowing non-anonymity could exacerbate factionalism. But it could also help overcome perceptions of factionalism. West Coast Postmodernists might be pleasantly surprised to find that an East Coast Traditionalist gave one of their papers a positive review.

    I agree that editors cannot be expected to know all the complexities of potential relationships between authors and potential reviewers, but I’m not seeing that allowing (while not requiring) reviewers to make their identities known makes the situation so much worse that it should be prohibited. With regard to corruption, a largely unenforceable request from the editors, who have little means for punishment, and little means to detect violations, seems likely to have little effect on someone who is truly nefarious. Such a person can contact the authors and take revenge if they tell the editors even if reviews are supposed to be anonymous. I do think, and hope you agree, that such nefarious behavior is fortunately rare, and concern about such behavior should therefore not play a major role in deciding the question.

    A lot of my reasoning is based on the assumption that non-anonymity would be chosen mostly only for positive reviews, and only for a subset of those reviews. Do you think that assumption reasonable?

    An interesting question on which I’d be delighted to get reader comments is “Which reviews should a reviewer consider making non-anonymous?”.

  5. I’m clearly not playing by the rules here.

    I was so bummed about my last submission (I don’t submit much) that I posted the (negative) reviews on my own blog (in the post Lacks a convincing comparison with simple non-Bayesian approaches).

    I’ve also contacted people whose papers I reviewed. The subject of “log-rolling” as so deftly described by Mark Bernstein, never even occurred to me (maybe because I don’t submit much any more). In one case, I wanted to know if I could cite the paper before it was published and get a link so I could blog about it (denied — the authors were traditionalists and didn’t want to “scoop” their own paper (?)). In other cases, I just wanted to kvetch about the review process (for papers I liked but were rejected). In some cases, I just wanted more direct feedback more quickly about an issue. In the recent cases, the authors seemed a bit shocked that I contacted them. Like I said, I hadn’t thought about possible conflicts and no one pointed them out to me.

    In an older case, there was a paper with results I really liked that I couldn’t quite understand. I knew one of the authors and actually had them walk me through the paper, at which point I helped them rewrite it. All while it was in process. It was like a super-speed revise-and-resubmit. I wouldn’t have done this if the paper wasn’t related to what I was working on actively myself, probably couldn’t have pulled it off if I didn’t know the authors, and wouldn’t have tried had I not know the authors did good work.

  6. Bob,

    > I’m clearly not playing by the rules here.

    Not clear. Many journals do not prohibit reviewers from contacting authors. Also, I haven’t seen definitive statements on who is free to republish reviews or who owns the copyright. (There is debate on the site I linked to in my post, but nothing definitive has yet emerged.)

    You allude to one of the better reasons I could think of for prohibiting reviewers from contacting authors: the editors understandably may well not want to handle arguments from authors saying “why did you reject my paper when Bob Carpenter thought it so good.” I’m still not sure it outweigh the potential positive benefits. You give a good example where it helped forward research by improving a paper and your understanding of the work.

    Your example does, however, get close to another concern I thought of. It seems reasonable to me for a journal to request that reviewers not contact authors until the review has been submitted. As one example of a potential problem, I don’t think it is appropriate for a reviewer, even with the best of intentions, to put the authors in the position of feeling that they must correspond with the reviewer in order to get a good review.

    I’m amused that you have shocked authors by contacting them. Guidelines as to when doing so is advisable, and when it is not, would be useful.

    To turn the general question around a bit, “When should a journal refuse a reviewer’s request for non-anonymity?”

  7. This is a great discussion on an important topic that I’ve just raised on my own blog as a result of reading about the recent Shakespeare Quarterly experiment on open reviewing. I believe the principle of blind reviewing is worthy but it requires participants to play by a set of assumed rules of ‘fair play’ that are difficult to assure in the real world. Editors can try to minimize the problems of blind reviewing but they cannot guarantee their removal, and in that space exists many review shortcomings.

    As we all realize, the alternatives have their own shortcomings. However, we live in a world where there is increasing demand on reviewers and the impression we all have is that refereeing is often hurried, sometimes poor, and even occasionally personal. I’d like to believe editors could exercise more vigilance but I doubt this is enough to rectify the situation. Open reviewing is at best part of a solution but even where it is deemed a success (as in the Shakespeare Quarterly experiment), there are downsides that were unforeseen, such as junior faculty not being willing to publicly comment on articles when they found themselves in disagreement with more senior commentators.

    Of course this does not answer the question just raised by Eleanor: “When should a journal refuse a request for non-anonymity” (in part because this question is too convoluted for my aging brain to sort out the conditionals ;) I’d prefer a simple solution of allowing reviewers to be named as a choice and not involving the journal beyond this. Of course, that assumes the editors are satisfied that nobody will complain ”why did you use this reviewer?” but there is no complaint-free editorial process that I know.

    My basic concern with all reviewing is that it occurs! By this I mean that the review is detailed, thoughtful and fair. I am frustrated (as an editor and author) by reviewers who think a one liner or a set of scores on a few scales constitutes a ‘review’. I am inclined, perhaps naively, to believe that if reviewers were expected to put their names to a review, they might also put a bit more thought into the process. It’s one thing for such reviewers to submit it to the editor without shame, I suspect they would be less inclined to do so if they were publicly named. And the record of reviews themselves could make an interesting addition to the research record.

    I am trying to utter these comments without blaming anyone (probably not successfully) and suggesting that the publication system has outgrown our ability to maintain a double-blind reviewing process of quality. Maybe the future is to offer a formal range of reviewing strategies for different venues, openly articulated, and modifiable as we go. Oh, and those of us in the education of future researchers might need to step up further to ensure the standards are learned.

  8. For reviewing to remain a viable (and valuable) part of academic discourse, we need to devise a set of processes that provide appropriate incentives for people to produce quality reviews. As with many other endeavors, a carrot-and-stick approach will likely be more successful than either of its parts.

    Stick: public shame in the form of published non-anonymous reviews. (Downside: will discourage some from reviewing, as Andrew Dillon pointed out.)

    Carrot: Recognize reviewers who produce consistently good results, as I suggested in an earlier post, Reviewing the reviewers. (Downside: requires a little bit of extra work for the authors/PC in the short run.)

Comments are closed.