Reviewing the reviewers


I’ve written about some alternatives to the current review process, and I believe one of ways in which the current process can be improved is by formal recognition of reviewers’ efforts. While many conferences and journals acknowledge reviewers by publishing their names, this does not reflect the quality of the effort put in by some reviewers. A more lasting and public recognition of quality reviewers may be one way to improve the quality of this volunteer effort.

Interestingly, the APS recently instituted a policy of recognizing referees who review the articles submitted to the various APS journals.

The basis for choosing the honorees was the quality, number, and timeliness of their reports, without regard for membership in the APS, country of origin, or field of research. Individuals with current or very recent direct connections to the journals, such as editors and editorial board members, were excluded.

The ACM would do well to consider a scheme for recognizing excellence in reviewing as well. Consider the following proposal: during a review process for a conference, the program committee or a few meta-reviewers rate each review for quality on a five point (1-5) scale. Reviews that score a four accrue one point for the reviewer who wrote them; reviews that score a five accrue two points. (Perhaps reviews that score a 1 should accrue -1.) Scores are aggregated for each reviewer, and reviewers in the top 10% (or 15% or some other cutoff) get official recognition by the SIG or by the ACM, who are responsible for maintaining and publicizing the list of these excellent reviewers. To account for improvement (or lack of participation), the score on which reviewers are ranked could be a (potentially weighted) sum of contributions over the latest three years during which the person reviewed at least one submission.

These details are somewhat arbitrary, but I believe are representative of a scheme that recognizes outstanding reviewing. Making reviewing more visible to the community should encourage people to write more thoughtful reviews. While this scheme imposes some additional work on the program committee to vet reviews, the improvement in review quality  should compensate by yielding a smoother-running and more reliable review process.

Share on: 


  1. Twitter Comment

    RT @HCIR_GeneG: Posted “Reviewing the reviewers” [link to post]

    Posted using Chat Catcher

  2. Twitter Comment

    Posted “Reviewing the reviewers” [link to post]

    Posted using Chat Catcher

  3. Twitter Comment

    great idea! RT @HCIR_GeneG: Posted “Reviewing the reviewers” [link to post]

    Posted using Chat Catcher

  4. James Landay says:

    I totally agree and would be glad to do this extra rating work for reviews that I’m handling. Some journals already do this, but then don’t publish the results (i.e., just us it for picking reviewers).

  5. I also think it would be really useful to have this rating history available at the time reviewers are being selected. That way you have a sense of what to expect.

  6. From my current experience with submitting articles for review with APS and some other journals I have to say that the quality of reviews is pretty pathetic. I am one of the top 1000 reviewers on Amazon, and even the second-rate review of some of the best Amazon reviewers beats what I’ve experienced through the peer-review process by a nautical mile. It is good that APS is trying to make some effort in improving their review process, but my sense is that the whole peer-review system is fundamentally flawed and unscalabel to the amount of scientific publishing that is going on today.

  7. […] ratings of reviews (as described earlier) to well-reviewed papers for each reviewer. Use the keywords of those papers to establish the […]

  8. Twitter Comment

    Wishes so much right now to be able to review the reviewers at #chi2010 as suggested by @HCIR_GeneG [link to post]

    Posted using Chat Catcher

  9. Twitter Comment

    says hmm, this is certainly worth thinking about: reviewing the reviewers: [link to post] via @addthis

    Posted using Chat Catcher

  10. […] Lennart Nacke expressed the desire to act on the suggestion in a blog post I wrote to review the reviewers. So why not? I would like to see if we can collect some data to […]

  11. […] course the adoption of this scheme does not preclude the incorporation of reviewer ratings, the adoption of which should be used to improve the quality of reviews as well. var […]

Comments are closed.