Reviewer Operating Characteristic

on

David Karger made an interesting proposal on the Haystack blog about the efficiency of CHI reviewing. Using an ROC analysis of reviewer scores for CHI 2010, he found that when there is consensus between the first two reviews that a paper in question scores below 2 of 5, that there is no need to solicit a third review for it.  While this method would have caused the rejection of 6 of about 300 papers that weren’t actually rejected, it would save almost 500 reviews.

The question is, is this tradeoff worth it?

The answer is probably yes, particularly if you consider the following points:

  1. Having fewer papers to reviews should, on average produce better reviews by reducing the per-reviewer load and by decreasing the demand for reviewers (thus decreasing the number of poor reviewers)
  2. Papers rejected based on two reviews are still entitled to a rebuttal, which would offer a chance to correct reviewing mistakes.
  3. The associate chair responsible for the paper has the authority to request a third review even if the first two scores are low if the reviews are inadequate.

Another reason to believe this strategy will be successful is that it is already adopted by the SIGMOD conference. It would be interesting to obtain data from a couple of other (earlier) CHI conferences to double-check the findings. If they hold up, this could be an easy fix to relieve pressure on an overloaded system.

Of  course the adoption of this scheme does not preclude the incorporation of reviewer ratings, the adoption of which should be used to improve the quality of reviews as well.

3 Comments

  1. Would love to see this sort of rationality applied to committee review processes in general!

  2. Jamie says:

    Has a CHI rebuttal ever made a difference? I don’t recall any of them even getting a serious reply.

  3. In my experience on the CHI PC and as an author, a rebuttal that addresses specific mis-readings of the paper by reviewers in a polite and unemotional manner can help a border-line paper. It doesn’t always, but it’s not wasted effort, either. What it isn’t good for is arguing with reviewers — that just pisses them off. But if reviewer 3 didn’t understand the data in table 2, and you promise to make that data more salient in the final version, that may be convincing to the meta-reviewer.

    It would be interesting to see some stats about when papers are influenced by rebuttals, but since the actual acceptance criteria guidelines are determined after the rebuttals, it’s a hard thing to measure. A better bet would be to interview reviewers and get their comments. Anybody up for that kind of study?

Comments are closed.