Does the CHI PC meeting matter?

on

Jofish reports some interesting numbers regarding the role that the associate chairs play in the outcome of CHI paper reviews. He analyzes the CHI 2010 data to reach the following conclusions:

  • Of the 302 submissions accepted, 57 or so were affected by decisions made at the meeting
  • The 1AC (the primary meta-reviewer) was instrumental in getting a paper rejected 31 times, but was not able to prevent rejection 111 times, and represented reviewers’ consensus 1199 times.
  • He also provides some more ammunition for the desk-reject debate.

It would be great to repeat this analysis on other years to see how reliable the patterns are.

An open question is whether the 57 or so papers whose fates were determined at the PC meeting deserved the outcome they received. (Obviously, the rejected papers’ authors would argue against this process.) It’s also interesting to note that it is not possible to replace the CHI PC process with an rule based on average scores, because both the reviewers and the ACs might then try to game the system by assigning extreme scores to marginal papers.

1 Comment

  1. […] Many people in the CHI community are aware of the range of problems associated with the CHI conference review process that tries to cram 1,300 or more submissions through a rather small reviewer pool with the goal of selecting the interesting and the important, while filtering out  inappropriate or unfinished work. Needless to say, the process is often imperfect. […]

Comments are closed.