Jofish reports some interesting numbers regarding the role that the associate chairs play in the outcome of CHI paper reviews. He analyzes the CHI 2010 data to reach the following conclusions:
- Of the 302 submissions accepted, 57 or so were affected by decisions made at the meeting
- The 1AC (the primary meta-reviewer) was instrumental in getting a paper rejected 31 times, but was not able to prevent rejection 111 times, and represented reviewers’ consensus 1199 times.
- He also provides some more ammunition for the desk-reject debate.
It would be great to repeat this analysis on other years to see how reliable the patterns are.
An open question is whether the 57 or so papers whose fates were determined at the PC meeting deserved the outcome they received. (Obviously, the rejected papers’ authors would argue against this process.) It’s also interesting to note that it is not possible to replace the CHI PC process with an rule based on average scores, because both the reviewers and the ACs might then try to game the system by assigning extreme scores to marginal papers.