Comments on the CHI reviewing process

on

In the aftermath of the CHI 2010 PC meeting, we had an interesting discussion of issues related to reviewing and managing the CHI conference submission process. Several interesting approaches to improving the outcomes were discussed, including reinstating mentoring, rating reviews, adding a desk-reject option for some papers, etc. The overall goal is to improve the quality of submissions and the quality of reviews. Simplifying the overall process was also brought up several times.

Improving the quality of submissions had in the past been addressed with mentoring: Mentoring was a program that helped first-time authors structure their papers to conform with expectations of the CHI community. Prospective authors were matched with volunteer mentors who helped them prepare the paper for submission. The program ran for a few years in early 2000s, but was canceled due to difficulties in finding mentors. It would be great to restart the program again.

Another way to improve the quality of papers that are subjected to review is to filter out those papers that contain obvious structural flaws. In the current CHI review process, papers are assigned to Associate Chairs (ACs) who recruit three external reviewers for each paper. In addition, one or more other ACs is assigned to read papers that have middling scores or high degree of disagreement among the external reviewers. Papers with scores toward the middle of the distribution (and those with considerable disagreement) are then discussed at the PC meeting. At any time, any AC can also nominate a paper for discussion regardless of its score, good or bad.

The problem is that papers with obvious flaws that will result in a clear rejection still consume as much effort on the part of the external reviewers as do papers that have a chance of being accepted. This dilutes the reviewer pool, overloading reviewers, and making it harder to obtain quality reviews. One proposed solution to this problem is to screen  submissions before they are reviewed. A number of ACs (say 3-4) read each submission to scan for obvious flaws that will preclude acceptance. If they all agree that a submission passes this inspection step, it gets added to the review pool; if not, it is returned to the authors with an explanation of what was lacking in the submission, along with a relevant example (based on a similar submission that was accepted to a prior conference). This process could be combined with the afore-mentioned mentoring process if the paper is submitted more than two months in advance of the regular deadline. The goal is to remove those papers that would wind up receiving uniformly low scores (less than 2 out of 5) from the pool of papers to which external reviewers will be assigned. The extra work up front by the ACs would result in a smaller number of papers that would need to be managed later. While this suggestion may seem unfair or undemocratic, it may improve the overall quality of the conference by allowing a more effective allocation of the scare reviewing resources.

In addition to increasing the quality of submissions considered for review, more attention needs to be paid to the quality of reviewing. One of the challenges to obtaining quality reviews is to establish a good match between the reviewer and the paper being reviewed. Typically, papers are characterized by a number of author-selected keywords, and reviewers describe their expertise through the same keywords; the system can then match reviewers to papers to assure that expertise is being applies correctly. Unfortunately, this approach doesn’t always work, in part because reviewers’ self-characterization may not reflect their actual expertise.

There are several possible ways to mitigate this mismatch.

  1. Obtain the publication record of each reviewer from the digital library (or through some other means such as Google Scholar). Use keywords from papers (perhaps weighted by author order) to characterize a reviewer’s expertise.
  2. Obtain ratings of reviews (as described earlier) to well-reviewed papers for each reviewer. Use the keywords of those papers to establish the expertise profile for each reviewer. Ratings  can be obtained either from ACs, or from the authors themselves (as suggested by Wendy Mackay). To control for bias, only ratings for rejected papers should be used to assess review quality. The process of inferring keywords based on review quality can also be used to assess the accuracy of self-reported expertise in a range of areas.
  3. Introduce bidding, a process where prospective reviewers get to bid on which papers they would like to review. This approach is used in a number of other conferences, and might be an effective solution to the competition for reviewers that occurs at the beginning of the CHI review period.

The CHI conference is growing (the number of submissions was up by 18% over last year), and it is important to make sure that we have an effective and efficient review process that balances the needs of the authors and the reviewers.

8 Comments

  1. Twitter Comment


    Posted “Comments on the CHI reviewing process” [link to post] #chi2010

    Posted using Chat Catcher

  2. Twitter Comment


    RT @HCIR_GeneG: Posted “Comments on the CHI reviewing process” [link to post] #chi2010

    Posted using Chat Catcher

  3. you mention scanning documents in advance for serious flaws that preclude publication. this would obviously be quite a superficial quick checklist? was there any discussion of what these critical flaws might be? certainly a push in the CHI community was to move away from documents fitting ‘the CHI format’.

    also – i would be worried about basing reviewer quality purely on disgruntled authors who would have had their papers rejected. it would depend which stage of grief these authors judged the reviews. if all authors rated them, then perhaps this would be countered out by the lull of security that accepted authors would be feeling.

  4. I haven’t devised a taxonomy of papers that would be subject to desk rejection, but these are the kinds of papers that get 1s and 2s when subject to reviews.

    I think it makes sense to have ratings issued by authors when writing their rebuttals. That way, there is an incentive not to be too nasty, since your paper is still being considered. Also, that would tend to pull ratings from people who received the longer, more thoughtful reviews. It may also be possible to tune things by weighting review scores based on the reviewers’ ratings of the paper (giving more weight to the middle scores), but that smacks of over-engineering.

  5. “A number of ACs (say 3-4) read each submission to scan for obvious flaws that will preclude acceptance. ” — you’ve just overloaded an even smaller pool than the reviewers — high quality PC members! This is the bigger problem. PC members shouldn’t have to be using keyword search to find reviewers. If they don’t personally know at least two people in the field who would be suitable for a paper, they really are probably not experienced enough for this job. The problem is that the load is so high that many senior people no longer want to do it and the large number of papers being submitted means that many junior people without this proper background are pulled in earlier in their careers than they might be otherwise. Not sure of the solution, but I would hope a quick reject by a smaller number of people (say 2 ACs) would work better (doesn’t increase the load, just moves it earlier).

  6. I think one reason that it’s hard to find reviewers is that the same people are being sought by different ACs for different papers. A bidding process (implemented at many other conferences) would address that. As to extra work, I had to manage 12 papers this time, several of which could have been desk-rejects. In the end, it took more time to manage them. The reason I proposed 3-4 ACs (the number is, of course, arbitrary) is to reduce the chance of an improper desk-reject.

  7. Tony says:

    Will any of these proposals be implemented for 2011, and if so, which ones?

  8. I am not on the conference or program committee for CHI 2011, and so cannot give you any definitive answers. My guess is that the most likely thing to change is moving to the bidding process for selecting reviewers because that feature is already available in the review management software. Other changes will require more investigation and probably buy-in from SIGCHI. If you are interested in seeing any of these (or other changes), please contact SIGCHI officers and let them know. The more people express their preferences, the more likely change is to happen.

Comments are closed.