Writing CHI Rebuttals

on

CHI rebuttals are due at the end of the week. What to do? What to write? How do you convince those reviewers (particularly Reviewer #3) that your work has merit, if only they would brush up on their understanding of regression analysis. I am not promising any miracles, but I’ve written and read a few rebuttals over the years. Here’s my take.

First, rebuttals matter. They matter because they show that you haven’t given up, that you haven’t conceded the reviewers’ criticisms. They matter because sometimes reviewers do get it wrong. They matter because sometimes the reviewers get it right, but the flaws can be corrected. And they matter because they help you articulate the merits of your work, which will not be wasted effort regardless the outcome.

So how does one write a rebuttal?

Start with a few deep breaths. As much as you might feel like grabbing the reviewers by the neck and shaking vigorously, that’s probably not going to improve your chances of getting accepted. A calm, rational response, on the other hand, might do the trick. Be factual. Rhetoric and invective will not help your case; they’ll just piss off the reviewers. Besides, you can always blog about the stupidity of the reviewers later.

The next step is to understand the big complaints reviewers might have had, and whether these are fatal flaws in your work, or aspects that can be corrected. Let’s assume the latter. Read the reviews carefully, and identify issues that you can reasonably address. You’re not going to run another experiment, but you might be able to expand your data analysis. Look to the meta-review to see what items the Associate Chair thought were important among the litany collected by the reviewers. The meta-review should highlight the important shortcomings; addressing those will improve your chances of acceptance. Don’t worry if the reviewers don’t exactly agree. Try to make sense of their comments with respect to the work you did, not the work the reviewers might have wished you had done.

Make sure reviewers know what the takeaway from your paper was (particularly if any of them expressed doubts). This is the elevator pitch for your paper, and you should have it nailed. Sometimes reviewers don’t read the paper as carefully as you’d like them to (after all, they may be reading lots of papers, and doing other things as well), so adding a bit of emphasis here may help alert them to what’s important about the work.

One useful tactic for writing a rebuttal is to offer concrete changes that are responsive to reviewers’ comments. This may apply to related work, to motivation, to analysis, etc.

If the reviewers think you’re short on related work, put in a paragraph or two that relate your work to the papers the reviewers suggested were missing.

If the reviewers didn’t understand your analysis, or wanted to see a different cut at it, put some details of the analysis (numbers help!) right into the rebuttal with the explanation that you’ll add it to the paper. We used this approach to shore up the qualitative analysis presented in the ReBoard paper last year. The reviewers wanted to see more than anecdotal quotes, so we analyzed our data and reported frequencies of activities in the rebuttal.

If reviewers disagree about the merits of the work, it may be possible to cite the opinion of a more supportive review to bolster your case. When doing this, however, be sure the argument is valid, and that the meta-reviewer has not dismissed that review. Sometimes a review with a positive (or negative) score doesn’t provide a strong argument to support the score, so appealing to those reviews is not useful.

When writing the rebuttal, start by thanking the reviewers. Then set out the points raised by the reviewers that you will address. Don’t waste too much space on the outline, but give the reader a clear sense of where you’re going. Don’t obsess about the things you cannot correct; focus on the aspects you can change. When you’re done, get someone impartial to read the reviews and your rebuttal, and ask them to comment on the tone as well as on the facts.

In summary, your goal is to convince the reviewers and the meta-reviewers that although as written your paper had some shortcomings, you are able to correct them if the paper gets accepted. Because reviewers are instructed to assess the paper as submitted, it may be difficult to salvage a paper that requires a lot of re-writing. On the other hand, if you can demonstrate your ability to correct the problems by offering the solutions in the rebuttal, that may persuade the reviewers that you can make the required changes. Sticking to the facts and offering concrete evidence of your competence to make the changes may help.

7 Comments

  1. I’m happy to hear that ReBoard was a survivor of the rebuttal process. I spoke with Stacy about that project at GROUP and it was one of my favorite papers at CHI 2010.

    Thanks for this post. The rebuttal process is starting to feel less like Fight Club. :)

  2. Thanks Gene! A helpful and informative post to help me along with my first rebuttal, that I am currently working on.

  3. […] This post was mentioned on Twitter by Nirmal Patel, Gene Golovchinsky, Nick Diakopoulos, Sarita Yardi, Max L. Wilson and others. Max L. Wilson said: RT @HCIR_GeneG: Posted "Writing CHI rebuttals" http://palblog.fxpal.com/?p=5001 #chi2011 […]

  4. Jason says:

    Thank you for the post. This is my first rebuttal. Is it customary to thank the reviewers before starting the review, or is that not necessary?

  5. Jason says:

    *should have been rebuttal, not review

  6. Does every submission get rebuttal, or only those borderline papers? Is it a build-in process?

  7. I believe every submission gets a chance to rebut CHI reviews. Of course if your scores are low (say, lower than 2.5) it is unlikely that the rebuttal will have any effect on the outcome, even if the reviewers do raise their scores slightly. I don’t expect large jumps in reviewers’ scores unless there was some simple misunderstanding. Typically, a well-argued rebuttal might cause reviewers to adjust their scores by 0.5-1.0 points, which may not be sufficient for acceptance if the starting average was too low.

Comments are closed.