Congratulations! You were asked to write a rebuttal response for your paper. That is a great achievement! It means your paper made it to round two of the reviewing process or the program committee discussion and the program committee considers accepting it for the conference. Writing a good rebuttal response is the next step and might help your paper to get accepted.

Keep in mind that while even an awesome rebuttal won’t save a submission with damning reviews, submitting an empty rebuttal might probably kill a submission accepted basically everywhere but on paper (Saw it happen a few times as revier - Dom). Do not expect too much from writing a rebuttal (unless the reviewers made serious errors), but still try your best.

No lone wolves. Rebuttal phase is a highly communicative phase. All communication with PC and reviewers is official communication by all authors, run all your planned messages and texts by your collaborators and especially the professor.

Steps#

Important Checks. As very first steps, check the following:

• Are there significant factual errors in the reviews that require contacting the PC? E.g., review for a different paper based on summary.
• Any requests that require additional experiments, data, or reaching out to collaborators? These will take time and should be started as soon as possible.

See also references. If you have the time, read the articles in the references below in detail - Andreas Zeller and Matthias Payer make some really good points about the rebuttal process.

1. Set up a shared document (Google Doc, Overleaf, etc.) for the rebuttal following the template below.
2. Follow the protocol below more or less closely to write the rebuttal.
3. See also examples for filler text and a worked example below.
4. Submit the rebuttal in the submission system (HotCRP, EasyChair, etc.) when everbody is happy with it.

Template#

We use a specific template for writing rebuttals that grows from the bottom up. The template is usually a Google Doc, but works also in any other collaborative writing tool. In general, it consists of the following headings, in Google Docs you can also use the tabs feature to separate the different sections of the rebuttal.

1. Metadata section
2. The actual rebuttal section to be filled later
3. The review table to be filled with individual points
4. The raw reviews from email or submission system

Metadata Section#

Provide metadata at the beginning of the document, mostly so

• People working on multiple rebuttals know on which one they are currently working.
• People can directly check limits like rebuttal deadline and word limits.
• People can find it, if they search for it years later in Google Drive.

Recommended metadata includes:

• (Submission) title of the paper
• Submission venue and year (so the document is searchable), e.g., “IEEE S&P Third Round (‘Oakland’) 2023”
• Rebuttal deadline (including first response deadline and actual deadline)
• Word limit
• Received rebuttal scores

Rebuttal Section#

Placeholder for the actual text of the rebuttal draft, after all steps are finished.

Ignore HotCRP for now. Do not submit any draft rebuttal responses in HotCRP. While in general, drafts should not be visible to anyone but the authors, a misconfigured HotCRP gives reviewers access to these drafts (which has happened in the past).

Review Table#

Table for the points made by the reviewers. This is the most important part of the rebuttal document, as it allows you to keep track of the points made by the reviewers and your responses to them (especially with multiple collaborators). The table should include the following columns:

This can also be type set as normal bullet points, but the table allows for easier collaboration and overview of the points made by the reviewers.

Tags assigned to responses usually include:

• [Requested] Points that are requested by the reviewers, e.g., as “Questions for the authors”.
• [Should Address] These points are vital for the review and should be answered, even if points in other categories have to be cut short or even completely from the rebuttal.
• [May Address] Still important points which may be cut for “Should” points.
• [Won't Address] Points so outlandish or gibberish that you decided explicitly to not address them.
• [Factual Errors] Factual errors committed by reviewers. Often easy to resolve (“You assumed X, we meant Y, we will rewrite Z to make this more clear”). Decide if to address those points on a case to case level based on severity.

Raw Reviews#

Actual full-text of the reviews (so that they are collected in the same place). Minimal styling like Google Doc headings to the individual reviews for easier scrolling.

Recommended Protocol#

As with any scientific activity, writing a rebuttal benefits from a structured approach (especially if you are not the only person working on it).

0. Collect Reviews#

Depending on the venue, you received the reviews by email, in a submission system, or any other way. For the following steps and to allow for easier collaboration, copy and paste the relevant parts of the reviews at the bottom of the template in a small font size.

Relevant parts include not only the strenght/weaknesses but also comments, summaries, and ratings (and any other part that is relevant for a review).

1. Extract Points#

The reviews are likely written as a flow text (at least partially). Extract individual points made by the reviewers and place them in the reviews table above the raw reviews. Tag each point with reviewer and their rating, e.g., [RevA, WA] for the weak accept of reviewer A and optionally color code them in some way (will be relevant later).

Collaboration: If working in a team, feel free to do multiple passes over the reviews, check each other’s extracted points for correctness.

2. Assign and Merge Topics#

Review the extracted points and assign topics to them (“Abstract - Overclaim in Abstract”, " Related Work - Missing References", etc.). These topics are later used to group multiple points so that providing a single answer is sufficient (which will save space and allows for addressing more points in a rebuttal).

Because you aim to later group points for each topic, you should look for topics that can be easily answered with a single answer, e.g.:

• “Criticising Methodology” is probably a bad topic for multiple points with different criticisms regarding the methodology, as you probably want to answer each one individually.
• “Badly formatted figure labels” is probably an okay topic, even if the reviewers talk about different figure labels, as you can concisely address it with “we will fix the figure labels”.

Collaboration: Feel free to get multiple opinions on assigned topics. You can go even further and actually “code” reviewers’ points with multiple collaborators.

3. Write Answers#

Try to write answers for each identified topic. This is step is fairly early in our protocol, because it allows for an easy check & control of earlier steps.

If you notice, that your assigned topics are not sufficient for a specific answer, feel free to reassign points or create new topics. You can write the answers in bullet points, but answers at this stage will likely still be longer than in your final rebuttal.

Start answering early! Reviewers’ points often include highly specific clarification requests for data subgroups or statistical approaches. These might require multiple days to compute & answer (or to contact the collaborator who can answer them).

Don’t be afraid to reach out. While not exactly common, there is nothing wrong with reaching out to the PC to ask reviewers for clarification on certain points (Talk to Professor beforehand).

E.g., if a reviewer claims your work “is not novel and has been published before in similar form” without providing any references, you might want to reach out and ask them to provide references or rescind their point.

Collaboration: Feel free to write alternative answers or revise existing ones. Ask collaborators if you can’t remember why we decided on X or did thing Y the way we did (This is one of the many points in time, where a clean research documentation pays itself off).

4. Decide Importance#

Mark the different topics in the table (or in the response field) with one of our usual categories:

• [Requested] Points that are requested by the reviewers, e.g., as “Questions for the authors”.
• [Should Address] These points are vital for the review and should be answered, even if points in other categories have to be cut short or even completely from the rebuttal.
• [May Address] Still important points which may be cut for “Should” points.
• [Won't Address] Points so outlandish or gibberish that you decided explicitly to not address them.
• [Factual Errors] Factual errors committed by reviewers. Often easy to resolve (“You assumed X, we meant Y, we will rewrite Z to make this more clear”). Decide if to address those points on a case to case level based on severity.

Extreme errors. In cases of extreme errors committed by reviewers that affected the review outcome, you might want to contact the PC before rebuttal deadline so that you might receive a more appropriate review (talk to Professor beforehand).

Keep in mind, that you are not ranking points based on how much you like them. An insulting but wrong reviewer’s point might not seem worth your time, but is often easily answered and should be addressed.

Collaboration: Feel free to discuss & reorder points in collaboration.

5. Order#

Order topics in the individual categories based on the planned final order in the rebuttal. This is a style choice, simplest suggestion is to go by order of (somewhat arbitrary) importance, e.g.:

1. Number of involved reviewers (RA,RB,RC,RD > RA).
2. Importance in paper (Criticism concerning Abstract > Criticism concerning some figure labels).
3. Type of comment (Criticism > Clarifying Questions > General Questions).

For an advanced psychological approach to addressing points, see Prof. Zeller’s text in the references below.

Collaboration: Feel free to discuss & reorder points in collaboration.

6. Write Rebuttal#

Now it is time to actually write the rebuttal.

1. Copy & Paste the pre-written topics & answers from table to the rebuttal area.

2. Reorder the topics in the rebuttal area to match the order you decided on in the previous step. Add bold intros to each topic, e.g.:

   [Topic summary] (Rev A, Rev B): [Answer …]

   You can also include in-line references to the reviewers, e.g.:

   [Topic summary] (Rev A, Rev B): [Answer 1] (RA). [Answer 2] (RA,RB).

3. Add filler text (Greetings, addressed points etc. See filler texts and worked example below).

4. Extend answers to actual flow text.

   Don’t worry to much about text limits at this point (it will be too long anyways. If it is not, address more points!).

Collaboration: Reiterate & collaborate on witing the rebuttal.

7. Shorten Rebuttal#

At this point, you should be above the word limit for a rebuttal. Attempt to shorten sentences or maybe decide on removing the very minor points.

Trying to trick the count system is generally not worth the time (which should instead be spend in rewriting answers) and often negatively impacts readability.

Still, a few common shortening hacks that don’t affect readability too much:

• Shorten reviewer identifiers (Reviewer A –> RA).
• Compact references ([1, 2, 3] –> [1,2,3]; counts as one word).
• Utilize reference numbers from paper (Don’t copy full references below your review, just write [13] if you referenced this work in your paper. Reviewers generally figure this out).

Collaboration: Help with shortening & deciding what points to keep.

Rebuttal Style#

Some general answer guidelines you should stick to:

• Do not just say you will “refocus/update/better clarify Section X.”! Give concrete examples on how you will update this section and what you hope to accomplish with these changes.
• Be friendly to the reviewers, but do not blindly agree with them on everything. Agreeing that your work “is not generalizable and lacks validity” is an easy reject.
• If possible cite other works that utilized the same or a similar approach to support the validity of your study.
• Do not propose to change/update/revise foundational concepts underlying your research (e.g., your research questions), because that is (a) not possible unless you have a time machine and (b) would require so much work that it would be an easy reject and resubmit for the reviewers. Focus your proposed change on better highlighting certain aspects or considerations for these concepts missed by the reviewers.

Filler Text Examples#

Some generic filler texts (greeting, intro, outro) for your rebuttal. Nice to have, but if you are low on space, better focus on actual rebuttal content.

One exception: if you are aiming for a revision or conditional accept with your rebuttal, you should use your intro filler text to convey to the reviewers that you are super incredibly convinced that you will be able to address all their concerns for the deadline / revision.

First of all, we would like to thank the reviewers for their detailed, constructive, thought-provoking reviews. We are currently editing our paper to address the helpful reviewer suggestions, and are confident that all suggestions can be addressed with minor rewriting and clarifications, as outlined below.

We thank the reviewers for their time, effort, and thoughtful insights. We’ll address all suggestions made by the reviewers to improve the paper; we excluded smaller comments below due to the word limit.

We are delighted to see that you see merit in our submission regarding security and trust in open source software and deeply appreciate your suggestions for improving our work even further.

Dear Reviewers, We thank you for your already invested time, your service to this community, and for providing us with your insightful and warranted reviews. Due to length requirements we were sadly not able to address every individual remark, but of course aim to incorporate them into our work nonetheless.

Aside from the major points listed above, we will of course also include minor points that we did not address specifically, such as spelling corrections and minor structural suggestions.

Worked Example#

Below is a worked example which goes over every sentence of a rebuttal that got the paper accepted and illustrates why it is there.

Before the rebuttal, the paper had 1 weak Accept and 3 weak Rejects.

Structure#

The rebuttal below was created following the protocol described above (but in a flow text instead of table, the table was introduced later to this process):

• Reviewers’ points were sorted into categories (so that one answer is sufficient, saves space).
• Points are shortly summarized (“Regarding your comments about our related work”) and linked to the specific reviewers (RA -> Reviewer A).
• Points were ordered by their perceived importance:
  1. Number of involved reviewers (RA,RB,RC,RD > RA).
  2. Importance in paper (Criticism concerning Abstract > Criticism concerning some figure labels).
  3. Type of comment (Criticism > Clarifying Questions > General Questions)
• References are either from the paper ([13]) or directly mentioned in the rebuttal (Acar et al. "Security developer studies with GitHub users: Exploring a convenience sample."), adding them as footnotes might be confused with paper references and cost you more space as you probably should add publication venue and year.

Actual Rebuttal#

Dear Reviewers,

Address the recipient(s) directly, “Dear X” is a common approach.

We thank you for your already invested time, your service to this community, and for providing us with your insightful and warranted reviews.

Be nice to your reviewers! They do it for free.

Due to length requirements we were sadly not able to address every individual remark, but of course aim to incorporate them into our work nonetheless.

You probably won’t be able to address every point (sufficiently) within the word limit. Don’t be afraid to point this out and reassure the reviewers that you will of course still address their highly important request for a slightly different wording on page 5 in paragraph 12.

About your concerns regarding the abstract (RA,RB,RD): In an attempt to shorten the abstract, we clearly overshot our target and will re-extend it with more in-depth specifications and more detailed and better aligned claims to remedy that.

Don’t be afraid to be honest about easy fixes. For this paper, we initially prepared the abstract for another conference with a more strict abstract word limit.

This is a fairly trivial thing to fix, so no need to die on this battlefield, just agree with the reviewers and move on.

Regarding your comments about our related work (RA,RB,RC), we will consolidate papers with similar topics within subsections to provide a better overview of topics and extend upon their position relative to our approach and findings.

The criticism wasn’t in-depth for these points, so no need to overthink the answer. Just agree that you will address X by doing Y and move on.

Regarding sampling, validity, and generalization (RA,RB): we aimed to improve our data quality by approaches described in [19] and filtering by manually removing low quality participants. Nonetheless, we acknowledge that crowdsourcing platforms are inherently biased and will improve the highlighting of this fact in our work. In addition, we will word our claims more carefully to not evoke the impression of an overgeneralization.

Criticism of sampling and validity of your studies are probably the most likely ones that get your paper rejected, so address them carefully and thoroughly.

All approaches involving sampling from a population are in some way flawed. Nothing wrong with conceding certain weaknesses of your approach to the reviewers, but always show confidence in the validity of your research (otherwise, you should not have selected this approach).

Office vs. Home (RA): We were aware of this distinction and aimed to address it by, e.g., including it as a (optional) factor in our regression models and by stating the scenarios accordingly, but did not find relevant and statistically significant effects. We speculate this might be due to the generally wide-spread usage in both office and home environments.

Later conversations showed, that we misunderstood the points here. Still, this appeared to be more of a “In your paper about X why didn’t you do Y” point, which can be fairly easily addressed with “because Z”.

Weak discussion (RA,RB,RC,RD): We agree with all of your points; we will distill specific recommendations for academics, industry, end users to allow a broader field of readers to profit from our work, as well as give recommendations for policy. We hope to thus deliver a more pointed conclusion.

Important point, but easily addressed. For bigger changes (and when the reviewers did not give specific recommendations) list the exact changes you plan to include as this gives your address more weight and shows your willingness to work with the reviewers. Do not forget to include why this will make your work better.

“(secure) email” (RB): Multiple participants specifically mentioned “secure email” or somewhat related terms such as “safe” or “trustworthy” , e.g., direct quote from P59: “Probably via a secure e-mail or an online notification to my system.”

Running out of actual criticism to address, now follow the clarification requests. This reviewer doubted that participants used the wording of “secure email”. Easily addressed by providing an actual quote from a participant with this wording.

Qualitative Coding (RD): In the “immediate resolving” approach, conflicts are resolved when they emerge by all coders in a consensus discussion and/or by introducing new codes (followed by recording all answers). As such, any reported inter-coder reliability would always be 1. Similar approaches were used, e.g., by Stevens et al. in “The battle for New York: A case study of applied digital threat modeling at the enterprise level.”

Unfamiliarity of used methods: you will waste too much space explaining your approach in-depth, just give a quick summary and provide examples of other research with similar approaches (preferably recent, highly cited & not from our group).

2x3 experimental design (RD): we chose this design due to (1) avoiding habituation effects by assigning the phrasing condition in an in-between group design and (2) achieving more reliable data for comparing the different scenarios in a within subject approach.

Out of criticism, just answering some general questions in the remaining space. Answering fairly general questions from reviewers not familiar with your field is not generally a bad thing, as these are often straight-forward to answer and will increase the confidence of this particular reviewer in you paper (but make sure to address more specific criticism from the expert reviewers first).

Selecting models by AIC (RD): Model selection by AIC/BIC is a common approach to generate (explorative) models with optional factors, e.g., also found in Vaidya et al. “Does being verified make you more credible? Account verification’s effect on tweet credibility.” and Acar et al. “Security developer studies with GitHub users: Exploring a convenience sample.”

Another unfamiliarity of used methods: this time even with three published examples of similar approaches (including one of our papers).

Our rebuttal did not include an ending due to space. You can end by sending your regards or similar if you have the space, but preferably address more points.

This is not the place for important things you want the reviewers to read. Put that at the beginning.

After the Rebuttal#

Take a deep breath and congratulate yourself on writing the most information dense 400 words text you probably will ever write in your life or at least this month.

The path forward is fairly similar regardless if your paper gets accepted or not:

• Go again over the reviewers’ points and decide whether addressing them will improve the paper or not (this is different from whether you should address the point in a rebuttal). Convert these points into actionable TODOs either for camera-ready or next submission.
• Try to identify sections that reviewers misunderstood. You should revisit these sections and rewrite them accordingly.
• Assess whether your work could benefit from additional experiments. If your work is accepted: consider them for a follow-up paper, else plan the next submission accordingly.

Change Lists#

Some conferences require change lists, below is an example for how a change list could look like.

Dear Shepherd,

Thank you again for your valuable feedback on our paper. Below, we summarize the changes we plan to implement in response to your main concerns, along with several other improvements. If everything proceeds as planned, we aim to share a revised draft by [XX].

The rest of the change list is structured similar to the rebuttal, be precise and concrete about the changes you plan to implement. Can also include headlines if you want to group the changes further.

In some cases the shepherd will highlight specific points that they want to see addressed in the rebuttal, absolutly address these points in the change list and be very specicic about how you will address them, e.g., addressing shepherd’s point #7:

#7 Clarifying the Threat Model (RA,RB): We will add a new section to the paper that explicitly defines our threat model, including assumptions about the adversary’s capabilities and goals. We will also clarify how our method can be applied in different contexts, such as in the analysis of real-world protocols.

After this, you can outline other changes you plan to implement in the paper. Usually no lenght limits for a change, so feel free to add all points by the reviewers. You are usually even allowed to include attachments such as revised figures or tables. The examples below are generated ones (not from a real paper) just to illustrate the format / wording.

Limitations of Applicability: We will add a dedicated section to the paper discussing the limitations of our approach, particularly its reliance on structured input data and controlled environments. We will note that the method may not generalize well to adversarial or highly dynamic settings. Additionally, we will clearly state that our method is not intended as a replacement for threat modeling frameworks, but as a complementary analysis tool for early-stage protocol evaluation.

Clarifying Evaluation Criteria: We will revise the evaluation section to clarify our use of F1 score and time-to-detection as primary metrics. We will justify their selection based on our threat model and empirical goals, and we will acknowledge that they may obscure false negative rates in highly imbalanced datasets. To address this, we will add a breakdown of errors by class and include a discussion of precision-recall tradeoffs.

…

The outro is similar to the one in the rebuttal and should hightlight that you are looking forward to additional feedback and are happy to incorporate any further suggestions or requirements.

We appreciate the thoughtful comments and guidance. The plan above reflects our current intentions, and we are happy to incorporate any further suggestions or requirements you may have. Please don’t hesitate to share additional feedback.

Best Regards, Authors

References#

1. Patterns for writing good rebuttals by Andreas Zeller

   Backup (page tends to disappear)

   Patterns for writing good rebuttals#

   by Andreas Zeller

   I compiled the following patterns for rebuttals (also known as author clarifications) for major software engineering conferences (ICSE, ESEC, FSE, ASE, ISSTA), having seen a number of rebuttals as PC chair of ESEC/FSE 2011 and having written a number of rebuttals for top conferences. These patterns may or may not be applicable in your context; use at your own risk.

   • Understand the decision process. Most SE conferences apply a process called Identify the Champion. Carefully read these process patterns, understanding the roles of the reviewers and the PC chair, and where your rebuttal can make a difference.
   • Identify the undecided. These are reviewers who may be swayed by your arguments. You can identify them (a) by generally liking your paper, but not the details, (b) because they provide their scores, or (c) because they explicitly state that they may be swayed. Focus your arguments on these reviewers. (And if you ever become a PC chair: include scores in the reviews, such that your authors know whom to focus upon.)
   • Identify the champion. The champion likes your paper anyway, so don’t bother too much with her review. If there’s no potential champion, chances of getting your paper accepted are slim.
   • Arm the champion. In the final discussion, your champion will need arguments against the detractors, especially strong detractors. Refute every single issue raised by the detractors to give your champion extra arguments for acceptance. Lower the confidence in the detractors’ reviews by pointing out mistakes.
   • Identify the detractors. A strong detractor can only be countered by a strong champion. Rather than trying to dissuade a strong detractor, your aim should be on arming the champion (see above).
   • Answer the questions. Some conferences want you to only answer the specific questions the reviewers have asked, and/or instruct their PC members to only focus on these answers. Try to relate every argument of yours to a specific question.
   • Write for the PC chair. If your reviewers have done a bad job, say so in the beginning, and the PC chair may assign you another reviewer. Stick to facts (review is just one paragraph, reviewer claims “no Foo” when Section 3 is named “All about Foo”, etc.) that the PC chair can check in less than a minute. If a reviewer completely misunderstood the paper, feel free to report this, but keep in mind it is your duty to ensure even a casual reader gets the message. If all reviewers misunderstood your paper, the problem is not the reviewers.
   • Write for the committee. When your paper is discussed during the PC meeting, reviewers will glance through the reviews, and also read your rebuttal. If there’s one thing you want the committee to know, say so in the very beginning (because that’s all most reviewers will read). This assumes that your paper will be discussed, so focus on getting it discussed first.
   • Convince. Convince the reviewers that you’ll be able to improve the paper. If the reviewer asks: “Do you have a formal definition of Foo?”, don’t say “Yes”. But don’t say “Thanks! We’ll add this.” either. Say “A Foo is a Bar within a Qux range; we’ll add a detailed definition.” You don’t need to provide the full change; just sketch the change you will make.
   • Choose comments wisely. Focus on major concerns - the ones where the reviewer assumes you may not be able to improve the paper. You do not need to convince the reviewer that you’re able to fix typos, straight-forward presentation issues, language issues, or anything else that can be fixed by simple proofreading. This is taken for granted.
   • Organize your rebuttal. Make it easy for reviewers to find their questions (and your arguments). Refer to reviewers by R1, R2, …; questions by R1Q1, R1Q2, … Use bullet points and short sentences to allow for quick scanning. Start with the most important and most common concerns, going down towards lesser issues.
   • No tricks. If the limit is 500 words, do_not_trick_the_process by_replacing_spaces_with_underscores or likewise; all that will happen is that the PC chair will delete your rebuttal. If it’s 2500 characters, do not assume that reviewers KAAYAYA and AYP because they FSITPOG (know all about you and your acronyms and accept your paper because they feel stupid in the presence of genius).
   • Thank the reviewers. A rebuttal is not a place to vent off your fumes, even though your work is so great and the reviewers are all so stupid. Reviewing is hard work and even the least significant review contains grains of truth which will be helpful for your future work.
   • Don’t expect too much. Your rebuttal will not save your paper. Your rebuttal will play a role only if your paper is on the edge, or if reviewers made very obvious mistakes. If the reviewers update their reviews to reply to your arguments, that’s already a lot. But regardless of the outcome, your rebuttal will help to improve your paper and eventually get it accepted - if not now, there’s always another deadline on the horizon.

2. How not to alienate your reviewers, aka writing a decent rebuttal by Matthias Payer