Reviewing

Reviewing#

At some point in your scholarly career, you likely will get asked to review a conference submission (or poster, see further below).

Over their career, each scholar develops their own approach and style of writing reviews. Nonetheless, This article is intended as an general introduction for newbies and goes over the common mindset and review steps.

This Document. The content of this document emerged from collaborative brainstorming across multiple research groups. The process involved (1) each group contributing ideas and questions into a shared Google Doc, followed by (2) a joint call to review and refine the entries. Contributions are often prefixed with the names of individuals, and occasional disagreements or different approach are included.

Mindset#

The best review mindset is probably to see the process from both sides: as reviewer and author.

  1. Your initial question should be: “As a reviewer, how can I convince the authors of my viewpoints?”.
  2. Your goal should be for the authors to say: “I agree with the reviewer’s points.”.

More specific as a mindset:

  • Be positive: Rejecting papers is easy and impresses no one.
  • Be constructive: Don’t be “that reviewer” we all complain about.
  • Separate technical vs. philosophical weaknesses: This will help in the PC discussion.
  • Questions for authors: Give R2 paper authors a chance to answer unknowables.
  • Keep confidentiality: Continue institutional trust in the process, particularly for vulnerabilities.
  • Assume Responsibility: You are responsible for your reviews and all their content (especially if you handed the review to an external reviewer).

Do

  • Constructive criticism.
  • Clear explanations of what is missing.
  • Knowledge about the topic or disclosure of not being knowledgeable.

Do Not

  • Unprofessional roughness.
  • Uninformed rejection, e.g. someone is uninterested in user studies and says that the paper is crap because user studies are crap.
  • Lack of explanation of the acceptance / rejection rating.
Novelty. Do not claim a work “is not novel and has been published before in similar form” without providing any references. Authors might reach out and ask you to provide references or rescind your point.

Approach#

The general review form usually covers the following four points in some form, sometimes in combined categories:

  1. Soundness (of research methods)
  2. Relevance (to the conference)
  3. Novelty (of research)
  4. Presentation (of paper)

Structure#

Most reviews follow a structure provided by the conference submission system (but generally cover the following points).

Basics#

  1. Is the paper “on topic”?
    • Check call for papers & maybe check in with others who have reviewed there before
  2. What’s the quality of the venue? (Top tier, some workshop, somewhere in between)
    • Some workshops really just act as work cemeteries. Someone did work, the idea and/or the method weren’t great, they will be unable to have a huge impact at a top conference, and it’s not worth redoing it all. These papers may end up at workshops (also as deterrents for other people to try the same idea, or try the same method), and it doesn’t make sense to have the “top tier” standards for workshop papers.
    • Top tier conferences mean the work has to be novel, the problem has to matter, and the method needs to be valid. Small writing/presentation issues can be fixed with shepherding; stats or method issues maybe with a revision. “Someone tried very hard but the idea just didn’t make a lot of sense from the start” is just not “good enough” for top tier.
    • Specialist venues (SOUPS) have high standards, too, at least for methods & “fit”.
    • Journals: seem to be not really important in our field. Reviewing is SLOW also due to terrible usability.

This informs (1) whether you even write a full review, or just check in with chairs and potentially write a nice suggestion to submit elsewhere; (2) how you review, e.g., is the paper more or less pre-accepted unless it’s utter bs (workshop)? Highlight positive things, ask to add to limitations, done. Or is this a potential top tier paper? Be very thorough (but encouraging) in your review, the bar is high :)

Summary#

  • Checks if you actually read and understood the paper.
  • Sums up the content in 4-5 sentences.
  • Don’t just copy paste rephrase from the abstract.

Dom: My basic template for this:

The paper investigates / introduces … For this, … The authors find … Based on their findings …

Strengths/Weaknesses#

  • Often lists (bullet points are okay).
  • Are often ordered by priority (e.g., “paper well-written” goes last).
  • Should be somewhat representative of your final decision (A reject likely has more weaknesses than strengths).

Potential Points:

  • Paper Structure (Is the order sensible? Is everything present you would expect?)
  • Are the sections well-developed? (do they seem too short/long?)
  • Is there any relevant related work missing?
  • Does the paper answer all questions they tried to answer?
  • Is the methodology clearly explained and fit to answer their question?
  • Is the article well-written and easily understandable?
  • Are there any minor errors you would also want to point out? (While you should not base your decision on this, highlighting e.g. typos or minor format errors is helpful for the final paper version!)

Comments for Authors#

This is likely the main part of your review.

  • What should they work on to make their paper better?
  • Are there any questions you want them to clarify (that are not necessarily positive/negative points)
  • Intro: Thank the authors for the work/effort they put into the paper and for their submission
  • Remember: the paper likely took PhD student(s) multiple months of work. Even if the paper ends up being rejected, appreciate their work!
  • Important: If you’re not an expert in the field of the paper, let the authors know, e.g. “I read this paper as an educated outsider.”

Yas: I usually go through from top to bottom, literally writing my review in parts that look like this:

  • Introduction: This introduction is well-written and motivates the research clearly. The research questions are clear, and well-motivated.
  • Working my way from introduction to conclusion. And I change up the style sometimes so people don’t recognize me.
  • I try to check for all these points here, and cushion my review in uplifting language. :)
  • The paper should …
    • be clear about the hypothesis and/or research questions.
    • clearly state the contributions and novelty.
    • clearly explain the experimental design, the execution, the results.
    • Special attention to a thorough methods section!
    • disclose limitations in detail
    • explain ethics + data protection
    • supplement with meaningful tables and diagrams
    • have sound statistics!
    • have sound related work
    • Results and discussion align with research questions/motivation of the paper
    • be well-written.
  • If points are missing, these are weaknesses I mention in my review.

Sascha: Continue with all the things you like about the work. Take your list of strengths and add details.

  • Talking about weaknesses is all about coaching the paper to the next level for a future re-submission: Point to the weaknesses and sketch ideas on how to improve, e.g., if you did not understand their methodology, tell them to provide the missing details
  • If you want a paper to be accepted, be clear about it in the review, e.g. “The paper fills an important research gap and the results are of high value for the community.”
  • If you need to reject a paper, still be nice to the authors, e.g. “While I think the paper is not there yet, I encourage the authors to continue their line of work and implement all the changes I mentioned above.” But only if true! You can recommend a more appropriate venue too.

Misc

  • Lucy/Jaron: Super detailed lovely helpful comments like typos, hard-to-understand paragraphs go in a “detailed comments” “nitpicks” section.
  • Yas: ask for missing explanations in rebuttal!
  • Yas: ask for specific changes for shepherding/revision

Questions for Authors#

(Not on every review form as distinct section, you can add them to the comments otherwise)

  • Yas: ask for missing explanations in rebuttal!
  • Yas: ask for specific changes for shepherding/revision
  • Dominik: Try to limit yourself to questions that would change mind/decision on recommendation (e.g., ethics, consent, debriefing), as authors will burn their precious rebuttal word count on answering your questions.

Comments for the PC#

  • Stuff you might want to discuss with other reviewers but not the authors
  • Rating:
    • Should it be Accepted/Revised/Rejected?
    • Will you fight for this or not?
    • Honestly disclose how familiar you are with the topic.

Yas: Here I disclose which parts I’m sure / unsure about.

  • For example, if someone does super fancy statistics or a crypto proof I don’t understand, I will disclose that here and ask the chairs / other reviewers to double check.
  • Papers are often collections of different methods, and I may only be comfortable with reviewing a part. I however want to make sure that the paper got a full review from someone.
  • Here I will also stress ethical concerns, e.g., about data protection, payment (don’t exploit people, duh), consent, debriefing.
Guidelines from CHI

Tips from CHI:

  • no paper is perfect
  • make every efforts to see the merits of a paper
  • respectfully direct the authors on how to raise the level of their discourse to leap above the bar of acceptance, or that of excellence
  • the success of a field, a conference, and the careers of the authors, depend entirely on YOU (the reviewer) and the quality of the work you undertake
  • one more time: no paper is perfect, don’t wait for perfection to give an accept, it might never happen!

And:

  • our brain makes fast decisions on paper acceptance (within seconds and true for all humans!)
  • you will then unconsciously search for elements to confirm your choice (because after all you cannot be wrong can you?)
  • be aware of that process and fight it!
  • search for elements that confirm but also reject your initial choice
  • embrace the mantra: it is ok to change your opinion!

And:

  • presentation matters: figures quality, organization and flow, how well the paper lives up to the promises it makes
  • if those are a little bit off, papers tend to fare poorly in reviews
  • yet reviewers often feel the paper is not entirely satisfactory – that something was off – and then search for reasons to explain why
  • → judge for the content first, propose solutions for improving the form, be kind to language as most authors are not english speakers!

And:

  • a paper does not have to be “ambitious” to be important and publishable. You never know the impact of a project at the outset
  • some ideas may be provocative, and spark whole new directions, even though they have some flaws and limitations
  • others will wither and disappear from attention, perhaps awaiting the day when a topic returns to the fore
  • it is important to realize that all these are okay, even desired
  • trap: don’t reject a paper because it does not appear to you trendy
  • trap: don’t reject a paper because you would have done it in a different way
  • Let literature figure it out

And:

  • a paper with a new methodology, or something that is quite different from what we are used to see in HCI, may be judged more critically
  • you should particularly make a point in being thorough as you are the gatekeeper of what appears at the conference and what could be reproduced in the future. New methodologies have to be robust, well done, well motivated, well validated
  • that being said, we don’t have to be over critical about it because it is different from what we are used to. Be nice to new ideas, if the authors make a good job explaining the whys and hows, we shall embrace those

And:

  • not all papers need a user study, rate the contribution for what it is
  • null hypothesis testing is not the gold standard: be open to other forms of statistical evaluations (https://aviz.fr/badstats/).
  • inconclusive results are fine, only accepting positive results leads to the file-drawer effect
  • celebrate pre-registration of experiments
  • be open to demographics: experiments done in a particular country are similarly valuable as experiments done in North America/Europe
  • replicating experiments is a great scientific practice
  • make sure that the authors claims are coming from the data

And:

  • being obvious does not mean not novel or not relevant
  • it is not because it is simple that it is not ingenious
  • an idea may seem obvious when we are presented to it, but that being said nobody thought about it before
  • also, it is not necessary to be groundbreaking for it to be make a valuable contribution. Recognise that most research is incremental.

Process#

  • Colorful pen, mark positive/negative points and content that goes in summary
  • Dominik: Prints out, takes everywhere and marks the text:
    • + to +++ for good stuff
    • - to --- for negative stuff
    • c for content
    • ? for checking
    • Then c go to summary, + goes to strengths, - go to weaknesses, ? that cannot be resolved are Q’s for authors, [XX] for publications I need to check next time I am online; uses offline reviewing & keeps text file for later.
  • Lucy: prints out + (helps me to write questions (etc) in the margins). I usually save the paper copies until the other reviews come back / a decision is made so that I remember my thoughts about the paper.
  • Dilara / Sascha / Lucy / Dominik (everyone but yas???): keep reviews + papers in neatly organized folders (helps as inspiration & if you have to review the same paper twice).

Duration#

A survey at Oakland 2018 has the median review time (including PC discussion etc.) at 4 hours. Note that as a novice reviewer, your time per review will likely be significantly longer, especially if you need to read many related work publications.

Worked Example#

“The idea behind this paper is really good, and I’m happy that someone is looking at the effects of research methodology on user behaviors.”

  • Novelty: Check
  • Contribution: Check

“However, I think that the authors make a crucial mistake in their definition of “consistency”. The problem is twofold: <explanation of two points>”

  • Constructive criticism: Check

“To conclude, I’m not sure if the effects the authors claim to have found are indeed due to methodological differences, or rather due to chance (point 1) or contextual differences (point 2).”

  • Usefulness? Questionable.
  • Results? Questionable

“Finally, I want to point the authors to their use of casual language (“way more”), contractions (“don’t”), mixed english (“generalized” and “behaviour”), and some missing references to Sections of the paper.”

  • Helps polish the paper - good comment. It’s such a pain writing papers with different people!

Other Types#

In addition to papers, there are a number of other formats you might be asked to review:

Academic Posters#

Posters are widely used in the academic community, and most conferences include poster presentations in their program. Research posters summarize information or research concisely and attractively to help publicize it and generate discussion. A poster is usually a mixture of a brief text mixed with tables, graphs, pictures, and other presentation formats.

At a conference, the researcher stands by the poster display (or in a Zoom room with shared screen) while other participants can come and view the presentation and interact with the author.

Reviewing a poster is similar to a paper, with a few exceptions:

  • Appearance of the poster is relevant.
  • Content is somewhat less relevant, compared to a paper:
    • Posters can be based on already published work (approach already reviewed by others).
    • Posters can describe early work (insufficient content for a full review).
    • Poster format conveys less content than a full paper.

Below are some poster-specific points for a review:

Appearance#

You can include a description of the poster itself in the summary section (“Landscape format with three columns and red primary color. Headlines include …”).

  1. Display attracts viewer’s attention.
  2. Words are easy to read from an appropriate distance (1–2 meters).
  3. Poster is well organized and easy to follow.
  4. Graphics and other visuals enhance presentation.
  5. The poster is neat and appealing to look at.

Content#

Dominik: You can re-review already published work regarding approach, results, etc. (e.g., if you want to practice reviews), but I’ve never seen this required.

  1. Content is clear and easy to understand.
  2. Content is relevant for other researchers (or authors could benefit from feedback).
  3. There is enough detail about methods for me to understand the general approach.
  4. The approach taken is appropriate for the problem and technically sound.
  5. Conclusions are stated clearly.

References#

Previous Recruitment Channels Data Analysis Plan Next