Reviewing other people's work

A good review answers three questions, in this order:

1. What is claimed? Summarise the contribution in your own words. If you cannot, the report is unclear — and that is itself the first finding.
2. Does the evidence support the claim? The baselines, the seeds, the uncertainty, the ablations, the contamination risk. Point at the table and the row, not at a feeling.
3. What would make it stronger? Concrete and feasible, not «do more».

Distinguish three categories and mark which one you mean:

• A methodological fault — the conclusion does not hold. It has to be fixed.
• A presentation problem — the result may be right but cannot be judged. It should be fixed.
• A matter of taste — you would have done it differently. It may be mentioned, it should not weigh.

A review template (it works for classmates, colleagues and conference submissions):

A summary of the contribution (3 sentences, in your own words)

Strengths
- …

Methodological faults (have to be fixed)
- [Table 2] The baseline is untuned while the method has been searched over 12 configurations.
  The comparison is therefore not fair. A suggestion: the same search budget for both.

Presentation (should be fixed)
- [Fig 3] The y-axis starts at 0.7, which exaggerates the difference. Start at 0.
- The number of seeds is not stated. State n and the spread in every table.

Matters of taste (not decisive)
- I would have chosen F1 over accuracy, but it is justified in the text.

Questions to the authors
- How was the overlap between the training and the test data in section 4.2 handled?

Receiving a review: respond to every point separately with fixed / partly fixed / not fixed + the reason. A reviewer who is told why you disagreed is happier than one who is ignored.

What makes reviewing hard is not finding faults — it is finding the right faults and formulating them so that they can be acted on. A review that only says «a weak contribution» is worthless however right it is.