Reading and analysing research papers

Three passes (Keshav):

1. 5 min: the title, the abstract, the figures, the conclusion. What is claimed? Do I care?
2. 30 min: the method and the experiments in detail, skipping the proofs. Write down: the claim → which table supports it → what it is compared against → how many seeds → which data.
3. 2 h (only if it matters): reproduce the reasoning, find the assumptions, look for what is not said.

The questions that expose weak papers:

• Did the baseline get the same tuning? Does it say?
• How many seeds, and is the spread reported?
• Is the benchmark older than the model's training data (contamination)?
• Are negative results and ablations reported?
• Is there code and an exact configuration?
• Is the improvement larger than the noise between seeds?

A paper with big claims and small tables is a hypothesis, not a result. That holds for papers from large labs too.

A reading card — fill it in for every paper (it takes 10 minutes and saves hours):

Field
The claim (one sentence)
The method (three sentences)
The main table + baselines
Seeds / spread
Data + contamination risk
Ablations supporting the mechanism
Weaknesses
Relevant to me because
Would I reproduce it? At what cost?

Try it on the LoRA paper (Hu et al. 2021): the claim is clear, the tables compare against several PEFT methods and a full fine-tune, but seeds and spread are reported sparsely — note that. Then try it on a paper that has just come out.