Planning a research project of your own

A research project usually fails on the bounding and the budget, not on the idea.

From an idea to a question: «Does memory help agents?» → «Does episodic memory (summaries, k = 4) give a higher solution rate than no memory over 20-session series with simulated users, on task family X?» The question has a measure, a control, data and a scope.

The plan (one page, pre-registered):

1. The hypothesis + what falsifies it.
2. The experiments in order: a pilot (1 day) → the main experiment → the ablations. Each with an expected outcome.
3. The compute budget: GPU hours per run × seeds × variants. If it exceeds what you have: shrink the model or the data, not the number of seeds.
4. Stopping criteria: «if the pilot does not show ≥ X, stop and report it negatively».
5. A timetable with half the time for the analysis and the writing.

If the plan is published (internally) before the run, nobody — not even you — can move the goalposts afterwards.

A budget calculation — an example:

The item	The number	The time each	The total
The pilot	1	2 GPU-h	2
The main: 2 conditions × 5 seeds	10	4 GPU-h	40
The ablation: 3 variants × 3 seeds	9	4 GPU-h	36
Reruns (a 30 % buffer)			23
The total			~100 GPU-h

Do you have 40 GPU-h? Then: a smaller model (half the time), 3 seeds in the main experiment, 2 ablations. Never one seed.

Write three things down that can go wrong (the data is missing, the baseline is unexpectedly strong, the environment is buggy) and what you do then. That is your risk plan.