Zenith: taking a Flash model into the frontier

Six systems cost more and score lower than Zenith and DeepSeek V4.1 Flash. Up and to the left is better.

On a full day of ML research, Zenith running DeepSeek V4.1 Flash passes GPT-5.6 Sol, for $2.82 a task.

In March, Andrej Karpathy put autoresearch on GitHub: a few hundred lines of Python that let an AI agent run machine-learning experiments while you sleep, keeping what works and throwing away what doesn't. More than 95,000 stars later, the Karpathy Loop has become shorthand for AI that does its own research.

From a night to a day

A night of five-minute experiments is the easy version. Real research takes days, and the result has to hold up on data the agent has never seen. That is where the gap opens. Open models now run close to closed ones on tasks that take minutes, but on benchmarks that run for hours or a full day, such as FrontierSWE and AutoLab, the best open model reaches between half and three-quarters of the leader's score.

A Flash model in the frontier

AutoResearchExam, from Bespoke Labs, gives an agent 24 hours on each of 29 ML research problems and grades the work on a test set it never sees. Five systems average above its strong-reference line. Four are the most expensive closed models on the board. The fifth is Zenith running DeepSeek V4.1 Flash, the small, fast tier of DeepSeek's newest model family, at $2.82 a task.

It passes GPT-5.6 Sol and beats Kimi K3, the largest open model in the comparison, at a small fraction of their cost. Against the leader, Claude Fable 5.1, it reaches nearly nine-tenths of the score for about a ninetieth of the price.

How Zenith does it

Zenith is our open-source harness for planning, running and evaluating experiments. We treat autonomous research as a control problem: balancing exploration of new approaches with exploitation of promising ones. Follow one direction for too long, and a run can get stuck at a local optimum. Explore too broadly, and it can exhaust its budget before developing any approach into a strong result. The artifacts from a run, including experiment records, code and measured results, provide the evidence for deciding when to keep going and when to change direction.

We developed this approach using a broad collection of research problems, excluding AutoResearchExam. Meta-Zenith used these problems to create an initial set of research skills: instructions for managing exploration and exploitation. We then ran Zenith with those skills on the development problems and fed the final rewards back to Meta-Zenith. Through repeated cycles of execution, feedback and revision, Meta-Zenith refined the initial skills into the final skill set.

Meta-Zenith refines research skills from the rewards of development runs; Zenith uses run artifacts to balance exploration and exploitation; the final skill set is evaluated on AutoResearchExam.

The AutoResearchExam results reported here come from Zenith running with that final skill set.

Why it matters

At $2.82 a task, the whole exam costs about $82 with Zenith, against about $7,500 with the leader. A day of machine research on an open ML problem now costs less than a London flat white. That buys ninety attempts for the price of one, and puts a research loop within reach of a university lab or a single graduate student.

It is also yours. The model weights are MIT-licensed and the harness is open source, so the whole system runs on hardware you control: a GPU cluster rather than a laptop, but your cluster.

When a day of research costs less than a coffee, the scarce input becomes the question. Deciding which problems deserve a day of machine attention, and which results deserve belief, is a human judgement, and it is becoming the most valuable one we make.

Coming soon

We will release the autonomous research configuration for Zenith soon.