Faraday Shows Smaller Agents Can Steer Bigger Ones • The Weighted Average

Inherent Labs has introduced Faraday, a 27-billion-parameter AI-scientist agent that uses a coding agent as a tool and reportedly beat Claude Opus 4.8 and GPT-5.5 on held-out paper-replication tasks. The edition’s GPU-kernel correctness analysis offers the engineering analogue: an agent’s result is only as useful as the evaluation contract around it. The important number is not the model size; it is the 60% held-out win rate on the AI-for-science split, measured by a rubric-based judge. Of the benchmark’s 310 tasks, 68 were held out—68 ÷ 310 = 21.9%—so the result is a domain-shift test, not a claim that a small model has become an autonomous scientist.

Research teams should treat Faraday as evidence for a new architecture: a smaller model can supply scientific judgment while delegating code execution to a larger coding agent. They should not yet replace review, replication, or domain expertise. The immediate pilot is to compare an agent’s research plan, resource use, and reproduction quality on private papers, with human review of every conclusion.

The scientific layer sits above the coding layer

The Inherent Labs announcement frames Faraday as an AI Scientist trained to replicate results rather than merely summarize them. Each Replica task redacts a results figure from a research paper and asks the agent to recreate it under a time and compute budget. The model must infer missing experimental details, write code, run the experiment, and decide when a scaled-down reproduction remains faithful to the paper’s claim.

The arXiv abstract calls Faraday a 27B-parameter agent that uses coding agents as tools and surpasses Claude Opus 4.8 and GPT-5.5 on held-out replication tasks. The full paper methods specify 242 training tasks and 68 test tasks drawn from 100 machine-learning and AI-for-science papers. Each task has a 60-minute limit and a one-seventh MIG slice of an H200 GPU. Those constraints matter: the agent is not being given an unlimited laboratory or a blank check for compute.

The architecture is economically interesting. Faraday is post-trained from Qwen3.6-27B, while the coding-agent tool supplies the implementation muscle. The larger tool can search, patch, and run code; Faraday decides which hypothesis to test, how to interpret incomplete evidence, and whether a result is faithful. That is a division of labor closer to a principal investigator and research assistant than to one monolithic model.

The result extends the archive’s reproducibility gate for AI-generated mathematics. There, the key question was whether a model’s mathematical claim could survive formal verification. Here, the test is broader but still grounded: can an agent recover enough of the omitted experimental process to recreate a figure without hard-coding the answer? In both cases, the durable capability is not fluent explanation. It is disciplined contact with an external check.

The benchmark’s scale gives the headline some texture. The held-out slice is 68 of 310 tasks, or 21.9% of Replica. In the paper’s results summary, Faraday beats the named baselines on 73% of in-distribution ML tasks and 60% of held-out AI-for-science tasks. The authors also report average held-out improvements of 6% over Claude and 8% over GPT-5.5. Those are judge scores, not raw scientific accuracy, and the paper does not publish a universal cost per successful replication.

A promising judge is still a judge

The scoring system is the experiment’s strength and its vulnerability. A task-specific rubric scores visual similarity, support for the scientific claim, whether the experiment implements the paper’s method, resource use, and scientific integrity. Claude Opus 4.7 generates the rubric; GPT-5.5 Codex judges the completed rollout. The paper’s reward-method section explains why a rubric is needed: replication is underspecified, and a simple numeric target can reward shortcuts.

But a rubric-based judge is not a laboratory instrument. The paper reports a human study in which 20 participants supplied 117 rankings, but the human comparison focused on a selected set of 41 rollouts where Faraday had a large judge advantage. The human-study discussion explicitly says that design does not establish that humans prefer Faraday on average. Full-scale evaluation of eight tasks also relied on the same judge without human validation.

That caveat defines the operator bar. A research organization can pilot Faraday-like systems when the work is reproducible, the artifacts are inspectable, and a domain expert signs off. It should not delegate claims whose only evidence is an LLM judge agreeing with another LLM judge. The agent may discover a better experimental design; it may also produce a visually convincing figure that fails to reproduce the original mechanism.

The strongest counterpoint is benchmark transfer. Replica asks for figure replication, not a complete literature review, wet-lab experiment, clinical study, or novel theorem. A model trained on this task could become excellent at recovering plots without becoming good at choosing important questions. The conclusion breaks if private-paper tests show poor transfer, if independent human assessors disagree with the judge, or if the coding-agent tool does most of the work while Faraday’s added judgment contributes little.

Evidence that would change the verdict is concrete: preregistered private-task evaluations, randomly sampled human ratings, released task artifacts, cost and time per accepted reproduction, and tests across fields the model never saw during training. The paper’s limitations and discussion point toward that next phase rather than claiming the phase is complete.

Research leaders should pilot the pattern on low-stakes, in-silico replication. Compare Faraday-like planning against a frontier coding agent alone, keeping the paper, time limit, and compute budget fixed.
Platform teams should preserve the entire evidence trail. Store prompts, tool calls, code, plots, failed runs, and judge rationale so a scientist can audit the path rather than only the final image.
Scientists should make human validation the release gate. A judge score can prioritize review; it cannot turn an unreplicated result into a discovery.

Faraday’s real contribution is architectural. It suggests that the scarce layer in scientific AI may be judgment about what to try and how to know when an answer is honest—not another model that merely writes more code. That is a useful direction, provided replication remains the first obligation.