Owning a Model Costs $124K per Benchmark Point • The Weighted Average

Fireworks made its Training API and Fireworks Lab generally available, opening a path where customers write their own training loop in Python while Fireworks runs the trainer, the rollouts, and the weight synchronization between them. The general-availability post says the company has operated reinforcement learning across more than 10,000 GPUs outside the frontier labs, and lists customer results: Harvey’s Tenet scored 19.7% all-pass on its LAB benchmark against 10.8% for base Kimi K3, Vercel’s v0 auto-fixer reached a 93% error-free generation rate with a 40x end-to-end latency improvement, and Factory’s small LoRA adapters caught roughly 70% of real secrets against about 59% for GPT-5.5 at a 5% false-alarm budget.

Stitch the Harvey figure to a cost and the specialization trade becomes priceable. This paper’s earlier arithmetic put Harvey’s disclosed training run at roughly $1.1 million of rented compute — about 150 B300s for two months. Divide that by the 8.9-point all-pass improvement Fireworks reports and you get a number nobody published: roughly $124,000 of compute per percentage point of benchmark gain, before data work, evaluation, or engineering salaries. That is the honest unit for “should we train our own model,” and it is high enough that only a narrow class of teams should say yes.

The pricing structure tells you who this is for

Fireworks splits the offering into two compute modes with genuinely different economics. The training product page describes serverless training as LoRA adapters on an always-on shared pool billed per token, with sampling in the same session and no idle GPU charge, against dedicated training provisioned per run and billed by time on the GPUs you hold, supporting full-parameter runs and the largest mixture-of-experts models. The operational detail that matters for a budget: dedicated trainers bill until you close the client, and Fireworks stops an idle trainer after ten minutes by default — a configurable knob that decides whether an abandoned experiment costs dollars or thousands.

That split maps cleanly onto two buyer profiles. A team validating whether specialization helps at all wants serverless: per-token billing means a failed experiment costs the tokens it burned and nothing more, and LoRA adapters deploy behind a single inference endpoint, so dozens of variants can be served without standing up dozens of deployments. A team that has already proven the gain and is now optimizing cost per served token wants dedicated, because sustained throughput on held GPUs is cheaper per unit of work than a shared pool once utilization is high. Choosing wrong in either direction is the most common way training budgets evaporate before a model reaches production.

The engineering argument underneath is that reinforcement learning collapses the old train-then-ship pipeline, because every step needs fresh rollouts from newly updated weights. Fireworks claims three things follow: numeric formats and kernels must match between trainer and rollout or the learning signal corrupts silently, expert selection in MoE models must be replayed to keep tokens on the same computational path, and weight transfer must be compressed — the post describes an XOR diff plus zstd giving up to a 10x reduction in transmission bandwidth. The company says customers report 2–4x more iterations on the same training budget. Every one of those figures is vendor-supplied.

When $124,000 a point is worth paying

The case for owning a model is not capability; it is control over a capability that is load-bearing for your product. Harvey’s own account of post-training Kimi K3 for long-horizon legal work describes a task where general models fail on structure rather than knowledge, and Vercel’s fine-tuned auto-fixer targets latency in an interactive loop where a frontier call is too slow at any price. Heidi Health’s clinical scribe reports moving from proof of concept to production in four weeks at 3.5x lower latency. The common thread is a narrow, high-volume, latency-sensitive task with a measurable success criterion — not general reasoning.

Three things could break the thesis. First, the benchmarks are the customers’ own: LAB is Harvey’s, and a vendor-reported all-pass rate on a private evaluation is not a comparable number. Second, the $124,000-per-point figure divides a compute estimate by a self-reported gain, and it excludes the expensive part — the data curation, reward design, and evaluation harness that Fireworks itself sells embedded researchers to help build. Third, the base model keeps moving: a specialized model that beats today’s frontier on one task can be overtaken by the next general release, which is precisely the pattern behind GLM-5.3-Flash selling Opus-class coding at a fraction of the price and behind the collapse in what a frontier-class input token costs on open weights.

The evidence that would change the verdict is a third-party evaluation of a Fireworks-trained model against its base on a public benchmark, or a customer disclosing total program cost rather than compute alone. Until then, the operator move is arithmetic before commitment: name the metric, measure the base model’s score on it, price the gap at roughly $124,000 a point plus your own engineering months, and compare that to the recurring inference savings the specialized model would deliver. If the answer is not obviously positive at a year’s volume, rent the frontier model — and if it is, remember that the buyer’s leverage still comes from measuring value delivered rather than capacity consumed, the same shift visible in today’s lead on ChatGPT’s dollar-a-user ad yield.