The hidden fleet: every setting explained.
An API provider serves the very large DeepSeek-V3 model on rented NVIDIA H200 GPUs ($2.50 per GPU-hour) to 241 requests a second of chat, batch work, live chat, reasoning and agents. Traffic has a sharp daily peak, 2.9 times the daily average.
01Tool by tool
Factory settings vs AlphaTango.
The same table as the diagram, with each setting explained. AlphaTango keeps every tool in place and controls only their settings, all tuned together for one goal for the whole service: the lowest cost per answer delivered within its speed promise.
| Tool | Factory settings (every tool as shipped) | AlphaTango (same tools, settings controlled by AlphaTango) |
|---|---|---|
| Front doorthe router that decides which copy of the model handles each request | Half of returning chats reach the copy that remembers themAbout half of returning turns go back to the copy that already holds the conversation in memory | 3 of 4 reach the copy that remembers themAbout three-quarters go there, so less of each prompt has to be recomputed |
| Autoscaleradds and removes copies of the model as traffic changes | Stops at 48 copies; waits 5 minutes to free GPUsStandard spare room; waits 5 minutes before releasing GPUs | 20% more room before the peak; frees GPUs at once20% more spare room, so it is ready before the peak arrives; releases GPUs at once when traffic falls |
| Enginethe serving engine that runs the model on the GPUs | No helper model; large batches of workNo drafting helper; standard work batches | A small helper model drafts words for it; smaller batchesDrafting helper on, with the draft length set by the load; conversation memory stored as 8-bit numbers; smaller work batches; agent memory kept for 5 minutes between turns |
| Scheduler & GPU managerdecides where copies run, how many GPUs each gets and when batch work runs | Batch jobs start on arrival, right in the peakBatch work released the moment it arrives | Batch jobs spread over an hour, still on timeBatch work released smoothly over 60 minutes, still inside its 4-hour deadline, so it fills the quiet hours instead of the peak |
| GPUs & powerGPU power limits and clock settings | As shipped | No change |
02Where factory settings fall short
Why the factory set-up costs more.
Each tool runs with its vendor’s default settings and knows nothing about the others. Where that falls short in this case:
The engine runs without a drafting helper (speculative decoding): a small model that drafts a few words ahead, which the big model then checks in one step. The answers are exactly the same, but each GPU produces them faster. It ships turned off.
Offline batch work is released the moment it arrives. Much of it arrives during the daily peak, so it competes with live users exactly when GPUs are scarcest.
The autoscaler may run at most 48 copies of the model (384 GPUs) and waits 5 minutes before releasing GPUs. At the afternoon peak the factory set-up needed about twice that, so it hit the ceiling and the answers ran late.
The fleet averaged 265 GPUs, about $16,421 a day, and still broke the speed promise on all ten test days. Without the 48-copy limit it would have needed 354 to 424 GPUs and still missed on some days.
03Whole-Service Gate
Kept only if the whole service improved.
✓ The gate for this case
A version with no safety margin would have been 75% cheaper, but it kept the speed promise on only 5 of 10 days, so it was rejected.
The extra 20% spare room that was kept costs 12% on its own, and it is what keeps the promise on all ten days.
Every candidate change is tested on the AlphaTango Digital Twin, on traffic days the settings had never seen.
04Result
3.4× fewer GPUs: 79, not 265
265 down to 79 on a daily average
about $4.2 million a year freed at this size
against 0 of 10 at factory settings
the diagram's “cost per answer 3.5× lower”
Other tests of the same idea
57% cheaper; 38 GPUs down to 19; promise 0 of 10 → 10 of 10 days
'3.4× fewer GPUs' compares GPUs switched on, averaged over the day. 'Lower cost per on-time answer' divides what the whole service costs (GPU rental plus electricity) by the work it delivered within its speed promise. Both compare AlphaTango with every tool left at the settings its vendor ships (factory settings). Results come from AlphaTango's whole-system simulator, built on its Digital Twin and run on public traffic recordings, scored on days the settings had never seen. GPU prices are rental list prices.