Many models, one pool: every setting explained.
An enterprise hosts 61 internal models on rented NVIDIA H200 GPUs, with the real popularity pattern of a large 2026 production AI service: a few models are busy, most are used only a few times an hour. Together they peak at 41 requests a second. The promise: the first word of 99% of answers within 5 seconds.
01Tool by tool
Factory settings vs AlphaTango.
The same table as the diagram, with each setting explained. AlphaTango keeps every tool in place and controls only their settings, all tuned together for one goal for the whole service: the lowest cost per answer delivered within its speed promise.
| Tool | Factory settings (every tool as shipped) | AlphaTango (same tools, settings controlled by AlphaTango) |
|---|---|---|
| Front doorthe router that decides which copy of the model handles each request | As shipped | No change |
| Autoscaleradds and removes copies of the model as traffic changes | One per model: 61 separate fleetsOne autoscaler per model, 61 in all | One shared pool for all the quiet modelsOne autoscaler for the shared pool (never below 1 GPU); busy models keep their own autoscalers |
| Enginethe serving engine that runs the model on the GPUs | One model per GPUOne model per GPU | Quiet models share GPUs, loaded fast from standbyQuiet models share GPUs and take turns producing words; their weights are swapped in quickly from standby (host) memory when needed |
| Scheduler & GPU managerdecides where copies run, how many GPUs each gets and when batch work runs | Every model gets its own GPUsOne GPU pool per model | Quiet models share one pool; busy ones keep theirsAll 61 low-traffic models placed in one shared GPU pool |
| GPUs & powerGPU power limits and clock settings | As shipped | No change |
02Where factory settings fall short
Why the factory set-up costs more.
Each tool runs with its vendor’s default settings and knows nothing about the others. Where that falls short in this case:
Every model is deployed on its own, each with its own autoscaler, as the tools are designed to be used.
Even a model asked a few times an hour keeps its own GPUs reserved all day.
So most of the fleet sits waiting for requests that rarely come, and is paid for all day.
The fleet averaged 122 GPUs, $7,320 a day. In this best case factory settings did keep the speed promise on all 4 test days; they simply paid for far more GPUs than the work needed.
03Whole-Service Gate
Kept only if the whole service improved.
✓ The gate for this case
Only models whose own copy would be less than half busy were pooled. Busy models keep their own GPUs, because pooling them would put the whole service's promise at risk.
Every candidate change is tested on the AlphaTango Digital Twin, on traffic days the settings had never seen.
04Result
9.3× fewer GPUs: 122 down to 13
122 down to 13 on a daily average
89% cheaper per on-time answer
for both; every model still keeps its speed promise
for the same spend
Other tests of the same idea
75% cheaper; $3,233 → $838 a day; promise 0 of 4 → 4 of 4 days
69% cheaper; 61 GPUs down to 19
factory settings could not keep the promise at any fleet size; AlphaTango 53% to 64% cheaper and kept it
'3.4× fewer GPUs' compares GPUs switched on, averaged over the day. 'Lower cost per on-time answer' divides what the whole service costs (GPU rental plus electricity) by the work it delivered within its speed promise. Both compare AlphaTango with every tool left at the settings its vendor ships (factory settings). Results come from AlphaTango's whole-system simulator, built on its Digital Twin and run on public traffic recordings, scored on days the settings had never seen. GPU prices are rental list prices.