Skip to main content
All posts
AI & Data18 min read

SLM On-Prem vs API: The TCO Break-Even Calculator

When does self-hosting a small language model beat paying per token? A transparent TCO formula, EU list prices from November 2026 and an xlsx calculator.

Published Updated: 15 November 2026

On 15 November 2026, one Azure NC40ads H100 v5 node in Germany West Central lists at €8.15 an hour. That is €5,946 a month if it runs all the time, or €3,271 a month on a three-year reservation. On the same day, a million input tokens to GPT-5.4 nano in the EU Data Zone cost €0.20. Every week a board somewhere asks us the same question: "Wouldn't it be cheaper and more sovereign to run a small model ourselves?" The honest answer depends on one number most teams have never measured, which is tokens per month.

This article is for the people who sign that decision: CTOs, CISOs, platform leads and architects. We give you a transparent formula, plug in list prices checked on the day of publishing, and show where the break-even sits. The calculator lets you replace every assumption. Our conclusion upfront: on cost alone, self-hosting a small language model rarely wins below tens of billions of tokens a month. Sovereignty and offline operation are better reasons, and deserve their own line in the business case.

TL;DR / Key takeaways

  • The floor decides, not the GPU. One H100 serving an 8B model has more capacity than most workloads use. Two nodes, platform and people form a fixed floor of about €11,900 a month on a three-year reservation.
  • Break-even is measured in billions. On our reference workload, self-hosting beats GPT-5.4 at 3.4 billion tokens a month, GPT-5.4 mini at 11.3 billion and GPT-5.4 nano at 41.6 billion.
  • Compare like with like. An 8B model competes with nano-class APIs on quality, not with frontier models. Run the evaluation first, then the maths.
  • The EU data-zone premium is small. Azure lists Data Zone EU at 10% above Global for the GPT-5.4 family. That premium is the cheapest sovereignty control you can buy.
  • Hybrid wins in practice. A gateway routing each task to the cheapest option that passes its evaluation beats any single choice.
Four ways to run a small language model and how each one is billed, from public API to edge hardware. Numbered callouts match the sections below.
Four ways to run a small language model and how each one is billed, from public API to edge hardware. Numbered callouts match the sections below.

Download: Editable draw.io file · Open in diagrams.net · Chart data (CSV) · TCO calculator (xlsx)

Why this matters now

Two pressures meet in 2026 budgets. First, small models have become good enough for most enterprise tasks. In April 2025, Gartner predicted that by 2027 organisations will use small, task-specific AI models three times more than general-purpose LLMs. Open-weight 8B to 30B models now handle classification, extraction, summarisation and routing well enough that a frontier model is overkill for most traffic.

The second pressure is sovereignty. In November 2025, Gartner reported that 61% of the 241 Western European CIOs and IT leaders it surveyed expect geopolitics to increase their reliance on local cloud providers. In DACH, most AI steering committees we work with now ask where the model runs, and who could be compelled to hand over the prompts.

Meanwhile API prices keep falling. Epoch AI found in March 2025 that the price of reaching a given performance level fell between 9× and 900× per year, depending on the benchmark. A three-year GPU reservation is a bet against that curve.

The four deployment options, layer by layer

The hero diagram orders the options by how much you control and how much you carry.

① Public model API

You call a provider's endpoint and pay per token. The provider runs the GPUs, scaling and upgrades. Fixed cost is close to zero. You control prompts, keys and the contract, and nothing else. It is the right default for exploration and spiky, low-volume work, and the wrong answer when policy rules out the provider's jurisdiction.

② Managed private endpoint

The model is still operated by the provider, but inside a defined boundary. On Azure, a Data Zone deployment processes prompts and responses only within the Microsoft-defined data zone. For the EU zone, that follows the EU Data Boundary, which can include EFTA countries. Data at rest stays in your chosen geography. You reach the deployment through a Private Endpoint in your own VNet, so no public network path is involved.

Billing is per token (Data Zone Standard) or per reserved throughput unit (Data Zone Provisioned), and Data Zone Batch gives 50% off asynchronous jobs. On 15 November 2026 the Azure Retail Prices API listed GPT-5.4 mini at €0.74 per million input and €4.44 per million output tokens in Data Zone EU, exactly 10% above Global. Check deployability first, because a price meter is not a deployable model: in mid-2026 Microsoft Q&A still described 5.4 mini and nano in Data Zone Standard as US-only, with no date for the EU zone.

③ AKS + vLLM in your subscription

You run an open-weight model on GPU nodes in your own subscription, served by vLLM on Kubernetes. You choose the weights, pin the version and keep the logs. The NC40ads H100 v5 size gives you one H100 NVL with 94 GB, enough for an 8B model with ample KV cache, or a 30B-class model at FP8. The cost model changes shape: GPU-hours times nodes, plus cluster, plus people. That is a step function, not a line. Our vLLM and KServe on AKS guide covers the serving architecture. Our GPU cost control guide covers keeping the nodes busy.

④ Edge and on-premises

The same stack runs on hardware you own: in a data centre, on a plant floor, or in a branch that must keep working when the WAN does not. vLLM and llama.cpp run fully offline. You control everything, including power, spares and the refresh cycle. The GPU-hour becomes capex divided by months, plus power, cooling, rack space and support. An EU dedicated host sits in between. Hetzner lists its GEX131 with one RTX PRO 6000 Blackwell Max-Q (96 GB) in Falkenstein and Helsinki from €1,499 a month, plus a €1,499 setup fee. With no published MLPerf result for that card on Llama 3.1 8B, we keep it out of the break-even curve; the calculator accepts your own measured throughput.

⑤ The gateway: route by task, not by ideology

The most cost-effective estates we see do not pick one option. They put an AI gateway in front of all of them and route each task to the cheapest deployment that passes its evaluation: bulk extraction to a self-hosted SLM once volume justifies it, hard reasoning to a frontier API, plant-floor assistants to the edge. The gateway also enforces quotas, attributes cost and filters personal data.

The formula

Every number below comes from this formula; the calculator implements it with live cells.

Text
share_in   = input_tokens / (input_tokens + output_tokens)
b_api      = share_in × p_in + (1 − share_in) × p_out          [€ per 1M tokens]
API(T)     = T / 1,000,000 × b_api

cap_node   = out_tok_per_s / output_tokens × (input_tokens + output_tokens)
             × utilisation × 3,600 × 730                       [tokens per month]
nodes(T)   = max(n_min, ceil(T / cap_node))
SELF(T)    = nodes(T) × node_price + platform + people

break-even T* = (n_min × node_price + platform + people) / b_api × 1,000,000
                (valid while T* ≤ n_min × cap_node)

The inputs we used, and where they come from:

  • Workload. The MLPerf Inference v5.1 Llama 3.1 8B profile: 778 input and 73 output tokens per request on average (CNN-DailyMail summarisation). A public throughput figure exists for it, and it is typical of enterprise SLM work: read a lot, write a little.
  • Throughput. Red Hat's MLPerf v5.1 submission: 5,104 tokens per second on one H100-SXM-80GB in the server scenario (2-second time to first token, 100 ms per output token), using vLLM 0.10.0 and an FP8 Llama 3.1 8B. Azure's NC40ads uses the H100 NVL variant, so treat this as an approximation and measure your own.
  • GPU node. NC40ads H100 v5, Germany West Central, Azure Retail Prices API: €8.1458 an hour on demand, €53,517.66 for one year reserved, €117,739.58 for three. That is €5,946, €4,460 and €3,271 a month.
  • Platform. AKS Standard tier (€0.0898 an hour) plus two D4s v5 system nodes (€0.2065 an hour each): €367 a month.
  • Our assumptions. 50% average utilisation, at least two GPU nodes for availability and rolling upgrades, and €5,000 a month for people (about half an engineer at a loaded €120,000 a year). Replace all three.
  • API prices. The GPT-5.4 family in Data Zone EU from the Azure Retail Prices API; Mistral Small 4 at USD 0.15 in and 0.60 out per million tokens, converted at the ECB rate of 1.1052 (13 November 2026).

What the numbers say

Break-even between API cost and a self-hosted 8B model on AKS with vLLM, tokens per month against euros per month, log scales
Break-even between API cost and a self-hosted 8B model on AKS with vLLM, tokens per month against euros per month, log scales

The self-hosted lines are flat for most of the chart, and that is the whole story. One H100 node at 50% load handles about 78 billion tokens a month on this workload, roughly 35 requests a second around the clock. With two nodes as the minimum, a third is needed only at about 156 billion tokens. Below that you pay the floor: €11,908 a month on a three-year reservation or €17,260 on demand.

Break-even volume in billions of tokens per month for each API comparator
Break-even volume in billions of tokens per month for each API comparator

Three readings matter for a decision.

First, the fair comparison is the nano line. An 8B model does not match GPT-5.4 on hard reasoning, so the 3.4 billion break-even only counts if your evaluation says the small model is good enough. Against GPT-5.4 nano, the closest match in quality, self-hosting pays off at 41.6 billion tokens a month: about 49 million requests, or 19 a second on average. Against Mistral Small 4, an Apache 2.0 mixture-of-experts model also sold as a cheap API, you need about 70 billion.

Second, utilisation barely matters below the floor. Halve it to 25% and none of the break-even points move. The fixed floor moves them. Remove the people cost and the nano break-even drops to 24.1 billion. A single node with no people, which we would not run in production, gets it to 12.7 billion. Double people cost to €10,000 and it rises to 59 billion. The decision is more sensitive to your operating model than to your GPU.

Third, workload shape is the hidden variable. Our profile reads 778 tokens and writes 73. Chat, agents and code write far more, and output tokens are what keep a GPU busy. Capacity per node can fall by an order of magnitude, the steps arrive much earlier, and the API often looks better than this chart suggests. Measure your own token mix before you buy anything.

What one GPU node costs per month at list prices
What one GPU node costs per month at list prices

A three-year commitment cuts the H100 node price by 45% against on demand. That is a large saving, and a three-year bet on a GPU generation. Spot at €1,099 a month can be evicted at any time, so it suits batch evaluation, not serving. The Hetzner line shows why dedicated EU hosting attracts sovereignty-driven teams; treat it as a price point until you have benchmarked the card.

In a recent engagement…

A regulated DACH organisation asked us to validate a business case for moving its document-processing assistants from a US-hosted API to self-hosted open weights. The case had a GPU quote and a token price, and nothing in between: no measured volume, token mix or latency target.

We instrumented the traffic through their gateway for four weeks. Volume was well below the floor in every scenario, and the workload was strongly input-heavy. On cost, self-hosting lost clearly. On sovereignty, the real driver, an EU data-zone deployment behind a Private Endpoint met the policy for every use case but one, an offline production site.

The outcome was hybrid: the API moved to the EU data zone with gateway routing and per-team cost attribution, a small edge deployment served the offline site, and a written trigger puts self-hosting back on the agenda once measured volume crosses the break-even. The longest discussion was not technical. It was agreeing who owns the evaluation set that decides whether a small model is good enough.

The decision table

Read each row as "if this is true, lean towards".

SituationLean towardsWhy
Volume unknown or below ~1 billion tokens/month① or ② APIYou are far below any fixed floor; measure first
Spiky traffic, many models, fast-changing needs② Data Zone StandardPay per token, EU processing, no capacity risk
Steady high volume, latency-sensitive, model must be from the provider② Data Zone ProvisionedReserved throughput without running GPUs
Volume above your break-even, and an SLM passes your evaluation③ AKS + vLLMFixed cost is spread over enough tokens
Policy requires European control of the weights and the stack③ on EU infrastructure, or ④Data zone alone may not satisfy it
Offline, air-gapped or plant-floor latency④ EdgeThe only option that works without the WAN
Output-heavy chat or agents at moderate volume① or ② APIDecode-heavy work fills GPUs fast; step costs bite early
Large asynchronous jobs② Data Zone Batch50% off list price, 24-hour target

Never compare a self-hosted model with an API it does not match on your evaluation set, and never reserve GPUs for three years without a plan for the model you will serve in year three.

Loading diagram...

Governance for European enterprises

GDPR. Every option except ④ involves a processor. Your Article 28 contract, records of processing and transfer assessment must match the deployment type you actually use: Global and Data Zone EU are different processing locations for the same model name. Pin the type with Azure Policy so nobody quietly switches to Global for the higher quota.

EU AI Act. Hosting does not change whether a use case is high-risk; that depends on what the system does, and the Annex III obligations now apply from 2 December 2027. Running open weights unchanged does not make you the provider of the general-purpose model; your obligations attach to the AI system you build on it. Fine-tuning makes you provider of a modified general-purpose model only if the change is significant. The Commission's July 2025 guidelines use more than a third of the original training compute as the indicative threshold, far above typical LoRA fine-tunes. Keep the evidence anyway: base model, data summary, compute used.

DORA. For financial entities, an API supporting a critical or important function is an ICT third-party service. It belongs in the register of information, and Article 28(8) requires an exit strategy. Self-hosted open weights are a credible exit path, and the calculator prices it. That alone can justify keeping option ③ warm below break-even. Our sovereign cloud comparison covers the infrastructure side of that exit.

A pragmatic adoption path

  1. Weeks 1–4: measure. Route all LLM traffic through one gateway and record tokens in and out per use case, latency targets and model. Our token cost engineering guide shows what to log.
  2. Weeks 3–6: build the evaluation set. For each high-volume task, collect a few hundred real examples. Run the frontier model, a nano-class API and one or two open-weight SLMs against it. Cost only decides between models that pass.
  3. Week 6: move to the right API tier. Switch EU-sensitive workloads to Data Zone, push asynchronous jobs to Batch, enforce deployment types with policy. This often saves more than self-hosting would.
  4. Quarter 2: run the calculator with your numbers. Replace workload, throughput (measured on your model and latency target), people cost and minimum nodes. If a workload is clearly above break-even, pilot AKS + vLLM on demand before reserving anything.
  5. Every quarter: re-run it. Prices fall, GPUs change, volume grows. A written trigger per workload means the decision is revisited because of data, not a vendor pitch.

The verdict

Small language models are a genuine shift, but self-hosting them is a fixed-cost business. Two nodes, a platform and the people to run them cost €12,000 to €17,000 a month before the first token, and most enterprise workloads come nowhere near filling them. On cost alone, an EU data-zone API wins below tens of billions of tokens a month, and its 10% premium is the cheapest sovereignty control on the market.

Self-hosting wins in three cases: measured volume clearly above break-even for a task a small model handles well, a policy requiring European control of the full stack, or a site that must work offline. Put a gateway in front, measure before you buy, and let each workload earn its way onto your own GPUs.

If you want a second pair of eyes on the numbers, our AI architecture team runs this calculation as a short, fixed-scope review on your measured traffic, including the evaluation set that decides which model is good enough.

FAQ

At what volume is self-hosting a small language model cheaper than an API?

With our inputs, billions of tokens a month. On a workload of 778 input and 73 output tokens per request, two Azure H100 nodes on a three-year reservation plus platform and half an engineer cost about €11,900 a month. That equals GPT-5.4 at 3.4 billion, GPT-5.4 mini at 11.3 billion and GPT-5.4 nano at 41.6 billion tokens a month, on Data Zone EU list prices of 15 November 2026.

What is the formula for comparing API and self-hosted LLM cost?

API cost is monthly tokens times a blended price that weights input and output prices by your token mix. Self-hosted cost is GPU nodes times node price, plus platform and people, where nodes are volume divided by capacity per node, rounded up, but never below your availability minimum. The break-even is the volume where both are equal.

Why does GPU throughput matter less than expected?

A small model on a modern GPU has more capacity than most enterprise workloads use. One H100 serving an 8B model handled about 5,100 output tokens per second in Red Hat's MLPerf v5.1 server run, roughly 78 billion tokens a month at 50% load on that workload. Most organisations never fill the two nodes they need for availability, so the fixed floor sets the break-even.

Is an EU data zone API sovereign enough, or do we need self-hosting?

Often it is. Microsoft processes Data Zone EU traffic within the EU Data Boundary and keeps data at rest in your geography, for a 10% premium over Global list prices. It does not remove a US provider's legal exposure, so a strict sovereignty policy may still require open weights on EU infrastructure or your own hardware.

Does self-hosting an open-weight model change our EU AI Act obligations?

Running a model unchanged does not make you the provider of the general-purpose model. Fine-tuning makes you provider of a modified general-purpose model only if the change is significant; the Commission's July 2025 guidelines use more than a third of the original training compute as the indicative threshold. High-risk obligations under Annex III, now applying from 2 December 2027, depend on the use, not the hosting.

What should we include in a self-hosted LLM TCO besides GPUs?

Cluster control plane and system nodes, monitoring and logs, at least two GPU nodes for availability, and people for patching, model upgrades, evaluation and on-call. On-premises adds amortised capex, power, cooling, rack space and support. Leaving out people is the most common error we see: half an engineer costs more per month than a reserved H100 node.

Sources

All prices and pages checked on 15 November 2026. List prices exclude VAT and negotiated discounts. Where we use our own assumptions, the text and the calculator say so.

Tier A: primary, vendor price lists and regulatory

Tier B: research and benchmarks

Tier C: directional

Topics

SLM vs LLM costsmall language model TCOself-hosted LLM vs API costLLM break-even calculatorvLLM on AKS costAzure OpenAI Data Zone EU pricingon-premises LLM inferencesovereign AI cost

Frequently Asked Questions

With our inputs, billions of tokens a month. On a workload of 778 input and 73 output tokens per request, two Azure H100 nodes on a three-year reservation plus platform and half an engineer cost about EUR 11,900 a month. That equals GPT-5.4 at 3.4 billion, GPT-5.4 mini at 11.3 billion and GPT-5.4 nano at 41.6 billion tokens a month, on Data Zone EU list prices of 15 November 2026.

Expert engagement

Need expert guidance?

Our team specializes in cloud architecture, security, AI platforms, and DevSecOps. Let's discuss how we can help your organization.

Get in touchNo commitment · No sales pressure

Related articles

All posts