vLLM vs Ollama vs SGLang: LLM Inference Compared 2026

In August our Helix fleet was serving Qwen3.8-27B to its coding agents on eight NVIDIA RTX PRO 6000 cards, one SGLang engine per GPU, and every engine stopped at 12 running requests. SGLang had sized the cache that holds each request’s recurrent state when it started, capped itself to fit, and printed “max_running_requests is capped to 12 by the mamba state cache” at every boot. Nobody had noticed. Storing that state in bfloat16 doubled the slots, and throughput at 192 concurrent requests rose from 3,833 to 7,883 tokens a second (HelixML). A month later our own benchmark on a single H100 hit the same cap.

vLLM and SGLang are GPU inference servers built to run many requests at once, vLLM around PagedAttention and SGLang around RadixAttention. TensorRT-LLM is NVIDIA’s server for NVIDIA GPUs. llama.cpp runs quantised GGUF models on CPUs, Apple silicon and GPUs, and Ollama wraps it, since June 2026, in a one-command local server.

We ran all five on one H100 against two models. On Llama 3.1 8B, from 2024, the fastest engine for one user came fourth of five at fifty, so concurrency picked the engine. On Qwen3.8-27B, released in August 2026, only vLLM and SGLang scaled with load, and each needed one default changed first. At 50 concurrent requests they served 1,725 and 1,610 tokens a second. Ollama served 33.

If you have been asked to self-host a model, or your Ollama prototype has just fallen over in staging, the numbers below say which engine to standardise on, and whether to run inference yourself at all.

The results on one H100

Every figure below comes from one RunPod H100 SXM 80GB, rented at $3.49 an hour on 25 September 2026, serving one engine at a time. Qwen3.8-27B ran at 8 bits on every engine: Qwen’s own FP8 checkpoint on the three GPU servers, and a Q8_0 GGUF file on llama.cpp and Ollama. Throughput is output tokens a second across 200 measured requests, and cost per million output tokens assumes the card serves that load all day (RunPod pricing).

SGLang’s row is its second run, with the state cache sized for 50 requests. On its defaults SGLang capped itself at 20 running requests and served 982 tokens a second at 50 concurrent, with a median wait of 10 seconds for the first token.

The same harness on Llama 3.1 8B, a 2024 model, ran FP16 weights on the three GPU servers and Q8_0 on the other two:

Two models, two different results

On Llama 3.1 8B, released in July 2024, the best engine depended on the load: llama.cpp was fastest for one user and fourth of five at fifty, the pattern Red Hat’s August 2025 comparison found on an A100. On Qwen3.8-27B, released in August 2026, the order held: SGLang and vLLM led at every level, and at fifty they served 52 times what Ollama did. llama.cpp beat Ollama on the identical GGUF file with both models, so Ollama’s gap is its own.

The architecture explains it. 48 of Qwen3.8’s 64 layers use Gated DeltaNet, a form of linear attention that keeps a fixed-size recurrent state for every request in flight, where a transformer keeps a growing cache of past tokens. The other 16 layers use full attention. Every engine has to hold that state for each running request, and each one sized it differently. Two needed a flag before they would use the card, one could not load the model, and one ran a single request at a time. So check your engine against your model’s architecture first, and your concurrency second.

What each engine did with a hybrid model

SGLang is strongest where requests share long prompts, and most of all on hybrid models: it keeps a dedicated prefix cache for them, where vLLM caches their state only at block boundaries. vLLM has cached shared prefixes by default since its V1 engine in early 2025, and has the larger ecosystem: llm-d and Red Hat AI Inference Server are built on it.

TensorRT-LLM runs only on NVIDIA GPUs, and since version 1.0 it no longer compiles engines ahead of time. On Llama it came within 13% of vLLM once its CUDA graphs covered the batch. Ollama batches other models through the llama.cpp server it has run underneath since May 2026, once OLLAMA_NUM_PARALLEL is raised from its default of one, and for one user on one machine it is still the quickest way to a running model.

How we ran the benchmark

• Hardware: one RunPod secure-cloud H100 SXM 80GB, driver 580.126.09 (CUDA 13.0), $3.49 an hour, one engine at a time. RunPod pods cannot run Docker inside them, so each engine was installed natively.
• Engines: vLLM 0.30.0, SGLang 0.5.20, TensorRT-LLM 1.2.1 (and 1.3.0rc28 on a second H100), llama.cpp b11179 and Ollama 0.34.4, each restarted at every concurrency level.
• Weights: for Qwen, Qwen’s own FP8 checkpoint and Unsloth’s Q8_0 GGUF, both 8-bit but different formats. For Llama, Unsloth’s ungated copy, since Meta gates its own, and bartowski’s Q8_0 GGUF. Ollama loaded the same GGUF file as llama.cpp.
• Load: identical completion requests of about 900 prompt tokens and 256 output tokens, 20 warm-up and 200 measured, at 1, 10 and 50 in flight. Token counts come from each engine’s own usage report.
• Fair settings: llama.cpp got one slot per request with 2,048 tokens each, Ollama got OLLAMA_NUM_PARALLEL at the concurrency level, and TensorRT-LLM got CUDA graphs up to batch 64, because its default of 32 halved its Llama throughput at 50.
• Repeatability: two full Llama runs agreed within 1% to 4% on throughput. Time to first token moved by up to 15%, so treat latency as approximate.
• Not measured: output quality, prefix caching (the prompts share only a short preamble) and multiple GPUs. Ollama’s OpenAI endpoint also applies the chat template, so it processed 948 prompt tokens to the others’ 896. llama.cpp ended one of the 200 Qwen prompts at once, the model’s choice rather than a server fault.
• Rerun it: the harness, pinned versions, results and server logs are in winderai/llm-inference-benchmark, and python3 main.py --model qwen3.8-27b --gpu-hourly-cost 3.49 reruns the Qwen set. The runs cost $25.37 in all.

When a card of your own beats an API

Cost per million output tokens is the card’s hourly price divided by the tokens it produces in an hour, times a million. For Qwen3.8-27B, with the H100 at $3.49 an hour and API prices read in September 2026:

A self-hosted card has to serve that load for about a third of the day before it undercuts the cheapest API. A June 2026 preprint found the same across models, with self-hosted cost varying up to 24 times with the request rate, and an earlier cost study found that break-even for owned hardware turns on monthly token volume. The RunPod price sat below Lambda’s $4.29 and at about half of AWS’s p5, $55.04 an hour for eight GPUs, so a dearer card raises every self-hosted figure in the table. If the arithmetic says you should buy tokens, AI integration is the work of wiring a managed API into your systems.

Choosing an engine for your model

Ask the questions in this order:

1. Does the engine’s latest stable release support your model’s architecture? New architectures reach main branches and release candidates months earlier, as TensorRT-LLM’s Qwen3.5 support shows.
2. For a hybrid model, or traffic that resends long shared prompts, use SGLang, after sizing its state cache for your concurrency.
3. For everything else on GPUs, or when the platform around the engine matters, use vLLM, after setting --max-num-seqs on hybrid models. llm-d and Red Hat AI Inference Server are built on it.
4. For an NVIDIA-only estate on a model its stable release supports, TensorRT-LLM came within 13% of vLLM at 50 concurrent on Llama, once its CUDA graphs covered the batch.
5. For one user, a laptop, Apple silicon or low-bit weights, use llama.cpp, or Ollama for the one-command setup, knowing that Ollama runs some architectures one request at a time.
6. For low or bursty traffic, buy tokens from an API, per the arithmetic above.

The model comes first, because it decides which engines can serve it. Choosing it, fine-tuning a smaller one, and deploying it on your own hardware is what LLM consulting covers. The engine is the bottom layer of the stack, and the rest of it, from gateways to evaluation, is in the wider LLMOps tooling landscape.

The log line

The four problems in the Qwen run share a shape. Each engine was built around plain transformers, and each reported the mismatch in a different place: SGLang in a warning at startup, vLLM in an error that stopped it starting, Ollama in a scheduler message, and TensorRT-LLM in a stack trace. A table of peak throughput shows none of it, and all of it decides what a card actually serves.

We run both of the engines that scaled, in production, and the model picks which. Since 21 September our eight-card machine, eight NVIDIA RTX PRO 6000 Blackwell GPUs, has served Qwen3.8-Flash-Next on vLLM across four cards and GLM-5.3-Flash on SGLang as two replicas of two cards each, with a small Qwen3.5-based model sharing one of the Qwen cards. GLM-5.3-Flash runs on a community SGLang build made for these Blackwell workstation cards, because vLLM’s packed FP8 path could not load the quantised checkpoint and its sparse attention layout. Ramjet, the inference load balancer we build for Helix, sits in front of all of them and sends each request to the engines that hold its model, and each GLM request to the replica already holding its prefix. Helix’s own GPU runner has Ollama and vLLM runtimes built in. In August the same machine ran Qwen3.8-27B as eight SGLang engines, one per card, and that fleet’s log line opened this article.

The hybrid models have kept finding defaults that assumed plain transformers. 34 of GLM-5.3-Flash’s 45 layers are linear attention, so to resume a cached conversation SGLang needs a snapshot of their state as well as the usual KV cache. On 25 September we found a single 313,000-token prompt producing 51 snapshots for 28 slots and evicting every other conversation’s, while the token cache sat only 72% full. Capping snapshots per conversation cut cold turns on long conversations from 7.6% to 1.1% (HelixML).

Wiring an engine into a serving stack that autoscales, is monitored and earns more than it costs is MLOps consulting, and it includes reading those lines before a client’s traffic finds them. Monitoring the engine once it is serving and scaling it past one card are the next problems. If your engine has a line like that in its log, we would like to hear about it.

Before trusting any benchmark with a new model, ours included, start the engine on your own card with your real concurrency and read its startup log to the end. On our H100, the line that mattered said the engine would run 20 requests where we had asked for 50.