SGLang vs vLLM: what we learned running both in production

If you self-host LLMs, you eventually pick between vLLM and SGLang. Both are fast, both speak the OpenAI API, both have committed maintainers, and the marketing benchmarks are basically a coin toss depending on whose graph you read.

We have run both. Not as a benchmark exercise, as actual production traffic. Some of it is small instruct models doing classification and extraction. Some of it is multi-turn agent loops with long, repeated system prompts. Some of it is embeddings. The results are not the same on every workload, and the engines are not interchangeable.

This is the version of the comparison we wish someone had handed us six months ago.

The thirty-second version

vLLM is the safer default. Bigger community, faster to add new model architectures, more deployment recipes online, more battle-tested with weird hardware. If you are setting up your first inference box and you do not yet know what your traffic looks like, start here.

SGLang is the throughput specialist. Its prefix caching is genuinely a different design point, not a tuning trick, and on the right workload it leaves vLLM behind. If you are running agents, RAG with stable retrieved chunks, or anything where many requests share a long prefix, the gap is real.

Neither is dramatically faster on a "send 200 unrelated prompts at a 70B model" benchmark. The interesting differences show up in the corners.

What vLLM gets right

The thing vLLM nailed first was PagedAttention. The KV cache is the tax you pay on every token of context, and naive allocation fragments memory in a way that wastes huge amounts of GPU. Paging the cache the way an OS pages physical memory was the right idea, and it is now table stakes everywhere, including SGLang.

Beyond that, the wins are mostly about ecosystem maturity:

• Model coverage moves fastest here. New architectures usually land in vLLM first or simultaneously, with the configs and parsers worked out.
• Hardware coverage is broader. AMD MI300, Inferentia, ARM, older Ampere cards. SGLang works on most of these too, but the rough edges show up sooner outside the H100 / A100 happy path.
• Tool calling has battle-tested parsers for the major model families. You still have to know which parser to pass on the command line for your specific model template, and getting that wrong silently breaks tool use, but the parser zoo is comprehensive.
• The OpenAI compatibility layer is the strictest. Edge cases like stream_options.include_usage, structured outputs, log probs, and the exact shape of finish reasons all behave the way client SDKs expect.
• Continuous batching is rock solid. Throughput under uneven request lengths is one of the parts of inference engineering that is genuinely hard, and vLLM has been hardening it for two years.

What vLLM is not good at: shared-prefix workloads. The KV cache reuse is request-scoped. If two requests send the same 4 KB system prompt, the engine recomputes the prefill for both. There are workarounds (prefix caching landed as a feature) but the implementation is more limited than what SGLang does, and the win is smaller in practice.

What SGLang gets right

SGLang's headline feature is RadixAttention. Instead of treating the KV cache as request-scoped state, it builds a radix tree across active requests and reuses every shared prefix automatically. This is not a "cache the system prompt" optimisation. Any common prefix across any pair of requests, including in-flight ones, is shared.

For some workloads this changes the economics:

• Agent loops that re-send the conversation history on every turn become dramatically cheaper. The prefill on turn 7 only costs the new tokens, not the previous six exchanges.
• Multi-tenant systems with one big shared system prompt stop paying that prompt's prefill cost over and over.
• RAG where retrieved chunks repeat across users (popular questions hit the same docs) gets the same treatment.

On these workloads we have seen 2–4x throughput against vLLM with prefix caching disabled, and a meaningful gap even against vLLM with its prefix caching turned on. On classic "200 unique prompts, no shared structure" benchmarks the gap is much smaller, sometimes zero, sometimes inverted.

A few other things SGLang does cleanly:

• Structured output is a first-class citizen. The constrained decoding integration is tighter than what you get bolting xgrammar onto vLLM. JSON-schema-conformant outputs come back faster because the constraint compilation is amortised across batched requests.
• Disaggregated prefill and decode. Splitting the two phases onto different workers (or different GPUs) is supported as a real deployment mode, not a research demo. For long-context workloads this matters.
• The server is genuinely lighter. Cold start is faster, the binary is smaller, the memory overhead before model weights load is lower.

What SGLang is not good at: being the safe choice. Smaller community, sparser docs, sharper edges. Some configuration knobs are documented in code comments only. Some new model architectures take a release or two to land. If you hit a weird bug at 3am, you are more likely to be the first person to file the issue.

The things that bit us, in both

A few honest gotchas that the docs underplay:

Tool call parsers are the silent failure mode. Both engines need you to specify a parser that matches the chat template of your model, and both will happily start, accept requests, and produce responses with no tool calls if the parser is wrong or missing. The model is generating the tokens, the engine is just not interpreting them. Always end-to-end test tool calling after any model or engine version bump.

OpenAI compatibility is a moving target. Both engines aim for it but neither is bit-exact. Streaming with usage included, log probs on streamed tokens, the exact wire shape of finish_reason: "tool_calls" versus the legacy function_call, all of these have been silently broken at various times in various versions. If your client expects a specific behaviour, pin the engine version.

Memory fragmentation is real on long-running servers. Both engines manage their own KV cache, but the GPU allocator underneath them does not always cooperate. After a few days of mixed-length traffic the effective free VRAM can drop. A scheduled rolling restart is not a sign of fragility, it is the correct operational posture.

Concurrency limits matter more than batch size. Both engines project continuously, so the right knob is usually "max concurrent requests" rather than "max batch size". Set this too high and you get out-of-memory crashes under tail traffic. Set it too low and you leave throughput on the table. Both engines expose this; both default it conservatively.

Healthchecks lie if you let them. A naive HTTP healthcheck on the engine port returns OK while the model is still loading weights or the GPU is wedged in a recoverable error state. Health endpoints that actually run a tiny inference are worth the few hundred milliseconds.

How to choose

Forget the benchmarks. Ask three questions about your traffic.

1. Do many requests share a long prefix? Same system prompt across users, same retrieved context across queries, multi-turn conversations re-sending history. If yes, SGLang's prefix tree is a structural advantage and you should at least pilot it.

2. Do you need the absolute newest models on the day they drop? vLLM tends to land them first. SGLang catches up but it is not always day one.

3. Are you running on H100/A100 or on something more exotic? vLLM has the broader hardware story by a noticeable margin. SGLang assumes modern data-centre GPUs as the default.

If you answered "yes, no, H100" the case for SGLang is the strongest. If you answered "no, yes, mixed hardware" stick with vLLM. If you are running mixed traffic at scale, the right answer might be both, with a router in front that sends agent and RAG traffic to SGLang and one-shot completions to vLLM.

Where we ended up

We run small instruct and embedding models behind an OpenAI-compatible API, much of the traffic is one-shot or shallow multi-turn, and the cost of operational complexity matters more to us than the last 15% of throughput on the most prefix-heavy workload. For most of what we do, vLLM has been the right choice and the answer was rarely close. SGLang is sitting in the wings for the agent-heavy traffic where the prefix-cache advantage actually shows up.

The best advice we can give is the most boring: do not pick based on a screenshot of someone else's benchmark. Run a one-day shadow deployment on your real traffic, measure tail latency and throughput on the workload that pays your bills, and let the numbers decide.

Whatever you pick, keep your client code unchanged and behind the OpenAI-compatible interface. Then the next time the answer flips, you swap the backend and not the application.