I run load tests on LLM stacks. Some of those stacks sit behind a vendor API. Most of them sit on hardware the client owns, in a data center the client controls, running an engine the client compiled. The second category is what this post is about.
If you are running vLLM, llama.cpp, Ollama, or TensorRT-LLM on your own boxes, the questions you need answered are different from the ones you ask about a hosted API. You are not sizing to stay under rate limits. You are sizing GPU count, memory per instance, and concurrency before the KV cache cliff kills throughput. You own the whole problem, which means you can measure the whole problem. It also means you have to.
Why local stacks are a different measurement problem
With a hosted API, your load test tells you whether the vendor can meet your SLO at your scale. That matters, but the stack itself is a black box. You cannot see queueing inside the engine. You cannot compare two serving configurations or two quantization levels on the same corpus. You cannot fix what breaks because you do not control it.
10|With a local stack, you control the engine, the model weights, the hardware, and the network path. That gives you four things a cloud API does not: visibility into KV-cache pressure, the ability to compare engines fairly on identical prompts, control over quantization and batching strategy, and a way to measure cost per unit of throughput on private hardware instead of cost per API call.
It also gives you more ways to break the system. Queueing happens inside the engine, not behind a load balancer you can scale horizontally. GPU memory is fixed per instance. Continuous batching is tunable but unforgiving. Prompt length variance that a hosted stack absorbs quietly will blow through your memory budget and crash throughput in three requests.
The measurement problem is not simpler. It is more direct.
The engines we test
The stacks we load test most often are vLLM, which dominates production deployments where throughput and batching matter; llama.cpp, which runs on CPUs, edge devices, and smaller VRAM budgets; Ollama, which wraps llama.cpp with an API layer and model management; and TensorRT-LLM, which is NVIDIA's optimized inference runtime for datacenter GPUs. Each has different memory layouts, batching strategies, and failure modes. A configuration that works on vLLM may not map cleanly to llama.cpp. Testing one is not testing all of them.
20|## Four walls: the local constraint
The pattern we call Four Walls is this: the load generator, the prompt corpus, every captured response, and the results database stay inside the client's network. No traffic to a SaaS load platform. No prompts sent to a vendor cloud for analysis. No results exported to someone else's dashboard.
This is not about preferences. For a hospital running a clinical model, a bank running fraud detection, or a manufacturer running a private research assistant, the data cannot leave. The corpus is patient records or proprietary documents. The responses contain exactly the thing being protected. Compliance will not allow external tools to touch any of it.
Running the whole test stack inside means we use the client's load tool (LoadRunner, NeoLoad, JMeter, k6, or Gatling), script it to their standards, store everything on their hardware, and hand them a method they can repeat without us. It takes more setup than pointing a cloud service at a URL. It is the only version of this work that regulated teams can actually do.
Two layers, both visible
30|There are two places to measure on a local stack, and you need both layers to find the real breakpoint.
The application path is what users see. This is where we drive load through whatever sits in front of the model: authentication, document retrieval, prompt assembly, response streaming. We script this in the client's existing load tool. It tells you what the whole system delivers, including everything that is not the model.
The inference path is the serving engine itself. Here we benchmark the vLLM or llama.cpp endpoint directly with NVIDIA AIPerf and correlate it with the engine's own metrics: time to first token, inter-token latency, tokens per second for prompt and completion separately, request queueing depth, KV-cache occupancy, and batch utilization. This layer tells you where the GPU or the engine ran out of room.
One without the other misleads. Measure only the application path and you cannot tell whether latency comes from the model or from the six things bolted in front of it. Measure only the inference path and you have a microbenchmark of a component, not a picture of what users experience. The report needs both on the same concurrency axis so when throughput falls off a cliff, the cause is visible.
The metrics that replace requests per second on local hardware
40|Requests per second still exists. It is not what you size on. These are:
Time to first token (TTFT). The wait before anything streams back. On a local engine this is dominated by prefill cost (processing the prompt) and by how many requests are queued in front of yours. It grows nonlinearly with prompt length and with engine load. A system can post acceptable mean TTFT and still feel broken if p95 is twice that, because the tail is what users notice.
Inter-token latency (ITL), also called time per output token. The gap between tokens as the completion streams. It tells you whether decode is smooth or choppy, and it degrades first when the engine starts batching more requests than it can comfortably serve. Watch this metric near saturation.
Tokens per second, measured separately for prompt throughput and completion throughput. This is how you compare two models, two engines, or two GPU configurations fairly, because it is the unit the hardware actually produces. An A100 and an H100 do not deliver the same tokens per second on the same model. Neither do two quantization levels. This is the number that makes those comparisons honest.
Goodput. Not how many requests finished, but how many finished inside the SLO. A run that completes every request but misses the latency target on forty percent of them has a goodput of sixty percent. This is the number that drives hardware sizing, because it combines throughput with the promise you made.
50|KV-cache utilization and memory pressure. On vLLM and TensorRT-LLM, you can see how much GPU memory the KV cache is using and how close you are to the limit. llama.cpp exposes similar data. When memory fills, the engine either refuses new requests or evicts cached context, and throughput craters. This is not a metric you see on a hosted API. On local hardware it is the single most useful signal for sizing concurrency and sequence length budgets.
Cost per unit of throughput. For a hosted API you measure cost per token or cost per request. For local hardware you measure cost per million tokens per hour, divided by the capital and power cost of the GPU. This tells you whether adding another A100 is cheaper than moving to an H100, or whether quantizing to int8 saves enough memory to double concurrency without buying more boxes.
Why local stacks break differently
Hosted APIs degrade gradually. Local engines hit cliffs. Here is why.
GPU memory is a hard limit. Each active request holds its prompt and its generated tokens in GPU VRAM as a KV cache. That memory grows with sequence length and with concurrency. When it fills, the engine cannot accept more work. With continuous batching, the engine tries to pack requests efficiently, but if your prompts are longer than you budgeted or your concurrency ramps too fast, you cross the memory line without warning and throughput stops. A test that works at eight concurrent users can fail completely at ten.
60|Quantization changes speed and quality. Loading a model at fp16, int8, or int4 changes both how much memory it needs and how it behaves under load. int8 can double your concurrency budget on the same GPU, but it also shifts output quality in ways that may not show on short prompts. If you test one quantization level and deploy another, you tested a different system. If you compare two engines or two models at different quantizations, the comparison is not fair. Load tests must hold quantization constant or report it clearly.
Continuous batching is tunable but unforgiving. vLLM and TensorRT-LLM batch requests dynamically to maximize throughput. The batching strategy has parameters: max batch size, timeout, preemption policy. Those knobs matter. Set max batch size too high and you blow out memory. Set it too low and you leave throughput on the table. Tuning those parameters requires measurement under realistic load. Guessing costs you either capacity or stability.
Prompt and output length variance is not averaged away. On a hosted API, variance in sequence length is the vendor's problem. On a local engine, it is yours. One 8,000-token prompt in a batch of 500-token prompts can stall the whole batch or blow the memory budget. A test corpus with narrow length distribution will not expose that. If your production traffic has outliers, your test corpus must too, or the breakpoint you measure is fiction.
Warm versus cold prefix cache changes everything. If your application uses a shared system prompt or a large document that appears in every request, a warm prefix cache can cut TTFT in half. A cold cache serves the same request far slower. Two identical load runs can produce very different latency curves depending on cache state. If you do not control for it, you do not know which version of the system you measured. We run both and report the delta, because the delta is often the single most useful number for whoever designs the prompt structure.
70|Soak time surfaces memory growth. A twenty-minute test tells you about twenty minutes. Fragmentation, connection leaks, and queue buildup show up after an hour of sustained load. On a local stack where you own uptime, a short test is not enough. Run soak long enough to see if memory grows or if throughput degrades over time.
Building realistic load for local stacks
Getting the metrics right is half. The load has to look like production or the metrics describe nothing.
Start with a corpus of prompts sampled from real traffic. Fifty to a hundred prompts is typical, chosen so the distribution of lengths matches what the system actually sees. Never one prompt repeated. If production data cannot be used directly, sanitize it or synthesize to the same shape, but keep the shape. Narrow test data produces narrow conclusions.
Test streaming as streaming. If the application uses server-sent events, the load script must parse server-sent events and capture the timing of every chunk. Substituting a non-streaming call because it is easier to write erases the two most important metrics: time to first token and inter-token latency.
80|Run the load patterns that expose different failure modes. A ramp finds the thresholds. A burst simulates the spike when a department all logs in at the same time. Sustained load at the target level proves the system holds. A soak surfaces memory leaks and fragmentation. Near saturation, search for the breakpoint in fine concurrency steps (one or two users at a time), because the failure is a cliff and a coarse ramp will step over the edge without seeing it.
Run warm-cache and cold-cache variants separately and report both. The delta tells you what caching buys, and it is often the number that decides whether a shared system prompt is worth the complexity.
Comparing engines and configurations fairly
One of the reasons to own the hardware is that you can compare options on the same corpus. But the comparison is only fair if you control everything except the thing you are testing.
If you want to know whether vLLM is faster than llama.cpp for your workload, run both on the same model, the same quantization, the same prompts, and the same GPU. If you want to know whether int8 quantization saves enough memory to double concurrency, run both at the same batch size and sequence length budget, on the same corpus, and measure throughput and quality together.
90|That last part matters. Quantization changes output, not just speed. A test that only measures tokens per second will tell you int8 is faster and stops there. A test that also checks quality under load will tell you whether int8 truncates more often, produces partial streams, or degrades accuracy when the system is full. You need both numbers to make the tradeoff honest.
Verified Under Load: quality on local stacks
Everything above is about speed. Speed is half the answer.
Under load, inference engines fail quietly. The status code is 200. Latency is acceptable. The output is wrong. I have seen completions truncate mid-sentence because a timeout returned a partial stream. I have seen a batch pipeline skip pages when the queue backed up and report success anyway. I have seen quantized models leak information that the fp16 version redacted correctly. None of those failures show on a latency graph.
Verified Under Load is our name for closing that gap. During the run, we capture the outputs the system produced and inspect them automatically for exactly those failure modes, using a checker that never shares a model or a code path with the system under test. For regulated work the default is one hundred percent of a defined slice of the output. Sampling is used only when the client chooses it, and the report says so.
100|The result is two breakpoints instead of one: the load level where latency crosses the SLO, and the load level where output quality does. They are rarely the same number, and the second one is usually the one that matters. On a local stack where you control the hardware, that distinction is cheaper to find early than to discover in production.
Practical notes for local operations
A few things that belong in this post more than in the general one:
GPU memory versus concurrency is a tradeoff, not a constant. More memory per GPU lets you serve longer sequences or higher concurrency, but it costs more. Splitting a model across two smaller GPUs instead of one large one changes latency and cost. These tradeoffs are specific to your traffic shape. Measure them.
Continuous batching is not automatic optimization. It has parameters. Default settings are rarely optimal for your workload. Tuning requires iteration under realistic load. Budget time for it.
110|Quantization is a speed-and-quality tradeoff. int8 and int4 save memory and increase throughput, sometimes dramatically. They also change model behavior. On some tasks the difference is unnoticeable. On others it breaks compliance or accuracy requirements. Test both speed and output correctness at every quantization level you consider. A test that measures only one is incomplete.
Comparing two engines or two GPU counts fairly means same corpus, same quantization, same sequence length budget. Otherwise you are comparing configurations, not engines, and the conclusion will not hold when you change the variables.
Local stacks let you see queueing and memory pressure that cloud APIs hide. Use that visibility. It is the advantage you paid for.
The two paths: application and inference
I said this earlier but it is worth repeating because it is the structure of every report we deliver.
The application path is what users touch. We measure it with the client's load tool (LoadRunner, NeoLoad, JMeter, k6, or Gatling), scripted to the client's standards, running inside the client's network. This path includes authentication, retrieval, orchestration, prompt assembly, response parsing, and the model call itself. It tells you what the whole system delivers.
120|The inference path is the engine: vLLM, llama.cpp, Ollama, or TensorRT-LLM, measured directly with NVIDIA AIPerf and correlated with engine metrics. This path tells you where the GPU ran out of memory, where batching failed, where the KV cache filled, and where the model itself became the bottleneck.
One report, two layers, same concurrency axis. When throughput falls, you know whether it was the model, the hardware, or the six things wrapped around it. That is what local visibility buys.
Where this leaves you
Load test the application path with the tool you already run. Load test the inference path with engine-native metrics. Keep all of it inside your network when the data requires it. Check that outputs are still correct when the box is full. Size GPU count and memory from measured throughput under realistic load, not from vendor benchmarks on synthetic prompts.
That is the practice in one paragraph. The full version, including how we size GPU capacity from real load curves and how Verified Under Load works on quantized models, is on the AI Performance Engineering page. The LLM-specific engagement structure is on the LLM load testing page.
If you want the general version of this argument (why requests per second is the wrong metric and what replaces it), that is in LLM load testing: why requests per second is the wrong number. This post is the local companion to that one. Together they cover the range: cloud APIs and self-hosted stacks.