I have been running performance tests on hospital and enterprise systems since 2000. The controller has changed names a few times, the protocols have changed more than that, but the job has stayed the same: find the point where the system stops keeping its promises, and say so with numbers before the users find it for you.
The first time I pointed that discipline at a streaming LLM endpoint, two things happened at once. The tooling mostly worked. The numbers mostly did not mean anything.
That is the whole problem with LLM load testing in one sentence. The mechanics of driving load are familiar. The measurements you are used to reading were designed for systems that return a whole answer at once, cost roughly the same per request, and degrade gradually. An inference stack does none of those things.
Five ways AI systems break differently
Streaming splits response time in two. A conventional request has one latency. A streaming completion has a time to first token, then a stream of inter-token gaps, then an end. Average those together and you get a number that describes neither the wait the user notices nor the stutter they notice after it. The practical consequence: a system can post a respectable mean response time while feeling broken, because the first token arrived late and the rest arrived in bursts.
Cost per request is not a constant. With a traditional API, one request costs about what the next one costs. With a model, a 40-token prompt and a 4,000-token prompt are different workloads by an order of magnitude, and output length compounds it. If your test corpus is one prompt repeated, or a handful of short prompts, you are measuring a system that does not exist. Every conclusion drawn from it is fiction with a decimal point.
Saturation is a cliff, not a slope. Most systems degrade gradually as you add users: latency climbs, then error rate climbs, and you have time to see it coming. A serving engine holds each active request's context in GPU memory as a KV cache. That memory fills nonlinearly with concurrency and sequence length. The system is fine at one concurrency level and failing a few users later, and the gap between the two can be narrow enough that a coarse ramp steps right over it.
Cache state changes the answer. A shared system prompt, a large document in context, or a common prefix can be served from a warm prefix cache far faster than the same request served cold. Two runs with identical load can produce very different curves depending on what the engine had cached. If you did not control for it, you do not know which system you measured.
Tails and time. Medians describe the user who got lucky. p95 and p99 describe the ones who will file the ticket. And short runs hide anything that accumulates: memory fragmentation, connection leaks, queue growth that only shows after an hour of sustained pressure. A twenty-minute test of a system that runs all day tells you about twenty minutes.
The metrics that replace requests per second
None of the above means requests per second is useless. It means it is not the number you size on. These are:
Time to first token (TTFT). How long the user waits before anything appears. It is dominated by prefill, the cost of processing the prompt, and it grows with prompt length and with queueing in front of the engine. This is the number that determines whether the assistant feels responsive.
Inter-token latency (ITL), also reported as time per output token. The gap between tokens as the answer streams. It determines whether the stream reads smoothly or stutters. It is the metric that gets worse first when the engine starts batching more requests than it comfortably can.
Tokens per second, measured separately for prompt tokens and completion tokens. Throughput in the unit the hardware actually produces. It is how you compare two models, two engines, or two GPU counts fairly.
Goodput. Not how many requests completed, but how many completed inside the SLO. A run that finishes every request but misses the latency target on a third of them has a goodput of two thirds. This is the number that drives sizing, because it is the only one that combines throughput with the promise you made.
p95 and p99 latency, end to end. The tails. On any system with a queue, they move first and they move most.
Cost per request on a hosted API, or cost per unit of throughput on private hardware. Because the answer to "how many users can it handle" is meaningless without "at what cost per answer."
Each of these has a moment where it matters most. TTFT and ITL matter for interactive assistants. Tokens per second and cost per unit matter for batch workloads and sizing. Goodput matters for everyone, because it is the one leadership can read.
Building realistic load
Getting the metrics right is half of it. The load has to look like production or the metrics describe nothing.
Start with a corpus of prompts sampled from real traffic, typically fifty to a hundred, chosen so the distribution of lengths matches what the system actually sees. Never one prompt repeated. If production data cannot be used directly, sanitize it or synthesize to the same shape, but keep the shape.
Test streaming as streaming. If the application uses server-sent events, the test drives server-sent events and captures the timing of every chunk. Substituting a non-streaming call because it is easier to script erases the two most important metrics.
Run warm-cache and cold-cache variants separately and report both. The delta between them is often the single most useful number for whoever designs the system prompt.
Then run the shapes that expose different failures: a ramp to find the thresholds; a burst to simulate the spike when a department all logs in at nine; sustained load at the target level; and a soak long enough to surface memory growth and fragmentation. Near saturation, search for the breakpoint in fine concurrency steps, because the failure is a cliff and a coarse ramp will step over the edge without noticing.
Two layers, one report
There are two places to measure, and you need both.
The application path is what the user sees. On this layer we work in whatever load tool the client already runs. LoadRunner, NeoLoad, JMeter, k6, and Gatling can all drive a streaming endpoint and capture time to first token, per-token latency, and tokens per second per virtual user; the tool matters far less than the script design, and a team that knows its own tool will keep running the tests after we leave. (If the tool is OpenText LoadRunner, we operate and resell it, but it is one option, not a requirement.) This layer shows you what the app stack adds. Authentication, orchestration, retrieval, and the front end all sit between the user and the model, and every one of them adds latency that the model did not cause.
The inference path is where the model and the hardware actually run out. On this layer we benchmark the serving endpoint directly with NVIDIA AIPerf and correlate it with the engine's own metrics from vLLM, llama.cpp, or whatever the client runs: TTFT, inter-token latency, tokens per second, request queueing, and KV-cache pressure.
One without the other misleads. Measure only the application path and you cannot tell whether the wait is the model or the plumbing. Measure only the inference path and you have a benchmark of a component, not a test of the system users touch. The report puts both on the same concurrency axis so the breakpoint has a cause attached to it.
The part nobody measures
Everything above is about speed. Speed is half the answer.
Under load, AI systems fail quietly. The status code is 200, the latency is inside the SLO, and the answer is wrong. Three failure modes I have seen in the field: a completion that truncates mid-sentence because a timeout returned a partial stream and the application accepted it; an OCR stage that skips pages when the queue backs up and reports the document as processed; a de-identification model that redacted every identifier at low volume and leaked one in a thousand when the batch size climbed. None of those show up on a latency graph.
Verified Under Load is our name for closing that gap. During the run, we capture the outputs the system produced and inspect them automatically for exactly those failure modes, using a checker that never shares a model or a code path with the system under test. For regulated work the default is one hundred percent of a defined slice of the output; sampling is used only when the client chooses it, and the report says so. The result is two breakpoints instead of one: the load level where latency crosses the SLO, and the load level where output quality does. They are rarely the same number, and the second one is usually the one that matters.
Inside the walls
Most of the load testing platforms built for the cloud era assume you will send them your traffic. For a hospital, a research institution, a bank, or an agency running a private model precisely because the data cannot leave, that assumption ends the conversation. The prompt corpus is patient records. The responses are the thing being protected.
Running the whole rig inside the client's network is what we already do for regulated performance testing, and the model does not change that. The load generator, the corpus, every captured response, and the results database run on the client's hardware with the client's load tool. No traffic to a SaaS load platform, no prompts in a vendor's cloud, no results exported for analysis. It takes more setup than pointing a hosted tool at a URL. It is also the only version of this work that the compliance officer will sign.
For applications built on hosted APIs, the cloud-API track exists too, with rate limits and cost per run modeled into the plan. But the private track is the one we lead with, because it is the one most of the people asking cannot do any other way.
Where this leaves you
Measure the application path in the tool you already run, measure the inference path the way the engine actually fails, keep all of it inside your network when the data requires it, and check that the answers are still right when the box is full.
That is the practice in one paragraph. The full description, including the LLM and assistant load testing engagement and capacity planning and GPU sizing from measured throughput, is on the AI Performance Engineering page.