LLM and assistant load testing
Find the concurrency your model actually supports, and what users see on the way there.
The numbers that replace requests per second
- Time to first token (TTFT)
- Perceived responsiveness; the prefill cost.
- Inter-token latency (ITL) / time per output token
- Generation smoothness.
- Tokens per second
- Prompt and completion tokens, separately.
- End-to-end latency
- At p50, p95, p99.
- Goodput
- The share of requests that met the SLO, the number that drives sizing.
- Error, timeout, and rate-limit rates
- Counted per concurrency step, not averaged across the run.
- Cost
- Cost per request (API track) or cost per unit of throughput (private track).
- KV-cache utilization and queue depth
- From the engine's own metrics.
Private models and hosted APIs
vLLM, llama.cpp, TensorRT-LLM, or the client's serving stack, tested on the client's hardware. Warm-cache and cold-cache runs measured separately, because shared system prompts and document context change the answer.
Rate limits mapped before the first run; concurrency ramped inside them; spend caps set per run; cost per request reported alongside latency.
Realistic load, not synthetic load
- [·]A corpus of production-sampled prompts (typically 50 to 100) that matches the real length distribution; never one prompt repeated.
- [·]Streaming endpoints tested as streaming (server-sent events), never a non-streaming stand-in.
- [·]Ramp to find thresholds, burst to simulate spikes, sustained load, and soak runs long enough to surface memory growth and fragmentation.
- [·]Breakpoint search with fine concurrency steps near saturation, because the failure is a cliff.
Two layers, one report
Scripts that drive the application the way users or upstream systems do, including streaming responses (server-sent events), with time to first token, per-token latency, and tokens per second captured per virtual user. We work in the load tool you already run: OpenText LoadRunner, Tricentis NeoLoad, Apache JMeter, Grafana k6, Gatling, or similar. If you have none, we bring the tooling.
Direct benchmarking of the serving endpoint with NVIDIA AIPerf, correlated with the engine's own metrics (vLLM, llama.cpp, or the client's stack): TTFT, inter-token latency, tokens per second, request queueing, and KV-cache pressure.
Why both: the application path shows what users experience and where the app stack adds latency; the inference path shows where the model and hardware actually run out. One without the other produces a number nobody can act on.
Agents and RAG
Assistants that retrieve, reason, and call tools do not fail like endpoints. Load has to be shaped like conversations, not requests: context that grows across turns until the eleventh message overflows the window; tool-call chains whose latencies cascade; token budgets that differ between two identical requests because the agent took a different path.
We test these as an extension of the LLM engagement: conversation-shaped scenarios, per-step latency and token accounting, retrieval latency and hit quality under load, and Verified Under Load applied to the final answer and to each dropped or skipped step.
Four Walls: nothing leaves your network
The load generator, the prompt corpus, every captured response, and the results database run on your network, on your hardware, with your load tool. No traffic to a SaaS load platform, no prompts in a vendor's cloud, no results exported for analysis. This is how we already run performance testing for regulated clients: the same tooling, the same discipline, pointed at the model instead of the EHR.
Who it is for: hospitals and research institutions, defense and government, finance, and any team running private models because the data cannot leave.
Second track: For applications built on hosted APIs, we run a cloud-API track with rate limits and cost per run modeled into the plan. Four Walls is the default we lead with; cloud API is the option.
Verified Under Load
Speed is half the answer. Under load, AI systems fail quietly: answers truncate, timeouts return partial results, OCR skips pages when the queue backs up, a de-identifier leaks an entity it caught at low volume, an agent drops a step. Verified Under Load means we measure throughput and latency AND we inspect the outputs the system produced during the run, automatically, for exactly those failure modes. For regulated work the default is one hundred percent of a defined slice of the output; sampling is used only when the client chooses it, and it is stated as such in the report.
- [·]A per-run evidence report (pass, fail, quarantine counts by failure mode).
- [·]The load level at which output quality starts to degrade, alongside the load level at which latency does.
- [·]The independence rule stated in writing (the checker never shares a model or a code path with the system under test).
How an engagement runs
What you walk away with
- [·]Latency and throughput curves per concurrency step.
- [·]The saturation point and which resource caused it.
- [·]Warm versus cold cache deltas.
- [·]Goodput at the target SLO.
- [·]The evidence report from Verified Under Load.
- [·]A recommendation on concurrency limits, batching, and hardware.
Straight answers
Start with the number you do not have
Bring us the system and the volume you need it to survive. We will scope the first run, tell you what it will measure, and put a date on the report.
844.PROTICOM
proticom.ai