AI Performance Engineering
Load, stress, and scalability testing for AI systems, measured the way the engine actually fails, and verified so the answers are still right when the box is full.
Why AI systems break differently
Streaming splits "response time" into time to first token and inter-token latency; averaging them hides both.
Cost per request varies by an order of magnitude with prompt and output length; uniform test payloads produce fiction.
Saturation is a cliff, not a slope: GPU memory for the KV cache fills nonlinearly, so the system is fine at one concurrency level and failing a few users later.
p95 and p99 matter more than medians, and soak runs expose memory growth that short runs never show.
These are measurable. They are just not measured by requests per second.
Three engagements, one method
Chat and completion endpoints, assistants, copilots. Private models and hosted APIs.
Document AI, OCR, de-identification, extraction, batch inference.
Throughput curves to hardware counts, pre-sizing on a sample.
Agents and RAG: conversation-shaped load, growing context, tool-call chains. Covered on the LLM and assistant load testing page.
Two layers, one report
Scripts that drive the application the way users or upstream systems do, including streaming responses (server-sent events), with time to first token, per-token latency, and tokens per second captured per virtual user. We work in the load tool you already run: OpenText LoadRunner, Tricentis NeoLoad, Apache JMeter, Grafana k6, Gatling, or similar. If you have none, we bring the tooling.
Direct benchmarking of the serving endpoint with NVIDIA AIPerf, correlated with the engine's own metrics (vLLM, llama.cpp, or the client's stack): TTFT, inter-token latency, tokens per second, request queueing, and KV-cache pressure.
Why both: the application path shows what users experience and where the app stack adds latency; the inference path shows where the model and hardware actually run out. One without the other produces a number nobody can act on.
Four Walls: nothing leaves your network
The load generator, the prompt corpus, every captured response, and the results database run on your network, on your hardware, with your load tool. No traffic to a SaaS load platform, no prompts in a vendor's cloud, no results exported for analysis. This is how we already run performance testing for regulated clients: the same tooling, the same discipline, pointed at the model instead of the EHR.
Who it is for: hospitals and research institutions, defense and government, finance, and any team running private models because the data cannot leave.
Second track: For applications built on hosted APIs, we run a cloud-API track with rate limits and cost per run modeled into the plan. Four Walls is the default we lead with; cloud API is the option.
Verified Under Load
Speed is half the answer. Under load, AI systems fail quietly: answers truncate, timeouts return partial results, OCR skips pages when the queue backs up, a de-identifier leaks an entity it caught at low volume, an agent drops a step. Verified Under Load means we measure throughput and latency AND we inspect the outputs the system produced during the run, automatically, for exactly those failure modes. For regulated work the default is one hundred percent of a defined slice of the output; sampling is used only when the client chooses it, and it is stated as such in the report.
- [·]A per-run evidence report (pass, fail, quarantine counts by failure mode).
- [·]The load level at which output quality starts to degrade, alongside the load level at which latency does.
- [·]The independence rule stated in writing (the checker never shares a model or a code path with the system under test).
What you walk away with
- [·]Throughput and latency curves per concurrency step (TTFT, ITL, tokens per second, p50/p95/p99).
- [·]The breakpoint: where latency, error rate, or output quality crosses the SLO, and which resource ran out first.
- [·]Sizing: the hardware count range for the target volume, with cost per unit of throughput.
- [·]The evidence report from Verified Under Load.
- [·]A written recommendation in plain language for the people who sign.
How an engagement runs
Built on what you already run
The application path runs in the load tool your team already knows. LoadRunner, NeoLoad, JMeter, k6, and Gatling can all drive streaming LLM endpoints and capture time to first token, per-token latency, and tokens per second; we script in whichever one you own, and we bring the tooling when you have none. On the inference side we use NVIDIA AIPerf and the serving engine's own metrics, the tools your platform team will recognize.
If you are on OpenText LoadRunner or Performance Center, we operate and resell it; see OpenText DevOps Cloud Services.
Straight answers
Start with the number you do not have
Bring us the system and the volume you need it to survive. We will scope the first run, tell you what it will measure, and put a date on the report.
844.PROTICOM
proticom.ai