Capacity planning and GPU sizing
How many boxes, which model, at what cost per unit of work. Measured, then extrapolated, never guessed.
Where sizing numbers usually come from, and why they are wrong
Vendor throughput figures assume short prompts, warm caches, and ideal batching. Your workload has none of those.
The gap between the marketing number and the measured number is routinely large, and it is the difference between the right purchase and a second purchase six months later.
Pre-sizing on a sample
What it is: you provide a sanitized or synthetic sample of your workload (prompts, documents, or images) and the candidate models or serving stacks. We run it on our own known hardware and return tokens per second (or items per hour), the saturation point, and a GPU count range for your target volume, before you commit capital.
Scope: a paid first step with a fixed scope and a dated report; credited against a larger engagement when one follows.
Four Walls note: for clients whose data cannot leave, the same method runs on their hardware as the first phase of a load-test engagement.
- [·]Throughput per device.
- [·]Saturation point.
- [·]GPU count range for the target volume.
- [·]Cost per unit of throughput.
- [·]The assumptions, stated.
Full sizing from load tests
When the system exists, sizing comes from the load tests described on the LLM and pipeline pages: measured curves at each concurrency, the breakpoint, and the resource that caused it, extrapolated to the target volume with a stated safety margin. Includes model and serving-engine comparisons at equal load, and quantization trade-offs measured for both throughput and output quality.
What goes into the number
- [·]Prompt and output length distribution.
- [·]Concurrency pattern (steady, bursty, batch windows).
- [·]Cache behavior.
- [·]SLO for TTFT and ITL, or for items per hour.
- [·]Goodput target.
- [·]Growth assumptions.
- [·]The cost model (capital, cloud, or hybrid).
Two layers, one report
Scripts that drive the application the way users or upstream systems do, including streaming responses (server-sent events), with time to first token, per-token latency, and tokens per second captured per virtual user. We work in the load tool you already run: OpenText LoadRunner, Tricentis NeoLoad, Apache JMeter, Grafana k6, Gatling, or similar. If you have none, we bring the tooling.
Direct benchmarking of the serving endpoint with NVIDIA AIPerf, correlated with the engine's own metrics (vLLM, llama.cpp, or the client's stack): TTFT, inter-token latency, tokens per second, request queueing, and KV-cache pressure.
Why both: the application path shows what users experience and where the app stack adds latency; the inference path shows where the model and hardware actually run out. One without the other produces a number nobody can act on.
On this pageApplication-path numbers set the SLO; inference-path numbers set the hardware.
Four Walls: nothing leaves your network
The load generator, the prompt corpus, every captured response, and the results database run on your network, on your hardware, with your load tool. No traffic to a SaaS load platform, no prompts in a vendor's cloud, no results exported for analysis. This is how we already run performance testing for regulated clients: the same tooling, the same discipline, pointed at the model instead of the EHR.
Who it is for: hospitals and research institutions, defense and government, finance, and any team running private models because the data cannot leave.
Second track: For applications built on hosted APIs, we run a cloud-API track with rate limits and cost per run modeled into the plan. Four Walls is the default we lead with; cloud API is the option.
Verified Under Load
Speed is half the answer. Under load, AI systems fail quietly: answers truncate, timeouts return partial results, OCR skips pages when the queue backs up, a de-identifier leaks an entity it caught at low volume, an agent drops a step. Verified Under Load means we measure throughput and latency AND we inspect the outputs the system produced during the run, automatically, for exactly those failure modes. For regulated work the default is one hundred percent of a defined slice of the output; sampling is used only when the client chooses it, and it is stated as such in the report.
- [·]A per-run evidence report (pass, fail, quarantine counts by failure mode).
- [·]The load level at which output quality starts to degrade, alongside the load level at which latency does.
- [·]The independence rule stated in writing (the checker never shares a model or a code path with the system under test).
On this pageThe count we recommend is the one at which outputs still pass, not just the one at which latency still passes.
How an engagement runs
What you walk away with
- [·]A sizing memo with the count range and its assumptions.
- [·]Cost per unit of throughput by option.
- [·]The measured curves behind it.
- [·]A procurement-ready summary.
Straight answers
Start with the number you do not have
Bring us the system and the volume you need it to survive. We will scope the first run, tell you what it will measure, and put a date on the report.
844.PROTICOM
proticom.ai