Document tray feeding a matte scanner unit: batch pipeline at volume

QA / TESTING · AI PERFORMANCE ENGINEERING

AI pipeline volume and scalability testing

A pipeline that works on ten thousand items is not evidence it works on a million. Measure it.

SCOPE

The pipelines we test

PIPELINE TYPES

[·]Document AI and extraction (contracts, claims, records, forms).
[·]OCR and scanned-document processing.
[·]De-identification and redaction of text and images (PHI, PII), including medical imaging pipelines.
[·]Classification, tagging, and embedding generation at scale.
[·]Image and video analysis pipelines.
[·]Any batch inference job with a queue in front of it.

WHAT WE MEASURE

[·]Items per hour at each concurrency and batch size (documents, pages, images, records).
[·]Per-stage latency: ingest, pre-processing, model inference, post-processing, write-back.
[·]Queue depth and growth over time; where the backlog forms.
[·]GPU, CPU, memory, and I/O utilization per stage; which one runs out first.
[·]Failure and retry rates by stage, and what happens to the items that failed.
[·]Cost per thousand items at the target throughput.

FAILURE MODES

How pipelines fail at volume

The bottleneck is usually not the model

I/O, OCR, and serialization stages saturate first.

Queues hide failures

A backlog looks like progress until it overflows.

Long runs expose what short runs cannot

Memory growth and connection exhaustion show up hours in, not minutes in.

Output quality degrades before the pipeline stops

Skipped pages, partial extractions, leaked entities. This is why Verified Under Load is part of every pipeline engagement.

METHOD

Two layers, one report

Layer 1: application path (your load tool)

Scripts that drive the application the way users or upstream systems do, including streaming responses (server-sent events), with time to first token, per-token latency, and tokens per second captured per virtual user. We work in the load tool you already run: OpenText LoadRunner, Tricentis NeoLoad, Apache JMeter, Grafana k6, Gatling, or similar. If you have none, we bring the tooling.

Layer 2: inference path (AIPerf + engine metrics)

Direct benchmarking of the serving endpoint with NVIDIA AIPerf, correlated with the engine's own metrics (vLLM, llama.cpp, or the client's stack): TTFT, inter-token latency, tokens per second, request queueing, and KV-cache pressure.

Why both: the application path shows what users experience and where the app stack adds latency; the inference path shows where the model and hardware actually run out. One without the other produces a number nobody can act on.

On this pageYour load tool drives the pipeline's real entry points and APIs at volume; AIPerf and engine metrics isolate the model stage.

DEPLOYMENT

Four Walls: nothing leaves your network

The load generator, the prompt corpus, every captured response, and the results database run on your network, on your hardware, with your load tool. No traffic to a SaaS load platform, no prompts in a vendor's cloud, no results exported for analysis. This is how we already run performance testing for regulated clients: the same tooling, the same discipline, pointed at the model instead of the EHR.

Who it is for: hospitals and research institutions, defense and government, finance, and any team running private models because the data cannot leave.

Second track: For applications built on hosted APIs, we run a cloud-API track with rate limits and cost per run modeled into the plan. Four Walls is the default we lead with; cloud API is the option.

On this pageThe test corpus is the client's own data, which is precisely why it cannot leave.

VALIDATION AT SCALE

Verified Under Load

Speed is half the answer. Under load, AI systems fail quietly: answers truncate, timeouts return partial results, OCR skips pages when the queue backs up, a de-identifier leaks an entity it caught at low volume, an agent drops a step. Verified Under Load means we measure throughput and latency AND we inspect the outputs the system produced during the run, automatically, for exactly those failure modes. For regulated work the default is one hundred percent of a defined slice of the output; sampling is used only when the client chooses it, and it is stated as such in the report.

WHAT YOU GET

[·]A per-run evidence report (pass, fail, quarantine counts by failure mode).
[·]The load level at which output quality starts to degrade, alongside the load level at which latency does.
[·]The independence rule stated in writing (the checker never shares a model or a code path with the system under test).

On this pagePipeline failure modes we inspect for: skipped pages, partial extractions, residual identifiers in "clean" output, inspected on one hundred percent of a defined slice.

ENGAGEMENT

How an engagement runs

Define

Target volume (a full backlog or a monthly intake), corpus sampled from production, per-stage SLOs, and pass criteria for output.

Build

Load scenarios in your tool and AIPerf profiles; warm-cache and cold-cache variants.

Run

Ramp, sustained load, burst, soak (long enough to expose memory growth), and breakpoint search with fine steps near saturation.

Verify

Automated inspection of outputs from the runs (Verified Under Load).

Report

Curves, breakpoints, sizing, and the evidence report, with a recommendation the client can put in front of leadership or procurement.

DELIVERABLES

What you walk away with

IN THE REPORT

[·]Throughput curves by stage.
[·]The breakpoint and the stage that caused it.
[·]Hardware sizing for the target volume.
[·]A queue and retry policy recommendation.
[·]The evidence report from Verified Under Load.
[·]A written go or no-go for the volume you asked about.

Performance engineering on hospital and enterprise systems since 2000LoadRunner and Performance Center operated for clients continuously since 2014Test automation practice across regulated environmentsOpenText partner and reseller

FAQ

Straight answers

We are evaluating vendors for this pipeline. Can you test all of them?

Yes, with the same corpus and the same volume ladder, so the comparison is fair and the leakage and throughput numbers are yours, not the vendor's.

Our data cannot leave the building.

See Four Walls; that is the default for this page.

What if the model is fine and the bottleneck is elsewhere?

That is the common case, and the report says so with the numbers, so the fix goes where the problem is.

ALSO IN THIS PRACTICE

LLM and assistant load testing[→]

Chat and completion endpoints, assistants, copilots. Private models and hosted APIs.

Capacity planning and GPU sizing[→]

Throughput curves to hardware counts, pre-sizing on a sample.

Back to the practice overview: AI Performance Engineering.

// Start with the number you do not have

Start with the number you do not have

Bring us the system and the volume you need it to survive. We will scope the first run, tell you what it will measure, and put a date on the report.

How we prove it first →

contact@proticom.com
844.PROTICOM
proticom.ai