LLM Testing: A QA Process for Output That Changes Every Run

LLM testing checks whether an application’s outputs stay correct across repeated runs, scored against a threshold instead of a string. The same input can return a different answer next run. A pass is a score, not an equality. This is the process our QA team runs on LLM features.

We wrote it for the QA lead who has been handed an LLM feature with no evals team behind them. The first suite we wrote for one of those features passed every case and shipped a broken feature. Everything below comes from what we changed after that.

What Is LLM Testing?

LLM testing is the validation of an LLM application’s outputs against defined criteria, across repeated runs, at a score threshold. The criteria cover correctness, faithfulness to source, safety, tone, and cost. The threshold replaces the assertion because no two runs return the same text.

Four borders keep that definition honest.

• Testing vs evaluation: An evaluation produces a score for one output. A test passes or fails a set of outputs at a threshold. Testing uses evaluation as its oracle.
• Application vs model: This page tests the feature users touch: prompt, retrieval, tools, and the model together. Validating the model on its own follows a 7-step manual validation framework.
• Process vs checklist: Our 30 manual QA checks for AI and LLM apps say what to check. This page is how we run those checks as a repeatable suite.
• Testing LLMs vs testing with LLMs: Using a model to generate or grade tests is AI in software testing. This page is about the LLM being the thing under test.

The engagement below is why we needed that definition.

Why Did Our First LLM Test Suite Fail?

Our first LLM test suite failed because it asserted on strings, and production never sent the strings it had seen. At temperature 0 on fixed inputs the outputs matched. On new inputs and default settings, they did not. The suite passed in CI on every run. The feature failed for users in the first week. Nothing in the suite was wrong; the suite was measuring the wrong property.

What the Suite Asserted On

The suite was a set of 40 input-output pairs with exact-match assertions on the output. The feature summarised support tickets into 3 fields: issue, severity, and next action. Each case held one ticket and the expected 3-field summary, written by the product owner.

In CI, the model ran at temperature 0 against the same 40 tickets. The outputs matched. The suite went green 12 times in a row across 2 weeks of prompt edits.

What Production Did Instead

Production ran the same prompt against tickets nobody had seen. The severity field came back as “High”, “high”, “HIGH”, and “High priority” across 4 tickets with the same content. A downstream rule matched only “High”. Three of the 4 tickets dropped into the wrong queue.

The summaries themselves were correct on every one of those tickets. A human reading them agreed with all 4. The suite had no case for a correct answer in an unexpected shape, because exact match cannot express that.

The Rule We Took From It

Every LLM test we write now scores a property of the output, never its text. The text is only stable under conditions production does not run in. For the severity field the property is “value is one of 3 allowed labels after normalisation”. For the summary the property is “issue matches the ticket’s stated problem”, scored by a judge. The suite went from 40 assertions to 40 criteria.

💡 Practitioner Note: The first suite a client shows us asserts on strings 8 times out of 10. It has never once predicted the production failure.

That rule set the order we test in.

What Do We Test in an LLM Application, and in What Order?

We test 7 layers in an LLM application, in the order production failures taught us to run them. The traditional pyramid flattens for LLM features. A unit test cannot isolate a prompt from the model, so every layer carries weight. Each layer below closes with what it missed on a real engagement.

1. Unit tests on single outputs.
2. Functional tests across a task.
3. Regression on every change.
4. Multi-turn and context.
5. Performance and cost per turn.
6. Responsibility.
7. Security.

Unit Tests on Single Outputs

A unit test in LLM testing is one input, one output, one criterion, scored across repeated runs. The criterion is a property of the output, and the score is compared to a threshold. Section 4 below is this layer in full.

What it missed: a unit test scores what it is told to score. Our ticket summariser passed correctness on every case and failed on label shape, because nobody wrote a shape criterion.

Functional Tests Across a Task

A functional test in LLM testing groups the unit tests for one capability and passes at an aggregate threshold. Summarisation is a task. Forty ticket summaries scored for correctness, with a task threshold of 90 percent, is its functional test.

What it missed: the aggregate hid the slice. A 92 percent pass across all tickets contained a 60 percent pass on billing tickets. We now report by slice, never by task alone.

Regression on Every Change

A regression test in LLM testing reruns the full dataset after any change to prompt, model, retrieval, or tooling. The trigger is any change, because a prompt edit is a code change with no diff the compiler can see. Section 9 covers the practice.

What it missed: the first regression suite reran only the cases tied to the changed prompt. The 3 cases that broke were tied to a different prompt in the same chain.

Multi-Turn and Context

Multi-turn testing verifies that facts stated in earlier turns survive into later replies and topic switches. This layer only exists for conversational features. Our full process for it is in how we test AI chatbots, including the 10-run practice this page borrows.

What it missed: a suite of single-turn cases passed a support assistant that forgot the order number by turn 5. No single-turn case can find a turn-5 failure.

Performance and Cost per Turn

Performance testing for an LLM feature measures latency to first token, latency to completion, and token cost. All 3 are measured per turn and under concurrency. Cost is a test dimension because a prompt edit that adds 400 tokens of context adds it to every call.

What it missed: latency was tested per call and passed. A second product started sharing the same inference endpoint, and p95 latency doubled with no change in our code.

Responsibility

Responsibility testing checks outputs for bias, toxicity, and unfair treatment across demographic variants of the same input. Two versions of the same request differ only in a name or a location. The outputs get compared on decision, tone, and refusal.

What it missed: the probe set used Western names only. The first production complaint came from a user whose name the model treated as a company.

What Is LLM Security Testing?

LLM security testing checks that untrusted content cannot redirect the application, reveal what it holds, or reach restricted systems. Prompt injection holds position 1 in the OWASP Top 10 for LLM Applications 2026, published August 2026. Sensitive information disclosure holds position 2. The 2026 edition renamed system prompt leakage to hidden context exposure. It is the first edition weighted partly on real incident data.

Our security set has 3 groups of inputs: direct overrides, role-play framings, and instructions hidden inside content the application reads. Every group runs against every feature that touches money, personal data, or account access.

What it missed: the first set tested only what the user typed. The injection that reached staging was inside a pasted email the feature was asked to summarise.

All 7 layers share one oracle, and it works on a single output.

How Do You Unit Test an LLM Application?

To unit test an LLM application, score one output against one criterion, repeat the run, and pass at a threshold. The criterion replaces the expected string. The runs replace the single execution. The threshold replaces the assertion. This is the unit every other layer is built from.

What the Unit Is

The unit in LLM testing is one input, one criterion, and the output the application returns for that input. It is not the model, not the prompt, and not the function that calls the model. A single input has one unit per criterion. The ticket summary above has 3.

The Four Ways to Score an Output

An output is scored 1 of 4 ways, and each one has a case where it lied to us.

• Exact match: The output equals or contains a reference string. It works for labels, IDs, and structured fields after normalisation. It lied on the severity field above, before we added normalisation.
• Semantic similarity: An embedding distance between output and reference passes above a cut-off. It works for paraphrase. It lied on a refund policy summary that was 0.91 similar to the reference and stated the wrong refund window.
• LLM-as-a-judge: A second model scores the output against a written rubric. It works for open-ended criteria. It lied when the judge prompt said “helpful” without defining it, and scored a polite wrong answer at 5.
• Human scoring: A reviewer scores against the same rubric. It works as ground truth and calibration. It lied when 2 reviewers scored the same output 2 and 4 with no anchors on the scale.

💡 Practitioner Note: Across our LLM engagements, semantic similarity is the scorer most often trusted and least often correct. A high similarity score means the output sounds like the reference.

Pointwise or Pairwise

Pointwise scoring rates one output on its own. Pairwise scoring compares 2 outputs to the same input and picks the better one. Pointwise is the unit test. Pairwise is how we compare a new prompt or model version against the current one. Same dataset, before the switch.

Pairwise has no threshold. Its output is a win rate. A new prompt that wins 55 percent of pairs is not an improvement we ship.

Calibrating the Judge

Judge calibration is the check that an LLM judge agrees with a human on a labelled sample first. Only then does it score unlabelled outputs. We run the judge on 50 outputs a human has already scored. Agreement below 85 percent means the rubric changes, not the threshold.

Two rules came from calibration failures. The judge returns an explanation with every score, so a disagreement is readable. The judge model is pinned to a version, because a judge that updates silently moves every score on the suite.

The Threshold That Replaces the Assertion

A threshold in LLM testing is the score a criterion has to reach across runs for the unit to pass. It is set per criterion, not per suite. Correctness on a refund amount sits at 100 percent. Tone on a greeting sits at 80.

In our process, the product owner approves the threshold with input from QA. Whoever owns the risk on that slice, security, compliance, or a domain lead, signs it too. It is written down before the first run. A threshold set after seeing the scores is a description of the model, not a test of it.

The Test Case, Shown

A complete LLM unit test has 6 fields. The table shows one from the ticket summariser, anonymised.

Run 7 is the reason for runs. A single execution had a 90 percent chance of passing this case. It had a 100 percent chance of teaching us nothing.

How Many Runs Before We Trust a Score

Our default is 10 runs per unit test before we read its score. It is an operating trade-off we chose, not a statistical rule. Ten runs give an intermittent failure a fair chance to show. They keep a routine suite cheap enough to run on every change.

Money and account-access criteria run 20. Everything else runs 10.

Which Metrics Do We Report?

We report 7 metrics on an LLM test run, and each one is reported per slice, not per suite. A suite-level number hides the slice that failed. A CTO reads the first 3 metrics; a QA lead reads all 7.

• Pass rate at threshold: Out of all runs on a slice, the share that met every criterion’s threshold.
• Faithfulness: For features that read from a source, the share of output claims traceable to that source.
• Hallucination rate: The share of outputs containing at least 1 claim absent from source and context.
• Correctness: Judge or human agreement that the output answers the input, on the anchored rubric.
• Completeness: The share of required elements present when the input asked for more than 1 thing.
• p95 latency and cost per call: Both per turn, both compared against the previous run.
• Judge agreement: How closely the judge tracked the human sample this cycle, so the other 6 numbers can be trusted.

We use accuracy, precision, recall, and F1-score where a component fits them: an intent classifier, a router, a retriever. They are not our primary measures for open-ended generation. For the model on its own, they belong to the validation framework linked in the first section.

How Do We Build the Test Dataset?

We build an LLM test dataset from logged failures first, then production traces, then synthetic cases. Each case carries pass criteria instead of an expected output. The dataset is versioned with the prompt it tests. Every row is a test scenario with its criteria attached.

The shares are our practice across engagements, not a rule.

Start From Failures, Not Features

The first 50 cases in every dataset we build are outputs the feature has already got wrong. One domain reviewer grades each pass or fail with a written reason. Starting from features produces cases the prompt was written to pass. Starting from failures produces cases the prompt has to earn.

The Test Prompt Set

A test prompt set is the fixed mix of input types every slice carries. Ours has 4 types:

• Direct requests.
• Edge inputs: empty, oversized, malformed, code-mixed.
• Adversarial inputs: injection, override, role-play.
• Out-of-scope requests the feature is meant to refuse.

A slice with no adversarial column has not been security tested, whatever the pass rate says. A slice with no out-of-scope column has not tested refusal.

Write Pass Criteria, Not Expected Outputs

A pass criterion states the property a correct output has; an expected output states one correct text. Criteria survive model changes; expected outputs do not. “The reply names the refund window in days” is a criterion. “Refunds are processed within 14 days” is one output that meets it.

Expected outputs still have a place: labels, IDs, and structured fields, where exact match is the criterion.

Generating Test Cases With an LLM

We generate synthetic cases with a model, and a human reads every one before it enters the dataset. The generator receives the slice definition and 5 real cases and returns 30. In our engagements, a human removes 8 to 12 of every 30. The reasons are duplication, impossibility, or the answer leaking into the input.

Generated cases carry a source tag. When a generated case fails, the reviewer reads the case before the output. Generated inputs are wrong more often than the feature is.

Versioning the Dataset With the Prompt

The dataset carries a version tag that changes with the prompt version it tests. A prompt at v7 runs against dataset v7. A regression comparison between prompt v6 and v7 runs both against dataset v7, never each against its own.

DeepEval’s dataset model separates the input and criteria from any single run’s output for the same reason. Langfuse’s golden dataset guide covers the maintenance loop in more depth.

The most expensive failure class gets its own section.

How Do We Test for Hallucination?

We test for hallucination by scoring each output claim against the source it was meant to come from. An unsupported claim is a failure whether or not it is true. A hallucination is not a wrong answer. It is a claim the application had no basis for.

The case that set this rule was a policy assistant. Asked about international shipping, it answered with a correct paragraph on customs handling. The knowledge base contained no customs content. The paragraph passed similarity, passed correctness, and was written from nothing. It was accurate that week and had no way to stay accurate.

Grounded vs Ungrounded Claims

A grounded claim traces to a passage in the source; an ungrounded claim does not. The judge receives the output and the source and returns each claim with a passage reference or “none”. One “none” fails the output.

The Nonexistent-Entity Probe Set

A nonexistent-entity probe asks about a product, policy, person, or document that does not exist. It passes only on a refusal or a clarification. Our set holds 30 probes per feature. A feature that answers 1 of the 30 has a hallucination rate of at least 3 percent. These are the easiest inputs to get right.

Faithfulness on Retrieved Context

Faithfulness on a retrieval-backed feature is the share of output claims supported by the retrieved passages. Not by the whole knowledge base, by the passages returned. A claim supported by a document the retriever never returned is a retrieval failure dressed as a correct answer. We score both, because the fix is different.

How Do We Run LLM Tests in CI and After Release?

We run the LLM suite as a CI gate on every change and as a sampled scorer on live traffic. The gate catches changes we make. The sampler catches changes made to us. Both feed failures back into the dataset.

The CI Gate and What Blocks a Merge

The CI gate runs the full dataset on every pull request touching a prompt, model, retrieval setting, or tool. A merge is blocked in 2 cases. Any criterion on a money, PII, or account-access slice drops below threshold. Any slice drops more than 5 points from the last main-branch run.

Runtime is the objection every team raises. So the gate runs 10 per case on critical slices, 3 on the rest, and the full 10 nightly.

Thresholds by Criticality

Thresholds are set by what the slice touches. Money, PII, and account access gate at 95 percent pass or higher across runs. Refusal on out-of-scope gates at 100. Tone, formatting, and length gate at 80.

Production Monitoring

Production monitoring scores a sample of live traffic with the same judge and rubric the suite uses. It alerts on a moving average. We sample 5 percent of calls, skewed toward the critical slices. The scores land in the same report as the CI run. A drop in production and a drop in CI read the same way.

Drift

Drift is a change in output quality with no change in the application’s code, prompt, or data. The cause is nearly always the provider updating the model behind a stable model name. We caught one when faithfulness on a summarisation slice fell from 96 to 88 across 9 days. There were 0 merges in the period.

The fix was not in our repo. The response was a pinned model version, a rerun of the suite against the new one, and a decision. Without the sampler, the first signal is a customer.

💡 Practitioner Note: The drift incident is what converts a client from “we test before release” to “we test continuously.” Nobody budgets for it before it happens.

Turning a Production Failure Into a Test Case

Every production failure the sampler flags becomes a dataset case within 1 sprint, with the criterion it failed. This is the loop that makes the dataset representative 6 months after launch. The logged-failures share in the table above is fed from here.

A prompt edit is the most common change the gate catches.

How Do We Regression-Test When the Prompt or the Model Changes?

We regression-test a prompt or model change by rerunning the full dataset and comparing every slice against the last run.

The pass condition is a tolerance band, not an exact score. The changed case is never the only case rerun. The judge is pinned so the comparison measures the change, not the judge.

What a Prompt Diff Breaks

A prompt diff is a text change to an instruction. It breaks cases that have no visible connection to the edit. We added one line to a summariser prompt to stop it inventing severity labels. It stopped. It then began dropping the “next action” field on 14 percent of tickets. The added line had pushed the field instruction lower in the prompt.

Rerunning the Full Set, Not the Changed Case

A regression run covers every slice in the dataset, whatever the change touched. The 3 cases we expected to change are the least useful cases in the run. We already know what they will do. The cases we did not expect to change are the test.

Tolerance Bands Instead of Exact Thresholds

A tolerance band is the range of score movement treated as noise rather than regression. Ours is 3 points on a 10-run score. A slice that moves from 94 to 92 is noise. A slice that moves from 94 to 90 is a regression, and the run fails.

A gate with no band fails on noise and gets switched off within a month.

Pinning the Judge Model

A pinned judge is a fixed model version used for every scoring run in a comparison. A judge that updates between runs moves the scores of both versions, and the comparison measures the judge. We pin the judge per dataset version and re-calibrate on the 50-case human sample when either changes.

Which LLM Testing Tools Do We Use, and Which Did We Drop?

We use one code-first framework for the CI gate and one platform for production tracing. Every engagement started in a spreadsheet. The two tool families are not alternatives. The framework runs the suite; the platform watches production and feeds failures back.

Code-First Frameworks

A code-first framework is a library the suite imports. LLM tests then run under the same test runner as everything else. DeepEval is pytest-native and holds the widest metric set. RAGAS scores retrieval and generation separately, which matters for the faithfulness split above. Promptfoo runs from YAML and carries the largest red-teaming set. Promptfoo agreed to be acquired by OpenAI in March 2026 and remains open source at the time of writing.

Eval-Plus-Observability Platforms

An eval-plus-observability platform traces every production call and scores samples of them with the same evaluators the suite uses. Langfuse and Arize Phoenix self-host. LangSmith and Braintrust are hosted. In our engagements the choice came down to data residency, not features.

What We Run Today

We run DeepEval in CI and Langfuse for production, self-hosted, on most engagements. We dropped 2 things. A hosted platform on a client with a data-residency requirement. A similarity-only scoring setup, after the refund-window case above. The first suite on every engagement is a spreadsheet, because the criteria have to exist before the tool does.

The ISTQB Certified Tester AI Testing v2.0 syllabus, released May 2026, covers generative AI testing, non-determinism, and red teaming. The practices above are what those topics look like inside a delivery report.

The questions below come up on nearly every engagement after this point.

What Else Do QA Leads Ask Us About LLM Testing?

QA leads raise 7 further questions about LLM testing once the process above is in place. Each answer comes from delivery, not from tool documentation. The final question covers working with a testing partner.

What Does an LLM Test Case Look Like?

An LLM test case has 6 fields: input, criterion, scorer, threshold, run count, and result. The table in section 4 shows one. The criterion replaces the expected output. The run count replaces the single execution. A test case with an expected string and 1 run is a conventional test case pointed at an LLM.

What Questions Should a QA Lead Ask Before Testing an LLM Feature?

A QA lead asks 5 questions before the first test case is written. What does the feature read from, so faithfulness has a source? What does it write to, so integration has an oracle? Which slices touch money, PII, or account access? Who owns the threshold per criterion? Which model version is pinned, and who is told when it changes?

A feature with no answer to the last question has no regression baseline.

How Do You Test an LLM Agent?

An LLM agent is tested at 2 levels. Each tool call is a unit; the trajectory is a functional test. The unit checks tool selection and arguments against the input. The trajectory check has 3 parts. The sequence reached the outcome. A failed step stopped the run. The agent’s claim of success matches the system of record. An agent that says “email sent” is tested against the mailbox, not the transcript.

How Many Test Cases Does a Suite Need?

Our suites start at 50 failure cases. They grow to 200 to 400 cases per feature within 3 months of production. The count per slice matters more than the total. A slice with fewer than 20 cases produces a pass rate that moves 5 points on 1 new case.

Can You Trust LLM-as-a-Judge?

An LLM judge is trusted after it matches a human on a 50-case labelled sample at 85 percent or higher. It stays trusted while that agreement holds. Below that, the rubric is the problem. Judge scores without explanations are not read in our reports, because a disagreement with no reason cannot be fixed.

Can We Start Without an Evals Platform?

Yes, every engagement we run starts without one. The criteria, the dataset, the rubric, and the run log live in a spreadsheet. That holds until the dataset passes 300 rows and the runs need scheduling. A platform speeds up execution. It does not write a criterion.

When Does a Team Need a Testing Partner?

A team needs a testing partner when the feature has shipped and the failures are arriving from users. Nobody on the team owns a criterion, a threshold, or a rerun. The gap is rarely tooling. It is the absence of a test discipline for outputs that change every run.

Testscenario’s AI testing services build the dataset, write the criteria, and calibrate the judge. The suite then runs in CI and against production as a managed engagement. The first deliverable is the slice-level pass-rate report on the feature you already have.