AI Testing: How to Test LLM and Agentic Applications

Map the system before writing prompts

Draw the path from user input through moderation, prompt construction, retrieval, model generation, tool selection, application rules, and persistence. Mark every source of nondeterminism and every boundary that can expose data or change state. This prevents the common mistake of treating a system defect as a mysterious model failure.

For a job-search assistant, separately test query interpretation, location normalization, inventory retrieval, ranking, explanation generation, and saved-job actions. A relevant-looking result may still violate a location filter or use a stale job record.

Build a risk-based evaluation set

Create cases from real tasks, domain boundaries, historic defects, and abuse scenarios. Store inputs, permitted context, expected invariants, rubric, severity, and tags. Include straightforward cases so usefulness does not collapse while the team focuses on attacks.

• Golden cases for common user goals and known-good source material.
• Edge cases with missing, conflicting, long, or malformed context.
• Adversarial cases for prompt injection, data exfiltration, and policy bypass.
• Regression cases copied from every meaningful production escape.

Choose the right oracle for each property

Use code for schema, required fields, citations, permissions, amounts, and forbidden actions. Use rubrics for relevance, completeness, and tone. Use qualified human reviewers where context or consequences are nuanced. Model-based graders can scale comparisons, but calibrate them against human judgments and do not let them grade the same blind spots they generate.

Run repeated trials for high-risk cases and report distributions or pass rates by slice. Preserve severe failures even when an aggregate score improves. A single unauthorized disclosure matters more than a small gain in average helpfulness.

Test RAG and agents at intermediate steps

For RAG, measure whether relevant chunks were retrieved, whether access controls were applied before retrieval, and whether claims are supported by cited evidence. For agents, assert allowed tools, argument constraints, confirmation points, stopping rules, and final state.

Inject tool timeouts, partial writes, malformed responses, rate limits, and untrusted instructions inside tool output. Verify that retries are safe, the user receives an honest status, and the audit trail explains what happened. A graceful apology is not recovery if the system already created a duplicate record.

Monitor production as part of the test strategy

Offline evaluations cannot represent every real input. Monitor refusal rates, tool failures, retrieval misses, user corrections, latency, cost, and safety events with privacy-aware sampling. Route high-severity signals to human review and promote confirmed failures into the evaluation set.

What should a QA engineer do next?

Select one narrow workflow, write a system map, and create 30 tagged cases. Automate its hard invariants, define a three-level rubric for the subjective qualities, and run the set across at least two prompt or model versions. Review disagreements instead of hiding them in an average.