How to test an LLM app against real conversations

How do you test an LLM app against real conversations?

You test an LLM app by building an eval set from real, anonymized conversations and attaching a pass/fail check to each one. Then you run your bot on every case, score the replies automatically, and rerun the whole set every time you change a prompt, a model or your retrieval setup. The goal is simple: catch regressions before your users do.

An eval set is a fixed list of inputs plus the rules a good answer must follow. An LLM app is any product where a large language model (LLM) writes part of the output, such as a support chatbot, a phone agent or a document assistant. Classic unit tests break down here because the same input can produce many valid answers. Evals solve this by checking properties of the answer instead of the exact text.

Why use real transcripts instead of made-up test inputs?

Real transcripts show how people actually talk to your bot. Users write short, messy, off-topic messages, change their mind halfway through, and ask things you never planned for. Inputs you invent at your desk tend to be clean and polite, so they miss the cases that break production.

To build the set, export a sample of conversations from your logs. Pick a mix: common requests, edge cases, conversations where users complained, and conversations where the bot clearly failed. Each of these becomes one test case with a short note on what a good reply must do.

Anonymize before you store anything

Transcripts often contain names, phone numbers, emails, addresses and account IDs. Replace these with placeholders like <NAME> or <PHONE> before the data goes into your test repo. Keep the structure of the message intact, because the bot's behavior often depends on it. If your users are covered by a privacy law or a contract, check what you are allowed to keep.

What kinds of checks should an LLM eval use?

LLM eval checks fall into three groups. Use the cheapest check that can reliably answer the question, and only move to a more expensive one when you must.

Exact checks should cover most of your set. They catch the failures that matter most in production: missing key facts, leaked internal text, broken JSON for a downstream parser, and forbidden topics.

Rubric checks break a vague goal like "good answer" into small, testable criteria. Each criterion should be a yes/no question. Several small criteria are easier to debug than one big score.

LLM-as-judge means asking a second model to grade the reply against a rubric. It is useful for tone, clarity and helpfulness, which are hard to test with string matching. Judges have known problems, though. They can favor longer answers, favor answers that sound confident, and give different scores on different runs. Ask the judge for a simple verdict (pass or fail) with a short reason, and compare its verdicts to human labels on a sample before you trust it.

What does a minimal eval script look like?

A minimal eval script needs three parts: a list of test cases, a function that calls your bot, and a loop that runs checks and prints results. The example below is provider-neutral. Replace run_bot with your own call to OpenAI, the Claude API, a local Llama model, or your full app pipeline.

import json

TEST_CASES = [
    {
        "id": "refund-basic",
        "input": "hi i want my money back for order <ORDER_ID>",
        "contains": ["refund"],
        "not_contains": ["I am an AI language model"],
    },
    {
        "id": "no-competitor",
        "input": "is your plan better than the other guys?",
        "contains": [],
        "not_contains": ["CompetitorName"],
    },
    {
        "id": "extract-json",
        "input": "Book a table for 2 at 7pm tomorrow. Reply as JSON.",
        "contains": [],
        "not_contains": [],
        "json_keys": ["party_size", "time"],
    },
]


def run_bot(user_input: str) -> str:
    # Placeholder: call your LLM app here and return the reply text.
    raise NotImplementedError


def check(case: dict, reply: str) -> list[str]:
    failures = []
    for text in case.get("contains", []):
        if text.lower() not in reply.lower():
            failures.append(f"missing '{text}'")
    for text in case.get("not_contains", []):
        if text.lower() in reply.lower():
            failures.append(f"found forbidden '{text}'")
    if "json_keys" in case:
        try:
            data = json.loads(reply)
        except json.JSONDecodeError:
            return failures + ["not valid JSON"]
        if not isinstance(data, dict):
            return failures + ["JSON is not an object"]
        for key in case["json_keys"]:
            if key not in data:
                failures.append(f"JSON missing key '{key}'")
    return failures


def main() -> None:
    passed = 0
    print(f"{'CASE':<16} {'RESULT':<6} DETAILS")
    for case in TEST_CASES:
        reply = run_bot(case["input"])
        failures = check(case, reply)
        result = "PASS" if not failures else "FAIL"
        passed += result == "PASS"
        print(f"{case['id']:<16} {result:<6} {'; '.join(failures)}")
    print(f"\n{passed}/{len(TEST_CASES)} passed")


if __name__ == "__main__":
    main()

The script prints one row per case, so you can see at a glance which behavior broke. Errors are not caught on purpose: if the bot call fails, the run should stop and tell you, not quietly count it as a failed answer.

How do you run regression tests on every prompt change?

A regression run means running the full eval set after any change that could affect output. That includes edits to the system prompt, a new model version, a change to temperature, new tools, and changes to the documents in a retrieval-augmented generation (RAG) index. Small prompt edits often fix one case and quietly break two others.

Treat prompts like code. Keep them in version control, give each version an ID, and run the eval set before you merge a change. Fast exact checks can run in continuous integration (CI) on every pull request. Judge-based checks cost more, so many teams run them nightly or before a release. The prompt engineering production checklist covers how to version and structure prompts so these runs stay easy.

Handle randomness on purpose

LLM output can change between runs even with the same input. For cases that flip between pass and fail, run them three to five times and record the pass rate. A case that passes four out of five times is a real signal that the prompt is fragile, and that is worth knowing.

How should you track eval results over time?

Save every eval run as a row of data: timestamp, prompt version, model name, case ID, pass or fail, and the failure reason. A CSV file or a small database table is enough to start. With this history you can answer the question that matters most after a bad release: "which change made this case start failing?"

Watch the overall pass rate, but also watch individual cases. A steady 90% pass rate can hide the fact that a different 10% fails each time. When a real bug reaches production, add that conversation to the eval set so the same bug cannot return unnoticed.

In the no-code AI bot framework I built, each bot could run on a different LLM and a different data source. That setup makes a shared eval set even more useful, because the same real questions can be replayed against every configuration.

Summary

Testing an LLM app starts with real, anonymized conversations turned into an eval set. Score replies with exact checks first, rubric checks next, and an LLM judge only for fuzzy qualities. Rerun the set on every prompt, model or retrieval change and keep a history of results. If you want help building an eval pipeline for your product, see AI applications.