Prompt engineering for production chatbots: a practical checklist

What makes a chatbot prompt production-ready?

A production-ready chatbot prompt is one that behaves the same way on the ten-thousandth conversation as on the first. It gives the model a clear role and scope, grounds answers in retrieved context, fixes the output format, and defines what to do when the answer is unknown. It is also versioned in git and tested against real conversations before every change goes live.

Prompt engineering is the practice of writing and refining the instructions sent to a large language model (LLM) so it produces the output you want. In a demo, a loose prompt is fine. In production, a vague prompt creates slow, unpredictable failures that users find before you do.

In my role as OpenAI & Prompt Engineer at Xobot, most of the work is exactly this: turning a working demo prompt into one that holds up under real traffic. The checklist below is the structure that makes that repeatable.

How should you structure a system prompt?

A system prompt is the hidden instruction block sent before the user's messages. It works best when it is split into clearly labeled sections in a fixed order. Fixed sections make prompts easier to read, review and change without breaking something else.

A reliable order is: role, scope, rules, context, output format and examples. Use headings or XML-style tags to separate the sections, so the model can tell instructions apart from retrieved documents.

SYSTEM_PROMPT_VERSION = "support-bot/v7"

SYSTEM_PROMPT = """
<role>
You are the support assistant for Acme Cloud. You help customers with
billing, account settings and product setup.
</role>

<scope>
Only answer questions about Acme Cloud. For anything else, reply:
"I can only help with Acme Cloud questions."
</scope>

<rules>
- Answer only from the information inside <context>.
- If <context> does not contain the answer, reply:
  "I don't know that yet. I can connect you with our support team."
- Never invent prices, dates, links or policy details.
</rules>

<context>
{context}
</context>

<output_format>
- Plain text, at most 4 short sentences.
- For step-by-step tasks, use a numbered list.
- End with the source title in the form: Source: <title>
</output_format>
"""


def build_system_prompt(chunks: list[dict]) -> str:
    context = "\n\n".join(f"[{c['title']}]\n{c['text']}" for c in chunks)
    return SYSTEM_PROMPT.format(context=context)

The prompt carries a version string so every logged answer can be traced back to the exact prompt that produced it.

How do you define role and scope?

The role tells the model who it is and who it serves. Keep the role short and concrete: the product, the audience and the jobs the bot handles. Long personality descriptions add little and can conflict with the rules.

The scope tells the model what it must not do. List the topics that are out of scope and give an exact sentence to use for them. An exact sentence is easier to test than a general instruction like "stay on topic."

How do you ground answers in retrieved context?

Grounding means the bot answers from documents you supply, not from its general training. In a retrieval-augmented generation (RAG) setup, the app retrieves relevant chunks and places them inside the context section of the prompt. The article What is RAG? explains how that retrieval works.

Tell the model plainly that the context is the only allowed source. Label each chunk with a title or ID so the model can cite it and so you can check which chunk an answer used. Keep instructions and context clearly separated, because retrieved text can contain sentences that look like instructions.

How do you handle "I don't know" and refusals?

The "I don't know" rule is the most important line in a production prompt. Without it, the model fills gaps with fluent guesses, and users cannot tell a guess from a fact. Give the model an exact fallback sentence and, where possible, a next step such as contacting support.

Refusals need the same precision. Define what the bot refuses, such as legal advice or account changes it cannot verify, and give it a polite fixed reply. Then check that the bot does not over-refuse normal questions, which is a common side effect of strict rules.

How do you control output format with examples?

Output format covers length, structure, tone and any machine-readable parts. State it as a short list of rules, not a paragraph. If another program reads the output, ask for strict JSON and validate it in code before using it.

Examples, also called few-shot examples, show the model what a good answer looks like. Two to five short examples usually work better than a long list of rules. Pick examples that cover the tricky cases: a missing answer, an out-of-scope question and a multi-step task.

How do you version and test prompts?

Store each prompt as a file or constant in git, next to the code that uses it. Every change goes through a normal pull request, so it gets a diff, a review and a way to roll back. Log the prompt version with each answer so you can tie a bad reply to the change that caused it.

git log --oneline -- prompts/support_bot.py
git diff HEAD~1 -- prompts/support_bot.py

Test every prompt change against real conversations. Save a set of anonymized user questions with the expected behavior, including edge cases and past failures, and run the new prompt against all of them before release. The article on testing LLM apps shows how to build this test set and score the results.

What is the full production prompt checklist?

Use this checklist before shipping a new chatbot or a prompt change.

Summary

A production chatbot prompt is a small program: structured sections, a clear scope, grounding in retrieved context, a fixed output format and an exact "I don't know" rule. Keep it in git, log its version with every answer, and test each change against real conversations. For a chatbot built and tuned this way on your data, see the AI applications service.