Most prompt engineering guides are written for demos. Production prompts face hallucinations, adversarial inputs, context length limits, and latency budgets. Here's what actually holds up.
Rahul Sharma
Head of AI Engineering
The single highest-leverage change in most production LLM systems is switching from free-text outputs to structured JSON outputs. Parse a model's free-text response and you're at the mercy of its phrasing choices. Define the output schema up front and the model produces something you can reliably parse, validate, and act on.
For Claude, use tool use with a schema to force structured output. For OpenAI, use Structured Outputs with response_format: { type: "json_schema" }. Both approaches cause the model to validate its own output against your schema before returning — dramatically reducing parse failures. This is a core pattern in our LLM development engagements.
Chain-of-thought (asking the model to "think step by step") improves accuracy on reasoning tasks. But where you place it matters for production use. Asking the model to reason in the final output increases token cost and latency. Instead, route reasoning to a thinking field in your structured output schema — the model reasons internally and returns only the conclusion in your main output fields.
On a classification task with 8 categories, we reduced average output tokens from 340 to 62 by moving chain-of-thought to a hidden scratchpad field. Accuracy improved 4 percentage points because the model had space to reason without trying to format its reasoning for human consumption simultaneously.
Hallucinations in production are mostly a prompting problem, not a model problem. Three techniques that reliably reduce them: ground the model in source documents before asking questions (RAG), explicitly tell the model to say "I don't know" rather than guess, and ask the model to cite its sources within the structured output schema.
For high-stakes outputs (medical, legal, financial), add a verification step: after the model produces its answer, run a second call with the prompt "Does this answer contradict the source documents?" and surface the answer only if the verification passes.
Context windows are larger than ever but not infinite — and every token in the context costs money and adds latency. Structure your prompts so the most important information comes first (models attend more to early context) and late. Instructions in the middle of a long prompt are reliably underweighted.
For multi-turn conversations, implement a sliding window with summarization: keep the last N turns verbatim and replace earlier turns with a compressed summary. This keeps context costs bounded while preserving the continuity the model needs.
Production systems rarely use one model for everything. We route by task complexity: a cheap, fast model (GPT-4o-mini or Claude Haiku) handles classification, extraction, and routing decisions, while a more capable model (GPT-4o or Claude Sonnet) only gets invoked for tasks that genuinely need deeper reasoning. On a customer support triage system we built, this routing cut average per-request cost by 71% with no measurable drop in resolution accuracy, because 80% of incoming tickets were simple enough for the cheaper model to classify correctly.
Latency budgets should be set explicitly per feature, not left implicit. A user-facing autocomplete feature has a very different acceptable latency (under 300ms) than a background report-generation job (30+ seconds is fine). We design the model and prompt strategy around the latency budget first, not the other way around — it's easy to build a prompt that produces great output and terrible latency, then discover the mismatch in production.
If your system prompt is ever concatenated with untrusted user content, you're vulnerable to prompt injection — the user's input hijacking your prompt's instructions. The primary defense: clearly delimit trusted (system prompt) and untrusted (user content) sections using XML tags, and instruct the model to treat the user section as data only. Secondary defense: validate outputs against a schema that would reject any response that includes system-level metadata or unusual formatting.
The most common production failure we debug isn't the model getting the wrong answer — it's the model returning a technically valid but practically useless response, like an empty array when no results should trigger a specific fallback message instead. Fix: always define explicit behavior for edge cases (zero results, ambiguous input, out-of-scope requests) in the prompt itself, not just the happy path.
A second recurring failure: prompts that work in testing but degrade after a few weeks in production as user input patterns drift. This usually isn't model drift — it's exposure to input types the original eval dataset didn't cover. We treat this as a signal to expand the eval set, not just patch the prompt reactively.
A third failure specific to tool-calling agents: models occasionally call a tool with subtly malformed arguments that pass schema validation but fail at the business-logic layer, like a valid-looking but nonexistent product ID. We now validate tool call arguments against live data before executing, not just against the JSON schema.
Production prompts need the same testing discipline as production code. Build an eval dataset of 50–200 representative inputs with expected outputs. Run this dataset against every prompt change before deploying. For tasks where correctness is subjective (writing, summarization), use an LLM judge — a second model call that scores the output against a rubric. This is far more scalable than human review for iteration speed. Teams without this workflow in place often hire LLM developers to build the eval harness before scaling prompt changes further.
Treat prompts as versioned artifacts, not throwaway strings embedded in application code. We store prompts with version numbers, run every version change through the eval dataset before deployment, and keep the previous version available for instant rollback if a new prompt regresses on a metric we track. This discipline sounds heavyweight for "just a string," but production incidents caused by an unreviewed prompt tweak are common enough that the overhead is worth it once you're past the prototype stage.
Tags
Continue Reading
A practical guide to building AI agents that actually work outside of demos — covering tool selection, memory, error recovery, and deployment patterns we've validated in production.
Two approaches, very different trade-offs. RAG is faster, cheaper, and keeps your data current. Fine-tuning gives you style and behaviour control. Here's when each wins — and when to use both.
Moving computer vision models from Jupyter notebooks to production serving infrastructure requires solving latency, throughput, model versioning, and data drift challenges. Here's our battle-tested playbook.
From AI agents to mobile apps — iMOBDEV builds what your business needs.
