On this page
When we went through 12 million logs, the number that stuck with us was 78%. That's the share of agent failures that are completely silent, wrong answers, hallucinations, users who give up and leave without ever throwing an error. None of that shows up in a latency dashboard. So when teams run prompt A/B tests that only measure rubric scores and response time, they're optimizing around the failures that are actually hurting them.
What You'll Have After Following This Guide
By the end, you'll have a working prompt A/B test running against real production traffic, with variants attributed to trace-level outcomes and a decision framework that goes well beyond latency and thumbs-up scores. This isn't about offline evals on a benchmark dataset. Those give you directional signal, but they can't replicate the user behavior distribution that produces silent failures in production. This is for engineering teams running AI agents who need to know whether a prompt change actually reduced a specific failure mode, not just whether it scored better on a rubric.
Get These Prerequisites in Place Before You Write a Single Variant
Confirm all of this before touching any prompt:
- • Your agent is already running in production with structured logging
- • You can inject a prompt version ID into request metadata at generation time (not reconstructed after the fact)
- • Your logging or tracing layer can accept and store per-request variant labels alongside trace IDs
- • You have defined at least one success or failure metric beyond latency, even a rough one
- • Your team knows what a stopping rule is before the experiment starts (we cover this in Step 6, if you don't have one pre-defined, you'll end the experiment at the wrong time)
Estimated time: 2 to 4 hours to wire up the experiment; 3 to 7 days minimum of production traffic to collect meaningful signal. Teams routinely underestimate that second number and declare winners on two days of data.
Knowledge level: Comfortable with LLM API calls and familiar with trace IDs and log aggregation. No statistics background required beyond being able to run a two-proportion z-test.
Tool compatibility: This guide is tool-agnostic. Langfuse, LangSmith, Braintrust, Portkey, and PromptLayer all support some version of this workflow. The principles apply everywhere; implementation details will vary. Gartner predicts 40% of enterprise applications will embed task-specific AI agents by end of 2026, and most teams aren't adding monitoring capacity at the same rate they're adding models. Getting this infrastructure in place now matters.
One thing worth flagging on evaluation approach: passing all logs through an LLM-as-judge system to classify outcomes is less reliable than most teams expect. With agents that can run hundreds of tool calls over minutes or hours, LLM judge accuracy degrades significantly. This is part of why we built our own approach at Guardy, but regardless of what you use, be honest with yourself about your evaluator's accuracy before you trust its output to decide a winning variant.
Step 1: Defining "Better" Comes Before Writing Variants
This is the step most teams skip entirely. They write variant B before they've defined what outcome would make variant B the winner. Don't do that.
Write your experiment hypothesis first. The format is: "If we change X in the system prompt, the rate of [named failure mode] will decrease by Y% as measured by [specific log signal]." Write that sentence down before any code changes. If you can't complete that sentence, you're not ready to run the experiment.
Distinguish proxy metrics from outcome metrics. Rubric scores, coherence ratings, and thumbs feedback are proxy metrics. Task completion rate, session abandonment rate, re-open rate, and downstream business indicators are outcome metrics. The problem with optimizing only for proxy metrics is that a prompt can score well on coherence rubrics while still hallucinating tool parameters or misrouting intent. In our analysis, hallucinations were the top failure category, user frustration was second, and agent forgetfulness or laziness was third. None of those show up reliably in a coherence score.
we worked with a Series B finance startup running agents to automate accounts receivable. Their agent was generating quotes from vendor PDFs. It looked fine on the surface, different quotes for different PDFs, approximately correct prices. But the agent wasn't actually ingesting the PDF. It was hallucinating the quote price using other context in the conversation. Rubric scores would have looked clean. Outcome metrics would have caught it.
Name the specific silent failure you're targeting. Before creating variants, write down the failure mode you're trying to reduce. Examples: "the rate of sessions where the agent confidently gives a wrong answer and the user doesn't push back," or "the rate of sessions where the agent forgets a constraint stated in the first turn." That named failure becomes your primary measurement target.
Expected output from this step: A one-line experiment hypothesis written down before any code changes, with a named failure mode, a specific log signal, and a numeric target.
Step 2: One Variable Per Variant, and Label Everything Before Traffic Starts
Change one variable at a time. This is obvious, and it's the most commonly violated rule in practice. Teams change tone, instruction structure, and few-shot examples all at once and then can't attribute the result to any specific change. If you want to test multiple variables, run sequential experiments or use a proper factorial design, don't bundle them.
Label every variant before traffic starts. Every request must carry a machine-readable variant ID, something like prompt_version: "prod-a" or prompt_version: "prod-b", written into the trace or log at generation time. Not reconstructed later. Not inferred from timestamps. Written at generation time. This is table stakes and most teams do it inconsistently.
If you're using a prompt management tool like Langfuse, LangSmith, or PromptLayer, set the version ID in that tool's prompt registry and confirm it flows through to your logging layer. If you're not using one, the minimal pattern is to inject the variant ID into the system message metadata or a request header, then assert it's captured in your logging layer with a test call before any traffic is routed.
Store your variants in version control or a prompt registry. Both variants, control and challenger, should be stored somewhere that isn't just a string in your codebase. You'll need the exact prompt text later for replay and debugging.
Expected output: Two named prompt versions in version control or a prompt registry, with a test call to each that returns a log entry containing the correct variant label.
Step 3: Hash on Session ID, Not Request ID
Hash on session ID, not request ID. This is the agent-specific wrinkle that generic A/B testing guides miss. If you assign variants randomly per request, a user will get variant A on turn 1 and variant B on turn 3. That poisons multi-turn attribution entirely. Hash your assignment on a stable user or session identifier so every turn in a session gets the same variant. Models handle individual tool calls reliably at 85-95% parameter accuracy but often fail once a conversation involves multiple turns, clarifications, or interruptions, which means multi-turn attribution is exactly where you need clean data.
Start with a 90/10 canary split. Don't go 50/50 immediately. The goal of the first few hundred sessions is to catch catastrophic regressions, a prompt that causes the agent to refuse all requests, or hallucinate tool calls on a class of inputs, before they hit the majority of traffic. Move to 50/50 only after the canary period shows no serious regressions.
Wire up your fallback logic carefully. If your LLM provider rate-limits or returns an error, your fallback must always route to the control variant. And that fallback invocation must be explicitly excluded from outcome attribution, tagged as assignment: "fallback-excluded" and filtered out before analysis. If you count fallback invocations as variant B outcomes, you'll see phantom failures on the challenger that have nothing to do with the prompt.
Expected output: A routing function that logs its assignment decision alongside the variant label, with a dry-run test showing 10 simulated requests distributed at roughly 90/10 and fallback requests flagged as excluded.
Step 4: Every Span Needs a Variant Label and a Trace ID
Every log entry for a generation must carry at minimum: variant label, trace ID (linking all turns in a session), span ID (for multi-step agents), timestamp, token count, latency, and the outcome signals you defined in Step 1.
A minimal log entry looks like this:
{
"trace_id": "abc-123",
"session_id": "user-456",
"span_id": "span-789",
"prompt_version": "prod-b",
"timestamp": "2026-03-15T14:32:01Z",
"latency_ms": 1240,
"token_count": 847,
"quality_score": 0.84,
"frustration_signal": false,
"task_completed": true
}
Without trace-level linkage, you'll know "variant B had more session abandonments" but not whether those came from the same session type, the same user segment, or a confounded input distribution. You need to be able to filter by prompt_version: "prod-b" and see every associated span and outcome signal for a single session end-to-end.
For multi-step agents, log every intermediate step. Each tool call, retrieval step, and generation within a session should carry the same trace ID and variant label. Teams that only log the final response lose the ability to diagnose which intermediate step caused a failure. Hallucination rates across modern LLMs range from 15% to 52%, with real-time conversational agents showing higher rates during multi-turn interactions, and you can't localize a hallucination to a specific step without span-level logging.
This also matters for detecting the behavioral signals that single-turn metrics miss. Session-level scorers detect incomplete answers, lost context, and rising user frustration that single-turn metrics cannot surface. Without the trace structure, you can't compute those session-level signals.
Expected output: A sample trace in your logging system where you can filter by variant_id: "prod-b" and see all associated spans, quality scores, and outcome signals for a single session end to end.
Step 5: Tier 1 and Tier 2 Metrics Won't Tell You What You Need to Know
Track three tiers of metrics:
Tier 1, Operational metrics: Latency, cost, token usage. These are table stakes and easy to get. Don't skip them, but don't stop here.
Tier 2, Quality and eval scores: Rubric scores, groundedness scores, instruction-following scores from your evaluator of choice. Useful for directional signal. Not sufficient on their own, a prompt variant that improves rubric scores while increasing the hallucination rate on edge cases is a net regression.
Tier 3, Outcome metrics and frustration signals: This is where the experiment actually lives. Measure task completion rate, session abandonment (user ends session without a confirmation action), repeated rephrasing (user sends the same request 3 or more times), and escalation rate. These signals detect the failures that coherence scores miss. Many AI conversations end prematurely with users abandoning interactions after just 2-3 dialogue turns, and if your metrics don't capture that, you won't see a difference between variants that produce it at different rates.
On evaluator consistency: Run each generation through your evaluator consistently across both variants. Don't mix human spot-checks with LLM-as-judge mid-experiment. And be careful about sampling, 57% of A/B tests called winners would not have reached statistical significance if run to their proper sample size, and stopping tests early can inflate false positive rates from 5% to 30%. Sampling evaluations compound this problem by adding variance on top of an already underpowered test. Full-log coverage eliminates this, but it requires classifiers that can operate at scale without manual review on every session.
At Guardy, we run per-customer post-trained classifiers that cover every log, not a sample. The built-in classifiers cover hallucinations, bad tool calls, agent forgetfulness, and jailbreak attempts. For domain-specific failure modes, like a finance company needing to detect mismatched GL codes across varied intermediate agent behavior, teams can instantiate a custom classifier by checking 3 to 4 example logs, and it's deployed in under a minute. The broader point holds regardless of what you use: calculate statistical power before the test launches, and make sure your evaluator is consistent across both variants.
Expected output: A per-variant summary table with columns for avg latency, avg cost, avg quality score, your Step 1 failure-mode rate, and session outcome rate, updated daily as traffic accumulates.
Step 6: Set Your Stopping Rule Before You See Any Results
"Looks better after two days" is not a stopping rule.
With non-deterministic models and variable traffic, early results are noise. The correct approach is to set your minimum sample size before the experiment starts, a practical lower bound is 500 sessions per variant for low-frequency failure modes, and higher for rarer signals. For your primary failure-mode metric, run a two-proportion z-test or chi-square test. If the difference isn't significant at p < 0.05 with your current sample size, extend the experiment.
Write your rollback trigger before traffic starts. This is separate from your winner-selection criterion. Define the condition that will cause you to immediately revert to the control regardless of statistical significance, for example: "if variant B shows a greater than 20% increase in failure-mode rate after 200 sessions, we roll back." The rollback trigger is a safety guardrail. It prevents a bad challenger from causing sustained harm while you wait for statistical significance.
The reason this matters is simple: most prompt testing is still done informally. Teams run evals that look clean, or do manual A/B tests, but neither approach tells you what actually improves agent performance in production. Deciding a winner on gut feel just re-introduces the same problem you were trying to solve.
Expected output: A written experiment brief with start date, minimum session count, primary metric, significance threshold, rollback trigger, and a scheduled review date, shared with the team before traffic starts.
Step 7: Ramp the Winner Slowly and Keep the Control Prompt
Don't flip to 100% immediately. Use a ramp schedule: 10% to 25% to 50% to 100%, with 48-hour observation windows between each step. Silent regressions often only surface at higher traffic volumes or with user segments that weren't well-represented in the initial canary. Tool misuse is the most common agent-specific failure mode in production, a single malformed argument at step 2 can silently corrupt every subsequent step in a multi-step workflow. Ramping gives you repeated chances to catch that before it becomes the default behavior for all users.
Do not delete the control prompt. Archive it in version control and your prompt registry. You need it as a baseline for the next experiment, and if a regression surfaces a week after full rollout, which happens, you need the exact prompt text to replay affected sessions against both versions.
The finance startup from Step 1 is worth coming back to here. Their agent "succeeded end-to-end from a surface perspective," and because they didn't have the tooling to check every step across every log, the broken PDF ingestion went undetected for weeks. The prompt looked fine. The outputs looked approximately right. The failure was invisible until it wasn't. Preserving your control prompt and ramping rollout is how you avoid being in that position.
After full rollout, archive the experiment results: variant labels, traffic volumes, metric deltas, significance values, and the rollback trigger threshold. This becomes the evidence base for future prompt decisions and prevents the institutional amnesia of "we already tried that."
Expected output: Production traffic 100% on the winning variant, experiment results documented, control prompt archived, and monitoring dashboards updated to baseline on the new prompt's failure-mode rates.
Most Prompt A/B Tests Break in One of These Five Ways
Mistake 1: Changing multiple prompt variables at once. You can't attribute a result to a specific change if you modified tone, instruction structure, and few-shot examples in the same variant. The fix is straightforward: one variable per experiment. Run sequential experiments or a factorial design if you need to test combinations.
Mistake 2: Per-request random assignment in multi-turn agents. A user gets variant A on turn 1 and variant B on turn 3. Attribution is poisoned. Hash on session ID at session start and hold the assignment for the session's lifetime. This is the most common structural error in agent-specific A/B tests because most teams adapt instructions written for single-turn web A/B testing, where per-request assignment is fine.
Mistake 3: Declaring a winner on rubric scores alone. Rubric scores don't catch hallucinations that are internally coherent, or user frustration that manifests as session abandonment two turns later. Our analysis shows hallucinations are the top failure category, user frustration is second, and agent forgetfulness is third, none of these show up cleanly in coherence scores. You have to instrument outcome metrics and frustration signals in Step 5 before the experiment runs. You can't add them retroactively without re-collecting data. Teams running at scale find that sampling evaluations, manually reviewing 5% of sessions, introduces enough variance to flip the apparent winner. Evaluating every log eliminates this. Guardy's per-customer post-trained classifiers do this without requiring manual review, but the principle applies to any approach: your evaluator needs to be consistent and thorough, not sampled.
Mistake 4: Counting fallback invocations as variant B outcomes. If your provider rate-limits and falls back to the control prompt, those sessions look like variant B failures in your data. Tag fallback invocations explicitly and exclude them from outcome attribution before analysis. This is an operational gap that almost no guide covers, and it can produce phantom regressions on an otherwise clean challenger.
Mistake 5: No pre-defined stopping rule or rollback trigger. Teams end experiments when they feel ready, which in practice means when the results look favorable. That introduces confirmation bias. Write the stopping rule into the experiment brief before traffic starts, and hold to it. Provider model updates are also invisible, when OpenAI or Anthropic updates their underlying models, a prompt tuned for a specific version may behave differently after the update. A pre-defined stopping rule also protects you from confounds introduced by model drift mid-experiment.
Where to Go After Your First Clean Prompt A/B Test
Set up continuous monitoring on your winning prompt's failure-mode rates. The baseline you established in this experiment is now your regression threshold. If a model provider updates their underlying model, your prompt's behavior can shift without any change on your end. You need a monitoring layer that will alert you when the failure-mode rate deviates from that baseline, not just when latency spikes. Our article on LLM observability and silent failures covers what that monitoring stack should look like.
Build a regression testing use for future prompt changes. Before a new prompt candidate hits production, run it against a fixed set of known-difficult inputs, the edge cases and failure modes you've already identified. Guardy's replay-and-fork capability lets you re-run any historical session against a new prompt candidate to check for introduced failures before traffic exposure, that's one option among several. The broader approach is covered in our guide to AI agent regression testing.
Expand your experiment vocabulary. Once you've run one clean A/B test, explore multi-armed bandit routing for faster convergence on low-traffic agents, or factorial designs for testing system prompt and few-shot combinations systematically.
Tools worth evaluating for this workflow:
- • Langfuse and LangSmith for trace management and prompt versioning
- • Braintrust for eval orchestration
- • Portkey for routing and fallback management
- • Guardy for production-side silent failure classification across all logs, prompt A/B testing with statistical rigor, and real-time alerting, particularly useful if sampling-based evaluation is leaving gaps in your outcome metrics
True agent monitoring is one of those things where you have to catch every issue that surfaces, not just the ones a handful of classifiers were pre-configured for. The teams that get this right treat it the way they treat application monitoring for traditional software: thorough, continuous, and not optional once users are in the loop.
Frequently Asked Questions
What metrics should I use to compare prompt variants?
Track three tiers: operational metrics (latency, cost, token usage), quality scores (rubric or groundedness scores from your evaluator), and outcome metrics (task completion rate, session abandonment, repeated rephrasing, escalation rate). Most teams stop at tier 1 or tier 2, but those won't distinguish variants with different silent failure rates. A prompt that improves rubric scores while increasing hallucination frequency on edge cases is a net regression. Tag frustration signals, sessions where the user rephrases the same request three or more times, or ends without completing a goal, and count those per variant. That's where the real difference between variants tends to live.
Should I test prompt variants on real users or before deployment?
Both have a role, but they answer different questions. Pre-deployment evals on a fixed benchmark tell you whether the variant handles your known failure modes. Production testing tells you whether it handles the actual distribution of inputs and user behaviors your agent encounters in the wild. Silent failures, hallucinations that pass coherence checks, user frustration that builds over a session, don't replicate reliably on benchmark datasets because the benchmark doesn't capture the long tail of real user behavior. Pre-deployment evals are a prerequisite for shipping; production A/B testing is how you know whether the change actually helped.
How do I do A/B testing for prompts in production?
The short version: inject a stable variant ID into every request at generation time (not reconstructed later), hash your assignment on session ID so multi-turn attribution stays clean, collect trace-level logs with the variant label attached to every span, measure your primary failure-mode metric alongside operational and quality metrics, and define your stopping rule before traffic starts. The steps in this guide cover each of those in detail. The non-obvious part is that the infrastructure work, consistent variant labeling, trace-level linkage, outcome signal instrumentation, has to be in place before the experiment runs. You can't add outcome metrics retroactively.
What's the right way to decide the winning prompt variant?
Not by feel, and not by stopping when results look favorable. Before the experiment starts, write down your minimum session count (at least 500 per variant for low-frequency failure modes), your primary metric, your significance threshold (p < 0.05 is standard), and a rollback trigger for catastrophic regressions. When you reach the minimum session count, run a two-proportion z-test on your primary failure-mode metric. If the result isn't significant, extend the experiment. The variant that shows a statistically significant reduction in your named failure mode, without a meaningful increase in operational costs or secondary failure modes, is the winner. Declare it on evidence, not on the dashboard looking slightly better after two days.
Share