Shadow mode for AI email agents: earn trust before a reply goes out
Evaluate your AI email agent on live support threads without sending its replies. Compare proposals, record failures and set criteria for human review.
Mailtender team
A support agent can write a convincing answer to a question it has misunderstood. You need to see those mistakes on your own support traffic before a customer receives one.
Shadow mode lets your agent propose replies while your team keeps answering customers. The proposals stay unsent. A reviewer checks them against the conversation, the account facts and the reply a person actually sent.
The useful output is a record of where the agent succeeds, where it guesses and which requests it should hand over. That record gives the support lead a basis for deciding what the agent may do next.
Keep the no-send boundary outside the agent
A prompt that says “do not send” still leaves the decision with the agent if it holds another working sending credential. Enforce shadow mode at the service that accepts its reply proposals. During the evaluation, remove alternative routes through SMTP, another email API or a tool that can send directly.
In Mailtender, an agent in shadow mode submits a proposed reply to an existing thread. The send gate records it as shadowed rather than queuing it for delivery. Your own application runs the model; Mailtender never calls one.
This boundary covers replies submitted through Mailtender. If your agent can also issue refunds or change account settings through other tools, control those actions separately before running it on live customer requests.
Choose one support question to evaluate first
Start with a category whose correct answer your team can verify. “Where can I download an invoice?” is a better first scope than “billing.” Billing also includes disputed charges, tax exceptions and refund promises.
Write down the scope before you collect results. A useful evaluation brief names the request category, the account information the agent may read, the current policy source and the circumstances that require a person.
Include awkward examples deliberately. For invoice questions, test a closed account, a user without billing access and a customer asking for another company's invoice. Keep a separate set of fictional challenge cases for rare situations that have not appeared in live traffic.
Those challenge cases help expose failures. Count them separately from live conversations so that a hand-picked test set does not become a claim about production performance.
Compare replies using the information available at the time
Suppose a customer reports being unable to sign in on a second computer. The agent proposes that the subscription permits one device. A person checks the current terms, finds no such restriction and escalates a possible session bug.
The failure is the invented restriction. A polite tone and clear troubleshooting steps do not make the proposal acceptable.
Use the human reply as a comparison, then verify both answers against the underlying facts. A person can make a mistake too. Also check whether the reviewer or human responder had information the agent could not see when it wrote the proposal.
| Check | What the reviewer looks for | Example failure note |
|---|---|---|
| Account facts | The reply uses the correct account, plan and permissions. | Assumed the requester was the billing owner. |
| Policy support | Each restriction or promise matches a current source. | Invented a one-device limit to explain a login error. |
| Escalation | The agent hands over when the evidence is missing or the request exceeds its scope. | Offered a refund instead of escalating a disputed charge. |
| Next action | The customer can follow the instructions in their actual account state. | Linked to a billing page the requester cannot access. |
Record a verdict with the specific reason. “Failed: promised a refund outside the published terms” gives the developer a reproducible problem. “Needs improvement” leaves the next reviewer guessing.
Make sure the human reply reaches the comparison
Mailtender pairs a shadowed proposal with the first subsequent human outbound reply in the same thread. A proposal waits for comparison until that reply is available. A supervisor can then record a pass or fail and a note.
If your team receives mail by forwarding and answers through another provider, those human replies also need to reach Mailtender. Configure an automatic Bcc to the mailbox's forwarding address. Otherwise the proposals can accumulate without the human answers needed for review.
Inspect the pair before scoring it. In a busy thread, the first subsequent reply may address a new customer message. Record that mismatch and evaluate the proposal against the question it actually answered.
Keep failures visible when you count passes
A single pass rate can hide the category you most need to inspect. An agent might answer invoice-location questions correctly and repeatedly invent refund eligibility. Separate those results before deciding where it may send.
Keep a small evaluation sheet alongside your review records. Include the proposal identifier, request category, agent version, policy version, verdict and failure reason. Record whether the example came from live traffic or a challenge case.
Count reviewed proposals and unreviewed proposals separately. Missing comparisons and skipped reviews are gaps in the evaluation. They are not passes.
Mailtender's trust metrics show the pass count and reviewed count from the latest 50 shadow reviews. That window is useful for recent behaviour. Keep the version history and category breakdown in your evaluation records when you need to compare releases.
Agree on the conditions for moving to review
Define acceptance criteria before looking at the results. For the invoice example, you could require correct access instructions across the account states in scope, escalation when ownership is unclear and no unsupported account disclosures.
Choose the sample size and review period for your own traffic and risk. Passing a sample does not prove the agent will handle every future request. A rare but serious failure may justify keeping a category under human control even when the overall pass rate looks good.
Old shadow proposals remain evaluation records. Changing the agent's mode does not turn them into held replies or send them to customers.
When you move to review, inspect the policy rules as well as the mode. In Mailtender, a matching allow rule can queue a reply in review mode. For a stage where every proposal needs approval, remove applicable allow rules or configure a matching force_review rule with the correct priority.
Assign a reviewer and check that person can find the current policy and account evidence. Approval adds value only when someone can check the proposed answer.
Return changed behaviour to evaluation
A new model, a changed prompt or a revised refund policy can invalidate earlier results. Record the change and run the affected categories through shadow again. Continue reviewed sending for categories whose evidence still holds.
Keep the failure cases that prompted each change. Repeat them after the fix, then look for the same failure in fresh traffic. A corrected example shows that you fixed one case; fresh comparisons help establish whether the change works beyond it.
See how Mailtender records shadow proposals and controls agent sending. For a concrete policy failure to include in your evaluation, read what Cursor's support bot teaches teams putting an agent on email.
