Your AI Feature Should Never Reach Production Without This Failure-Mode Register

An AI failure-mode register is a written table, completed before launch, that lists every way your AI feature can produce a wrong or harmful result, what the user sees when it happens, how the system detects it, what it automatically falls back to, and which named person owns the response. A row with blanks is a feature that isn't ready.

Nobody ships a payment service without deciding what happens when the gateway times out. The same team will then ship an AI summariser with no written answer to "what if it's wrong?" The register closes that gap in an afternoon.

Why does an AI feature fail differently from ordinary software?

Ordinary software fails loudly. It throws an exception, returns a 500, or stops responding, and monitoring catches it within seconds. AI features usually fail quietly. The system returns a fluent, well-formatted, confidently worded answer that happens to be wrong. Nothing crashes, no alarm fires, and the user gets no signal that the output is unreliable.

A stack trace is a gift: file, line number, timestamp. Wrong-but-plausible output arrives with none of that. Your error rate stays at zero while the feature tells a customer their policy covers something it doesn't, and a user pastes that into a client email. It's also why a demo that works is a warning sign rather than a green light: demos get run by people who already know the right answer.

Which failure modes belong in the register?

Ten failure modes cover most production AI features: hallucinated fact, stale knowledge, wrong retrieval, prompt injection, PII leakage into the output or the logs, biased or unfair output, silent quality drift, cost blowout from a runaway loop, upstream provider outage, and latency spike. Register all ten first, then add the ones specific to your domain.

Six produce a wrong answer. Four produce a wrong service, and a 40-second response is a failure too, just one users report as "it's broken". Two get confused constantly: a hallucinated fact is invented from nothing, while wrong retrieval is the model faithfully summarising the wrong document. Same symptom, different fix, separate rows. Injection belongs here even if your feature only reads internal documents, because any text the model sees is an instruction channel. OWASP's GenAI LLM Top 10, published 4 August 2026, still ranks prompt injection first, at LLM01.

What does the register actually look like?

The register is a five-column table: failure mode, what the user sees, how it's detected, automatic fallback, human owner. One row per failure mode. The detection column must name a mechanism that exists in your system today, not one you intend to build. The owner column must contain a person's name, never a team.

Fill it in as a group, with engineering, product and support in the room. Two people will disagree about what happens when the model is unavailable, and nobody will have considered the logs a leak path.

What detection can you actually implement?

Five detector types are realistic for most teams: output validators and schema checks, confidence or relevance thresholds, a canary question set with known answers, a user-feedback signal, and cost or rate alarms. Each catches a different class of failure. A register listing one detector across all ten rows is decoration.

Schema validation is cheapest, so start there: check enums against real values and verify any ID the model produced exists. An invented product code gets caught by a lookup, not a prompt. Then freeze 30 to 50 canary questions with verified answers, several the system should refuse, and run them before every prompt or model change. Drift shows up as a slope long before anyone complains. It's the argument for QA on generated code, where the review layer has to grow with the output.

How should the fallback behave when detection fires?

A fallback has three legitimate destinations: deterministic behaviour that doesn't involve a model, a human, or an honest statement that the system can't answer. Never fall back to a plausible guess, a cached answer presented as current, or a generic response that hides the failure. Degrading to silence beats degrading to fiction.

Deterministic is best where it exists. If AI search fails, show keyword results: users don't love it, and they trust it, because it behaves the same way every time. Handing to a human works when volume is low and stakes are high, provided the queue has an SLA and a name on it. The honest refusal is hardest to get past a stakeholder and worth the fight, because "I can't answer that from the documents I have" protects trust in every answer that follows. AI error states deserve real design work, not a red toast borrowed from form validation.

Who owns each failure mode, and what's the escalation path?

Every row needs one named human owner, a trigger threshold, and a response time. The owner isn't necessarily the person who fixes it; they're the person accountable for the response. Write escalation as a sequence with time bounds, ending in explicit authority to disable the AI path entirely without waiting for a deploy.

"The AI team" is not an owner. Teams don't get paged and teams don't make calls at 11pm. Put a name in the cell, a backup beside it, and tell both before launch. Keep the path short: detector fires, fallback engages, owner notified within the hour, immediately for anything touching PII. If nobody confirms a fix inside that window, the feature flag goes off. If turning your AI feature off requires a release, you don't have a mitigation, you have an intention.

What should you log, and what must never be logged?

Log the prompt template version, model identifier and version, retrieval document IDs, token counts, latency, every validator result, whether a fallback fired, and user feedback. Never log raw personal data, credentials, API keys, full restricted-document text, or special-category data such as health and financial detail, particularly into third-party observability tools.

Without version stamps you can't answer the only question that matters after a bad answer: what exactly ran? Model IDs change under you and prompts get edited on a Friday, so "AI response generated" turns an investigation into archaeology. The second half of the rule stops a quality problem becoming a legal one. Full prompts make debugging easy, and full prompts are where user-submitted personal data ends up, copied into a vendor's cloud and outside your GDPR or DPDP record of processing. Log IDs and hashes, and redact at capture.

What's the pre-launch review gate, and who signs it?

The gate is a scheduled review, before launch, where the completed register is read aloud and four people sign: the engineering owner, the product owner, whoever is accountable for security and data protection, and the support lead. Any row missing a detector, a fallback or a named owner blocks the launch.

Support signing changes the conversation, because the person answering the angry email asks the sharpest question: can we switch this off for one account? Here's the position I'll defend. "The model will get better" is not a mitigation. It's a forecast about someone else's roadmap, with no date and no test. A mitigation is something you can trigger and verify today. Working to a CMMI Level 5 standard means this review has a template and a signature list instead of depending on who remembered, which is why our AI development engagements put the register in week one.

How do you build the register in one sitting?

Two hours, one room, one shared document. Work through the ten steps in order, and treat anything you can't answer as a build task with an owner and a date rather than a note in the margin. Expect the disagreements to be the useful part.

1. Write the feature's job in one sentence: who uses it, and for what.
2. Copy the ten failure modes above into rows, deleting none yet.
3. Add your domain-specific rows, the ones a regulator would name first.
4. Fill "what the user sees" first, in plain language, from their point of view.
5. Name the detector for each row, marking any that doesn't exist as a build task.
6. Delete every fallback that's really a guess, replacing it with deterministic behaviour, a human, or a refusal.
7. Put a person's name in every owner cell, plus a backup, and tell them.
8. Set the escalation clock: notify within, decide within, kill switch after.
9. Write the logging rules: the fields you capture, and the fields banned outright.
10. Book the sign-off review and invite support; it isn't done until four people sign.

Keep it in the repo beside the code, and revisit it whenever the model, the prompt or the data source changes. If you'd rather someone ran that session with you, book a 30-minute call and bring your feature spec.

Frequently Asked Questions

Isn't a failure-mode register overkill for a small internal feature?

Scale it, don't skip it. A tool used by five people can have a five-row register. The rows you can't drop are the ones with legal or financial consequences: PII in logs, cost runaway, and wrong answers that leave the building.

How is this different from a standard risk register?

A standard risk register scores likelihood against impact and produces a heat map. This one produces behaviour. Every row ends in something the system does automatically and someone who gets notified. If you already run a corporate risk process, bolt these columns onto it.

Can't we just add guardrails and call it done?

Guardrails are detectors, so they belong in column three. They don't tell you what the product does when they trip, who hears about it, or how you'd know it had stopped working. A content filter with no fallback turns a wrong answer into a blank screen.

What if we're integrating a third-party AI feature rather than building one?

You still own every row, because your users can't see the vendor. What changes is the fallback column: you can't fix their model, so your mitigations are timeouts, output validation on your side, and a deterministic path.

Does having a human in the loop remove the need for this?

It reduces some risks and creates others. People approving AI output at volume start rubber-stamping, so approval fatigue deserves its own row. Measure the override rate: if reviewers approve nearly everything, the loop is decorative.

Who owns the register if we don't have an ML team?

Whoever owns the feature in production. The detectors are ordinary engineering: schema validation, threshold alarms, scheduled test runs, a feedback table.