Background

Production RAG systems rarely fail through a single catastrophic event. More commonly, reliability erodes through a sequence of operational changes: documentation evolves, retrieval behavior shifts, prompts are revised, dependencies change, and evaluation datasets become stale.

Traditional engineering practices classify failures by system components—retrievers, prompts, vector databases, or language models. While useful for implementation, this perspective provides limited guidance for operating production AI systems over time.

This article proposes a reliability framework based on three complementary dimensions:

• Failure Dynamics — how reliability changes over time
• Reliability Control Surface — where engineers can observe and intervene
• Detectability — how easily the failure is discovered before users are affected

To illustrate the framework, a controlled experiment simulates seven weeks of gradual documentation evolution in a production-style RAG system. The experiment demonstrates one representative failure class—Gradual Knowledge Drift—and shows why this class of failure frequently escapes traditional operational monitoring.

---

1. Introduction — AI Systems Rarely Fail the Way Traditional Software Does

Modern software systems fail in ways that operations teams understand well. A bad deployment increases error rates. A database outage causes requests to fail. A networking issue adds latency. Infrastructure becomes unavailable. These failures are disruptive, but they are also highly visible. Dashboards turn red, alerts fire, and engineers know where to start investigating.

Retrieval-Augmented Generation (RAG) systems introduce a different class of failure. Usually , a production RAG application can appear perfectly healthy from an operational perspective. Requests complete successfully, APIs return HTTP 200 responses, latency remains within service-level objectives, and every component in the architecture is online. Traditional monitoring tools report a healthy system. Yet users begin to lose confidence in the answers.

Fundamentally, we are trying to solve the AI reliability problem instead of the traditional software reliability problem.

---

From the graph, the key differences is that traditional software failures are around discrete events and gives immediate feedback; while RAG systems degrades gradually and usually invisible to infrastructure-level monitoring. Fundamentally, traditional software’s reliability is typically judged by correctness and availability: either the service works or it doesn’t. RAG systems add another dimension—knowledge quality. A system can achieve excellent uptime while steadily becoming less reliable.

This reframes reliability from a problem of system correctness to a problem of sustained knowledge quality.

---

2. Why Existing Classifications Are Insufficient

What do we know about RAG system failures. Perhaps newly published documentation isn’t being retrieved. Maybe document metadata has drifted, reducing retrieval accuracy. An embedding model has changed, but only part of the corpus has been re-indexed…

Current discussions usually classify failures by components, some of the examples are :

Component	Typical failures
Embedding model	Poor semantic representations, embedding drift after model changes, domain mismatch, multilingual mismatch
Vector database	Low recall, indexing errors, stale or missing vectors, incorrect filtering, ANN search inaccuracies
Chunking	Chunks too large/small, broken context boundaries, duplicated information, loss of semantic coherence
Retriever	Irrelevant documents retrieved, low recall, poor ranking, metadata filtering mistakes
Reranker	Relevant documents demoted, irrelevant documents promoted, unstable ranking
Prompt	Hallucinations, ignored context, prompt injection, poor instruction following, format inconsistencies
LLM / Generator	Hallucination, incorrect synthesis, unsupported claims, reasoning errors, overconfidence
Knowledge base	Outdated documents, incomplete corpus, inconsistent information, stale data
Ingestion pipeline	Failed indexing, partial ingestion, parsing/OCR errors, metadata extraction failures

These do explain where failures originate. However, they hardly explain:

• how failures evolve
• when engineers discover them
• which operational strategy is appropriate

Production RAG system operations require a reliability model, not only an architecture model.

---

3. A Reliability Framework for Production AI Systems

Imagine an engineer receiving the following incident report:

“The RAG system is hallucinating more than usual.”

Although the statement describes a symptom, it immediately raises several unanswered questions.

Has the system failed suddenly after a deployment, or has answer quality been declining for weeks? Is the root cause likely to be in the knowledge base, the retrieval pipeline, or the generation stage? Should engineers inspect operational dashboards, rerun evaluation suites, or begin a deeper investigation?

The difficulty is not a lack of observability—it is a lack of structure for reasoning about production AI failures.

From examining recurring production incidents, I found that most failures can be described along three complementary dimensions:

• Failure Dynamics describe how reliability changes over time.
• Reliability Control Surfaces identify where corrective action is most effective.
• Detectability characterizes how easily the failure is discovered before affecting users.

Rather than treating every incident as unique, these dimensions provide a common language for understanding, classifying, and responding to production AI failures.

Dimension 1 — Failure Dynamics

When a RAG incident occurs, the first question engineers should ask is not what failed, but how reliability changed over time.

Traditional software systems are typically designed around discrete failures. A deployment introduces a regression, a dependency fails, or a resource becomes exhausted. Reliability changes are usually tied to identifiable events, allowing engineers to reason about incidents as immediate failures.

Production RAG systems behave differently. Reliability often changes continuously rather than discretely. Documentation evolves, retrieval behavior shifts, prompts are revised, and evaluation datasets become stale. Individually, these changes appear harmless; collectively, they reshape the behavior of the system. As a result, understanding a production AI incident begins with a different question:

How did reliability evolve over time?

This leads to the first dimension of the framework: Failure Dynamics.

Immediate Immediate failures appear immediately after a discrete system change or unexpected input. They are typically associated with deployments, prompt revisions, tool misconfiguration, or invalid context injection. Engineers usually observe an immediate drop in correctness or task completion

Gradual Gradual failures emerge through a sequence of individually harmless changes. Documentation evolves, retrieval behavior shifts, evaluation datasets become stale, or models are upgraded incrementally. No single change is sufficient to trigger an incident, but their cumulative effect steadily erodes reliability.

Threshold Threshold failures remain latent until accumulated changes push the system beyond a critical operating boundary. Reliability appears stable until a tipping point is reached, after which performance degrades abruptly.

Oscillating Oscillating failures exhibit inconsistent reliability under similar operating conditions. Performance alternates between successful and unsuccessful outcomes because the underlying system behavior depends on input distribution, retrieval ordering, model stochasticity, or changing operational conditions.

Cascading Cascading failures originate from a local defect that propagates through downstream workflow stages. A retrieval error may influence planning, which affects tool selection, memory updates, and ultimately produces a significantly larger end-user failure than the original defect alone.

---

Dimension 2 — Reliability Control Surface

Once the failure dynamics have been identified, the next engineering question is:

Where should engineers intervene?

Failure Dynamics describe how reliability changes. Reliability Control Surfaces describe where reliability can be observed, influenced, and improved.

In traditional software systems, the answer is often localized. Engineers scale infrastructure to address resource contention, upgrade dependencies to resolve compatibility issues, or adjust service-level trade-offs between latency, availability, and consistency. The intervention point is usually well-defined because the system itself is deterministic.

Production RAG systems are different. A single user-visible failure may emerge from interactions across multiple stages of the pipeline. Corrective actions therefore require engineers to identify the control surface where reliability can be most effectively improved.

We define five primary Reliability Control Surfaces.

Knowledge The knowledge surface governs the quality of the information available to the system. Engineers intervene here by improving the corpus itself: removing stale documents, eliminating duplicates, correcting inconsistencies, or refining document organization. If the system retrieves incorrect knowledge, no downstream component can reliably recover the correct answer.

Retrieval The retrieval surface determines which knowledge reaches the model. Engineers adjust retrieval algorithms, chunking strategies, embedding models, metadata filters, rerankers, and search parameters to improve the relevance and completeness of retrieved context.

Generation The generation surface governs how retrieved context is transformed into an answer. Prompt design, model selection, decoding strategies, and structured output constraints all influence whether the model produces accurate, complete, and faithful responses.

Evaluation The evaluation surface determines how reliability is measured and enforced. Rather than improving answers directly, evaluation establishes quality gates through automated benchmarks, regression tests, and production monitoring. It answers the question: Has reliability changed enough to require intervention?

Operations The operations surface coordinates how the entire system behaves in production. Version management, deployment policies, rollout strategies, monitoring, traffic routing, and incident response all influence the long-term reliability of the application, even when individual components remain unchanged.

---

Dimension 3 — Detectability

The previous dimension answered where engineers should intervene. Detectability answers a different operational question:

How likely is this failure to be discovered before users experience it?

Not all failures are equally visible. Some immediately trigger monitoring systems, while others remain hidden behind apparently successful requests and fluent model responses. From an operational perspective, the cost of a failure depends not only on its severity but also on how long it remains undetected.

Traditional software systems have benefited from decades of investment in observability. Infrastructure failures, resource exhaustion, deployment regressions, and service interruptions typically produce measurable signals that monitoring systems can detect automatically.

Production AI systems introduce a different class of reliability problems. A request may complete successfully, latency may remain stable, and no infrastructure alarms may fire, yet answer quality can still deteriorate. In these cases, correctness—not availability—becomes the primary operational concern.

We therefore classify production AI failures according to their detectability.

D0 — Immediately observable Failures are directly visible through conventional operational signals or obvious incorrect behavior. Engineers are typically alerted immediately through monitoring systems or user-facing errors.

D1 — Operationally observable Failures become apparent through changes in production telemetry, deployment behavior, or runtime characteristics. Although the application continues functioning, operational metrics indicate that reliability has changed.

D2 — Evaluation observable Failures cannot be detected reliably through infrastructure monitoring alone. Instead, they require scheduled or continuous evaluation using representative workloads to identify declining correctness, retrieval quality, or answer fidelity before users notice the regression.

D3 — Investigation observable Failures remain operationally invisible until a specific customer incident triggers investigation. Root cause identification requires manual analysis, reproduction, and engineering judgment, making this class the most expensive and operationally disruptive.

---

The Complete AI Reliability Framework

The three dimensions introduced in this section are intended to be used together rather than independently.

Failure Dynamics describe how reliability changes over time. Reliability Control Surfaces identify where engineers should investigate and intervene. Detectability determines how failures become visible during operation. Together, these dimensions transform an isolated production incident into a structured reliability problem with an appropriate operational response.

Figure 3 summarizes the complete AI Reliability Framework. Rather than treating AI failures as disconnected symptoms such as hallucination, retrieval drift, or poor answer quality, the framework provides a systematic reasoning process—from identifying the failure dynamics to selecting the most appropriate operational response.

---

4. Controlled Experiment

To illustrate how the proposed framework can be applied in practice, we conducted a controlled experiment on a representative production RAG failure. Rather than attempting to validate every combination of failure dynamics, control surfaces, and detectability levels, this article focuses on a single, realistic scenario that demonstrates how the framework guides diagnosis and operational response.

The selected failure represents a Gradual failure dynamic, occurring on the Knowledge control surface with D2 (Evaluation Observable) detectability. This class of failure was chosen because it is both common in production RAG systems and operationally expensive. Reliability degrades incrementally as the knowledge corpus evolves, yet the decline often remains invisible until systematic evaluation reveals a measurable regression.

The experimental system consists of a Retrieval-Augmented Generation (RAG) application serving API documentation. Over a seven-week period, the underlying documentation corpus is progressively modified to simulate realistic knowledge evolution while the retrieval pipeline, model configuration, and evaluation dataset remain unchanged. This isolates knowledge evolution as the primary independent variable.

Each week, the system is evaluated against the same benchmark using four reliability metrics:

• Source Hit@k — whether the correct supporting documents are successfully retrieved.
• Answer Pass Rate — proportion of responses meeting predefined correctness criteria.
• Faithfulness — degree to which generated answers are supported by retrieved evidence.
• Unsupported Answer Rate — frequency of responses containing unsupported or fabricated information.

Figure 4 illustrates the overall experimental design.

---

5. Result

The controlled experiment produced three observations that characterize Gradual Knowledge Drift in production RAG systems.

Observation 1 — User-visible behavior changed before the system appeared to fail

Figure 5 follows a single production question across seven weeks.

Although the retrieval pipeline, evaluation dataset, and application remained unchanged, the answer gradually shifted from a correct response to repeated uncertainty before later converging on an incomplete explanation.

Importantly, the application never failed operationally. Every request completed successfully, yet the quality of the answer changed substantially as the knowledge corpus evolved.

week	question	answer
week_0	How should clients authenticate API requests?	Clients must include an Authorization: Bearer <api_key> header in all API requests.
week_1	How should clients authenticate API requests?	Unknown from the available documentation.
week_2	How should clients authenticate API requests?	Unknown from the available documentation.
week_3	How should clients authenticate API requests?	Unknown from the available documentation.
week_4	How should clients authenticate API requests?	Unknown from the available documentation.
week_5	How should clients authenticate API requests?	Clients authenticate API requests using a token, as indicated by the per-token rate limits and the 401 status code returned when authentication fails.
week_6	How should clients authenticate API requests?	Clients authenticate API requests using a token, as indicated by the per-token rate limits and the 401 status code returned when authentication fails.

Figure 5 demonstrates that user-visible reliability can deteriorate even when the system appears operationally healthy.

Observation 2 — Reliability degraded without operational failures

To determine whether this qualitative change reflected a measurable reliability decline, we evaluated the system weekly using four metrics.

Figure 6 records the evaluation results throughout the experiment.

Several observations stand out.

• Source Hit@4 remained consistently high for most weeks.
• Faithfulness changed only slightly.
• No infrastructure failures or retrieval outages occurred.
• However, Answer Pass Rate declined significantly, reaching its lowest point during Week 3.

The degradation therefore cannot be explained by infrastructure instability or catastrophic retrieval failure. Instead, the experiment demonstrates a gradual reduction in answer quality while conventional operational indicators remained largely stable.

Week	Source Hit@4	Answer Pass Rate	Faithfulness Pass Rate	Unsupported Answer Rate	Signal
week_0	1	0.85	1	0.05	Early drift visible
week_1	0.95	0.8	0.95	0.1	Early drift visible
week_2	0.95	0.85	0.95	0.05	Early drift visible
week_3	0.95	0.6	0.95	0.1	Material decay
week_4	0.95	0.8	0.95	0.05	Early drift visible
week_5	0.95	0.8	1	0	Early drift visible
week_6	0.85	0.75	0.95	0.05	Early drift visible

Observation 3 — Progressive degradation only becomes obvious longitudinally

Figure 7 plots the evaluation metrics over time.

Viewed week-by-week, each individual regression appears relatively minor. None would typically justify an operational incident on its own.

Viewed longitudinally, however, the pattern becomes unmistakable. Small fluctuations accumulate into a sustained decline in answer quality despite stable infrastructure and largely unchanged retrieval metrics.

This illustrates an important characteristic of Gradual Knowledge Drift: the operational signal emerges from the trend rather than any individual observation.

Interpretation

The experiment supports the hypothesis that some production RAG failures evolve gradually rather than immediately.

Unlike a deployment regression, where a single event produces an obvious operational change, gradual knowledge drift emerges through a sequence of individually reasonable modifications to the knowledge corpus.

Each weekly change appears harmless in isolation. Collectively, however, they alter the behavior of the retrieval-generation system enough to produce measurable reliability degradation.

Consequently, conventional monitoring—which is designed to detect discrete failures—provides little warning before users begin experiencing degraded answers.

The appropriate operational response is therefore fundamentally different. Immediate failures require incident response and rollback. Gradual failures require continuous evaluation, longitudinal monitoring, and periodic reliability assessment to identify trends before they become customer-visible.

---

6. Operational Principals

Principle 1: Reliability should be monitored longitudinally rather than episodically.

The experiment demonstrates that no individual weekly change was sufficient to trigger an operational incident. Instead, reliability declined through a sequence of small, individually reasonable modifications to the knowledge corpus. This suggests that production AI reliability is fundamentally temporal. The reliability state of the system cannot be inferred from a single evaluation or deployment event; it emerges from trends observed over time.

Consequently, evaluation should not be treated as a periodic validation activity performed before release. It should become a continuous operational process capable of identifying long-term changes in system behavior before they become customer-visible.

---

Principle 2: Operational strategies should depend on failure dynamics rather than implementation components.

The observed degradation originated from changes to the knowledge corpus, yet the operational response was determined by the behavior of the failure rather than its technical origin. An immediate deployment regression and a gradual knowledge drift may involve the same retrieval pipeline, but they require fundamentally different operational strategies. One favors rollback and incident response; the other favors continuous evaluation and trend monitoring.

---

Principle 3: Different detectability classes require different operational controls.

Not every reliability failure can be discovered through conventional monitoring. The experiment illustrates a D2 (Evaluation Observable) failure. Operational metrics remained largely stable while answer quality gradually declined. Traditional infrastructure monitoring therefore provided little indication that reliability had changed.

Operational controls should therefore be matched to the detectability class of the failure. Immediately observable failures benefit from alerting and monitoring, whereas evaluation-observable failures require continuous benchmark evaluation capable of detecting subtle quality regressions before users encounter them.

---

Principle 4: Knowledge evolution is an operational concern, not merely a documentation concern.

Throughout the experiment, the application code remained unchanged. Reliability changed because the knowledge available to the system changed. This observation highlights an important distinction between traditional software systems and production AI systems. Documentation is no longer passive reference material consumed only by engineers; it has become executable operational state consumed directly by the application.

---

7. Limitations

This article intentionally evaluates a single representative failure profile rather than the complete AI Reliability Framework.

The objective of this experiment is not to validate every possible combination of Failure Dynamics, Reliability Control Surfaces, and Detectability levels. Instead, it demonstrates how the proposed framework can be applied to reason about a realistic production incident under controlled experimental conditions.

The selected scenario—Gradual failure dynamics on the Knowledge control surface with D2 (Evaluation Observable) detectability—was chosen because it represents a common production failure that is difficult to identify using conventional operational monitoring.

Future work will extend the experimental series to additional failure profiles, including:

• Threshold failures
• Oscillating failures
• Cascading failures

using the same controlled experimental methodology. Evaluating multiple failure classes will allow the proposed framework to be assessed across a broader range of production AI reliability scenarios.

---

8. Conclusion

Production AI systems cannot be understood solely through their architectural components. Models, retrievers, prompts, and vector databases explain how an AI system is built, but they do not explain how reliability changes once the system enters production.

This article proposed an operational reliability framework that complements architectural thinking by introducing three additional dimensions: Failure Dynamics, Reliability Control Surfaces, and Detectability. Together, these dimensions provide a structured way to reason from a production incident toward an appropriate engineering response.

The controlled experiment on Gradual Knowledge Drift demonstrates one representative failure class within this framework. More importantly, it illustrates a broader operational reality: many production AI failures emerge not through catastrophic regressions, but through the accumulation of individually reasonable changes that gradually alter system behavior.

Traditional software engineering matured by developing shared languages for correctness, availability, and performance. As AI systems become long-lived production systems, reliability engineering will likewise require new operational concepts that describe how AI behavior evolves, how failures should be classified, and how reliability should be managed over time.

The framework presented here is one step toward that operational vocabulary.

---

Appendix A — Experiment Details

Corpus composition

• auth.md - Explains bearer-token authentication and how to interpret 401 and 403 responses
• errors.md - Summarizes HTTP error handling, including 429 rate limits, 5xx retries, and auth-related errors.
• pagination.md - Describes cursor-based pagination using limit, next_cursor, and cursor.
• rate_limits.md - Defines rate-limit responses, relevant headers, and worker coordination after 429s.
• retry_behavior.md - Specifies retry rules for 429, 5xx, idempotent requests, jitter, backoff, and Retry-After.
• troubleshooting.m - Lists support-debugging details, timeout guidance, and webhook retry behavior.

Weekly mutations

• week 1 - Added overlapping troubleshooting content
• week 2 - Added stale migration guide with outdated retry behavior
• week 3 - Re-indexed retry behavior doc with worse chunking
• week 4 - Added legacy SDK docs with similar wording
• week 5 - Switched to looser prompt version
• week 6 - Added noisy FAQ content competing with correct source

Support questions

When should clients retry after a 429 error?
What should an API client do when it receives 429 Too Many Requests?
Should clients use Retry-After or retry immediately after a 429?
How should workers behave after a rate limit retry?
Is a fixed 60 seconds the current retry policy for 429 responses?
Which header tells clients how long to wait after a rate limit?
Which operations should be retried automatically?
How should clients handle 5xx server errors?
How should clients authenticate API requests?
What does a 401 response mean?
What does a 403 response mean?
How do clients request the next page of results?
When should clients stop paginating?
What should a client do after a timeout?
What information should be included when contacting support?
Are webhook retries the same as client retry behavior?
What should clients do to request volume after receiving 429s?
Which headers describe the rate limit budget?
Should clients automatically retry requests that create side effects?
Which response value helps support investigate failures?

Repository link

rag-decay-experiment

---

Appendix B — Framework Summary

A summary the three-dimension reliability framework:

Failure Dynamics	Typical Control Surface	Typical Detectability
Immediate	Generation	D0
Gradual	Knowledge	D2
Threshold	Retrieval	D1–D2
Oscillating	Operations	D3
Cascading	Cross-surface	D3