Azure AI
From RAG Demo to Production: Building Secure AI Applications on Azure
A production architecture for retrieval-augmented generation that covers identity, ingestion, retrieval quality, evaluation, observability, security, and cost.
A RAG demonstration can be assembled quickly: split a document, create embeddings, retrieve a few chunks, and send them to a model. Production changes the question. The system must return authorized, relevant, traceable, and cost-controlled answers across changing data and unpredictable user prompts.
The production workload is therefore much larger than the model call. It includes content ingestion, access control, retrieval, orchestration, evaluation, safety, observability, feedback, and operational governance. If any link is weak, the final answer becomes unreliable even when the model is capable.
1. Start with a bounded decision or workflow
Define what the assistant is allowed to answer, which sources are authoritative, who may access them, and what happens when evidence is missing. A support assistant, policy search tool, and mortgage-document workflow require different retrieval strategies and different failure behavior.
Create an evaluation set before choosing chunk sizes or models. Include typical questions, ambiguous language, missing-answer cases, conflicting documents, permission boundaries, and adversarial prompts. The evaluation set becomes the stable contract for comparing changes.
2. Design ingestion as a governed data product
Ingestion should preserve document identity, source version, ownership, effective date, sensitivity, and access metadata. Extraction and chunking must be repeatable so a document update can replace earlier chunks without leaving stale content behind.
Chunking is not a universal number. Structure-aware chunks often outperform arbitrary token windows for policies, contracts, or technical manuals. Store enough metadata to filter by tenant, department, document type, status, and version before semantic ranking occurs.
- Use deterministic document and chunk identifiers
- Track ingestion status, failures, source version, and deletion
- Apply authorization metadata at index time and query time
- Separate raw content, processed content, and searchable projections
- Re-evaluate retrieval after changing embedding models or chunking
3. Keep retrieval observable and testable
Log the rewritten query, filters, candidate documents, scores, selected chunks, and citations for each trace—subject to privacy requirements. Without retrieval telemetry, teams often tune prompts while the actual problem is missing or irrelevant context.
Standard RAG follows a fixed retrieval sequence and is usually easier to evaluate and operate. Agentic RAG is valuable when the system must decompose questions, select among sources, perform several retrieval steps, or call business tools. Use that flexibility only when the use case needs it because every reasoning loop adds latency, cost, and new failure modes.
4. Evaluate the whole answer, not model fluency
A polished answer can still be unsupported. Measure retrieval relevance and coverage together with groundedness, completeness, correctness, citation quality, safety, latency, and cost. Different workloads will weight those metrics differently.
Run evaluations in CI/CD against a controlled dataset, then monitor sampled production traces and user feedback. Model outputs are nondeterministic, so use distributions and thresholds rather than expecting identical text. Review failure clusters to improve ingestion, retrieval, instructions, or product behavior.
- Fail safely when authoritative evidence is absent
- Require citations for claims that come from enterprise content
- Measure access-control leakage as a release-blocking failure
- Track task completion for agents that call tools
- Retest when documents, models, prompts, or indexes change
5. Build security around identity and data flow
Use Microsoft Entra ID and managed identities so services do not depend on embedded credentials. Keep secrets in Key Vault, restrict service networking where the threat model requires it, and enforce document permissions in retrieval—not only in the user interface.
Treat retrieved content as untrusted input. Documents can contain instructions intended to override the system prompt or manipulate tool use. Separate system instructions from data, constrain tools, validate tool parameters, apply least privilege, and require approval for consequential business actions.
6. Operate quality, latency, and cost together
Trace the request across gateway, orchestration, search, model, tools, and downstream APIs. Record token consumption, search latency, tool latency, retries, safety outcomes, and end-to-end result quality. A cheaper model is not cheaper if low quality increases retries or human review.
Use budgets and quotas, cache safe repeatable results, reduce unnecessary context, and route simple tasks to smaller models when evaluation supports it. Maintain a model lifecycle plan because versions, availability, behavior, and pricing change over time.
Key takeaways
What to carry into your next decision
- Build an evaluation set before optimizing prompts or choosing architecture.
- Treat ingestion, authorization, retrieval, and observability as first-class services.
- Prefer standard RAG until the use case demonstrates a need for agentic retrieval.
- Operate quality, safety, latency, and cost as one production system.
Sources and further reading
Version-specific and platform guidance was checked against these primary sources.