What is multimodal RAG?

Multimodal RAG enterprise systems extend retrieval-augmented generation beyond plain text to PDFs with tables, scanned images, and structured database queries. A router picks the right retriever per query, then blends results for the model.

Real enterprise content is not clean text. A clinical note has charts. An insurance claim has photos. A regulatory filing has tables. Text-only RAG misses most of the answer. The NIST AI Risk Management Framework Map function calls out data governance across modalities as a core control, and HIPAA, 42 CFR Part 2, SOX, and the EU AI Act all push the same direction.

How do you handle PDFs with tables and diagrams?

Use layout-aware parsing to detect text blocks, tables, and figures. Convert tables to markdown or JSON, caption figures with a vision model, and link child chunks back to the parent page for context.

Tools like Unstructured, LlamaParse, or Azure Document Intelligence preserve reading order. Store the original page reference so the model can cite the source. For SR 11-7 model documentation and SOX-relevant tables, audit every parsed value against the source PDF.

How do you retrieve from images and scanned documents?

Run OCR on scanned text, then index two parallel chunks per image: an OCR text chunk and a vision-language embedding for the image itself. Caption diagrams so semantic search can find them by description.

Tesseract or AWS Textract handles OCR. CLIP-style or SigLIP embeddings handle visual search. For HIPAA-protected imagery and biometric data covered under California CCPA/CPRA, GDPR special-category rules, and India DPDP, apply access controls at the chunk level before retrieval.

How do you combine RAG with structured database queries?

Use text-to-SQL with schema retrieval. The router sends quantitative questions to SQL, qualitative questions to vector search, and merges both into one grounded answer. Log every generated query for audit.

For FDIC and OCC examiners, NAIC Model AI Bulletin reviewers, and Singapore MAS FEAT auditors, the SQL audit trail matters as much as the answer. Pair structured outputs with FHIR resources for clinical data, or with the source database row IDs for financial reporting.

What enterprise use cases fit multimodal RAG?

Clinical documents with charts, insurance claims with photos and structured fields, regulatory filings with tables, and engineering specs with diagrams all need it. Each example mixes at least two modalities the model has to reconcile.

Healthcare teams under HIPAA, HITECH, and FDA SaMD guidance use it for chart-heavy clinical notes. BFSI teams under SR 11-7, SOX, and the NY DFS Circular Letter No. 7 use it for claims packets and regulatory filings. UAE PDPL, DIFC, Canada PIPEDA, and UK GDPR add similar controls in their regions. ISO/IEC 42001 sets the cross-border baseline.

What to do next

Audit your top three content types by modality. If two of them are not plain text, scope a multimodal pilot with a router pattern before adding more sources to a text-only index.

Read next: Enterprise RAG Architecture: The Reference Model

The post Multimodal RAG: Documents, Images, Structured Data appeared first on Data, AI, Automation & Enterprise App Delivery with a Quality-First Partner.

How do you evaluate enterprise RAG quality?

Enterprise RAG evaluation runs on four core RAG evaluation metrics: retrieval precision, retrieval recall, groundedness, and answer quality. Each has an automated scoring method. Combined, they catch the main failure modes before users see them.

A retrieval-augmented generation system can fail in four ways. It pulls the wrong chunks. It misses chunks it should have pulled. It writes claims the chunks do not support. Or it ships a fluent answer that fails the user’s task. The NIST AI Risk Management Framework Measure function and Federal Reserve SR 11-7 model validation guidance both push teams toward continuous, documented testing. State laws like the Colorado AI Act, NY DFS Circular Letter No. 7, Utah AI Policy Act, and Texas TRAIGA add accuracy and fairness pressure. Regulated workloads under HIPAA, SOX, and FCRA raise the bar further. The EU AI Act and GDPR data-quality principle add accuracy obligations for cross-border systems.

What is retrieval precision and how do you measure it?

Retrieval precision is the fraction of retrieved chunks that are actually relevant to the user’s query. Score it with a labeled golden set plus an LLM-as-judge rubric on every release.

Build a golden set of 200 to 500 queries with human-labeled relevant chunk IDs. On each evaluation run, compute precision at k (k = 5 or 10 for most enterprise RAG). Augment with an LLM-as-judge that scores each retrieved chunk as relevant, partial, or irrelevant. Track the score over time and alert on regressions.

What is retrieval recall and how do you catch missed context?

Retrieval recall is the fraction of relevant chunks in the knowledge base that the retriever actually returned. It matters most in high-stakes domains where missing context creates real harm.

Recall requires a known answer set. For each golden query, label every chunk in the corpus that contains relevant information. Then compute recall at k. Healthcare, financial services, and legal use cases need high recall because a missed regulation or contraindication can produce a confidently wrong answer that violates HIPAA, FCRA, or NAIC Model AI Bulletin expectations.

What is groundedness and how do you detect hallucinations?

Groundedness is the property that every claim in the generated answer traces back to a retrieved chunk. Score it sentence by sentence with an entailment model plus attribution checks.

Split the answer into atomic claims. For each claim, run a natural language inference model against the retrieved context. Score entailed, neutral, or contradicted. Compute the share of claims that are entailed. This is the strongest signal for hallucination detection in production. The FTC Section 5 deceptive-output posture and the Colorado AI Act both treat unsupported AI outputs as enforcement risk.

How do you score answer quality at scale?

Answer quality is whether the response actually solves the user’s task. Score it with a task-specific rubric, an LLM-as-judge scorecard, and human spot-checks on a sampled subset.

Define a scorecard per use case: completeness, correctness, format adherence, tone, citation accuracy. Run an LLM-as-judge on every release. Sample 1 to 5 percent of production traffic for human review. This mirrors how ISO/IEC 42001, Singapore MAS FEAT, India RBI, UAE PDPL, and Canada AIDA frame ongoing evaluation duties.

How often should you re-evaluate RAG quality?

Run sampled scoring on production traffic continuously. Run the full golden-set suite on every release. Run adversarial and red-team prompts at least quarterly to catch new failure modes.

Eighty percent or more of enterprise AI projects fail to reach production, and a weak evaluation harness is a top reason teams stall or ship unsafe systems.

What to do next

Stand up the four metrics this quarter. Start with a 200-query golden set, an LLM-as-judge, and an entailment-based groundedness check wired to your release pipeline.

Read next: Enterprise RAG Architecture: The Reference Model

The post Evaluating RAG Quality: Groundedness and Hallucination appeared first on Data, AI, Automation & Enterprise App Delivery with a Quality-First Partner.

How do you design a vector search knowledge base?

Enterprise vector search quality depends on four design choices: chunking strategy, embedding model, index pattern, and freshness mechanism. These decide retrieval quality more than the LLM does.

Get them wrong and even GPT-4 class models return irrelevant or stale context. Roughly 70% of enterprises still operate with siloed data, so the knowledge base is also where unification happens. Architecture-first beats prompt-first every time.

What chunking strategies fit enterprise documents?

Chunking splits source documents into retrievable units. Fixed-size chunks (256 to 1024 tokens) work for clean prose. Structural chunking by heading, clause, or section preserves meaning in legal, medical, and financial documents.

Use a parent-child pattern for long policies: embed small child chunks for precision, return larger parent chunks for context. Add 10 to 20% overlap so cross-boundary facts survive. For SEC filings or HIPAA policies, chunk by clause or numbered section, not arbitrary token windows.

How do you choose an embedding model?

Pick an embedding model on five criteria: domain fit, dimension count, latency, cost, and license. Open-weight models like BGE or E5 fit private deployments. API models like OpenAI text-embedding-3 fit fast time-to-value.

Higher dimensions (1536, 3072) raise recall but cost more storage and query time. For regulated workloads under SOX, HIPAA, or GLBA, license terms and data residency matter as much as benchmark scores. Lock the model version. Re-embedding the entire corpus after a model swap is the most expensive maintenance task in RAG.

What index patterns fit enterprise scale?

HNSW gives the best recall-latency trade-off for most enterprise corpora. IVF suits very large indexes where memory is constrained. Flat indexes work only at small scale or for exact-match audits.

Combine dense vectors with BM25 keyword search for hybrid retrieval, then re-rank the top 50 with a cross-encoder. Hybrid plus re-rank closes most relevance gaps that pure vector search misses on acronyms, product codes, and exact identifiers. For multi-tenant data, prefer per-tenant indexes or strict metadata filters so retrieval respects access boundaries from the start.

How do you keep the knowledge base fresh?

Stale context is the most common RAG failure in regulated industries. Use change-data-capture from source systems to trigger incremental upserts. Reserve full reindex for embedding model upgrades or schema changes.

Version every chunk with a source ID, hash, and effective date so auditors can reconstruct what the model saw on a given day. Snowflake, Databricks, and Oracle all expose CDC streams that feed cleanly into a vector pipeline. Freshness is a governance requirement under FINRA recordkeeping and HIPAA, not just a quality concern.

What to do next

Audit your current RAG stack against these four decisions. If chunking, embeddings, index pattern, or freshness was inherited from a demo, it is the bottleneck.

Read next: Enterprise RAG Architecture: The Reference Model

The post Enterprise Vector Search and RAG Knowledge Base Design appeared first on Data, AI, Automation & Enterprise App Delivery with a Quality-First Partner.

What is permission-aware RAG?

Permission-aware RAG is a retrieval architecture that enforces user identity and access rights at the retrieval layer, before results reach the LLM. Document and field permissions are captured at ingestion and re-checked at query time, with every retrieval logged for audit.

Most enterprise RAG leaks happen because teams put access control at the UI render layer. By then the model has already seen restricted text. HIPAA minimum-necessary, GLBA Safeguards Rule, FCRA accuracy duties, SR 11-7 data lineage, and 42 CFR Part 2 substance-use isolation all assume the system never reads what the user cannot see. Permission-aware RAG moves the filter to where it belongs.

Where do identity checks happen in the retrieval pipeline?

Identity checks belong between the retriever and the LLM. The query layer pulls user context, the retriever pre-filters the vector store by ACL tags, the re-ranker applies field-level redaction, and only then does the prompt assembler send chunks to the model.

The order matters. Ingestion tags every document and chunk with owner, classification, and ACL group. Query time fetches the caller’s identity, role, jurisdiction, and consent flags from the IdP. The vector search runs as a filtered query, not a post-filter on raw results. NIST AI RMF Manage function and NY DFS Part 500 access controls both treat retrieval as an access decision, not a UI concern.

How do you model row-level security for vector search?

Row-level security for vector search means storing ACL metadata alongside each embedding and filtering at query time. Pre-filter cuts the candidate set by permission first, then ranks by similarity. Post-filter ranks first, then drops disallowed rows.

Pre-filter is correct for regulated data. Post-filter looks faster but breaks recall: if every top-k result is denied, the user gets a blank or hallucinated answer. For multi-tenant deployments, isolate tenants in separate indexes or namespaces. Shared indexes with metadata filters are acceptable only when the index engine enforces filters server-side. The Colorado AI Act and Utah AI Policy Act both push toward documented isolation between consumer cohorts.

How do you handle document-level and field-level permissions?

Document-level permissions are binary: a user gets the chunk or does not. Field-level permissions are per-attribute: PHI, account numbers, or SSNs are stripped from the chunk before the LLM sees it, based on the caller’s role.

HIPAA Privacy Rule minimum-necessary, FCRA accuracy, GLBA Safeguards, and California CPRA access-to-data rights all push past binary access. A claims analyst may read a chart note but not the substance-use section governed by 42 CFR Part 2. The chunker should mark sensitive spans at ingestion. The re-ranker masks them at query time using deterministic redaction, not model judgment. EU GDPR Article 5 data minimization frames the same idea at concept level.

What logging and audit does permission-aware RAG require?

Permission-aware RAG logs user ID, query text, retrieved document IDs, permission decisions, redactions applied, model output, and timestamp for every retrieval. Logs go to a tamper-evident store with retention aligned to the source-system rules.

SR 11-7 model risk management, the NAIC Model AI Bulletin, SOX access controls, and NY DFS Part 500 all require the same thing: prove who saw what, when, and why. The audit trail should reconstruct the answer end to end. Singapore MAS FEAT, India DPDP Act 2023, UAE PDPL, and ISO/IEC 42001 add similar duties for institutions operating across 40-plus jurisdictions, where retention and disclosure rules vary by region.

What to do next

Audit your current RAG stack for the filter location. If permissions live at the UI or in a post-retrieval check, move them between the retriever and the LLM, tag chunks at ingestion, and stand up the audit log before the next regulator visit.

Read next: Enterprise RAG Architecture: The Reference Model

The post Permission-Aware RAG Architecture for Regulated Firms appeared first on Data, AI, Automation & Enterprise App Delivery with a Quality-First Partner.

Last Updated: May 20, 2026

What is enterprise RAG architecture?

Enterprise RAG architecture is a production-grade retrieval-augmented generation stack built for regulated data, enterprise identity, and audit requirements. It extends basic RAG with four layers: permission-aware retrieval, multimodal ingestion, groundedness evaluation, and compliance overlay. Consumer RAG tutorials miss all four and fail at enterprise rollout.

Most failed enterprise RAG projects look the same. A team builds a clean demo, the executive review goes well, and then security asks who can see what, how PII is handled, what happens when the model hallucinates a salary figure, and where the audit trail lives. The demo cannot answer any of these, and the project stalls.

Consumer RAG patterns do not scale into a regulated enterprise. A bank, hospital, insurer, or government agency needs different controls baked into retrieval, not bolted on after generation. This pillar lays out the reference architecture, the four layers that separate it from a demo, regulatory framing under NIST AI RMF, SR 11-7, HIPAA, GLBA, and NY DFS Part 500, and a phased program plan from pilot to multi-domain rollout.

What’s in this article

• Why does enterprise RAG need permission-aware retrieval?
• What does the enterprise RAG stack look like?
• How do you design the knowledge base?
• How do you evaluate RAG quality in production?
• How does multimodal RAG handle documents, images, and structured data?
• How does RAG intersect with AI governance?
• What deployment patterns fit a regulated enterprise?
• How do you sequence an enterprise RAG program?
• Frequently asked questions

Why does enterprise RAG need permission-aware retrieval?

Permission-aware retrieval filters retrieved chunks against the user’s identity, role, and entitlements before any text reaches the model. Without it, the LLM can surface data the user is not authorized to see.

Most teams filter in the UI. The retriever pulls every relevant chunk, the model reads them all, and the application hides what the user should not see. By then the data has already left its security perimeter. The model has read salary records, patient notes, or material non-public information, and the response can leak fragments through summarization or follow-up questions.

Production enterprise RAG enforces row-level and document-level security at the retriever. The vector store carries access metadata for every chunk. The retrieval call passes the caller’s identity and group membership, and only authorized chunks reach the LLM. SR 11-7, HIPAA minimum-necessary, GLBA Safeguards Rule, and 42 CFR Part 2 all point to the same control: data access tied to a verified identity at the moment of use.

For the deeper architecture pattern, see Permission-Aware RAG Architecture for Regulated Firms.

What does the enterprise RAG stack look like?

The enterprise RAG stack is a pipeline: ingestion, parsing, chunking, embedding, indexing, retrieval, permission filtering, reranking, generation, groundedness check, and audit logging. Each stage carries security and observability controls.

Source systems feed an ingestion layer that parses PDFs, Office files, scans, images, transcripts, and database extracts. Chunking splits content into semantic units with metadata for source, owner, classification, and access policy. An embedding model writes vectors to a private index. At query time the retriever pulls candidates with hybrid search (BM25 plus dense vectors) and applies permission filters using the caller’s identity. A reranker, often a cross-encoder or ColBERT-style scorer, narrows the set. The LLM generates an answer grounded in the surviving chunks. A groundedness check scores the answer, and an audit log captures the prompt, chunk IDs, model version, and final response.

Consumer RAG usually stops at retrieval, generation, and a UI.

Knowledge base design is the area most teams underestimate. See Enterprise Vector Search and RAG Knowledge Base Design for the full pattern.

How do you design the knowledge base?

Enterprise knowledge base design covers chunking strategy, embedding selection, index topology, hybrid search, reranking, and freshness policy. Each choice changes retrieval precision and recall in measurable ways.

Chunking is not one-size-fits-all. Contracts and policies need section-aware chunking to keep clauses intact. Tables need row or row-group chunking with column headers preserved. Long-form research uses sliding-window chunks with overlap. Transcripts need speaker-turn chunks. Pick chunking per content type, not per project.

A single embedding model rarely fits every domain. Many enterprises use one model for general text, a domain-tuned model for medical or legal content, and a separate strategy for code or structured data. Hybrid search beats dense alone because exact terms like CPT codes, ticker symbols, or part numbers carry meaning a vector blurs.

Freshness matters more than teams expect. A vector index that lags the source by 24 hours surfaces stale policy text the day after a regulator update. Build incremental ingestion, not full nightly rebuilds, and tag every chunk with a version and effective date.

How do you evaluate RAG quality in production?

RAG evaluation tracks four metric families: retrieval precision and recall, groundedness, answer relevance, and safety. Each is measured continuously against a labeled evaluation set, not a one-time benchmark.

Retrieval metrics tell you whether the right chunks were found. Precision at k, recall at k, and mean reciprocal rank show whether the retriever is the bottleneck. Groundedness, sometimes called faithfulness, scores how well each claim is supported by the retrieved chunks. Answer relevance asks whether the response addresses the question. Safety covers PII leakage, refusal accuracy, and toxicity.

A nightly pipeline runs the live system against a frozen test set, alerts on regressions, and feeds low-groundedness samples into a human review queue. NIST AI RMF Measure functions and SR 11-7 ongoing monitoring point to the same practice. For metric definitions and harness patterns, see Evaluating RAG Quality: Groundedness and Hallucination.

How does multimodal RAG handle documents, images, and structured data?

Multimodal RAG ingests documents, scans, images, charts, tables, and database rows into a unified retrieval layer. The retriever blends results across modalities so a single answer can cite a contract clause, a chart, and a database row together.

Real enterprise content is not clean text. A claims file combines a scanned form, an adjuster note, a damage photo, and a policy database row. A clinical note combines free text, structured vitals, and a lab PDF. Treating only the text strips out most of the signal.

The working pattern is modality-specific extraction feeding a shared semantic layer. Layout-aware parsers handle PDFs and scans. Vision models extract structure from images and charts. Text-to-SQL or schema-aware retrieval handles structured data, often through Snowflake or Databricks where the data already lives. Each extraction lands as chunks with consistent metadata. For the design tradeoffs, see Multimodal RAG: Documents, Images, Structured Data.

How does RAG intersect with AI governance?

RAG sits inside the AI governance program. It needs the same controls as any production AI: data lineage, PII classification, retention, audit logging, human review, and incident response.

Treat the vector index as a regulated data store. Every chunk carries source lineage, classification, retention, and access policy. PII is detected and tagged at ingestion. Audit logs capture the prompt, chunk IDs, model and embedding versions, the answer, the groundedness score, and the user identity. SR 11-7, HIPAA, FCRA, NY DFS Part 500, GLBA, SOX, and the NAIC Model AI Bulletin map cleanly. The Colorado AI Act, Utah AI Policy Act, Texas TRAIGA, NIST AI RMF, EU AI Act, India’s DPDP Act, UAE PDPL, Singapore’s Model AI Governance Framework, Canada’s PIPEDA, and ISO/IEC 42001 reinforce the same direction across jurisdictions.

For the broader program RAG plugs into, see Enterprise AI Governance Framework. For how RAG feeds agents, see Agentic AI for Enterprise.

What deployment patterns fit a regulated enterprise?

Three deployment patterns dominate: closed model with private vector store, hybrid with hosted embeddings and private generation, and fully hosted inside a VPC with sovereign region controls. The right choice depends on data sensitivity, latency, and regulator posture.

Pattern one is the strictest. Models like Llama, Mistral, or a private OpenAI deployment run inside the enterprise network or a sovereign region. Vector store, embedding service, and audit log sit behind the same perimeter. This fits HIPAA-covered workloads, FCRA decisioning, material non-public information, and 42 CFR Part 2 records.

Pattern two trades some control for capability. Embeddings run on a hosted service under a strong data processing agreement, often Snowflake Cortex or Databricks Mosaic, while generation uses a closed model. Internal knowledge assistants often fit this pattern.

Pattern three is fully hosted inside a customer-controlled VPC with private networking, customer-managed keys, and a sovereign region. Oracle and OpenAI enterprise offer variants. The control surface is smaller but the operating burden drops. Risk teams treat this as a managed third party under SR 11-7 and GLBA service provider rules.

How do you sequence an enterprise RAG program?

An enterprise RAG program runs in three phases: a single-domain pilot with the permission model in place by day 60, multimodal ingestion and an evaluation harness by day 180, and multi-domain rollout with full governance integration by day 360.

Phase one, days 0 to 60, picks a single domain with clean ownership. Common picks: internal policy search, an HR knowledge assistant, or contract clause lookup. The non-negotiables are permission-aware retrieval from day one, an audit log, and a labeled evaluation set of at least 200 queries. Skip permission and you will rebuild later.

Phase two, days 60 to 180, extends ingestion to multimodal sources, stands up the continuous evaluation harness, and adds human review for low-groundedness answers. Most of the real engineering happens here.

Phase three, days 180 to 360, rolls out additional domains, integrates with the AI governance program, and feeds agentic workflows. Roughly 80 percent of enterprise AI projects fail to reach production. The most common reason is skipping phase one controls to chase a faster phase three.

What to do next

Three next steps. Download the W7 Enterprise RAG Reference Architecture whitepaper for full diagrams and control mappings. Take the Scadea AI Readiness Assessment to find where data, identity, or governance gaps will block a rollout. Read the Closed LLM and Sovereign AI Deployment Patterns pillar if data residency applies.

Related reading

• Permission-Aware RAG Architecture for Regulated Firms
• Enterprise Vector Search and RAG Knowledge Base Design
• Evaluating RAG Quality: Groundedness and Hallucination
• Multimodal RAG: Documents, Images, Structured Data
• Enterprise AI Governance Framework
• Agentic AI for Enterprise

Frequently asked questions

What is the difference between enterprise RAG and consumer RAG?

Enterprise RAG adds permission-aware retrieval, multimodal ingestion, groundedness evaluation, and an audit-grade compliance overlay. Consumer RAG generates an answer with no identity check, no evaluation, and no audit trail.

Where should permission filtering happen in a RAG pipeline?

At retrieval, before chunks reach the LLM. Filtering in the UI is unsafe because the model has already read restricted text and can leak it through summarization or follow-up answers.

What regulations apply to enterprise RAG in the United States?

Common references include NIST AI RMF, SR 11-7, HIPAA, HITECH, 42 CFR Part 2, GLBA, FCRA, SOX, NAIC Model AI Bulletin, NY DFS Part 500 and Circular Letter No. 7, the Colorado AI Act, Utah AI Policy Act, Texas TRAIGA, and FTC Section 5. Obligations vary by jurisdiction and use case.

Do you need a separate vector database for enterprise RAG?

Not always. Many enterprises start with a vector index inside Snowflake, Databricks, or Oracle. A standalone vector store makes sense when scale, hybrid search, or specialized rerankers justify the operating cost.

How do you measure hallucinations in a RAG system?

Groundedness scoring compares each claim against the retrieved chunks. Automated scorers, often a smaller LLM acting as a judge, run against a labeled evaluation set. Low-groundedness answers route to human review.

Can RAG handle scanned documents and images, not just text?

Yes. Multimodal RAG uses layout-aware parsers, vision models, and structured data connectors to ingest scans, charts, photos, and database rows. Each modality lands as chunks with shared metadata so the retriever can rank across all of them.

How does RAG fit into an AI governance program?

RAG inherits the same controls as any production AI: data lineage, PII classification, retention, audit logs, human review for low-confidence answers, and an incident response path. The vector index is a regulated data store under SR 11-7, HIPAA, and GLBA.

What is the typical timeline to reach production with enterprise RAG?

A realistic plan runs 12 months. A single-domain pilot with permission-aware retrieval lands in 60 days. Multimodal ingestion and a continuous evaluation harness land by day 180. Multi-domain rollout completes by day 360.

Which deployment pattern fits HIPAA or FCRA workloads?

The closed-model pattern. Model, vector store, embedding service, and audit log sit inside the enterprise perimeter or a sovereign cloud region. Hosted services are limited to roles under a strong data processing agreement.

How do international rules like the EU AI Act, India’s DPDP Act, or Singapore’s Model AI Governance Framework apply?

Each addresses data governance, accuracy, and accountability with details that vary by jurisdiction. Enterprise RAG programs map controls to NIST AI RMF and ISO/IEC 42001, then layer regional rules through data residency, retention, and consent.

The post Enterprise RAG Architecture: The Reference Model appeared first on Data, AI, Automation & Enterprise App Delivery with a Quality-First Partner.