RAG Architecture Patterns: Chunking, Embedding, and Retrieval Strategies

Last Updated: March 20, 2026

Most RAG pipelines underperform because of decisions made before the model ever sees a query. The three core RAG architecture patterns — chunking, embedding, and retrieval — interact in ways most engineering teams don’t account for at design time. A February 2026 benchmark found recursive 512-token splitting outperformed semantic chunking on end-to-end accuracy by 15 points (69% vs. 54%). Hybrid retrieval with cross-encoder reranking consistently beats single-method retrieval by 10-30%. This article covers all three architectural layers and how to sequence your decisions.

What chunking strategy works best for production RAG?

Recursive character splitting at 400-512 tokens with 10-20% overlap is the most reliable baseline for production RAG across general enterprise document types.

LangChain’s RecursiveCharacterTextSplitter and LlamaIndex’s equivalent both implement this pattern. In a February 2026 benchmark across 50 academic papers, it scored 69% end-to-end accuracy. Semantic chunking scored higher on isolated recall (91.9% in Chroma Research’s evaluation) but only 54% end-to-end. That gap shows how isolated recall metrics miss downstream pipeline behavior.

A NAACL 2025 paper concluded the computational overhead of semantic chunking isn’t justified by consistent gains. Fixed 200-word chunks matched or beat semantic chunking across retrieval and generation tasks in their tests.

The exception is domain-specific clinical or legal documents with clear logical structure. A 2025 clinical decision support study found adaptive chunking aligned to topic boundaries hit 87% accuracy versus 13% for a fixed-size baseline. For healthcare EHR notes or structured regulatory filings, document-structure-aware chunking outperforms fixed splits.

Optimal chunk size also varies by query type. Factoid queries work best with 256-512 tokens. Multi-hop analytical queries benefit from 512-1,024 tokens. Keep assembled context under 8K tokens per call. A January 2026 analysis found a “context cliff” around 2,500 tokens where response quality drops measurably.

Which embedding model should I use for enterprise document retrieval?

Select embedding models using MTEB retrieval subtask scores, not overall MTEB scores, because two models with similar overall scores can perform very differently on retrieval tasks.

As of early 2026, top performers on MTEB retrieval subtasks are OpenAI text-embedding-3-large (55.4%) and Cohere English v3 (55.0%). For multilingual deployments, BGE-M3 supports 100+ languages and is the standard open-source choice. E5-Mistral fuses Mistral encoders with E5’s contrastive objective, making it a compact option for self-hosted regulated environments.

Domain-specific fine-tuned embeddings consistently outperform general-purpose models on narrow retrieval tasks. If your corpus is primarily HIPAA-regulated clinical notes or SOX-era financial filings, fine-tuning BGE-M3 on internal documents beats any off-the-shelf option.

What is hybrid retrieval in RAG and why does it outperform dense-only search?

Hybrid retrieval combines dense vector search (semantic similarity) with sparse BM25 keyword search, then fuses results using Reciprocal Rank Fusion (RRF) to consistently outperform either method alone.

On keyword-heavy queries, dense-only retrieval scores 0.58 NDCG. BM25 alone scores 0.88. Hybrid RRF reaches 0.89. For complex mixed queries, hybrid RRF scores 0.85, while the full pipeline with a cross-encoder reranker reaches 0.93. RRF is parameter-free and treats dense and sparse signals equally by converting raw scores to ranks before merging.

Azure AI Search implements native hybrid search with RRF fusion and Microsoft Entra access control out of the box, making it the default choice for Microsoft-stack enterprises. Vertex AI Search (Google Cloud) offers a managed equivalent for GCP deployments.

Does adding a reranker actually improve RAG accuracy?

Yes. Cross-encoder reranking after hybrid retrieval improves accuracy by 33-40% and adds roughly 120ms of latency on average, making it the highest-precision gain available without re-architecting the pipeline.

The standard pattern is to retrieve 50-100 candidates, then rerank to 10. Databricks research shows reranking alone can improve retrieval quality by up to 48%. Cohere Rerank 4 Pro scores 1,627 ELO (vendor-reported) with a 32K context window and support for 100+ languages. ColBERT is the leading open-weights reranker for self-hosted stacks.

Which vector database fits a regulated enterprise RAG stack?

The right vector database depends on your latency requirements, data volume, compliance obligations, and existing infrastructure. Benchmark throughput scores alone won’t tell you the answer.

For regulated industries handling HIPAA-covered data or SOX-era financial records, metadata filtering is the primary access-control mechanism. Tag each chunk with document classification, department, and sensitivity level. Apply those filters before vector similarity is computed. This prevents cross-tenant retrieval errors, a risk that grows sharply in multi-tenant deployments.

On the framework side: LangChain and LangGraph work well for prototyping and agentic orchestration. LlamaIndex adds 35% retrieval accuracy in document-heavy pipelines versus LangChain in 2025 benchmarks. Haystack achieves 99.9% uptime in production reliability tests and is preferred in regulated environments because it supports testable pipeline contracts. A common production pattern is LangChain for early development, LangGraph for orchestration, and Haystack at the evaluation and production layer.

What to do next

Start with recursive chunking at 512 tokens. Run baseline retrieval benchmarks on your own corpus, then layer in hybrid search and a reranker before optimizing embedding models. That sequence surfaces the biggest accuracy gains fastest.

Read next: Retrieval-Augmented Generation (RAG) for Enterprise AI Systems