Most RAG systems look solid in demos and degrade in production.

The gap is rarely the model. It is usually retrieval quality, data design, and lack of evaluation. In practice, over 70% of failures trace back to poor retrieval or chunking decisions rather than the LLM itself.

A production setup has two distinct phases.

Offline indexing handles ingestion, cleaning, chunking, embeddings, and storage. Online query flow handles retrieval, optional query rewriting, reranking, prompt construction, and generation.

Teams that explicitly separate these two stages report 20–40% better debugging efficiency because failures become easier to isolate.

For implementation:

Chunking is the highest leverage decision in most systems.

A practical baseline:

Teams that move from naive large chunks to this range often see +15–25% improvement in retrieval accuracy.

Large chunks dilute relevance. Small chunks break meaning.

For structured sources like APIs, legal text, or code, use structure-aware parsing instead of naive splitting. LlamaIndex supports multiple node parsers:
https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/

Switching models rarely fixes incorrect answers. Retrieval quality does.

A strong default is hybrid search combining semantic vectors with keyword matching. This typically gives +10–30% recall improvement over dense-only retrieval.

Vector databases commonly used:

Add a reranker on top. A cross-encoder typically improves top-k precision by 15–25%, especially when k > 5.

Embedding quality directly impacts retrieval.

Common options:

Switching from weaker embeddings to stronger ones can yield +10–20% retrieval gains without touching the LLM.

Without evaluation, there is no reliable way to improve the system.

You need to measure:

In many real systems, baseline retrieval hit rate starts around 0.6–0.7, which means 30–40% of queries already fail before generation begins.

A practical tool is RAGAS
Docs: https://docs.ragas.io

You also need a dataset. Options include synthetic Q&A generation or public benchmarks:

The common failure path is consistent. Retrieval brings partially relevant context, too many chunks are passed to the model, and the model fills gaps with guesses.

Observed patterns:

Other frequent issues:

These are typical, not edge cases.

A realistic production setup:

Teams using this baseline with tuning typically reach 80–90% answer relevance on internal datasets.

[Image: stack diagram showing each layer from ingestion to evaluation with arrows]

RAG performance is largely determined before the model is called. If results are weak, adjusting chunking, retrieval, and evaluation usually gives 2–3x improvement before any model change is needed.