How to Debug a Bad RAG Pipeline
A step-by-step approach to isolating whether a RAG failure comes from retrieval or generation, with the specific things worth logging along the way.
Separate Retrieval From Generation Before You Do Anything Else
A RAG pipeline has two distinct stages, retrieval and generation, and a bad final answer can come from either one. Debugging effectively means splitting the problem in two immediately: look at what was retrieved before you look at what was generated.
It is tempting to iterate on the prompt when an answer looks wrong, but if the retrieved context genuinely did not contain the answer, no amount of prompt engineering will fix it. Checking retrieval first saves you from optimizing the wrong stage.
What to Log and Inspect
At minimum, log the original query, the retrieved chunks with their similarity scores, and the final generated answer for every request, or at least a sampled subset in production. This turns a vague complaint like the answer was wrong into a specific, inspectable trace you can actually diagnose.
Once you can see the retrieved chunks, manually check whether the answer to the failing query is actually present in them. If it is present but the model ignored it, that is a generation or prompt problem; if it is absent, that is a retrieval problem, and no prompt change will fix it.
- Log query, retrieved chunks, similarity scores, and final answer together
- Manually verify whether the correct answer exists in the retrieved chunks
- Test generation in isolation by manually inserting the correct chunk
- Test retrieval in isolation by scoring recall at k against a labeled query set
Building a Repeatable Debugging Loop
One-off debugging of individual failures does not scale once you have real traffic. Collecting failing queries into a small evaluation set lets you rerun the same test cases every time you change chunking, the embedding model, or the prompt, so you can see whether a fix actually helped or just shifted the failure elsewhere.
Over time this evaluation set becomes one of the most valuable assets in the project, because it is the only thing that turns is this better than before into a question with a real answer instead of a guess based on a handful of manual spot checks.
Key takeaways
- Apply one concrete change from this post before collecting more reading.
- Prefer browser-side tools when the work involves secrets, tokens, or PII.
- Document the why next to the how so the next reviewer inherits context.
FAQ
- Who is this guide on rag for?
- Working developers who need a practical take on how to debug a bad rag pipeline — not a marketing overview. Skim the sections, apply one tip, then come back when you hit an edge case.
- Do I need an account to use the related tools?
- No. code.live tools run in your browser with no signup. Nothing you paste is uploaded to a server for the client-side utilities linked from this post.
- How often is this article updated?
- This post was published November 19, 2026. Fundamentals stay stable; check linked tool pages and official docs when version-specific behavior matters.