A RAG demo takes an afternoon. A RAG system your customers can trust takes discipline. Here's the checklist we use.
Retrieval-augmented generation (RAG) lets a language model answer questions from your own documents. It is the most common enterprise AI pattern we build. It is also the easiest to get subtly wrong.
1. Start with an evaluation set
Before you tune anything, collect 50–200 real questions with known good answers. Every change you make to chunking, retrieval or prompting should be measured against this set. Without it, you're guessing.
2. Get the documents right
- Clean extraction matters more than model choice: tables, headers and scanned PDFs need care.
- Chunk by meaning (sections, headings), not just by character count.
- Store metadata such as source, date and access level alongside every chunk.
3. Retrieval is where most quality is won
Hybrid search (keyword plus vector) consistently beats either on its own. Add a re-ranking step for the top results, and filter by user permissions before anything reaches the model.
4. Ground and cite
Instruct the model to answer only from retrieved context and to cite its sources. Show those citations in the UI so users can verify answers in one click.
5. Plan for cost and latency
Cache frequent queries, choose the smallest model that passes your evaluation set, and track cost per conversation from day one.
6. Monitor in production
Log questions, retrieved chunks and user feedback. Review low-rated answers weekly and add them to your evaluation set. That loop is what makes the system better over time.