RAG explained: how to build a chatbot that answers from your own documents
A plain-language guide to retrieval-augmented generation: how it works, when you need it, what it costs to build, and the mistakes that make AI assistants give wrong answers.

Ask a general AI model about your company's refund policy and it will either refuse or, worse, invent a plausible answer. It simply hasn't seen your documents. Retrieval-augmented generation (RAG) solves this: before answering, the system finds the relevant pieces of your own content and gives them to the model as context.
How RAG works
- Preparation (once, then on every update): documents are split into fragments (chunks), and each fragment is turned into a numeric vector (an embedding) that captures its meaning. Vectors are stored in a vector database.
- Search: the user's question is turned into a vector too, and the database returns the fragments closest in meaning — often combined with classic keyword search.
- Generation: the model receives the question plus the found fragments and instructions such as “answer only from the context, cite sources, say you don't know if the answer isn't there”.
When you need RAG — and when you don't
| Situation | Good fit? |
|---|---|
| Support bot answering from help-centre articles and policies | Yes |
| Internal assistant searching wikis, contracts, technical docs | Yes |
| Sales assistant answering from product catalogues and price lists | Yes, with fresh data sync |
| Calculations, order status, account data | Better via direct API calls (tools), not RAG |
| Creative writing without company facts | Not needed |
What determines answer quality
- Source quality: outdated or contradictory documents produce outdated or contradictory answers. Cleaning the knowledge base is often half the project.
- Chunking: fragments that are too small lose context; too large ones dilute the search. Splitting by headings and sections usually works better than by fixed length.
- Search: hybrid search (meaning + keywords) and re-ranking of results noticeably improve relevance, especially for product codes and names.
- Instructions: the model must be told to stay within the context and admit when it doesn't know.
- Access rights: a user must only get answers from documents they are allowed to see.
Typical tech stack
| Layer | Options |
|---|---|
| Models | OpenAI GPT, Anthropic Claude, Google Gemini, or open-source models such as Llama or Mistral |
| Embeddings and vector storage | pgvector in PostgreSQL, Qdrant, Pinecone, Weaviate |
| Orchestration | LangChain, LlamaIndex or a lean custom service |
| Interface | Website widget, Telegram or Slack bot, internal portal |
| Quality | Evaluation set, logging, feedback buttons, monitoring |
How we build a RAG assistant
- Discovery: which questions should it answer, from which sources, for whom.
- Knowledge audit: remove duplicates and outdated documents, define update rules.
- Prototype in 1–2 weeks on real documents and real questions.
- Evaluation: 50–150 test questions with expected answers; we measure accuracy, groundedness and “I don't know” behaviour.
- Integration and launch: widget or bot, access rights, analytics.
- Improvement loop: every bad answer from production becomes a new test case.
Costs and privacy
Running costs depend mostly on the number of questions and the model chosen; for most small and medium businesses they are modest compared to support staff time. If data must not leave your infrastructure, the whole stack can run on your servers with open-source models.


