← All articles
AI · 10 min read

RAG explained: how to build a chatbot that answers from your own documents

A plain-language guide to retrieval-augmented generation: how it works, when you need it, what it costs to build, and the mistakes that make AI assistants give wrong answers.

Ask a general AI model about your company's refund policy and it will either refuse or, worse, invent a plausible answer. It simply hasn't seen your documents. Retrieval-augmented generation (RAG) solves this: before answering, the system finds the relevant pieces of your own content and gives them to the model as context.

How RAG works

QuestionSearchTop fragmentsPrompt + contextLLM answerSources
A RAG request: find the relevant fragments first, then let the model answer using only them.
  1. Preparation (once, then on every update): documents are split into fragments (chunks), and each fragment is turned into a numeric vector (an embedding) that captures its meaning. Vectors are stored in a vector database.
  2. Search: the user's question is turned into a vector too, and the database returns the fragments closest in meaning — often combined with classic keyword search.
  3. Generation: the model receives the question plus the found fragments and instructions such as “answer only from the context, cite sources, say you don't know if the answer isn't there”.

When you need RAG — and when you don't

SituationGood fit?
Support bot answering from help-centre articles and policiesYes
Internal assistant searching wikis, contracts, technical docsYes
Sales assistant answering from product catalogues and price listsYes, with fresh data sync
Calculations, order status, account dataBetter via direct API calls (tools), not RAG
Creative writing without company factsNot needed

What determines answer quality

  • Source quality: outdated or contradictory documents produce outdated or contradictory answers. Cleaning the knowledge base is often half the project.
  • Chunking: fragments that are too small lose context; too large ones dilute the search. Splitting by headings and sections usually works better than by fixed length.
  • Search: hybrid search (meaning + keywords) and re-ranking of results noticeably improve relevance, especially for product codes and names.
  • Instructions: the model must be told to stay within the context and admit when it doesn't know.
  • Access rights: a user must only get answers from documents they are allowed to see.
The most common failure is not the model — it's retrieval. If the right fragment isn't found, even the best model can't answer correctly.

Typical tech stack

LayerOptions
ModelsOpenAI GPT, Anthropic Claude, Google Gemini, or open-source models such as Llama or Mistral
Embeddings and vector storagepgvector in PostgreSQL, Qdrant, Pinecone, Weaviate
OrchestrationLangChain, LlamaIndex or a lean custom service
InterfaceWebsite widget, Telegram or Slack bot, internal portal
QualityEvaluation set, logging, feedback buttons, monitoring

How we build a RAG assistant

  1. Discovery: which questions should it answer, from which sources, for whom.
  2. Knowledge audit: remove duplicates and outdated documents, define update rules.
  3. Prototype in 1–2 weeks on real documents and real questions.
  4. Evaluation: 50–150 test questions with expected answers; we measure accuracy, groundedness and “I don't know” behaviour.
  5. Integration and launch: widget or bot, access rights, analytics.
  6. Improvement loop: every bad answer from production becomes a new test case.

Costs and privacy

Running costs depend mostly on the number of questions and the model chosen; for most small and medium businesses they are modest compared to support staff time. If data must not leave your infrastructure, the whole stack can run on your servers with open-source models.

RAG turns a general AI into your expert: it answers with your facts, shows where they came from, and admits when it doesn't know. That's the difference between a demo and a tool your customers can trust.