Knowledge AI with RAG: making company knowledge usable under the GDPR

In almost every company, the most important knowledge sits in manuals, contracts, policies, tickets and emails, spread across shared drives, wikis and inboxes. Anyone who needs an answer searches for a long time or asks the colleague who "has always known". A general-purpose AI chat does not help here: it does not know your documents and, when in doubt, invents a plausible answer. Retrieval-augmented generation, RAG for short, solves exactly this problem. This guide explains how RAG works, what matters for data protection and how to introduce a knowledge AI safely.

In short: For every question, RAG first finds the matching sections in your own documents and lets the language model answer only on that basis, with a source citation. That makes answers verifiable and hallucinations much rarer. A knowledge AI becomes GDPR-compliant through hosting in the EU or in your own data centre, a data processing agreement and access rights that also apply to search.

What RAG is

The term comes from the research paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis et al. from 2020. The idea: a language model does not rely solely on the knowledge that ended up in its weights during training, but fetches matching passages from an external, searchable index for every query. The authors showed that this produces more specific and more factual answers than generation without retrieval.

For companies this brings three practical advantages:

• Freshness: New documents can be used immediately, without retraining a model.
• Traceability: Every answer can point to the passages it comes from.
• Control: You decide which documents go into the index and who may see which content.

Terms such as embedding or language model are briefly explained in the glossary (in German).

How a RAG pipeline works

A knowledge AI has two phases: preparing the documents and answering questions.

Phase 1: Preparing the documents

Ingestion and cleaning. PDFs, Office files, wiki pages and emails are converted to text. Tables, headers and scanned pages are the typical stumbling blocks here. Whatever is lost during ingestion, the AI cannot find later.

Chunking. The texts are split into sections, so-called chunks. Chunks that are too large dilute the search; chunks that are too small tear context apart. A proven approach is to split along the document structure, by chapters, paragraphs or clauses, and to carry metadata such as document title, page and section.

Embeddings. An embedding model translates each chunk into a numerical vector that represents its meaning. Texts with similar content sit close together in vector space, even if they use different words.

Vector database. The vectors are stored together with their metadata in a vector database. Qdrant, for example, describes itself in its documentation as a vector search database that supports lexical and hybrid search alongside semantic search, can filter by metadata and, being open source, can also run in your own data centre.

Phase 2: Answering questions

1. The question is also translated into a vector.
2. The vector database returns the most similar chunks, filtered by the user's permissions.
3. Optionally, a reranker sorts the hits again by relevance.
4. The language model receives the question plus the retrieved sections and the instruction to answer only on that basis.
5. The answer is delivered with source citations, ideally down to document, page and section.

Source citations: the most important safeguard against hallucinations

Language models write fluently even when they are wrong. RAG reduces this risk but does not eliminate it completely. Three rules are decisive:

• Answer only from sources. The model is instructed to use only the sections provided.
• An honest "I don't know". If the search finds nothing suitable, the AI says so instead of guessing.
• A verifiable reference. Every statement points to where it was found, so users can check it with one click.

The last point in particular changes acceptance. Employees do not have to trust the AI blindly; they can check it. In legal departments, quality management and customer service, that is often the precondition for the system being used at all.

Data protection and hosting

As soon as documents contain personal data, and that is almost always the case, the GDPR applies. For a knowledge AI this mainly leads to the following tasks:

• Data processing: You need a data processing agreement with every service provider that processes data, such as the hosting or model provider.
• Third-country transfers: If data leaves the EU, the transfer must be safeguarded. The simplest approach is to run the index, embeddings and language model in the EU.
• Purpose limitation and deletion: Deleted or outdated documents must also disappear from the index.
• Access rights: Anyone not allowed to see a document in the original must not be able to see it via the AI either.

The last point is often underestimated. Under Vector and Embedding Weaknesses, the OWASP project describes exactly these risks: insufficient access controls, data leakage between tenants in shared vector databases and manipulated content entering the knowledge base. As countermeasures it lists permission-aware vector stores, validating documents before they are ingested and complete logs of retrievals.

Which deployment model fits?

Deployment model	Control over data	Effort	Suitable for
SaaS with EU hosting	contractually safeguarded	low	most companies, quick start
Private cloud in the EU	high, own instance	medium	sensitive data, own security requirements
On-premises	complete, in your own data centre	high	critical infrastructure (KRITIS), hospitals, law firms, classified information

The role digital sovereignty plays here is explored in the article AI sovereignty in the enterprise (in German). The fundamentals of ethics and data protection are covered in AI ethics and data protection (in German).

How to recognise a good knowledge AI

Whether you compare vendors or build your own, these criteria help:

Quality of source citations. Does the answer point only to a document, or to the specific page and section? Can the reference be opened with one click?

Permissions in search. Are permissions taken over from your systems and applied to every search? Is there a connection to your identity management via SSO, SAML or SCIM?

Model choice. Are you tied to one language model, or can you choose between providers depending on the data class, for example European models for particularly sensitive content?

Indexing. How quickly are new documents searchable? How well are tables, scans and long documents processed?

Response time under load. Don't look at the average, look at the slowest responses, for example the 99th percentile. An AI that occasionally takes ten seconds will not be used.

Deployment models. Is it possible to move from SaaS to on-premises if requirements change?

A general checklist for choosing a vendor is offered in the article Choosing an AI vendor (in German).

Getting started in four steps

1. Choose one knowledge area

Do not start with "the whole shared drive", but with a clearly defined collection that attracts many questions, such as product manuals for support or internal policies for HR.

2. Review the data

Outdated, duplicate and contradictory documents are the most common cause of poor answers. Clean up the starting collection and decide who maintains the documents. Guidance on this is in the article AI data strategy (in German).

3. Define test questions

Collect 50 to 100 real questions with the correct answers and source locations. Use them to measure whether the knowledge AI finds the right source and answers correctly. Without such a test set, every assessment remains gut feeling. Which metrics to derive from it, from Recall@k to citation accuracy, is shown in the article Measuring RAG quality.

4. Pilot with real users

Give the knowledge AI to a small group, collect feedback on every answer and improve chunking, search and instructions. Only then expand to further areas. How a knowledge AI fits into a broader knowledge strategy is described in the article AI in knowledge management (in German). Combined with agents, it becomes the foundation for automated workflows: an agent that answers customer enquiries or prepares cases draws on the same evidence-backed knowledge base. How to introduce agents with clear permissions and approvals is described in the guide AI agents in business.

Common mistakes in knowledge AI projects

Indexing everything at once. If you load the entire shared drive into the index, you bring outdated drafts, duplicates and contradictory versions into the system. The AI then finds something, but often the wrong thing. A curated starting collection delivers better results than a complete one.

Trying to solve permissions afterwards. If access rights are only added after the pilot, the architecture is usually not designed for them. Permissions belong in the metadata of every chunk and in every search query from the start.

Looking only at the language model. Most quality problems do not arise in the model but in ingesting and splitting the documents. A lost table or an unreadable scan cannot be compensated for by any model, however good.

No ownership for upkeep. A knowledge AI is only as current as its documents. Decide who approves new content and removes outdated content, otherwise answer quality declines gradually. How to anchor such roles and approvals in your organisation for the long term is described in the AI department guide.

From my own products

With anirag.io I built a GDPR-compliant knowledge AI, and the most important design decision was the source citation: every answer points down to page and section, so users can check it instead of having to believe it. I use Qdrant as the vector database. An LLM router chooses between Anthropic, OpenAI EU, Mistral and Aleph Alpha depending on requirements, so sensitive content does not have to go to every model. 12,000 PDFs are indexed in around 30 seconds, and response time at the 99th percentile is under 800 ms. For companies with identity management there is SSO, SAML and SCIM, and for critical infrastructure (KRITIS), hospitals and law firms an on-premises variant. My lesson: users do not accept a knowledge AI because of clever wording, but because they can verify every statement.

If you want to make your company knowledge usable in this way, you will find my approach under Knowledge AI with RAG.

Frequently asked questions

What does RAG mean?

RAG stands for retrieval-augmented generation. For each question, the method first searches a knowledge base for matching sections and then lets the language model answer on that basis. This allows the AI to use your company knowledge without training a model of your own.

Does RAG prevent hallucinations completely?

No, but it reduces the risk considerably. If the model may only answer from retrieved sources, honestly declines when there are no hits and backs every statement with a reference, errors can be spotted quickly. A test set of real questions helps measure quality continuously.

Can a knowledge AI with RAG be GDPR-compliant?

Yes. The prerequisites are a legal basis for the processing, data processing agreements with all service providers, hosting in the EU or in your own data centre where possible, deletion concepts for the index and access rights that also apply to search.

Do I need my own language model for RAG?

No. RAG works with existing language models, because the company knowledge is supplied from the index with every query. You neither have to train the model nor hand over your documents to the model provider for training. What matters is that the provider contractually guarantees not to use your data for training.

What is the difference between RAG and fine-tuning?

Fine-tuning retrains a model on additional data to adapt its style or domain language. RAG leaves the model unchanged and supplies current facts from your documents with every question. For company knowledge that changes and has to be backed by evidence, RAG is usually the better choice.