Lessons in private RAG deployment

Versions and access before vectors

Tie each chunk to its document, version and source permissions. Design update and deletion without leaving accessible copies.

Test strategies against real questions

Semantic, lexical and hybrid search may behave differently for names, codes and technical language. Compare methods on an authorised set of questions and expected sources.

Citations, abstention and errors

Check whether answers cite enough evidence, decline unsupported questions and respect access profiles. Measure time and cost at expected load, not on one query.

What to record in a comparison

Prepare questions involving exact names, ambiguous terms and document version changes. For each retrieval approach, retain identifiers of returned passages and check whether the right source appears before judging the generated wording. This separates a retrieval failure from a model-writing failure.

Repeat with users who have different permissions and after withdrawing a document. Check whether the index or cache still exposes copies. A good aggregate answer score can hide one serious access failure; do not reduce that decision to an average accuracy number.

Deployment and capacity

If retrieval and generation run in an owned environment, size them separately. A single-user test does not show what happens when several people ask long questions at once. Observe memory, queues, response time and index size; plan backup and restoration for documents and metadata. Separating services can make model or index changes easier, but adds operational work. The right boundary depends on workload and the team maintaining the system.

A selection record, not a supplier list

To compare a search store or embedding model, keep the same authorised corpus, questions and access configuration. Record for each option: correct documents among the first results, exact-name performance, index size, update and deletion time, operating cost and backup/restore method. Avoid changing several components at once if you need to attribute an improvement. Microsoft’s retrieval guidance explains why the search stage deserves assessment before judging the final answer.

A retrieved document may also carry malicious instructions for the assistant. In a controlled test, place a fictional instruction inside a lower-trust source and check that the system treats it as data, not an order to reveal documents or take action. This differs from a relevance failure; OWASP documents prompt injection in LLM applications. Passing this test does not establish complete security; it defines one behaviour to recheck after changes.

The RAG architecture guide covers source preparation and a synthetic acceptance matrix. This article focuses on choosing and operating components once a document use case is defined.