A guide to local-first RAG

Prepare traceable documents

Inventory formats, versions, owners and permissions. Keep identifiers that support citing a section and removing a document unambiguously.

Compare methods rather than assume one

Test chunking, lexical and semantic search on representative questions. Preserve enough context to interpret an answer and apply access filters before presenting it.

Define when to decline

Include answerable, unanswerable and restricted-document questions. Check citations, freshness, abstention and per-role failures. Repeat after changing the model, index or sources.

A small test before expanding

Use synthetic or authorised documents. Prepare one policy available to the whole team, one note restricted to a role and one question with no answer in any source. Write down the expected retrieved passage and acceptable response before running the test. Ask the same questions under different identities. The test fails if an unauthorised role sees restricted content, even when the final answer paraphrases it instead of citing it.

Record document versions, retrieval settings, model and date. After changing chunking or embeddings, run the set again; an improvement on one question may worsen another. The purpose is to learn where the assistant is correct, declines to answer and crosses an access boundary.

Chunking and retrieval

Compare section or paragraph chunks with fixed-size chunks on the same questions. Preserve title, version and text position; without context a citation may be impossible to verify. For exact identifiers, project names or codes, test lexical as well as semantic search. If combining them, tune weights against the evaluation set rather than adopting a universal recommended number.

An illustrative acceptance matrix

This example is synthetic and has not been executed. Imagine two sources: P-1, a travel policy available to all staff, and R-2, a budget note restricted to finance. P-0 names a policy that does not exist. The application recognises staff and finance roles. Before testing, write down the expected source for each question and which output would count as a failure.

Then withdraw R-2 and run the tests again: index, cache and citations should reflect the change. Record retrieved sources, final answer and access decision separately so a failure can be located in retrieval, authorisation or generation. Microsoft’s guides on retrieval evaluation and answer evaluation separate those layers. Agree on thresholds for the actual task rather than inheriting them from this example.