Build a private document assistant with local AI

Choose one collection and one audience

Start with a controlled collection such as staff policies, product manuals, approved procedures or public reports. Name the users and the questions the assistant should answer. Do not ingest an entire shared drive.

Remove duplicates, expired versions, unsupported scans and documents with unclear ownership. A smaller trustworthy collection produces a clearer test than a large messy archive.

How retrieval works

The system extracts document text, breaks it into chunks, converts chunks into searchable representations and stores them in an index. At question time it retrieves relevant passages and sends only those passages to the model.

Open WebUI describes this as retrieval-augmented generation. LM Studio also supports attaching common document formats for local chat. Retrieval is not training; the underlying model does not permanently learn the documents.

Build the minimum pipeline

• Store approved source documents locally.
• Extract text and inspect tables, headings and scanned pages.
• Chunk by meaning and preserve source metadata.
• Create a local search index.
• Retrieve a small set of relevant passages.
• Prompt the model to answer only from evidence and cite sources.
• Show the source passage and allow “not found” as an answer.

Evaluate retrieval before the model

Create questions with known source passages. First check whether the correct passage appears in the retrieved results. If it does not, changing the language model may not fix the problem.

Then evaluate the answer for faithfulness, completeness and citation accuracy. Include questions whose answer is absent and permission tests where a user must not see a document.

Move from one person to a team

• Add user identity and role-based collection access.
• Back up source documents and indexes securely.
• Log access and failures without storing unnecessary sensitive text.
• Protect the model and retrieval APIs behind the application.
• Create a document-review and deletion process.
• Re-evaluate after every model, embedding, chunking or prompt change.

Decide whether to build or get help

A single-user experiment can be assembled with tools such as LM Studio or Open WebUI. A production system needs access control, monitoring, evaluation, recovery and ownership.

Use LocalLens to choose the model and launch plan. If the assistant will hold important institutional knowledge, connect with Deviceterra for a scoped private deployment.