A generation model writes the answer. An embedding model decides which evidence it gets to see. Poor retrieval can make a strong language model look unreliable.
What an embedding model does
An embedding model converts text, images or other inputs into numerical vectors that represent meaning. Similar items should sit near each other in that vector space.
A document assistant embeds stored chunks and the user's question. The system searches for nearby vectors, retrieves candidate passages and sends those passages to the generation model.
Embedding model versus language model
A language model predicts and generates tokens. An embedding model produces a compact representation for search, clustering, recommendations and similarity. They are separate responsibilities and often separate models.
Using a larger chat model cannot recover a relevant paragraph that the embedding search never retrieved.
Evaluate retrieval independently before changing the generation model.
How to choose an embedding model
- Language coverage: use a multilingual model when users and documents span languages.
- Domain fit: legal, code, scientific and general text may rank differently.
- Retrieval direction: confirm whether the model expects query and document prefixes.
- Context and chunk size: long passages may be truncated.
- Vector dimensions: larger vectors consume more storage and search memory.
- License and local runtime support.
- Measured retrieval quality on your own questions.
Practical local candidates
Sentence Transformers provides a broad ecosystem of dense, sparse and reranker models. Compact multilingual and English retrieval models are useful starting candidates for CPU deployments.
EmbeddingGemma is a compact multilingual embedding model designed for on-device use. Static embedding models can be dramatically faster on CPU for some retrieval and similarity workloads, trading some quality for efficiency. Do not select from a leaderboard alone; test your language and documents.
Why reranking improves results
Vector search quickly finds a broad candidate set. A reranker then scores the query and each candidate together, often improving which passages reach the language model.
Reranking costs extra latency, so measure whether it improves top-result accuracy enough for the workflow. A common pattern retrieves a larger set cheaply and reranks a smaller final set.
Build a retrieval evaluation
- Create 30 realistic questions with known source passages.
- Measure whether the correct passage appears in the top 1, top 3 and top 5 results.
- Include abbreviations, paraphrases, multilingual questions and exact terms.
- Test questions whose answer is absent.
- Compare chunking and embedding choices separately.
- Rebuild the index completely when changing embedding models.
Avoid silent incompatibility
Do not mix document vectors from one embedding model with queries from another. Vector dimensions, geometry and learned meaning differ. Store the embedding model name and version with the index.
When the model, preprocessing or chunking changes, create a new index and rerun the same retrieval evaluation. Deviceterra can help institutions turn this into a controlled private knowledge system.
Find a model your computer can run.
The LocalLens advisor applies conservative memory rules and tells you when the evidence is insufficient.
Run the free compatibility check →This guide is educational and reviewed for practical accuracy. Model software, licenses and hardware support change; verify official sources before a production deployment.