# Using all-MiniLM-L6-v2 for Local Vector Memory in MCP Servers

> Why SynaBun chose all-MiniLM-L6-v2 over OpenAI embeddings for local AI memory: 384 dimensions, runs on CPU, no API key, with the numbers SynaBun can show.

Source: https://synabun.ai/blog/all-minilm-l6-v2-mcp-vector-memory · By Danilo Khury · Published 2026-04-29 · Updated 2026-10-21

*Updated October 21, 2026: model facts checked against the model card, prices against OpenAI's documentation, and SynaBun's details against its source code. The first version quoted a benchmark against OpenAI that cannot be reproduced from anything published, so it has been replaced by the benchmark SynaBun uses to test its memory engine.*

When I started SynaBun I had to pick an embedding model. The default move would have been `text-embedding-3-small` or `ada-002`. OpenAI hosts them. They are good. They are easy.

I picked a small model called `all-MiniLM-L6-v2` instead. It runs on the user's CPU. It has no API key. It works on a plane.

This post is the long version of why, what the tradeoffs are, and how it actually performs. For how it fits next to other memory servers, see the [MCP memory servers comparison](https://synabun.ai/blog/mcp-memory-servers-comparison) and [the comparison page](https://synabun.ai/compare).

## What `all-MiniLM-L6-v2` is

`all-MiniLM-L6-v2` is a sentence embedding model from the [sentence-transformers](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) project. It was fine-tuned from `MiniLM-L6-H384-uncased` on about 1 billion sentence pairs during a Hugging Face community week. The relevant numbers, from its model card:

- **Architecture:** 6-layer MiniLM (a distilled BERT variant)
- **Embedding dimensions:** 384
- **Max sequence length:** 256 word pieces (longer inputs are truncated)
- **License:** Apache 2.0

SynaBun runs it through Transformers.js with the full-precision ONNX weights (`Xenova/all-MiniLM-L6-v2`, about 90 MB), downloaded once on first use.

The training pairs come from across the web (Reddit, S2ORC, MS MARCO, Yahoo Answers, Quora, NLI datasets and more). It is a general-purpose encoder, not a code-specific one.

## Why local matters for MCP memory

The [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) lets AI coding assistants like Claude Code, Codex, OpenCode, and Gemini call out to external tools. Memory is one of the most important tools because every "remember this" or "recall what I said about X" call has to round-trip somewhere.

When that somewhere is a cloud API, three things happen:

1. **Latency tax.** Every recall waits on a network round trip before the model even sees the result.
2. **Cost tax.** Embeddings are cheap individually, but a heavy coding session calls recall again and again.
3. **Privacy tax.** Your code, project notes, debugging traces, and design decisions get sent to a third party.

For a developer using Claude Code all day, those compound fast. Latency alone is the killer — you cannot stay in flow if every memory recall stalls the session.

Local embeddings remove all three. The cost is on-device compute, which on a modern laptop is small.

## The OpenAI alternative — and why it loses for this job

Here is the head-to-head, as of October 2026:

| | all-MiniLM-L6-v2 | text-embedding-3-small | ada-002 |
|--|--|--|--|
| Dimensions | 384 | 1536 (can be shortened with `dimensions`) | 1536 |
| Cost | $0 | $0.02 per 1M tokens | $0.10 per 1M tokens |
| Runs offline | Yes | No | No |
| API key | None | Required | Required |
| License | Apache 2.0 | Proprietary | Proprietary |

OpenAI's `text-embedding-3-small` scores higher on public benchmarks such as MTEB. For developer memory, recall quality is the one real argument for it — so the useful question is how good recall is with the small local model in practice.

## Benchmark setup

SynaBun has a benchmark that every change to its memory engine must pass (`benchmarks/memory-v2.mjs`, not yet in the public repository). It indexes a fixed corpus (SynaBun's own documentation plus synthetic regression fixtures, 59 documents), runs 23 labeled queries, and compares two recall pipelines that both use `all-MiniLM-L6-v2`:

- **Vector-only:** the original recall — cosine similarity over document vectors, with keyword search only as a fallback
- **Hybrid:** the current recall — keyword search, document vectors and passage refinement, fused by rank

It measures latency, the tokens the results cost the agent, and relevance as nDCG@10 against the labels. A second run adds 10,000 unrelated synthetic documents to test scale. The results below are from September 6, 2026, on an Apple M4 with Node 24. The corpus is engineering fixtures, not production traffic.

## Results

| Recall pipeline | nDCG@10 | p50 latency | p95 latency | Tokens per result set |
|--|--|--|--|--|
| Vector-only | 0.64 | 4.5 ms | 6.4 ms | about 4,700 |
| Hybrid | 0.87 | 1.6 ms | 2.4 ms | about 1,500 |

With 10,000 documents, hybrid recall stayed at 5.8 ms (repeated queries) and 10.5 ms (fresh queries, embedding included) at p95.

Three takeaways:

1. With the same small model, how you retrieve matters as much as which model you use: adding keyword search and passage refinement lifted relevance from 0.64 to 0.87.
2. Local recall costs milliseconds, including the query embedding.
3. This benchmark does not compare against OpenAI's models. If you need that comparison for your own corpus, run your queries against both — the model swap is a few lines.

## Why 384 dimensions is enough

The conventional wisdom is "more dimensions = better recall". It always trades against memory, storage and compute.

For a SynaBun user with 100,000 memories:

- 384 dims × 4 bytes × 100k memories = **about 150 MB** of vectors
- 1536 dims × 4 bytes × 100k memories = **about 600 MB** of vectors

The 4x storage difference matters when the server compares the query against stored vectors in its own process, as SynaBun does (more on that below).

## Storage — plain SQLite instead of a vector DB

SynaBun stores everything in one SQLite file through Node's built-in `node:sqlite`. No Pinecone. No Weaviate. No Qdrant. No extension to compile. Just a single SQLite file at `~/.synabun/mcp-data/memory.db`.

Why:

- **Zero infra.** Users do not run a service.
- **Backupable.** Copy the file. Done.
- **Inspectable.** SQLite is a real database. You can `.dump` it.
- **Embeddable.** The Node MCP server opens it natively.

The core of the schema looks like this (trimmed):

```sql
CREATE TABLE IF NOT EXISTS memories (
  id            TEXT PRIMARY KEY,
  vector        BLOB NOT NULL,              -- 384 floats packed as 1536 bytes
  content       TEXT NOT NULL,
  category      TEXT NOT NULL,
  project       TEXT NOT NULL,
  tags          TEXT NOT NULL DEFAULT '[]', -- JSON array
  importance    INTEGER NOT NULL DEFAULT 5,
  created_at    TEXT NOT NULL,
  accessed_at   TEXT NOT NULL,
  access_count  INTEGER NOT NULL DEFAULT 0
  -- plus subcategory, source, related files, checksums, trash state
);

CREATE VIRTUAL TABLE memory_search USING fts5(
  id UNINDEXED, content, category, project, tags, files,
  tokenize='unicode61 remove_diacritics 2'
);
```

Vectors live in a plain BLOB column and cosine similarity is computed in the server process. As of SynaBun 2.0.0 (Memory v2), recall is hybrid by default: an FTS5 keyword channel, a document-vector channel, and a passage-refinement pass over the best candidates, combined with reciprocal rank fusion:

```js
// each channel contributes by rank, not by raw score
score += 1 / (60 + rank);
```

Rank fusion means an exact identifier found by keyword search and a paraphrase found by the embedding both surface, without having to put BM25 and cosine on the same scale. If the embedding model is missing or fails, keyword recall keeps working. The [docs on the memory engine](https://synabun.ai/docs#section-memory-v2) go through every channel.

## The recency boost

A pure relevance ranking has one big failure mode for developer memory: yesterday's decision and a 6-month-old decision look equally relevant. SynaBun adds an optional recency weight with a 14-day half-life:

```python
def score(fused_score, memory, now):
    age_days = (now - memory.created_at) / 86400
    recency = 0.5 ** (age_days / 14)  # half-life of 14 days
    return fused_score * (1 + 0.15 * recency)
```

Toggleable per query with `recency_boost`, with a configurable default. With hybrid retrieval the boost lifts a brand-new memory by up to 15% and fades with the half-life. The older vector-only path blends 55% recency with 35% similarity plus a small access-count bonus.

## Limitations of `all-MiniLM-L6-v2`

It is not magic. Three things to know:

1. **256 word pieces max.** Anything longer gets truncated. SynaBun splits long memories into tokenizer-checked passages on save and keeps an aggregate document vector for the whole memory.
2. **English-heavy.** It was trained on English data, so recall in Portuguese, Spanish, or Mandarin is weaker. For multilingual users, a multilingual model such as `paraphrase-multilingual-MiniLM-L12-v2` is the usual upgrade.
3. **No code-specific training.** For pure code search (e.g., "find the function that does X"), a code-specific encoder is designed for the job. But SynaBun memories are usually English notes about code, not code itself, so a general-purpose encoder fits.

## When you should use OpenAI instead

I am not religious about this. Use cloud embeddings if:

- Your corpus is very large and recall quality is your bottleneck.
- You are running on infrastructure that cannot host a local model (rare).
- You need cross-lingual recall and do not want to ship a larger multilingual model.

For most MCP memory use cases — a single developer with a few thousand session notes — local is the right answer.

## How to use it in your own MCP server

If you are building your own MCP server and want local embeddings, the path is short. This is the same call SynaBun makes:

```javascript
import { pipeline } from '@huggingface/transformers';

const embedder = await pipeline(
  'feature-extraction',
  'Xenova/all-MiniLM-L6-v2',
  { dtype: 'fp32' }
);

async function embed(text) {
  const output = await embedder(text, {
    pooling: 'mean',
    normalize: true
  });
  return Array.from(output.data);
}
```

That is the whole integration. The model downloads on first use and is cached, and from then on every embed call is local and fast.

## Closing

The cloud-first default for AI tooling is leaving real value on the table. A small model that runs on any recent laptop gets you good recall with no network round trip and no cost per call. For developer memory specifically — where the corpus is small, the queries are frequent, and the privacy stakes matter — local wins.

If you want to see this in action, [SynaBun](https://synabun.ai) is open source under Apache 2.0. The memory pipeline lives in `mcp-server/src/services/` of [its repository](https://github.com/danilokhury/Synabun). Pull it apart, fork it, ship your own — or [install it](https://synabun.ai/docs#section-quick-start) and connect it to your agent.

Related reading:

- [MCP Memory Servers Comparison: SynaBun vs mem0 vs Letta vs Zep](https://synabun.ai/blog/mcp-memory-servers-comparison)
- [Self-Hosted AI Memory: Why Local Beats Cloud for Coding Agents](https://synabun.ai/blog/self-hosted-ai-memory-vs-cloud)
- [SynaBun vs Mem0](https://synabun.ai/synabun-vs-mem0) and [SynaBun vs Letta](https://synabun.ai/synabun-vs-letta)
- [External Models as Agents](https://synabun.ai/blog/external-agents-memory-pipeline)