Making OpenMemory Production-Ready: Hybrid RAG/CAG and Security Hardening

OpenMemory is the open-source, self-hostable memory layer that ships with mem0. It gives an LLM a persistent place to store and recall facts about you, exposed over the Model Context Protocol so clients like Claude, Cursor, and Codex can read and write memories on every turn.

I run it as a real service behind mem0-mcp.trybabble.io, and over two days I took it from “works on my laptop with Docker” to something I am comfortable putting on the public internet with multiple clients hitting it. This post is a tour of what I changed and, more importantly, why. Everything here is something I built and tested against the actual code; where I am generalizing past what I measured, I say so.

The themes were the same ones you hit on any retrieval system that graduates to production: retrieval quality, latency, correctness under concurrency, and not leaking data.

The problem: the MCP server was bypassing the good retrieval

mem0 already supports advanced retrieval. The surprise was that the MCP server in front of it was not using it. Search went straight to a dense vector lookup in Qdrant and returned the top matches. That is fine for a demo and noticeably mediocre the moment your queries are keyword-heavy (“what is my AWS account id”) rather than purely semantic.

A hosted reranker would have been the easy win here, and it would have been better out of the box. But it would also have made “self-hostable” a lie. So the first goal was to restore the retrieval pipeline the underlying library can support, while keeping the default deployment fully self-hostable. No mandatory calls to a hosted reranker, no torch, no GPU.

Hybrid retrieval: dense + BM25, fused with RRF

Dense vectors are great at meaning and bad at exact tokens. BM25 is the opposite. The fix is to run both and merge them.

• Dense retrieval is a semantic kNN search in Qdrant, filtered by user_id on the payload.
• Sparse retrieval is BM25 over a lemmatized query, matched against a lemmatized index so “running” and “ran” land on the same posting. (BM25 needs a Qdrant collection created with a sparse slot; an older dense-only collection silently falls back to dense retrieval, which is one reason the re-embed migration further down matters.)

The interesting part is the merge. Cosine similarity and BM25 scores live on completely different scales, so averaging them is meaningless. Instead I used Reciprocal Rank Fusion, which throws away the raw scores and merges purely on rank position:

score(d) = sum over each ranked list of  1 / (k + rank(d))

with k = 60, the value the original RRF paper settled on. A document that shows up near the top of either list gets a strong score, and you never have to normalize one scoring scheme against the other.

Reranking without torch

Fusion gets you a good candidate pool. The single biggest precision lever after that is a cross-encoder reranker, which re-reads the query and each candidate together (instead of comparing pre-computed vectors) and re-sorts the pool.

The catch is that most rerankers drag in PyTorch and a multi-gigabyte model. For a self-hosted default that is a non-starter. I used a FastEmbed ONNX cross-encoder (Xenova/ms-marco-MiniLM-L-6-v2) that runs in-process with no torch dependency. Cohere and an LLM-based reranker are still there as opt-in upgrades, but it works well out of the box.

One production detail that bit me: FastEmbed has no environment variable for its model cache, so on every cold start it would re-download the ONNX weights. I pass FASTEMBED_CACHE_DIR through explicitly so the model persists across restarts.

A subtle ordering rule runs through the whole pipeline: truncate last. The full fused-and-reranked pool is carried all the way through, and only after ACL and active-state filtering do we cut to the final result limit. Cut too early and a user with restricted access gets a half-empty page.

Here is the whole read path, the part that runs on essentially every user turn:

The thing to notice is that the policy gate (ACL plus active-state) sits after the cache on both paths, hit and miss. That placement is the whole safety argument for the cache, which is the next section.

A CAG cache, and why it is safe

Search is called on essentially every user turn, so it is the hot path. The retrieval pipeline above (embed, two Qdrant queries, fuse, rerank) is not free. So I added a CAG (cache-augmented generation) cache.

What it caches is deliberate: the raw user-scoped candidate pool, post-rerank but pre-policy. The default backend is an in-process LRU with TTL per worker; set OPENMEMORY_REDIS_URL and it transparently moves to Redis so the cache survives restarts and is shared across workers. (It is a dedicated variable on purpose - REDIS_URL already selects something else in this stack, so the cache never piggybacks on it.)

The thing I want to call out is the safety argument, because caching authorization-sensitive data is exactly how leaks happen. To be clear, the cache is not an authorization shortcut: it never stores a final result list. ACL and active-state are re-applied live on every call, even on a cache hit:

• A memory you paused or revoked after it was cached can never surface, because the live filter runs against Postgres after the cache read.
• There is no cross-app leak, because the pool is only ever filtered by user_id, and the per-app accessible set is applied live afterwards.

Invalidation is therefore cheap. It happens on add, delete, and content updates. Pause and archive need no invalidation at all, because the live state filter already handles them.

The critical bug: search was leaking paused and archived memories

This is the one I am most glad I caught. While rewriting search I had dropped the per-memory state == active check. Memories in OpenMemory can be paused or archived, but pausing or archiving does not remove their vectors from Qdrant. Only a hard delete does that.

The result: for any app that had no explicit ACL rules, those paused and archived vectors were still live in the index and were being surfaced in search results. A memory you thought you had hidden could come back. That is a data leak, plainly.

The fix is a live, candidate-scoped active-state filter (filter_results_by_active_state) that runs on every search, including cache hits, and drops anything not currently active. It is one indexed Postgres query scoped to the candidate ids. I backed it with regression tests in test_acl.py so it cannot silently regress again.

While I was in there, I also killed an N+1: ACL resolution used to do a Postgres lookup per memory. Now resolve_accessible_ids does it once per request. None means full access, an empty set short-circuits to “return nothing,” and a populated set scopes the results.

Moving categorization off the request path

Every memory gets an LLM-generated category. The original code did that categorization call synchronously, inside the SQLAlchemy flush of the write. So every write blocked on an LLM round-trip, and worse, it re-fired on every state change, so pausing a memory triggered a fresh categorization call.

I moved it to a post-commit threadpool. The write commits, returns to the client, and categorization happens afterward on its own database session. I also decoupled it from any specific provider and made it lazily initialized.

For bulk imports there is now OPENMEMORY_DISABLE_CATEGORIZATION=true, which skips the per-write LLM call entirely. Migrating thousands of memories should not burn through your provider’s daily request quota on category labels.

Making it genuinely self-hostable

A few changes were about removing hardcoded assumptions that only worked for the original hosted setup.

Embedder choice

Qdrant’s collection defaulted to 1536 dimensions (OpenAI’s size), so the moment you point it at any other embedder every upsert fails on a dimension mismatch. The code change was to make the embedding dimensions and collection name env-driven (the out-of-the-box default is still OpenAI). My production deployment uses that to run a local FastEmbed model (arctic-embed-m, 768d) instead, so in my setup embeddings never leave the box.

Migration

Changing your embedder means re-embedding everything. scripts/reembed.py does an idempotent dual-collection re-embed that also activates the BM25 sparse slot. An early version only re-embedded active memories, which would have stranded paused and archived ones with no vector and broken restorability. It now migrates everything that is not hard-deleted.

Qdrant Cloud

The vector-store auto-detect only passed host and port, which cannot authenticate against Qdrant Cloud. Added a QDRANT_URL + QDRANT_API_KEY branch for the TLS endpoint.

Concurrency and the relational store

Two bugs here, both classic “worked on SQLite in dev, fell over in prod” stories.

Running under --workers 4, the default SQLite journal mode locks the entire database on every write. Metadata inserts were failing with database is locked while the Qdrant vector still landed, leaving orphan vectors with no metadata row. Enabling WAL mode (concurrent readers plus one writer) with a 30s busy_timeout made writers wait instead of erroring.

SQLite is the dev fallback. In production the engine now branches: PostgreSQL gets a bounded, validated connection pool (pool_pre_ping, pool_size, max_overflow, pool_recycle, all env-tunable) suited to multiple uvicorn workers.

The sneakiest one: several queries used DISTINCT ON (Memory.id) together with ORDER BY created_at. SQLite shrugs at this; Postgres rejects it outright with “SELECT DISTINCT ON expressions must match initial ORDER BY expressions.” The DISTINCT only existed to collapse rows multiplied by a category join. The real fix was to stop multiplying rows in the first place: filter categories with an EXISTS subquery and load them with selectinload (a separate query) instead of a joined load. No DISTINCT needed, pagination stays correct, and you can order by any column.

Security and the zero-trust edge

The data plane is self-hosted on Railway, fronted by a Cloudflare Access zero-trust edge on the public hostnames. Two layers of auth back that up:

• An opt-in, constant-time API-key middleware on /mcp and /api/ routes (OPENMEMORY_API_KEY). Constant-time comparison so the check does not leak key length or prefix through timing.
• Env-driven CORS, with credentials disabled whenever the origin is a wildcard, because Access-Control-Allow-Credentials: true with * is both invalid and dangerous.

The dashboard needed care too. Cloudflare Access wants a service token, and that token must never reach the browser. So the Next.js UI now talks only to its own origin at /api/*; a server-side proxy route forwards each request to the real API and injects the CF-Access-Client-Id and CF-Access-Client-Secret headers server-side. Same-origin means no CORS and no cross-domain cookie problems, and the secret stays on the server. (Getting that proxy right also meant following FastAPI’s trailing-slash 307 redirects across same-origin hops, and routing every call through the server’s own configured origin rather than any base URL the browser could see.)

The shape of it

Putting it together, here is the production topology:

• Clients (Claude / Cursor / Codex) speak MCP over HTTP with a Cloudflare service token.
• Cloudflare Access validates the token at the edge and forwards to the origin.
• FastAPI runs the API-key and CORS checks, then the routes.
• The data plane is co-located and self-hosted: Postgres for metadata, ACL, and audit; Qdrant for dense plus BM25 sparse vectors; FastEmbed for both embeddings and reranking, in-process via ONNX; and the CAG cache.
• In my production setup the only hop off-box is to OpenAI’s gpt-4o-mini (the default reasoning model, overridable by env) for the actual reasoning: extracting atomic facts from text on write, and categorization. Because I run a local embedder and reranker, everything else - embeddings, retrieval, fusion, and reranking - stays on-box.

I wrote 34 unit tests across the cache, retrieval, ACL, and reranker, and a deep architecture document with the full write-path and read-path sequence diagrams. The retrieval leak in particular is something I never want to rediscover by accident.

Takeaways

Three things stuck with me:

1. Caching authorization-sensitive data is a footgun unless you separate “what to retrieve” from “what you are allowed to see.” Caching the raw pool and applying policy live on every read is what makes the CAG cache both fast and safe.
2. The dev database lies to you. WAL, DISTINCT ON, and connection pooling all behaved fine on SQLite and only revealed themselves on Postgres under real concurrency. Test against the engine you ship.
3. Self-hostable is a feature you have to defend. It is easy to let an OpenAI embedding dimension or a hosted reranker quietly become load-bearing. Keeping the default fully local took deliberate effort and was worth it.