Benchmark landscape
The agent-memory benchmark landscape.
Before writing our own memory-quality benchmark we read the ones the field already uses. The short version: almost every headline number is the vendor’s own paper, none has been cleanly reproduced by a neutral third party, and almost none of them measure the thing we care about — analysis over memory. This page is the honest map, with every number linked to its source.
What each benchmark measures
| Benchmark | Measures | Scale | Grading | Isolates analysis? |
|---|---|---|---|---|
| Deep Memory Retrieval (DMR) | Whether a memory system can retrieve one specific fact planted earlier in a long conversation — single-fact recall, end to end. | 500 conversations, one target fact each. | Answer-correctness against the planted fact (LLM-judge). | ✗ No |
| LoCoMo | End-to-end QA over very long multi-session dialogue: single-hop, multi-hop, temporal, and open-domain questions. | 50 dialogues, ~300 turns over up to 35 sessions; 1,540 QA samples. | String-match F1 and LLM-as-judge on the final answer. | ✗ No |
| LongMemEval | Five separately-scored abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. | 500 curated questions embedded in scalable chat histories. | Per-ability answer accuracy (LLM-judge). | ✗ No |
The headline numbers — and who reported them
| System | Benchmark | Claim | Status |
|---|---|---|---|
| Mem0 | LoCoMo | 26% relative improvement in LLM-as-judge accuracy over OpenAI's memory. Mem0 (own paper, ECAI 2025) — arXiv:2504.19413 | Self-reported |
| Zep | Deep Memory Retrieval (DMR) | 94.8% accuracy, above MemGPT's 93.4%. Zep (own paper) — arXiv:2501.13956 | Self-reported |
| Zep | LongMemEval | 63.8%, versus 49.0% attributed to Mem0 — a comparison Mem0 disputes. Zep (own paper) — arXiv:2501.13956 | Disputed |
The reproducibility reality
The most-cited suite, LoCoMo, was independently audited in 2025 — the first neutral third-party check of any of these numbers. It does not hold up as a settled scoreboard:
- ≈6.4% of the answer key is simply wrong — 99 score-corrupting errors across 1,540 questions.
- The LLM judge accepts up to 63% of intentionally-wrong answers, so a high score can mean the grader is lenient, not the system accurate.
- The honest ceiling on the suite as published is roughly 93–94%, not 100% — reported near-perfect scores are inside the noise of a flawed key.
Penfield Labs — “We audited LoCoMo” . The two leading vendors also publicly dispute each other’s LoCoMo methodology — treat every leaderboard number here as directional, not settled.
The thing none of them measure: analysis over memory
Every system in the field stores facts and retrieves them. A vector store can recall "Alice has a $50k deal." What none of these benchmarks isolate — and most fuzzy fact stores structurally cannot run — is analysis over that memory: "the top 5 deals by value, grouped by stage, for enterprise accounts only." That is a GROUP BY with a HAVING and a JOIN, and a similarity index has no query planner for it.
We could not find a single published benchmark that isolates analytical queries over episodic memory against vector or graph memory on identical data. The field measures recall of facts, not reasoning across them — which is why every row in the analysis column above is a ✗.
There is even evidence for the gap inside LoCoMo's own results: restructuring raw dialogue into a database of assertions lifted temporal-question F1 from 21.3 to 41.9. Structure helps most exactly where LLMs are weakest.
Where a database does not win
A real database is not a free win everywhere. For fuzzy semantic recall over unstructured text — "find the thing I said that's kind of like this" — embedding similarity still beats exact SQL, and every serious system relies on it. The honest wedge is analytical memory, not memory without embeddings.
What nlqdb is adding — and what we will not claim yet
We are adding an agent-memory-quality eval to the same harness we already run for text-to-SQL accuracy: four axes (retrieval precision/recall against relevance labels we define, temporal reasoning, forgetting and contradiction resolution, consolidation) plus the analytical task nobody else runs — aggregation and ordering over episodic memory, head-to-head against a vector-recall baseline on identical data.
We have not published our own numbers, so none appear on this page. When they land they will be reproducible and reported honestly, including the questions where a pure-SQL store loses. Until then, treat every leaderboard number in this space — ours included — as directional.
Landscape re-verified against the cited sources on 2026-08-06.