Public benchmark
Measured on real memory tasks with two checks
Engram is measured two ways: first, whether it retrieves the right evidence; second, whether an LLM can answer correctly from that retrieved context.
We keep retrieval and answer accuracy separate. Both use public LoCoMo, but they measure different parts of the memory loop.
LoCoMo evidence retrieval
Full public run. Scores show whether the required source evidence appears in the retrieved memories.
| System | MRR | R@1 | R@50 | R@200 |
|---|---|---|---|---|
| Engram | 0.5345 | 40.63% | 92.19% | 97.66% |
| LangMem / LangGraph | 0.4002 | 28.32% | 83.98% | 94.53% |
| Mem0 OSS | 0.3999 | 28.32% | 83.92% | 94.27% |
| AgentMemory | 0.4270 | 33.20% | 79.10% | 84.96% |
| BM25 | 0.4676 | 34.57% | 78.19% | 85.74% |
LoCoMo answer-level judge
Codex/self-reviewed diagnostic on the same 98 tasks. Scores show whether the generated answer is correct from retrieved context.
| System | Top-50 answer | Top-200 answer | vs base model |
|---|---|---|---|
| Engram | 66.33% | 59.18% | +56.13pp |
| Session-file BM25 | 61.22% | 53.06% | +51.02pp |
| Mem0 OSS | 55.10% | 57.14% | +44.90pp |
| BM25 | 48.98% | 44.90% | +38.78pp |
| LangMem / LangGraph | 46.94% | 52.04% | +36.74pp |
| No memory | 10.20% | 9.18% | baseline |
How to read R@200 and Top-200
Evidence retrieval is cumulative: R@200 can include everything R@50 found plus more, so it should not be lower. Answer accuracy is different. More retrieved context can also add noise or conflicting memories, so Top-50 may answer cleaner while Top-200 tests whether packaging stays useful with more evidence.