Does better memory mean better answers?

01

Two tests. Two different questions.

Imagine an agent checking whether a production refund needs an administrator's approval. A retrieval test asks whether the relevant policy appears in the returned memories. An answer test asks whether the agent actually gives the correct approval rule. Finding the policy helps, but does not guarantee that the agent reads it correctly or chooses it over an outdated note.

Recall@50 measures whether the required evidence appears among the first 50 results. Mean reciprocal rank rewards finding it earlier. Answer accuracy scores the answer produced from that context. Each metric answers a different question; a strong memory system needs both useful retrieval and evidence that the retrieved context improves the answer.

02

What we measured on 1,536 questions

We used the full set of 1,536 evidence-bearing questions in our LoCoMo diagnostic. Every compared system used the same question set and top-50 context cutoff. Codex generated answers, and a separate blind judging pass scored them using the same model family. The answer generator did not receive gold answers or system names; the judge did not receive the memory system's identity or retrieval context. Compatible prior blind judgments were reused for 2,086 of the 10,752 evaluated system-question pairs.

This is an Engram-run diagnostic, not an official leaderboard or independent endorsement. The downloadable artifact records the results. The table below uses the same source as our public benchmark page, so its numbers remain consistent with the published evidence.

03

Why more retrieved context can produce worse answers

A larger candidate set can include the missing fact and raise recall. It can also add adjacent project history, duplicate notes, and older decisions. The agent then has to distinguish the current rule from information that is related but no longer applicable. The extra evidence and the extra noise arrive together.

That is why a top-200 retrieval score can exceed top-50 while top-200 answer accuracy falls. This is not a contradiction. It is a reminder to evaluate the answer as well as the search. Our product work focuses on scope, relevance, and current decisions, with deeper retrieval available when a focused result is insufficient.

More context is not the same as better context.

04

A fair comparison needs more than percentages

Use the same dataset, split, corpus, query, cutoff, answer model, judge, and scoring rule. Publish sample counts and failures. Label results that came from a vendor, a different protocol, or a partial run. A missing score means unmeasured; it should not be displayed as zero.

Measure latency at the same boundary. A local in-process search and a hosted API with a network round trip are different deployment choices. Report cold and warm behavior, median and tail latency, and the hardware or hosting path. A hosted service can add team sharing and managed operations, but those benefits do not erase its latency cost.

05

What this benchmark does not prove

LoCoMo tests questions over conversation history. It does not directly prove that an agent will ship better code, recover a production incident faster, or use fewer tokens. Our current numbers also do not establish that Engram beats every memory product: the comparison covers the systems and versions represented in the shared run.

Helpful memory reports are valuable product signals, but they are self-selected and are not precision measurements. Token and time savings need a paired task experiment: the same model, repository snapshot, tool budget, and task with memory enabled and disabled. We plan to report those measurements separately once the experiment is complete.

06

Choose memory by the work you need it to preserve

For a long-running project, test whether the agent recalls the current architectural decision, resumes open work after compaction, and avoids a previously documented gotcha. For a team, test whether a lesson from one repository is useful in another without exposing private data outside the workspace.

Start with a real decision your agent has already made. Save the durable lesson, begin a fresh session, and ask a concrete question about it. Verify the recalled guidance against the repository or live system. That small check tells you something a headline benchmark percentage cannot: whether the memory helps with your work.