Benchmark ifunguye
Byapimwe ku mirimo nyayo ya memory mu bipimo bibiri
Dupima Engram mu buryo bubiri: niba ibona evidence ikwiye, kandi niba LLM isubiza neza ikoresheje iyo context.
Dutandukanya evidence retrieval na answer accuracy. Byombi bikoresha LoCoMo ifunguye, ariko bipima ibice bitandukanye bya memory loop.
Gushaka ibimenyetso muri LoCoMo
Run yuzuye ifunguye. Scores zerekana niba evidence ikenewe iri muri memories zagaruwe.
| Sisitemu | MRR | R@1 | R@50 | R@200 |
|---|---|---|---|---|
| Engram | 0.5345 | 40.63% | 92.19% | 97.66% |
| LangMem / LangGraph | 0.4002 | 28.32% | 83.98% | 94.53% |
| Mem0 OSS | 0.3999 | 28.32% | 83.92% | 94.27% |
| AgentMemory | 0.4270 | 33.20% | 79.10% | 84.96% |
| BM25 | 0.4676 | 34.57% | 78.19% | 85.74% |
Gusuzuma ibisubizo muri LoCoMo
Isuzuma rya Codex/self-review ku mirimo 98 imwe. Ipima niba igisubizo cyakozwe kivuye muri context yagaruwe ari cyo.
| Sisitemu | Igisubizo top-50 | Igisubizo top-200 | ugereranyije na model y'ibanze |
|---|---|---|---|
| Engram | 66.33% | 59.18% | +56.13pp |
| Session-file BM25 | 61.22% | 53.06% | +51.02pp |
| Mem0 OSS | 55.10% | 57.14% | +44.90pp |
| BM25 | 48.98% | 44.90% | +38.78pp |
| LangMem / LangGraph | 46.94% | 52.04% | +36.74pp |
| No memory | 10.20% | 9.18% | baseline |
Uko wasoma R@200 na Top-200
Evidence retrieval iba cumulative: R@200 ishobora kubamo ibyo R@50 yabonye n’ibindi, bityo ntikagombye kuba hasi. Answer accuracy iratandukanye. Context nyinshi ishobora kuzana noise cyangwa memories zivuguruzanya; Top-50 rimwe na rimwe itanga igisubizo gisukuye, Top-200 ikareba niba packaging igifasha iyo evidence ibaye nyinshi.