Benchmark ifunguye
Byapimwe ku mirimo nyayo ya memory mu bipimo bibiri
Dupima Engram mu buryo bubiri: niba ibona evidence ikwiye, kandi niba LLM isubiza neza ikoresheje iyo context.
Dutandukanya evidence retrieval na answer accuracy. Byombi bikoresha LoCoMo ifunguye, ariko bipima ibice bitandukanye bya memory loop.
Gushaka ibimenyetso muri LoCoMo
Run yuzuye ifunguye. Scores zerekana niba evidence ikenewe iri muri memories zagaruwe.
| Sisitemu | MRR | R@1 | R@50 | R@200 |
|---|---|---|---|---|
| Engram | 0.5427 | 40.95% | 93.29% | 97.59% |
| Mem0 OSS | 0.4314 | 31.25% | 85.16% | 94.79% |
| LangMem / LangGraph | 0.4006 | 28.39% | 83.98% | 94.53% |
| AgentMemory | 0.4627 | 35.68% | 83.33% | N/A |
| Session-file BM25 | 0.1858 | 9.11% | 81.90% | 95.05% |
| BM25 | 0.4676 | 36.39% | 78.19% | 85.74% |
Gusuzuma ibisubizo muri LoCoMo
Buri murongo ukoresha imirimo 1,536 yose ifite ibimenyetso n’umupaka umwe wa top-50. Codex yakoze ibisubizo, umusuzumyi wihariye utazi sisitemu arabipima. Iri ni isuzuma ryakozwe na Engram, si urutonde rwemewe rwa LoCoMo.
| Sisitemu | Ibisubizo byiza / N | Igisubizo top-50 |
|---|---|---|
| Engram | 1,377 / 1,536 | 89.65% |
| Mem0 OSS | 1,294 / 1,536 | 84.24% |
| AgentMemory | 1,280 / 1,536 | 83.33% |
| LangMem / LangGraph | 1,279 / 1,536 | 83.27% |
| Session-file BM25 | 1,235 / 1,536 | 80.40% |
| BM25 | 1,227 / 1,536 | 79.88% |
Uko wasoma imbonerahamwe zombi
Ibipimo bibiri byatangajwe bikoresha imirimo 1,536 yose ifite ibimenyetso. Gushaka ibimenyetso bisaba ibiranga inkomoko, kandi buri sisitemu ikoreshwa na model zimwe zo gusubiza no gusuzuma. Sisitemu zituzuza amasezerano amwe y’isuzuma zandikwa ukwazo aho kuvangwa ku rutonde.
Uko wasoma imbonerahamwe zombi