Benchmark ya wazi
Imepimwa kwenye kazi halisi za memory kwa vipimo viwili
Tunapima Engram kwa njia mbili: kama inapata evidence sahihi, na kama LLM inaweza kujibu kwa usahihi kutoka kwenye context hiyo.
Tunatenganisha evidence retrieval na usahihi wa jibu. Zote zinatumia LoCoMo ya wazi, lakini hupima sehemu tofauti za memory loop.
Utafutaji wa ushahidi wa LoCoMo
Run kamili ya wazi. Scores zinaonyesha kama source evidence inayohitajika ipo ndani ya memories zilizorejeshwa.
| Mfumo | MRR | R@1 | R@50 | R@200 |
|---|---|---|---|---|
| Engram | 0.5345 | 40.63% | 92.19% | 97.66% |
| LangMem / LangGraph | 0.4002 | 28.32% | 83.98% | 94.53% |
| Mem0 OSS | 0.3999 | 28.32% | 83.92% | 94.27% |
| AgentMemory | 0.4270 | 33.20% | 79.10% | 84.96% |
| BM25 | 0.4676 | 34.57% | 78.19% | 85.74% |
Ukaguzi wa majibu wa LoCoMo
Uchunguzi wa Codex/self-review kwenye kazi zilezile 98. Hupima kama jibu lililotengenezwa kutoka context iliyorejeshwa ni sahihi.
| Mfumo | Jibu top-50 | Jibu top-200 | dhidi ya modeli ya msingi |
|---|---|---|---|
| Engram | 66.33% | 59.18% | +56.13pp |
| Session-file BM25 | 61.22% | 53.06% | +51.02pp |
| Mem0 OSS | 55.10% | 57.14% | +44.90pp |
| BM25 | 48.98% | 44.90% | +38.78pp |
| LangMem / LangGraph | 46.94% | 52.04% | +36.74pp |
| No memory | 10.20% | 9.18% | baseline |
Jinsi ya kusoma R@200 na Top-200
Evidence retrieval ni cumulative: R@200 inaweza kujumuisha yote ya R@50 pamoja na zaidi, kwa hiyo haipaswi kuwa chini. Answer accuracy ni tofauti. Context nyingi inaweza kuongeza noise au memories zinazopingana; Top-50 wakati mwingine hujibu kwa usafi zaidi, huku Top-200 ikipima kama packaging bado inasaidia evidence inapoongezeka.