Benchmark ya wazi
Imepimwa kwenye kazi halisi za memory kwa vipimo viwili
Tunapima Engram kwa njia mbili: kama inapata evidence sahihi, na kama LLM inaweza kujibu kwa usahihi kutoka kwenye context hiyo.
Tunatenganisha evidence retrieval na usahihi wa jibu. Zote zinatumia LoCoMo ya wazi, lakini hupima sehemu tofauti za memory loop.
Utafutaji wa ushahidi wa LoCoMo
Run kamili ya wazi. Scores zinaonyesha kama source evidence inayohitajika ipo ndani ya memories zilizorejeshwa.
| Mfumo | MRR | R@1 | R@50 | R@200 |
|---|---|---|---|---|
| Engram | 0.5427 | 40.95% | 93.29% | 97.59% |
| Mem0 OSS | 0.4314 | 31.25% | 85.16% | 94.79% |
| LangMem / LangGraph | 0.4006 | 28.39% | 83.98% | 94.53% |
| AgentMemory | 0.4627 | 35.68% | 83.33% | N/A |
| Session-file BM25 | 0.1858 | 9.11% | 81.90% | 95.05% |
| BM25 | 0.4676 | 36.39% | 78.19% | 85.74% |
Ukaguzi wa majibu wa LoCoMo
Kila mstari unatumia kazi zote 1,536 zenye ushahidi na kikomo kilekile cha top-50. Codex ilitengeneza majibu na mkaguzi tofauti asiyejua mfumo akayapima. Huu ni uchunguzi ulioendeshwa na Engram, si orodha rasmi ya LoCoMo.
| Mfumo | Sahihi / N | Jibu top-50 |
|---|---|---|
| Engram | 1,377 / 1,536 | 89.65% |
| Mem0 OSS | 1,294 / 1,536 | 84.24% |
| AgentMemory | 1,280 / 1,536 | 83.33% |
| LangMem / LangGraph | 1,279 / 1,536 | 83.27% |
| Session-file BM25 | 1,235 / 1,536 | 80.40% |
| BM25 | 1,227 / 1,536 | 79.88% |
Jinsi ya kusoma majedwali mawili
Vipimo viwili vilivyochapishwa hutumia kazi zote 1,536 zenye ushahidi. Urejeshaji unahitaji vitambulisho vya chanzo, na modeli zilezile za majibu na tathmini hutumiwa kwa kila mfumo uliojumuishwa. Mifumo isiyotimiza masharti sawa huandikwa kando badala ya kuchanganywa kwenye orodha.
Jinsi ya kusoma majedwali mawili