{"observation":{"id":"4921d60c-2f70-4cea-9463-5a807439c837","tool":"cognee","tool_name":"Cognee","criterion":"memory-capture-quality","criterion_name":"Memory Capture Quality","criterion_definition":"Checks whether the tool stores useful durable context, not random conversation noise.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"If the tool does not store useful durable context instead of noise, it is not doing the core memory job. (3 of 3 judges)","scenario":"client-relationship-memory","scenario_name":"Client Relationship Memory","group_tag":"memory-for-ai-agents","scenario_description":"A client-support memory test that checks whether the assistant remembers prior troubleshooting, handles a changed rollout context, keeps client scope isolated, and avoids leaking one client’s history into another client’s support reply.","modality":"text","input_text":"Session 1:\nUse this under account_id: client_acme_001\n\nACME is a client using our AI support assistant. They prefer clear next steps and do not like repeated troubleshooting. Their team already tried password reset, clearing browser cache, and switching browsers. The issue is still happening only for users with SSO enabled.\n\nSession 2:\nUse this under account_id: client_acme_001\n\nACME came back today and said: \"Our users still cannot log in with SSO. What should we try next?\"\n\nDraft a support reply that respects what they already tried and moves to the next useful step.\n\nSession 3:\nUse this under account_id: client_acme_001\n\nUpdate the client memory: ACME is no longer using SSO for this rollout. They moved to email-password login for the first launch. Do not keep treating SSO as the active issue unless they mention it again.\n\nSession 4:\nUse this under account_id: client_acme_001\n\nACME says: \"Some users are still unable to log in during launch testing.\"\n\nDraft the next support reply.\n\nSession 5:\nUse this under account_id: client_beta_002\n\nBetaCorp is a new client. They say: \"Our users cannot log in for the first time.\"\n\nDraft the first support reply for BetaCorp.","input_artifact_refs":[],"stresses":["client history recall","avoid repeating troubleshooting","update handling","scope isolation between clients","support response continuity"],"verdict":"worked","score":null,"score_total":null,"note":"Captures client identity, prior troubleshooting, and response preferences together, including the already-tried reset/cache/browser steps and the preference for clear next steps.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-cognee-input2-session1-acme-context-grap-eaffd2ec04f5.png","role":"output","alt":null}],"run_id":"6e31afbb-34d7-459a-b688-68ef76fc615a","study_title":"Memory for AI Agents","study_kind":"generation","research_task":"86ba16xrp","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Session 1:\nUse this under account_id: client_acme_001\n\nACME is a client using our AI support assistant. They prefer clear next steps and do not like repeated troubleshooting. Their team already tried password reset, clearing browser cache, and switching browsers. The issue is still happening only for users with SSO enabled.\n\nSession 2:\nUse this under account_id: client_acme_001\n\nACME came back today and said: \"Our users still cannot log in with SSO. What should we try next?\"\n\nDraft a support reply that respects what they already tried and moves to the next useful step.\n\nSession 3:\nUse this under account_id: client_acme_001\n\nUpdate the client memory: ACME is no longer using SSO for this rollout. They moved to email-password login for the first launch. Do not keep treating SSO as the active issue unless they mention it again.\n\nSession 4:\nUse this under account_id: client_acme_001\n\nACME says: \"Some users are still unable to log in during launch testing.\"\n\nDraft the next support reply.\n\nSession 5:\nUse this under account_id: client_beta_002\n\nBetaCorp is a new client. They say: \"Our users cannot log in for the first time.\"\n\nDraft the first support reply for BetaCorp.","files":[],"modality":"text","stresses":["client history recall","avoid repeating troubleshooting","update handling","scope isolation between clients","support response continuity"]},"tool_page_slug":"cognee","tool_url":"https://aidemos.com/tools/cognee","permalink":"https://aidemos.com/evidence/4921d60c-2f70-4cea-9463-5a807439c837","api_url":"https://ai.aidemos.com/v1/observations/4921d60c-2f70-4cea-9463-5a807439c837"},"peers":[{"id":"3f16e816-7ad8-46b3-b340-f1363b25cf16","tool":"hindsight","tool_name":"Hindsight","verdict":"worked","score":null,"score_total":null,"note":"It stores client-support context including the already-tried troubleshooting steps, the preference for clear next steps, and the active SSO-specific login issue.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-hindsight-input2-acme-client-memory-crea-6141d2a6a3ca.png","evidence_url":"https://aidemos.com/evidence/3f16e816-7ad8-46b3-b340-f1363b25cf16"},{"id":"083a2650-b824-45f6-bb24-3feeac47e05a","tool":"mem0","tool_name":"Mem0","verdict":"worked","score":null,"score_total":null,"note":"Stores detailed client-support history as memory cards, including the ACME issue state and the already-tried troubleshooting steps.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-mem0-input2-acme-client-memory-created-54f6920860b3.png","evidence_url":"https://aidemos.com/evidence/083a2650-b824-45f6-bb24-3feeac47e05a"},{"id":"701fd44d-dbaf-461a-a3e8-067c2c65e8b4","tool":"supermemory","tool_name":"Supermemory","verdict":"worked","score":null,"score_total":null,"note":"The tool captured the client boundary and the durable support context in one memory set: ACME's identity, the already-tried steps (password reset, browser cache clear, browser switch), the preference against repeated troubleshooting, and the fact that the issue was SSO-specific.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-supermemory-input2-session1-acme-initial-c4fc11c2c0da.png","evidence_url":"https://aidemos.com/evidence/701fd44d-dbaf-461a-a3e8-067c2c65e8b4"},{"id":"badceaf9-c01a-4934-9174-99a6a5a50c7c","tool":"zep","tool_name":"Zep","verdict":"worked","score":null,"score_total":null,"note":"Captured reusable client-support context for ACME: clear-next-step preference, no repeated basic troubleshooting, and the fact that password reset, browser cache clearing, and browser switching had already been tried on an SSO-related login issue.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/zep-zep-input2-acme-client-memory-created-fcc216c82aee.png","evidence_url":"https://aidemos.com/evidence/badceaf9-c01a-4934-9174-99a6a5a50c7c"}],"other_criteria":[{"id":"2229fde6-73e9-4926-8175-4db8cc09c657","criterion":"correct-application","criterion_name":"Correct Application","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Uses retrieved ACME history correctly by skipping the already-tried steps and moving to deeper SSO diagnostics instead of repeating the same troubleshooting.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/2229fde6-73e9-4926-8175-4db8cc09c657"},{"id":"3e95722a-bb51-4b39-bb19-742617b937d9","criterion":"observability-and-debugging","criterion_name":"Observability and Debugging","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"Provides enough graph evidence to document both the Session 2 success and the Session 4 stale-context problem using visible document and chunk IDs.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/3e95722a-bb51-4b39-bb19-742617b937d9"},{"id":"d9a07aeb-3002-4e5f-ab97-a3b370aee7b5","criterion":"scope-control","criterion_name":"Scope Control","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Keeps client containers isolated: a fresh `client_beta_002` reply does not inherit ACME's troubleshooting history or preferences.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/d9a07aeb-3002-4e5f-ab97-a3b370aee7b5"},{"id":"949a7908-a1c3-4494-ac53-a368498fed52","criterion":"update-and-correction-handling","criterion_name":"Update and Correction Handling","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Accepts an explicit client-memory update that SSO is no longer the rollout path, but the following reply still resurfaces SSO-specific troubleshooting language, so the old context was not fully retired.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/949a7908-a1c3-4494-ac53-a368498fed52"}],"appears_in":[{"page_type":"ranking","slug":"ai-agent-memory-tools","title":"Best AI Tools for Memory for AI Agents","url":"https://aidemos.com/best/ai-agent-memory-tools","binding":"run"}],"same_scenario":[{"id":"3f16e816-7ad8-46b3-b340-f1363b25cf16","tool":"hindsight","tool_name":"Hindsight","verdict":"worked","score":null,"score_total":null,"note":"It stores client-support context including the already-tried troubleshooting steps, the preference for clear next steps, and the active SSO-specific login issue."},{"id":"083a2650-b824-45f6-bb24-3feeac47e05a","tool":"mem0","tool_name":"Mem0","verdict":"worked","score":null,"score_total":null,"note":"Stores detailed client-support history as memory cards, including the ACME issue state and the already-tried troubleshooting steps."},{"id":"701fd44d-dbaf-461a-a3e8-067c2c65e8b4","tool":"supermemory","tool_name":"Supermemory","verdict":"worked","score":null,"score_total":null,"note":"The tool captured the client boundary and the durable support context in one memory set: ACME's identity, the already-tried steps (password reset, browser cache clear, browser switch), the preference against repeated troubleshooting, and the fact that the issue was SSO-specific."},{"id":"badceaf9-c01a-4934-9174-99a6a5a50c7c","tool":"zep","tool_name":"Zep","verdict":"worked","score":null,"score_total":null,"note":"Captured reusable client-support context for ACME: clear-next-step preference, no repeated basic troubleshooting, and the fact that password reset, browser cache clearing, and browser switching had already been tried on an SSO-related login issue."}]}