{"observation":{"id":"93a526cd-3eea-4b8b-84d0-5e3221af8c8d","tool":"zep","tool_name":"Zep","criterion":"scope-control","criterion_name":"Scope Control","criterion_definition":"Checks whether memory is separated correctly across user, project, customer, workspace, or agent boundaries.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"Correct separation across users, projects, workspaces, or agents is fundamental to whether retrieved memory is actually the right memory. (3 of 3 judges)","scenario":"client-relationship-memory","scenario_name":"Client Relationship Memory","group_tag":"memory-for-ai-agents","scenario_description":"A client-support memory test that checks whether the assistant remembers prior troubleshooting, handles a changed rollout context, keeps client scope isolated, and avoids leaking one client’s history into another client’s support reply.","modality":"text","input_text":"Session 1:\nUse this under account_id: client_acme_001\n\nACME is a client using our AI support assistant. They prefer clear next steps and do not like repeated troubleshooting. Their team already tried password reset, clearing browser cache, and switching browsers. The issue is still happening only for users with SSO enabled.\n\nSession 2:\nUse this under account_id: client_acme_001\n\nACME came back today and said: \"Our users still cannot log in with SSO. What should we try next?\"\n\nDraft a support reply that respects what they already tried and moves to the next useful step.\n\nSession 3:\nUse this under account_id: client_acme_001\n\nUpdate the client memory: ACME is no longer using SSO for this rollout. They moved to email-password login for the first launch. Do not keep treating SSO as the active issue unless they mention it again.\n\nSession 4:\nUse this under account_id: client_acme_001\n\nACME says: \"Some users are still unable to log in during launch testing.\"\n\nDraft the next support reply.\n\nSession 5:\nUse this under account_id: client_beta_002\n\nBetaCorp is a new client. They say: \"Our users cannot log in for the first time.\"\n\nDraft the first support reply for BetaCorp.","input_artifact_refs":[],"stresses":["client history recall","avoid repeating troubleshooting","update handling","scope isolation between clients","support response continuity"],"verdict":"worked","score":null,"score_total":null,"note":"Kept client scope isolated: BetaCorp under a different account ID did not inherit ACME's troubleshooting history and received a first-touch support reply instead.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/zep-zep-input2-betacorp-scope-isolation-1fdc416168e7.png","role":"output","alt":null}],"run_id":"6e31afbb-34d7-459a-b688-68ef76fc615a","study_title":"Memory for AI Agents","study_kind":"generation","research_task":"86ba16xrp","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Session 1:\nUse this under account_id: client_acme_001\n\nACME is a client using our AI support assistant. They prefer clear next steps and do not like repeated troubleshooting. Their team already tried password reset, clearing browser cache, and switching browsers. The issue is still happening only for users with SSO enabled.\n\nSession 2:\nUse this under account_id: client_acme_001\n\nACME came back today and said: \"Our users still cannot log in with SSO. What should we try next?\"\n\nDraft a support reply that respects what they already tried and moves to the next useful step.\n\nSession 3:\nUse this under account_id: client_acme_001\n\nUpdate the client memory: ACME is no longer using SSO for this rollout. They moved to email-password login for the first launch. Do not keep treating SSO as the active issue unless they mention it again.\n\nSession 4:\nUse this under account_id: client_acme_001\n\nACME says: \"Some users are still unable to log in during launch testing.\"\n\nDraft the next support reply.\n\nSession 5:\nUse this under account_id: client_beta_002\n\nBetaCorp is a new client. They say: \"Our users cannot log in for the first time.\"\n\nDraft the first support reply for BetaCorp.","files":[],"modality":"text","stresses":["client history recall","avoid repeating troubleshooting","update handling","scope isolation between clients","support response continuity"]},"tool_page_slug":"zep","tool_url":"https://aidemos.com/tools/zep","permalink":"https://aidemos.com/evidence/93a526cd-3eea-4b8b-84d0-5e3221af8c8d","api_url":"https://ai.aidemos.com/v1/observations/93a526cd-3eea-4b8b-84d0-5e3221af8c8d"},"peers":[{"id":"d9a07aeb-3002-4e5f-ab97-a3b370aee7b5","tool":"cognee","tool_name":"Cognee","verdict":"worked","score":null,"score_total":null,"note":"Keeps client containers isolated: a fresh `client_beta_002` reply does not inherit ACME's troubleshooting history or preferences.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-cognee-input2-session5-betacorp-scope-is-83a68ad5f357.png","evidence_url":"https://aidemos.com/evidence/d9a07aeb-3002-4e5f-ab97-a3b370aee7b5"},{"id":"92c2bced-a602-47d0-b908-31556b9d054a","tool":"hindsight","tool_name":"Hindsight","verdict":"worked","score":null,"score_total":null,"note":"It keeps BetaCorp isolated as a new client bank, so ACME's troubleshooting history does not leak into the first BetaCorp reply.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-hindsight-input2-betacorp-scope-isolatio-5008d90d2a2b.png","evidence_url":"https://aidemos.com/evidence/92c2bced-a602-47d0-b908-31556b9d054a"},{"id":"cd37895b-f945-41a5-bd15-694b8e29ce52","tool":"mem0","tool_name":"Mem0","verdict":"worked","score":null,"score_total":null,"note":"Keeps client scope separated so BetaCorp does not inherit ACME's prior troubleshooting history or SSO context.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-mem0-input2-betacorp-scope-isolation-4d43e1d65825.png","evidence_url":"https://aidemos.com/evidence/cd37895b-f945-41a5-bd15-694b8e29ce52"},{"id":"d121c23e-d6c3-4b68-b261-ae56ec91ddf7","tool":"supermemory","tool_name":"Supermemory","verdict":"worked","score":null,"score_total":null,"note":"The tool kept client memories isolated: the separate BetaCorp container showed 'No memories retrieved yet' and did not inherit ACME's SSO history, troubleshooting steps, or support preferences.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-supermemory-input2-session5-betacorp-sco-bb3cb7769629.png","evidence_url":"https://aidemos.com/evidence/d121c23e-d6c3-4b68-b261-ae56ec91ddf7"}],"other_criteria":[{"id":"badceaf9-c01a-4934-9174-99a6a5a50c7c","criterion":"memory-capture-quality","criterion_name":"Memory Capture Quality","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Captured reusable client-support context for ACME: clear-next-step preference, no repeated basic troubleshooting, and the fact that password reset, browser cache clearing, and browser switching had already been tried on an SSO-related login issue.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/badceaf9-c01a-4934-9174-99a6a5a50c7c"},{"id":"4845da75-9170-4894-9f2a-e69e56444b0d","criterion":"relevant-retrieval","criterion_name":"Relevant Retrieval","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"On the follow-up, it retrieved the right ACME history and skipped the three already-tried basics, moving straight to higher-value SSO/IdP troubleshooting.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/4845da75-9170-4894-9f2a-e69e56444b0d"},{"id":"8a3bac73-a767-41c2-a295-f2f33f5aacf4","criterion":"update-and-correction-handling","criterion_name":"Update and Correction Handling","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"After a correction that ACME had moved from SSO to email-password login, the next reply still reused SSO/SAML/IdP troubleshooting, showing that one explicit update did not retire the stale context.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/8a3bac73-a767-41c2-a295-f2f33f5aacf4"}],"appears_in":[{"page_type":"ranking","slug":"ai-agent-memory-tools","title":"Best AI Tools for Memory for AI Agents","url":"https://aidemos.com/best/ai-agent-memory-tools","binding":"run"}],"same_scenario":[{"id":"d9a07aeb-3002-4e5f-ab97-a3b370aee7b5","tool":"cognee","tool_name":"Cognee","verdict":"worked","score":null,"score_total":null,"note":"Keeps client containers isolated: a fresh `client_beta_002` reply does not inherit ACME's troubleshooting history or preferences."},{"id":"92c2bced-a602-47d0-b908-31556b9d054a","tool":"hindsight","tool_name":"Hindsight","verdict":"worked","score":null,"score_total":null,"note":"It keeps BetaCorp isolated as a new client bank, so ACME's troubleshooting history does not leak into the first BetaCorp reply."},{"id":"cd37895b-f945-41a5-bd15-694b8e29ce52","tool":"mem0","tool_name":"Mem0","verdict":"worked","score":null,"score_total":null,"note":"Keeps client scope separated so BetaCorp does not inherit ACME's prior troubleshooting history or SSO context."},{"id":"d121c23e-d6c3-4b68-b261-ae56ec91ddf7","tool":"supermemory","tool_name":"Supermemory","verdict":"worked","score":null,"score_total":null,"note":"The tool kept client memories isolated: the separate BetaCorp container showed 'No memories retrieved yet' and did not inherit ACME's SSO history, troubleshooting steps, or support preferences."}]}