{"observation":{"id":"4ac77e78-6ef0-411b-9306-dd49a5425c07","tool":"zep","tool_name":"Zep","criterion":"scope-control","criterion_name":"Scope Control","criterion_definition":"Checks whether memory is separated correctly across user, project, customer, workspace, or agent boundaries.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"Correct separation across users, projects, workspaces, or agents is fundamental to whether retrieved memory is actually the right memory. (3 of 3 judges)","scenario":"team-handoff-project-continuity-memory","scenario_name":"Team Handoff / Project Continuity Memory","group_tag":"memory-for-ai-agents","scenario_description":"A project continuity and handoff test where the assistant must remember project goals, project rules, and a changed testing direction, then produce a useful handoff note for an unavailable team member without leaking context into an unrelated project.","modality":"text","input_text":"Session 1:\nUse this under project_id: ai_demos_memory_use_case\n\nWe are working on an AI Demos use case called Memory for AI Agents. The goal is to help users understand which memory tools are actually useful for real agent workflows. We are not promoting any tool. We are testing whether memory can help with real continuity: personal work brain, client relationship memory, and team handoff.\n\nSession 2:\nUse this under project_id: ai_demos_memory_use_case\n\nImportant project rules:\n- No observation without proof.\n- Screenshots and artifacts are primary evidence.\n- Inputs must help rank tools, not just prove that tools can store one fact.\n- Memory should be checked for retrieval, update handling, scope control, deletion or retirement, and observability.\n- The page should stay practical and user-facing, not only technical.\n\nSession 3:\nUse this under project_id: ai_demos_memory_use_case\n\nProject direction changed slightly. The old input set was too QA-style and not relatable enough. The new direction is to use real workflows: personal work brain memory, client relationship memory, and team handoff/project continuity memory.\n\nSession 4:\nUse this under project_id: ai_demos_memory_use_case\n\nI am unavailable tomorrow. Create a handoff note for an intern who needs to continue this use case. The note should explain:\n1. What this use case is about.\n2. What the current testing direction is.\n3. What rules they must follow before writing observations.\n4. What artifacts they need to capture while testing.\n\nSession 5:\nUse this under project_id: unrelated_sales_agent_project\n\nWe are building a sales email agent for a different project. Create a short kickoff note for the team.","input_artifact_refs":[],"stresses":["project-level memory","decision and rule retention","changed direction handling","handoff continuity","scope separation across projects"],"verdict":"worked","score":null,"score_total":null,"note":"Kept unrelated project memory separate: the sales-agent kickoff did not inherit the AI Demos memory-tool testing rules or project context.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/zep-zep-input3-project-direction-updated-e2d8864deb8e.png","role":"input","alt":"Zep input3 project direction updated.png"},{"url":"https://d3epheqghktydj.cloudfront.net/zep-zep-input3-unrelated-project-scope-isola-830b91ac67ea.png","role":"output","alt":null}],"run_id":"6e31afbb-34d7-459a-b688-68ef76fc615a","study_title":"Memory for AI Agents","study_kind":"generation","research_task":"86ba16xrp","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Session 1:\nUse this under project_id: ai_demos_memory_use_case\n\nWe are working on an AI Demos use case called Memory for AI Agents. The goal is to help users understand which memory tools are actually useful for real agent workflows. We are not promoting any tool. We are testing whether memory can help with real continuity: personal work brain, client relationship memory, and team handoff.\n\nSession 2:\nUse this under project_id: ai_demos_memory_use_case\n\nImportant project rules:\n- No observation without proof.\n- Screenshots and artifacts are primary evidence.\n- Inputs must help rank tools, not just prove that tools can store one fact.\n- Memory should be checked for retrieval, update handling, scope control, deletion or retirement, and observability.\n- The page should stay practical and user-facing, not only technical.\n\nSession 3:\nUse this under project_id: ai_demos_memory_use_case\n\nProject direction changed slightly. The old input set was too QA-style and not relatable enough. The new direction is to use real workflows: personal work brain memory, client relationship memory, and team handoff/project continuity memory.\n\nSession 4:\nUse this under project_id: ai_demos_memory_use_case\n\nI am unavailable tomorrow. Create a handoff note for an intern who needs to continue this use case. The note should explain:\n1. What this use case is about.\n2. What the current testing direction is.\n3. What rules they must follow before writing observations.\n4. What artifacts they need to capture while testing.\n\nSession 5:\nUse this under project_id: unrelated_sales_agent_project\n\nWe are building a sales email agent for a different project. Create a short kickoff note for the team.","files":[],"modality":"text","stresses":["project-level memory","decision and rule retention","changed direction handling","handoff continuity","scope separation across projects"]},"tool_page_slug":"zep","tool_url":"https://aidemos.com/tools/zep","permalink":"https://aidemos.com/evidence/4ac77e78-6ef0-411b-9306-dd49a5425c07","api_url":"https://ai.aidemos.com/v1/observations/4ac77e78-6ef0-411b-9306-dd49a5425c07"},"peers":[{"id":"867be637-ec54-428e-ae07-df3a446b5a91","tool":"cognee","tool_name":"Cognee","verdict":"worked","score":null,"score_total":null,"note":"Keeps an unrelated `unrelated_sales_agent_project` container separate from the AI Demos project, with the kickoff note staying on the sales-agent topic and not inheriting AI Demos or BetaCorp context.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-cognee-input3-session5-unrelated-project-383c32601f17.png","evidence_url":"https://aidemos.com/evidence/867be637-ec54-428e-ae07-df3a446b5a91"},{"id":"2cb22d1a-734c-4a92-b9b9-783b1320d1f7","tool":"hindsight","tool_name":"Hindsight","verdict":"worked","score":null,"score_total":null,"note":"It keeps the unrelated sales-agent project separate, so the AI Demos memory rules do not bleed into a different project bank.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-hindsight-input3-unrelated-project-scope-413787ca4ea3.png","evidence_url":"https://aidemos.com/evidence/2cb22d1a-734c-4a92-b9b9-783b1320d1f7"},{"id":"c55a0b58-7ae4-4fbc-9faa-0102880ce84b","tool":"mem0","tool_name":"Mem0","verdict":"worked","score":null,"score_total":null,"note":"Keeps an unrelated project isolated, so the sales-agent kickoff note does not inherit the AI Demos memory rules.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-mem0-input3-unrelated-project-scope-isol-b2593f270962.png","evidence_url":"https://aidemos.com/evidence/c55a0b58-7ae4-4fbc-9faa-0102880ce84b"},{"id":"7d020390-5977-43bb-bbb8-52189e7d3b5c","tool":"supermemory","tool_name":"Supermemory","verdict":"worked","score":null,"score_total":null,"note":"A separate unrelated project container stayed cleanly isolated: the kickoff note for the sales-email project contained no trace of the AI Demos memory categories, rules, or terminology.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-supermemory-input3-session5-unrelated-pr-a99632bb75a5.png","evidence_url":"https://aidemos.com/evidence/7d020390-5977-43bb-bbb8-52189e7d3b5c"}],"other_criteria":[{"id":"724dae08-39dc-4526-8d82-66495d788467","criterion":"memory-capture-quality","criterion_name":"Memory Capture Quality","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Captured project continuity context including the Memory for AI Agents goal, the three real-work scenarios, and proof-first evaluation rules covering screenshots, ranking value, retrieval, update handling, scope control, deletion or retirement, and observability.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/724dae08-39dc-4526-8d82-66495d788467"},{"id":"44e46e9f-c98b-4f26-b917-be56a69c2043","criterion":"relevant-retrieval","criterion_name":"Relevant Retrieval","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Retrieved earlier project context for the handoff and reused the use case, current direction, rules, and artifact requirements in the generated note.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/44e46e9f-c98b-4f26-b917-be56a69c2043"},{"id":"1236e92d-a34e-4711-a40e-e90c9446a247","criterion":"update-and-correction-handling","criterion_name":"Update and Correction Handling","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"After the project direction changed from QA-style inputs to real workflows, the handoff reflected the newer direction rather than the older test framing.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/1236e92d-a34e-4711-a40e-e90c9446a247"}],"appears_in":[{"page_type":"ranking","slug":"ai-agent-memory-tools","title":"Best AI Tools for Memory for AI Agents","url":"https://aidemos.com/best/ai-agent-memory-tools","binding":"run"}],"same_scenario":[{"id":"867be637-ec54-428e-ae07-df3a446b5a91","tool":"cognee","tool_name":"Cognee","verdict":"worked","score":null,"score_total":null,"note":"Keeps an unrelated `unrelated_sales_agent_project` container separate from the AI Demos project, with the kickoff note staying on the sales-agent topic and not inheriting AI Demos or BetaCorp context."},{"id":"2cb22d1a-734c-4a92-b9b9-783b1320d1f7","tool":"hindsight","tool_name":"Hindsight","verdict":"worked","score":null,"score_total":null,"note":"It keeps the unrelated sales-agent project separate, so the AI Demos memory rules do not bleed into a different project bank."},{"id":"c55a0b58-7ae4-4fbc-9faa-0102880ce84b","tool":"mem0","tool_name":"Mem0","verdict":"worked","score":null,"score_total":null,"note":"Keeps an unrelated project isolated, so the sales-agent kickoff note does not inherit the AI Demos memory rules."},{"id":"7d020390-5977-43bb-bbb8-52189e7d3b5c","tool":"supermemory","tool_name":"Supermemory","verdict":"worked","score":null,"score_total":null,"note":"A separate unrelated project container stayed cleanly isolated: the kickoff note for the sales-email project contained no trace of the AI Demos memory categories, rules, or terminology."}]}