{"observation":{"id":"296fccb1-080f-4f96-af1b-0af153818878","tool":"supermemory","tool_name":"Supermemory","criterion":"update-and-correction-handling","criterion_name":"Update and Correction Handling","criterion_definition":"Checks whether the tool can replace or invalidate outdated memories when corrected.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"A memory system has to replace or invalidate outdated information, or it will keep feeding the agent wrong context. (3 of 3 judges)","scenario":"client-relationship-memory","scenario_name":"Client Relationship Memory","group_tag":"memory-for-ai-agents","scenario_description":"A client-support memory test that checks whether the assistant remembers prior troubleshooting, handles a changed rollout context, keeps client scope isolated, and avoids leaking one client’s history into another client’s support reply.","modality":"text","input_text":"Session 1:\nUse this under account_id: client_acme_001\n\nACME is a client using our AI support assistant. They prefer clear next steps and do not like repeated troubleshooting. Their team already tried password reset, clearing browser cache, and switching browsers. The issue is still happening only for users with SSO enabled.\n\nSession 2:\nUse this under account_id: client_acme_001\n\nACME came back today and said: \"Our users still cannot log in with SSO. What should we try next?\"\n\nDraft a support reply that respects what they already tried and moves to the next useful step.\n\nSession 3:\nUse this under account_id: client_acme_001\n\nUpdate the client memory: ACME is no longer using SSO for this rollout. They moved to email-password login for the first launch. Do not keep treating SSO as the active issue unless they mention it again.\n\nSession 4:\nUse this under account_id: client_acme_001\n\nACME says: \"Some users are still unable to log in during launch testing.\"\n\nDraft the next support reply.\n\nSession 5:\nUse this under account_id: client_beta_002\n\nBetaCorp is a new client. They say: \"Our users cannot log in for the first time.\"\n\nDraft the first support reply for BetaCorp.","input_artifact_refs":[],"stresses":["client history recall","avoid repeating troubleshooting","update handling","scope isolation between clients","support response continuity"],"verdict":"struggled","score":null,"score_total":null,"note":"After ACME's rollout context was updated away from SSO, the UI still showed a '0 memory used' tag while the side panel continued to surface the old SSO memory at 71% and then 79% relevance on later replies, so it is not possible to confirm from the UI that the update reliably superseded the stale context.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-supermemory-input2-session3-acme-memory--fd8e39cbf155.png","role":"output","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-supermemory-input2-session4-acme-launch--75f2538cb5ac.png","role":"output","alt":null}],"run_id":"6e31afbb-34d7-459a-b688-68ef76fc615a","study_title":"Memory for AI Agents","study_kind":"generation","research_task":"86ba16xrp","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Session 1:\nUse this under account_id: client_acme_001\n\nACME is a client using our AI support assistant. They prefer clear next steps and do not like repeated troubleshooting. Their team already tried password reset, clearing browser cache, and switching browsers. The issue is still happening only for users with SSO enabled.\n\nSession 2:\nUse this under account_id: client_acme_001\n\nACME came back today and said: \"Our users still cannot log in with SSO. What should we try next?\"\n\nDraft a support reply that respects what they already tried and moves to the next useful step.\n\nSession 3:\nUse this under account_id: client_acme_001\n\nUpdate the client memory: ACME is no longer using SSO for this rollout. They moved to email-password login for the first launch. Do not keep treating SSO as the active issue unless they mention it again.\n\nSession 4:\nUse this under account_id: client_acme_001\n\nACME says: \"Some users are still unable to log in during launch testing.\"\n\nDraft the next support reply.\n\nSession 5:\nUse this under account_id: client_beta_002\n\nBetaCorp is a new client. They say: \"Our users cannot log in for the first time.\"\n\nDraft the first support reply for BetaCorp.","files":[],"modality":"text","stresses":["client history recall","avoid repeating troubleshooting","update handling","scope isolation between clients","support response continuity"]},"tool_page_slug":"supermemory","tool_url":"https://aidemos.com/tools/supermemory","permalink":"https://aidemos.com/evidence/296fccb1-080f-4f96-af1b-0af153818878","api_url":"https://ai.aidemos.com/v1/observations/296fccb1-080f-4f96-af1b-0af153818878"},"peers":[{"id":"949a7908-a1c3-4494-ac53-a368498fed52","tool":"cognee","tool_name":"Cognee","verdict":"failed","score":null,"score_total":null,"note":"Accepts an explicit client-memory update that SSO is no longer the rollout path, but the following reply still resurfaces SSO-specific troubleshooting language, so the old context was not fully retired.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-cognee-input2-session3-acme-memory-updat-2d684845392f.png","evidence_url":"https://aidemos.com/evidence/949a7908-a1c3-4494-ac53-a368498fed52"},{"id":"b328e35e-796b-46ed-8164-c24bf6307147","tool":"hindsight","tool_name":"Hindsight","verdict":"mixed","score":null,"score_total":null,"note":"When the later login-testing reply is generated, stale SSO context is still visible in retrieval, but it does not dominate the final answer and the newer rollout context stays primary.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-hindsight-input2-acme-update-handled-wit-0acd0a998b24.png","evidence_url":"https://aidemos.com/evidence/b328e35e-796b-46ed-8164-c24bf6307147"},{"id":"133f3e43-ac89-437b-8b33-251e882a8ed0","tool":"mem0","tool_name":"Mem0","verdict":"mixed","score":null,"score_total":null,"note":"Accepts the ACME correction and stores the rollout change, but does not retire the older SSO memory strongly enough to stop later reuse.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-mem0-input2-acme-update-memory-stored-393afdeb968e.png","evidence_url":"https://aidemos.com/evidence/133f3e43-ac89-437b-8b33-251e882a8ed0"},{"id":"8a3bac73-a767-41c2-a295-f2f33f5aacf4","tool":"zep","tool_name":"Zep","verdict":"failed","score":null,"score_total":null,"note":"After a correction that ACME had moved from SSO to email-password login, the next reply still reused SSO/SAML/IdP troubleshooting, showing that one explicit update did not retire the stale context.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/zep-zep-input2-acme-memory-update-acknowledg-3b9d5bcbd3a3.png","evidence_url":"https://aidemos.com/evidence/8a3bac73-a767-41c2-a295-f2f33f5aacf4"}],"other_criteria":[{"id":"25402b97-0e10-4fd2-bf59-da04f19e9704","criterion":"correct-application","criterion_name":"Correct Application","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Because the prior cache-clear memory was not used, the drafted reply recommended 'Clear Cache and Cookies' again, which directly repeats one of ACME's already-tried troubleshooting actions and violates the client's request to avoid repeated troubleshooting.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/25402b97-0e10-4fd2-bf59-da04f19e9704"},{"id":"701fd44d-dbaf-461a-a3e8-067c2c65e8b4","criterion":"memory-capture-quality","criterion_name":"Memory Capture Quality","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The tool captured the client boundary and the durable support context in one memory set: ACME's identity, the already-tried steps (password reset, browser cache clear, browser switch), the preference against repeated troubleshooting, and the fact that the issue was SSO-specific.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/701fd44d-dbaf-461a-a3e8-067c2c65e8b4"},{"id":"2c5ee1b7-2278-420d-9b43-b47a7bbb1a04","criterion":"relevant-retrieval","criterion_name":"Relevant Retrieval","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"In the next support turn, the reply panel showed 'No memories retrieved yet,' so the assistant did not surface ACME's prior troubleshooting history and instead repeated a cache-clearing step that had already been tried.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/2c5ee1b7-2278-420d-9b43-b47a7bbb1a04"},{"id":"d121c23e-d6c3-4b68-b261-ae56ec91ddf7","criterion":"scope-control","criterion_name":"Scope Control","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The tool kept client memories isolated: the separate BetaCorp container showed 'No memories retrieved yet' and did not inherit ACME's SSO history, troubleshooting steps, or support preferences.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/d121c23e-d6c3-4b68-b261-ae56ec91ddf7"}],"appears_in":[{"page_type":"ranking","slug":"ai-agent-memory-tools","title":"Best AI Tools for Memory for AI Agents","url":"https://aidemos.com/best/ai-agent-memory-tools","binding":"run"}],"same_scenario":[{"id":"949a7908-a1c3-4494-ac53-a368498fed52","tool":"cognee","tool_name":"Cognee","verdict":"failed","score":null,"score_total":null,"note":"Accepts an explicit client-memory update that SSO is no longer the rollout path, but the following reply still resurfaces SSO-specific troubleshooting language, so the old context was not fully retired."},{"id":"b328e35e-796b-46ed-8164-c24bf6307147","tool":"hindsight","tool_name":"Hindsight","verdict":"mixed","score":null,"score_total":null,"note":"When the later login-testing reply is generated, stale SSO context is still visible in retrieval, but it does not dominate the final answer and the newer rollout context stays primary."},{"id":"133f3e43-ac89-437b-8b33-251e882a8ed0","tool":"mem0","tool_name":"Mem0","verdict":"mixed","score":null,"score_total":null,"note":"Accepts the ACME correction and stores the rollout change, but does not retire the older SSO memory strongly enough to stop later reuse."},{"id":"8a3bac73-a767-41c2-a295-f2f33f5aacf4","tool":"zep","tool_name":"Zep","verdict":"failed","score":null,"score_total":null,"note":"After a correction that ACME had moved from SSO to email-password login, the next reply still reused SSO/SAML/IdP troubleshooting, showing that one explicit update did not retire the stale context."}]}