{"observation":{"id":"27465bd7-7b93-4cb4-9ab6-7ca5a8841ac6","tool":"hindsight","tool_name":"Hindsight","criterion":"correct-application","criterion_name":"Correct Application","criterion_definition":"Checks whether the agent actually uses the retrieved memory correctly.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"For agent memory, it is not enough to retrieve facts; the system must help the agent use them correctly in task execution. (2 of 3 judges)","scenario":"client-relationship-memory","scenario_name":"Client Relationship Memory","group_tag":"memory-for-ai-agents","scenario_description":"A client-support memory test that checks whether the assistant remembers prior troubleshooting, handles a changed rollout context, keeps client scope isolated, and avoids leaking one client’s history into another client’s support reply.","modality":"text","input_text":"Session 1:\nUse this under account_id: client_acme_001\n\nACME is a client using our AI support assistant. They prefer clear next steps and do not like repeated troubleshooting. Their team already tried password reset, clearing browser cache, and switching browsers. The issue is still happening only for users with SSO enabled.\n\nSession 2:\nUse this under account_id: client_acme_001\n\nACME came back today and said: \"Our users still cannot log in with SSO. What should we try next?\"\n\nDraft a support reply that respects what they already tried and moves to the next useful step.\n\nSession 3:\nUse this under account_id: client_acme_001\n\nUpdate the client memory: ACME is no longer using SSO for this rollout. They moved to email-password login for the first launch. Do not keep treating SSO as the active issue unless they mention it again.\n\nSession 4:\nUse this under account_id: client_acme_001\n\nACME says: \"Some users are still unable to log in during launch testing.\"\n\nDraft the next support reply.\n\nSession 5:\nUse this under account_id: client_beta_002\n\nBetaCorp is a new client. They say: \"Our users cannot log in for the first time.\"\n\nDraft the first support reply for BetaCorp.","input_artifact_refs":[],"stresses":["client history recall","avoid repeating troubleshooting","update handling","scope isolation between clients","support response continuity"],"verdict":"worked","score":null,"score_total":null,"note":"It uses the stored ACME troubleshooting history to avoid repeating basic reset/cache/browser advice and moves to more advanced next-step investigation.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-hindsight-input2-acme-sso-followup-no-re-205dd01f1a75.png","role":"output","alt":null}],"run_id":"6e31afbb-34d7-459a-b688-68ef76fc615a","study_title":"Memory for AI Agents","study_kind":"generation","research_task":"86ba16xrp","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Session 1:\nUse this under account_id: client_acme_001\n\nACME is a client using our AI support assistant. They prefer clear next steps and do not like repeated troubleshooting. Their team already tried password reset, clearing browser cache, and switching browsers. The issue is still happening only for users with SSO enabled.\n\nSession 2:\nUse this under account_id: client_acme_001\n\nACME came back today and said: \"Our users still cannot log in with SSO. What should we try next?\"\n\nDraft a support reply that respects what they already tried and moves to the next useful step.\n\nSession 3:\nUse this under account_id: client_acme_001\n\nUpdate the client memory: ACME is no longer using SSO for this rollout. They moved to email-password login for the first launch. Do not keep treating SSO as the active issue unless they mention it again.\n\nSession 4:\nUse this under account_id: client_acme_001\n\nACME says: \"Some users are still unable to log in during launch testing.\"\n\nDraft the next support reply.\n\nSession 5:\nUse this under account_id: client_beta_002\n\nBetaCorp is a new client. They say: \"Our users cannot log in for the first time.\"\n\nDraft the first support reply for BetaCorp.","files":[],"modality":"text","stresses":["client history recall","avoid repeating troubleshooting","update handling","scope isolation between clients","support response continuity"]},"tool_page_slug":"hindsight","tool_url":"https://aidemos.com/tools/hindsight","permalink":"https://aidemos.com/evidence/27465bd7-7b93-4cb4-9ab6-7ca5a8841ac6","api_url":"https://ai.aidemos.com/v1/observations/27465bd7-7b93-4cb4-9ab6-7ca5a8841ac6"},"peers":[{"id":"2229fde6-73e9-4926-8175-4db8cc09c657","tool":"cognee","tool_name":"Cognee","verdict":"worked","score":null,"score_total":null,"note":"Uses retrieved ACME history correctly by skipping the already-tried steps and moving to deeper SSO diagnostics instead of repeating the same troubleshooting.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-cognee-input2-session2-acme-deeper-sso-t-3afeeefe72f7.png","evidence_url":"https://aidemos.com/evidence/2229fde6-73e9-4926-8175-4db8cc09c657"},{"id":"f1e7d71f-ff71-48c3-987c-e6347e3375af","tool":"mem0","tool_name":"Mem0","verdict":"failed","score":null,"score_total":null,"note":"Keeps applying old ACME SSO context after the rollout changed, so the later reply still falls back to SSO troubleshooting instead of the new email-password path.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-mem0-input2-acme-stale-sso-context-reuse-8b6c8f404b41.png","evidence_url":"https://aidemos.com/evidence/f1e7d71f-ff71-48c3-987c-e6347e3375af"},{"id":"25402b97-0e10-4fd2-bf59-da04f19e9704","tool":"supermemory","tool_name":"Supermemory","verdict":"failed","score":null,"score_total":null,"note":"Because the prior cache-clear memory was not used, the drafted reply recommended 'Clear Cache and Cookies' again, which directly repeats one of ACME's already-tried troubleshooting actions and violates the client's request to avoid repeated troubleshooting.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/memory-for-ai-agents-supermemory-input2-session2-acme-repeate-116a5c13e552.png","evidence_url":"https://aidemos.com/evidence/25402b97-0e10-4fd2-bf59-da04f19e9704"}],"other_criteria":[{"id":"3f16e816-7ad8-46b3-b340-f1363b25cf16","criterion":"memory-capture-quality","criterion_name":"Memory Capture Quality","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"It stores client-support context including the already-tried troubleshooting steps, the preference for clear next steps, and the active SSO-specific login issue.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/3f16e816-7ad8-46b3-b340-f1363b25cf16"},{"id":"92c2bced-a602-47d0-b908-31556b9d054a","criterion":"scope-control","criterion_name":"Scope Control","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"It keeps BetaCorp isolated as a new client bank, so ACME's troubleshooting history does not leak into the first BetaCorp reply.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/92c2bced-a602-47d0-b908-31556b9d054a"},{"id":"b6142486-5ab1-451c-b7d7-5c700089d64e","criterion":"update-and-correction-handling","criterion_name":"Update and Correction Handling","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"After the rollout changed, it records that ACME moved to email-password login and stops treating SSO as the active issue by default.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/b6142486-5ab1-451c-b7d7-5c700089d64e"},{"id":"b328e35e-796b-46ed-8164-c24bf6307147","criterion":"update-and-correction-handling","criterion_name":"Update and Correction Handling","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"When the later login-testing reply is generated, stale SSO context is still visible in retrieval, but it does not dominate the final answer and the newer rollout context stays primary.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/b328e35e-796b-46ed-8164-c24bf6307147"}],"appears_in":[{"page_type":"ranking","slug":"ai-agent-memory-tools","title":"Best AI Tools for Memory for AI Agents","url":"https://aidemos.com/best/ai-agent-memory-tools","binding":"run"}],"same_scenario":[{"id":"2229fde6-73e9-4926-8175-4db8cc09c657","tool":"cognee","tool_name":"Cognee","verdict":"worked","score":null,"score_total":null,"note":"Uses retrieved ACME history correctly by skipping the already-tried steps and moving to deeper SSO diagnostics instead of repeating the same troubleshooting."},{"id":"f1e7d71f-ff71-48c3-987c-e6347e3375af","tool":"mem0","tool_name":"Mem0","verdict":"failed","score":null,"score_total":null,"note":"Keeps applying old ACME SSO context after the rollout changed, so the later reply still falls back to SSO troubleshooting instead of the new email-password path."},{"id":"25402b97-0e10-4fd2-bf59-da04f19e9704","tool":"supermemory","tool_name":"Supermemory","verdict":"failed","score":null,"score_total":null,"note":"Because the prior cache-clear memory was not used, the drafted reply recommended 'Clear Cache and Cookies' again, which directly repeats one of ACME's already-tried troubleshooting actions and violates the client's request to avoid repeated troubleshooting."}]}