{"observation":{"id":"2cef815d-d145-47f1-8b06-9a7b16d79c4a","tool":"ai-for-database","tool_name":"AI for Database","criterion":"follow-up-context","criterion_name":"Follow-Up Context","criterion_definition":"Does the tool remember previous answers correctly?","criterion_evidence_type":"transformation","criterion_rank_role":"context","criterion_rank_role_reason":"Remembering prior turns improves workflow, but a tool can still do the core job without strong conversation memory. (2 of 3 judges)","scenario":"best-customers-with-unpaid-order-and-payment-method-follow-ups","scenario_name":"Best customers with unpaid-order and payment-method follow-ups","group_tag":"live-database-plain-english-queries","scenario_description":"A conversational multi-table customer analysis with follow-up questions. It asks for the best customers by both order volume and spend, then drills into unpaid orders for the top 3 and their usual payment methods. Designed to test ranking logic, join-heavy analysis, and follow-up context retention.","modality":"text","input_text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","input_artifact_refs":[],"stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"],"verdict":"failed","score":null,"score_total":null,"note":"On the first follow-up it dropped part of the top-3 context: the query result returned 4 rows for Deepak and Rahul only, yet the answer still asserted that Karan Joshi had no unpaid orders even though his row was absent.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/beb3be5f949c403083b813a7d7d8ed80.png?v=1","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/03bafb9e9463451aaacd59707100c57c.png?v=1","role":"output","alt":null}],"run_id":"af2abc96-3311-484b-a19d-854a2fdd2bf3","study_title":"Query Live Databases Using Plain English with AI","study_kind":"generation","research_task":"86b9y6c99","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","files":[],"modality":"text","stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"]},"tool_page_slug":"ai-for-database","tool_url":"https://aidemos.com/tools/ai-for-database","permalink":"https://aidemos.com/evidence/2cef815d-d145-47f1-8b06-9a7b16d79c4a","api_url":"https://ai.aidemos.com/v1/observations/2cef815d-d145-47f1-8b06-9a7b16d79c4a"},"peers":[{"id":"accb8a5a-dc90-44ad-a432-27d91bf9b956","tool":"anomaly-ai","tool_name":"Anomaly AI","verdict":"failed","score":null,"score_total":null,"note":"It did not keep the referent of 'the top 3' stable across turns: Follow-up 1 answered for Rahul Sharma, Priya Patel, and Deepak Kulkarni, but Follow-up 2 silently switched to Deepak Kulkarni, Rahul Sharma, and Karan Joshi.","artifact_count":6,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/37e5963be794432da14f9cdcf638885f.png?v=1","evidence_url":"https://aidemos.com/evidence/accb8a5a-dc90-44ad-a432-27d91bf9b956"},{"id":"71867d8f-4b4e-4263-b1df-786dd813435f","tool":"askyourdatabase","tool_name":"AskYourDatabase","verdict":"worked","score":null,"score_total":null,"note":"It preserves the selected top 3 across turns and reuses the earlier result set for the payment-method follow-up without issuing a new SQL query.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/b566115874cd4ee687494cc959e0239c.png?v=1","evidence_url":"https://aidemos.com/evidence/71867d8f-4b4e-4263-b1df-786dd813435f"},{"id":"56f455ce-aca2-4d40-bfe0-67dd5955602e","tool":"basedash","tool_name":"Basedash","verdict":"mixed","score":null,"score_total":null,"note":"It remembered enough of the prior answer to check both ranking lists, but it still narrowed the follow-up instead of preserving the user's intended scope cleanly.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1d5ee66912954067955529095f49f9ef.png?v=1","evidence_url":"https://aidemos.com/evidence/56f455ce-aca2-4d40-bfe0-67dd5955602e"},{"id":"605fba10-6623-4553-8e48-c23ed0c6c03e","tool":"blazesql","tool_name":"BlazeSQL","verdict":"worked","score":null,"score_total":null,"note":"It retained the corrected top-3 customer set across both follow-ups, correctly using Deepak Kulkarni, Rahul Sharma, and Karan Joshi for the unpaid-order and payment-method lookups.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/8659db3b02aa49d68b26dd7e6267c998.png?v=1","evidence_url":"https://aidemos.com/evidence/605fba10-6623-4553-8e48-c23ed0c6c03e"},{"id":"108a9c89-c164-42ed-877f-c67c7082b7cd","tool":"camelai","tool_name":"camelAI","verdict":"worked","score":null,"score_total":null,"note":"It kept the follow-up scope correctly anchored to the top 3 highest-spending customers across both follow-ups, checking unpaid orders and payment methods only for Deepak Kulkarni, Rahul Sharma, and Karan Joshi.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/ce8a71702b5d416bbf3cce77acb2f7ec.png?v=1","evidence_url":"https://aidemos.com/evidence/108a9c89-c164-42ed-877f-c67c7082b7cd"},{"id":"84d433cb-4227-4de7-ba67-e1a80fda66e4","tool":"definite","tool_name":"Definite","verdict":"worked","score":null,"score_total":null,"note":"It retained the top-3 customer context across both follow-ups, answering unpaid-order and payment-method questions for the same three customers from the initial list.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/4ecf960c56f643f48609e0a54c971906.png?v=1","evidence_url":"https://aidemos.com/evidence/84d433cb-4227-4de7-ba67-e1a80fda66e4"},{"id":"40517473-404e-484a-8697-bc2d5f2d8e13","tool":"dot","tool_name":"Dot","verdict":"worked","score":null,"score_total":null,"note":"It carried the top 3 customers through both follow-ups and, on the payment-method question, correctly did not inherit the main query's paid/non-cancelled filter.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/e94fab739e414924b69bd67beb632d2f.png?v=1","evidence_url":"https://aidemos.com/evidence/40517473-404e-484a-8697-bc2d5f2d8e13"},{"id":"1c7f2960-e2a9-4aa3-9dee-dc969f6f3371","tool":"draxlr","tool_name":"Draxlr","verdict":"worked","score":null,"score_total":null,"note":"It kept the top 3 from that list context intact across both follow-ups and continued analyzing the same three customers.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/565a041698d44f05ad75ba27fe1fd48d.png?v=1","evidence_url":"https://aidemos.com/evidence/1c7f2960-e2a9-4aa3-9dee-dc969f6f3371"},{"id":"f1273e5e-f18a-444a-8beb-b3b8b303f544","tool":"futuresmart-nl2sql-agent","tool_name":"FutureSmart NL2SQL Agent","verdict":"worked","score":null,"score_total":null,"note":"Carries the same top-3 customer trio forward into both follow-ups and reuses those same three customers in the later SQL filters.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/7c00dceab6d3435e8be7cde11f43f1be.png?v=1","evidence_url":"https://aidemos.com/evidence/f1273e5e-f18a-444a-8beb-b3b8b303f544"},{"id":"4870c6a7-1647-459e-aae0-8b4c04b62cae","tool":"querio","tool_name":"Querio","verdict":"worked","score":null,"score_total":null,"note":"Retains follow-up context perfectly, reusing the same top-3 customer UUIDs in both later questions.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/a2212df102a643b4bd41951fd064776d.png?v=1","evidence_url":"https://aidemos.com/evidence/4870c6a7-1647-459e-aae0-8b4c04b62cae"}],"other_criteria":[{"id":"104bbb90-a9d0-4ffc-bfae-0e96eac9e7fe","criterion":"ambiguity-handling","criterion_name":"Ambiguity Handling","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"It resolves the 'order the most AND spend the most' ambiguity by showing both rankings side by side instead of silently collapsing the request into one metric, and it names the customer who tops both lists.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/104bbb90-a9d0-4ffc-bfae-0e96eac9e7fe"}],"appears_in":[{"page_type":"ranking","slug":"ai-database-query-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/ai-database-query-tools","binding":"run"},{"page_type":"ranking","slug":"text-to-sql-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/text-to-sql-tools","binding":"study"},{"page_type":"use-case","slug":"query-live-databases","title":"Query Live Databases Using Plain English with AI","url":"https://aidemos.com/use-cases/query-live-databases","binding":"study"}],"same_scenario":[{"id":"accb8a5a-dc90-44ad-a432-27d91bf9b956","tool":"anomaly-ai","tool_name":"Anomaly AI","verdict":"failed","score":null,"score_total":null,"note":"It did not keep the referent of 'the top 3' stable across turns: Follow-up 1 answered for Rahul Sharma, Priya Patel, and Deepak Kulkarni, but Follow-up 2 silently switched to Deepak Kulkarni, Rahul Sharma, and Karan Joshi."},{"id":"71867d8f-4b4e-4263-b1df-786dd813435f","tool":"askyourdatabase","tool_name":"AskYourDatabase","verdict":"worked","score":null,"score_total":null,"note":"It preserves the selected top 3 across turns and reuses the earlier result set for the payment-method follow-up without issuing a new SQL query."},{"id":"56f455ce-aca2-4d40-bfe0-67dd5955602e","tool":"basedash","tool_name":"Basedash","verdict":"mixed","score":null,"score_total":null,"note":"It remembered enough of the prior answer to check both ranking lists, but it still narrowed the follow-up instead of preserving the user's intended scope cleanly."},{"id":"605fba10-6623-4553-8e48-c23ed0c6c03e","tool":"blazesql","tool_name":"BlazeSQL","verdict":"worked","score":null,"score_total":null,"note":"It retained the corrected top-3 customer set across both follow-ups, correctly using Deepak Kulkarni, Rahul Sharma, and Karan Joshi for the unpaid-order and payment-method lookups."},{"id":"108a9c89-c164-42ed-877f-c67c7082b7cd","tool":"camelai","tool_name":"camelAI","verdict":"worked","score":null,"score_total":null,"note":"It kept the follow-up scope correctly anchored to the top 3 highest-spending customers across both follow-ups, checking unpaid orders and payment methods only for Deepak Kulkarni, Rahul Sharma, and Karan Joshi."},{"id":"84d433cb-4227-4de7-ba67-e1a80fda66e4","tool":"definite","tool_name":"Definite","verdict":"worked","score":null,"score_total":null,"note":"It retained the top-3 customer context across both follow-ups, answering unpaid-order and payment-method questions for the same three customers from the initial list."},{"id":"40517473-404e-484a-8697-bc2d5f2d8e13","tool":"dot","tool_name":"Dot","verdict":"worked","score":null,"score_total":null,"note":"It carried the top 3 customers through both follow-ups and, on the payment-method question, correctly did not inherit the main query's paid/non-cancelled filter."},{"id":"1c7f2960-e2a9-4aa3-9dee-dc969f6f3371","tool":"draxlr","tool_name":"Draxlr","verdict":"worked","score":null,"score_total":null,"note":"It kept the top 3 from that list context intact across both follow-ups and continued analyzing the same three customers."},{"id":"f1273e5e-f18a-444a-8beb-b3b8b303f544","tool":"futuresmart-nl2sql-agent","tool_name":"FutureSmart NL2SQL Agent","verdict":"worked","score":null,"score_total":null,"note":"Carries the same top-3 customer trio forward into both follow-ups and reuses those same three customers in the later SQL filters."},{"id":"4870c6a7-1647-459e-aae0-8b4c04b62cae","tool":"querio","tool_name":"Querio","verdict":"worked","score":null,"score_total":null,"note":"Retains follow-up context perfectly, reusing the same top-3 customer UUIDs in both later questions."}]}