{"observation":{"id":"4870c6a7-1647-459e-aae0-8b4c04b62cae","tool":"querio","tool_name":"Querio","criterion":"follow-up-context","criterion_name":"Follow-Up Context","criterion_definition":"Does the tool remember previous answers correctly?","criterion_evidence_type":"transformation","criterion_rank_role":"context","criterion_rank_role_reason":"Remembering prior turns improves workflow, but a tool can still do the core job without strong conversation memory. (2 of 3 judges)","scenario":"best-customers-with-unpaid-order-and-payment-method-follow-ups","scenario_name":"Best customers with unpaid-order and payment-method follow-ups","group_tag":"live-database-plain-english-queries","scenario_description":"A conversational multi-table customer analysis with follow-up questions. It asks for the best customers by both order volume and spend, then drills into unpaid orders for the top 3 and their usual payment methods. Designed to test ranking logic, join-heavy analysis, and follow-up context retention.","modality":"text","input_text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","input_artifact_refs":[],"stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"],"verdict":"worked","score":null,"score_total":null,"note":"Retains follow-up context perfectly, reusing the same top-3 customer UUIDs in both later questions.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/a2212df102a643b4bd41951fd064776d.png?v=1","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/8a2ee356efd3402183b54fd83212199a.png?v=1","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/624e3e2ae9164c9fbf5e9fce2100ca60.png?v=1","role":"output","alt":null}],"run_id":"af2abc96-3311-484b-a19d-854a2fdd2bf3","study_title":"Query Live Databases Using Plain English with AI","study_kind":"generation","research_task":"86b9y6c99","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","files":[],"modality":"text","stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"]},"tool_page_slug":"querio","tool_url":"https://aidemos.com/tools/querio","permalink":"https://aidemos.com/evidence/4870c6a7-1647-459e-aae0-8b4c04b62cae","api_url":"https://ai.aidemos.com/v1/observations/4870c6a7-1647-459e-aae0-8b4c04b62cae"},"peers":[{"id":"2cef815d-d145-47f1-8b06-9a7b16d79c4a","tool":"ai-for-database","tool_name":"AI for Database","verdict":"failed","score":null,"score_total":null,"note":"On the first follow-up it dropped part of the top-3 context: the query result returned 4 rows for Deepak and Rahul only, yet the answer still asserted that Karan Joshi had no unpaid orders even though his row was absent.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/beb3be5f949c403083b813a7d7d8ed80.png?v=1","evidence_url":"https://aidemos.com/evidence/2cef815d-d145-47f1-8b06-9a7b16d79c4a"},{"id":"accb8a5a-dc90-44ad-a432-27d91bf9b956","tool":"anomaly-ai","tool_name":"Anomaly AI","verdict":"failed","score":null,"score_total":null,"note":"It did not keep the referent of 'the top 3' stable across turns: Follow-up 1 answered for Rahul Sharma, Priya Patel, and Deepak Kulkarni, but Follow-up 2 silently switched to Deepak Kulkarni, Rahul Sharma, and Karan Joshi.","artifact_count":6,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/37e5963be794432da14f9cdcf638885f.png?v=1","evidence_url":"https://aidemos.com/evidence/accb8a5a-dc90-44ad-a432-27d91bf9b956"},{"id":"71867d8f-4b4e-4263-b1df-786dd813435f","tool":"askyourdatabase","tool_name":"AskYourDatabase","verdict":"worked","score":null,"score_total":null,"note":"It preserves the selected top 3 across turns and reuses the earlier result set for the payment-method follow-up without issuing a new SQL query.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/b566115874cd4ee687494cc959e0239c.png?v=1","evidence_url":"https://aidemos.com/evidence/71867d8f-4b4e-4263-b1df-786dd813435f"},{"id":"56f455ce-aca2-4d40-bfe0-67dd5955602e","tool":"basedash","tool_name":"Basedash","verdict":"mixed","score":null,"score_total":null,"note":"It remembered enough of the prior answer to check both ranking lists, but it still narrowed the follow-up instead of preserving the user's intended scope cleanly.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1d5ee66912954067955529095f49f9ef.png?v=1","evidence_url":"https://aidemos.com/evidence/56f455ce-aca2-4d40-bfe0-67dd5955602e"},{"id":"605fba10-6623-4553-8e48-c23ed0c6c03e","tool":"blazesql","tool_name":"BlazeSQL","verdict":"worked","score":null,"score_total":null,"note":"It retained the corrected top-3 customer set across both follow-ups, correctly using Deepak Kulkarni, Rahul Sharma, and Karan Joshi for the unpaid-order and payment-method lookups.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/8659db3b02aa49d68b26dd7e6267c998.png?v=1","evidence_url":"https://aidemos.com/evidence/605fba10-6623-4553-8e48-c23ed0c6c03e"},{"id":"108a9c89-c164-42ed-877f-c67c7082b7cd","tool":"camelai","tool_name":"camelAI","verdict":"worked","score":null,"score_total":null,"note":"It kept the follow-up scope correctly anchored to the top 3 highest-spending customers across both follow-ups, checking unpaid orders and payment methods only for Deepak Kulkarni, Rahul Sharma, and Karan Joshi.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/ce8a71702b5d416bbf3cce77acb2f7ec.png?v=1","evidence_url":"https://aidemos.com/evidence/108a9c89-c164-42ed-877f-c67c7082b7cd"},{"id":"84d433cb-4227-4de7-ba67-e1a80fda66e4","tool":"definite","tool_name":"Definite","verdict":"worked","score":null,"score_total":null,"note":"It retained the top-3 customer context across both follow-ups, answering unpaid-order and payment-method questions for the same three customers from the initial list.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/4ecf960c56f643f48609e0a54c971906.png?v=1","evidence_url":"https://aidemos.com/evidence/84d433cb-4227-4de7-ba67-e1a80fda66e4"},{"id":"40517473-404e-484a-8697-bc2d5f2d8e13","tool":"dot","tool_name":"Dot","verdict":"worked","score":null,"score_total":null,"note":"It carried the top 3 customers through both follow-ups and, on the payment-method question, correctly did not inherit the main query's paid/non-cancelled filter.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/e94fab739e414924b69bd67beb632d2f.png?v=1","evidence_url":"https://aidemos.com/evidence/40517473-404e-484a-8697-bc2d5f2d8e13"},{"id":"1c7f2960-e2a9-4aa3-9dee-dc969f6f3371","tool":"draxlr","tool_name":"Draxlr","verdict":"worked","score":null,"score_total":null,"note":"It kept the top 3 from that list context intact across both follow-ups and continued analyzing the same three customers.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/565a041698d44f05ad75ba27fe1fd48d.png?v=1","evidence_url":"https://aidemos.com/evidence/1c7f2960-e2a9-4aa3-9dee-dc969f6f3371"},{"id":"f1273e5e-f18a-444a-8beb-b3b8b303f544","tool":"futuresmart-nl2sql-agent","tool_name":"FutureSmart NL2SQL Agent","verdict":"worked","score":null,"score_total":null,"note":"Carries the same top-3 customer trio forward into both follow-ups and reuses those same three customers in the later SQL filters.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/7c00dceab6d3435e8be7cde11f43f1be.png?v=1","evidence_url":"https://aidemos.com/evidence/f1273e5e-f18a-444a-8beb-b3b8b303f544"}],"other_criteria":[{"id":"471feddc-e3aa-49ae-982a-15b4f964a920","criterion":"ambiguity-handling","criterion_name":"Ambiguity Handling","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Resolves the ambiguous phrase 'best customers' by answering with both top-by-orders and top-by-spend views rather than silently choosing one.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/471feddc-e3aa-49ae-982a-15b4f964a920"},{"id":"78125c79-ab09-4a31-9c8e-246e8474d64b","criterion":"plain-english-query-handling","criterion_name":"Plain English Query Handling","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Handles an informal conversational request and splits 'order the most' and 'spend the most' into two separate ranking dimensions instead of guessing.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/78125c79-ab09-4a31-9c8e-246e8474d64b"},{"id":"ac7d4ce7-d376-4989-bb44-e8abdd293fbf","criterion":"result-readability","criterion_name":"Result Readability","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"The answer is usable, but it is less readable than it could be because the output centers customer UUIDs instead of human names.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/ac7d4ce7-d376-4989-bb44-e8abdd293fbf"},{"id":"a5ea07fa-27b9-4a05-a6f4-ae6b245cefe3","criterion":"sql-generation","criterion_name":"SQL Generation","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Generates the ranking SQL and the follow-up SQL correctly across the three-turn customer analysis chain.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/a5ea07fa-27b9-4a05-a6f4-ae6b245cefe3"}],"appears_in":[{"page_type":"ranking","slug":"ai-database-query-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/ai-database-query-tools","binding":"run"},{"page_type":"ranking","slug":"text-to-sql-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/text-to-sql-tools","binding":"study"},{"page_type":"use-case","slug":"query-live-databases","title":"Query Live Databases Using Plain English with AI","url":"https://aidemos.com/use-cases/query-live-databases","binding":"study"}],"same_scenario":[{"id":"2cef815d-d145-47f1-8b06-9a7b16d79c4a","tool":"ai-for-database","tool_name":"AI for Database","verdict":"failed","score":null,"score_total":null,"note":"On the first follow-up it dropped part of the top-3 context: the query result returned 4 rows for Deepak and Rahul only, yet the answer still asserted that Karan Joshi had no unpaid orders even though his row was absent."},{"id":"accb8a5a-dc90-44ad-a432-27d91bf9b956","tool":"anomaly-ai","tool_name":"Anomaly AI","verdict":"failed","score":null,"score_total":null,"note":"It did not keep the referent of 'the top 3' stable across turns: Follow-up 1 answered for Rahul Sharma, Priya Patel, and Deepak Kulkarni, but Follow-up 2 silently switched to Deepak Kulkarni, Rahul Sharma, and Karan Joshi."},{"id":"71867d8f-4b4e-4263-b1df-786dd813435f","tool":"askyourdatabase","tool_name":"AskYourDatabase","verdict":"worked","score":null,"score_total":null,"note":"It preserves the selected top 3 across turns and reuses the earlier result set for the payment-method follow-up without issuing a new SQL query."},{"id":"56f455ce-aca2-4d40-bfe0-67dd5955602e","tool":"basedash","tool_name":"Basedash","verdict":"mixed","score":null,"score_total":null,"note":"It remembered enough of the prior answer to check both ranking lists, but it still narrowed the follow-up instead of preserving the user's intended scope cleanly."},{"id":"605fba10-6623-4553-8e48-c23ed0c6c03e","tool":"blazesql","tool_name":"BlazeSQL","verdict":"worked","score":null,"score_total":null,"note":"It retained the corrected top-3 customer set across both follow-ups, correctly using Deepak Kulkarni, Rahul Sharma, and Karan Joshi for the unpaid-order and payment-method lookups."},{"id":"108a9c89-c164-42ed-877f-c67c7082b7cd","tool":"camelai","tool_name":"camelAI","verdict":"worked","score":null,"score_total":null,"note":"It kept the follow-up scope correctly anchored to the top 3 highest-spending customers across both follow-ups, checking unpaid orders and payment methods only for Deepak Kulkarni, Rahul Sharma, and Karan Joshi."},{"id":"84d433cb-4227-4de7-ba67-e1a80fda66e4","tool":"definite","tool_name":"Definite","verdict":"worked","score":null,"score_total":null,"note":"It retained the top-3 customer context across both follow-ups, answering unpaid-order and payment-method questions for the same three customers from the initial list."},{"id":"40517473-404e-484a-8697-bc2d5f2d8e13","tool":"dot","tool_name":"Dot","verdict":"worked","score":null,"score_total":null,"note":"It carried the top 3 customers through both follow-ups and, on the payment-method question, correctly did not inherit the main query's paid/non-cancelled filter."},{"id":"1c7f2960-e2a9-4aa3-9dee-dc969f6f3371","tool":"draxlr","tool_name":"Draxlr","verdict":"worked","score":null,"score_total":null,"note":"It kept the top 3 from that list context intact across both follow-ups and continued analyzing the same three customers."},{"id":"f1273e5e-f18a-444a-8beb-b3b8b303f544","tool":"futuresmart-nl2sql-agent","tool_name":"FutureSmart NL2SQL Agent","verdict":"worked","score":null,"score_total":null,"note":"Carries the same top-3 customer trio forward into both follow-ups and reuses those same three customers in the later SQL filters."}]}