{"observation":{"id":"95c5c519-d20c-492e-a27e-9429632db30e","tool":"blazesql","tool_name":"BlazeSQL","criterion":"sql-generation","criterion_name":"SQL Generation","criterion_definition":"Does it generate database-backed SQL correctly?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"If the tool cannot produce correct SQL against the database, it cannot reliably answer the question being asked. (3 of 3 judges)","scenario":"best-customers-with-unpaid-order-and-payment-method-follow-ups","scenario_name":"Best customers with unpaid-order and payment-method follow-ups","group_tag":"live-database-plain-english-queries","scenario_description":"A conversational multi-table customer analysis with follow-up questions. It asks for the best customers by both order volume and spend, then drills into unpaid orders for the top 3 and their usual payment methods. Designed to test ranking logic, join-heavy analysis, and follow-up context retention.","modality":"text","input_text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","input_artifact_refs":[],"stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"],"verdict":"worked","score":null,"score_total":null,"note":"It generated a refined ranking query that joins customers to orders, filters out CANCELLED rows, requires PAID payment_status, and ranks customers by total_spent then total_orders.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/7b4a67af644f4cf4b3cb9c3ab3e11265.png?v=1","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/61dd209afcaf4ed2ad85aca9e7780580.png?v=1","role":"output","alt":null}],"run_id":"af2abc96-3311-484b-a19d-854a2fdd2bf3","study_title":"Query Live Databases Using Plain English with AI","study_kind":"generation","research_task":"86b9y6c99","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","files":[],"modality":"text","stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"]},"tool_page_slug":"blazesql","tool_url":"https://aidemos.com/tools/blazesql","permalink":"https://aidemos.com/evidence/95c5c519-d20c-492e-a27e-9429632db30e","api_url":"https://ai.aidemos.com/v1/observations/95c5c519-d20c-492e-a27e-9429632db30e"},"peers":[{"id":"c81647fc-befc-4195-abe0-94c81a794ac3","tool":"askyourdatabase","tool_name":"AskYourDatabase","verdict":"worked","score":null,"score_total":null,"note":"It can generate follow-up SQL constrained to exactly the top 3 customers, using hardcoded customer IDs in the WHERE clause.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/b566115874cd4ee687494cc959e0239c.png?v=1","evidence_url":"https://aidemos.com/evidence/c81647fc-befc-4195-abe0-94c81a794ac3"},{"id":"bff55c46-ffad-430e-a91c-0a9b7651909a","tool":"basedash","tool_name":"Basedash","verdict":"worked","score":null,"score_total":null,"note":"It successfully generated the ranking queries and follow-up lookups, including the top spenders, top order-count customers, unpaid-order status, and usual payment methods.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/b32adc19ccfc434bb534d9c3684fee55.png?v=1","evidence_url":"https://aidemos.com/evidence/bff55c46-ffad-430e-a91c-0a9b7651909a"},{"id":"120ec9a3-5150-43ee-9111-4f936f2f833c","tool":"dot","tool_name":"Dot","verdict":"worked","score":null,"score_total":null,"note":"It generated CTE-based SQL that joins customers to paid, non-cancelled orders, aggregates order_count and total_spend, and ranks customers across both measures.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/660535357d404e5ea29b115bbc94f21f.png?v=1","evidence_url":"https://aidemos.com/evidence/120ec9a3-5150-43ee-9111-4f936f2f833c"},{"id":"4e02ad7d-1469-43d8-a397-ee2961f2a39d","tool":"draxlr","tool_name":"Draxlr","verdict":"worked","score":null,"score_total":null,"note":"Across the three turns it generated production-grade SQL, including ranking logic, unpaid-order checks, and payment-method aggregation with CTEs, ROW_NUMBER, COALESCE, and LEFT JOIN.","artifact_count":4,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/ab41b8340d0a45188a25417725b1cede.png?v=1","evidence_url":"https://aidemos.com/evidence/4e02ad7d-1469-43d8-a397-ee2961f2a39d"},{"id":"dd395705-737a-4727-80d8-173b337b5bbf","tool":"futuresmart-nl2sql-agent","tool_name":"FutureSmart NL2SQL Agent","verdict":"worked","score":null,"score_total":null,"note":"Generates a clean JOIN/GROUP BY/ORDER BY/LIMIT query for the top-customers ranking, showing the database-backed answer path is functioning on this turn.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/f70a13085b6b468c83d0816c24d18f84.png?v=1","evidence_url":"https://aidemos.com/evidence/dd395705-737a-4727-80d8-173b337b5bbf"},{"id":"a5ea07fa-27b9-4a05-a6f4-ae6b245cefe3","tool":"querio","tool_name":"Querio","verdict":"worked","score":null,"score_total":null,"note":"Generates the ranking SQL and the follow-up SQL correctly across the three-turn customer analysis chain.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/364651fb80d04b7299109e6465d39e8d.png?v=1","evidence_url":"https://aidemos.com/evidence/a5ea07fa-27b9-4a05-a6f4-ae6b245cefe3"}],"other_criteria":[{"id":"90d5508d-c64e-4aac-bd0b-4fabb0ceb785","criterion":"ambiguity-handling","criterion_name":"Ambiguity Handling","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"It clarifies the ambiguous phrase 'best customers' by detecting that cancelled and failed orders could skew the ranking, then rerunning the analysis on paid, non-cancelled orders instead of silently keeping the first pass.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/90d5508d-c64e-4aac-bd0b-4fabb0ceb785"},{"id":"bc2aee6c-97d9-4eb4-845a-d4e6cb8db1df","criterion":"business-insight","criterion_name":"Business Insight","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"It explains the follow-up data in business terms, including that Rahul Sharma has 1 unpaid order worth $2,199 in CONFIRMED status while Deepak Kulkarni and Karan Joshi are fully paid up.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/bc2aee6c-97d9-4eb4-845a-d4e6cb8db1df"},{"id":"605fba10-6623-4553-8e48-c23ed0c6c03e","criterion":"follow-up-context","criterion_name":"Follow-Up Context","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"It retained the corrected top-3 customer set across both follow-ups, correctly using Deepak Kulkarni, Rahul Sharma, and Karan Joshi for the unpaid-order and payment-method lookups.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/605fba10-6623-4553-8e48-c23ed0c6c03e"}],"appears_in":[{"page_type":"ranking","slug":"ai-database-query-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/ai-database-query-tools","binding":"run"},{"page_type":"ranking","slug":"text-to-sql-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/text-to-sql-tools","binding":"study"},{"page_type":"use-case","slug":"query-live-databases","title":"Query Live Databases Using Plain English with AI","url":"https://aidemos.com/use-cases/query-live-databases","binding":"study"}],"same_scenario":[{"id":"c81647fc-befc-4195-abe0-94c81a794ac3","tool":"askyourdatabase","tool_name":"AskYourDatabase","verdict":"worked","score":null,"score_total":null,"note":"It can generate follow-up SQL constrained to exactly the top 3 customers, using hardcoded customer IDs in the WHERE clause."},{"id":"bff55c46-ffad-430e-a91c-0a9b7651909a","tool":"basedash","tool_name":"Basedash","verdict":"worked","score":null,"score_total":null,"note":"It successfully generated the ranking queries and follow-up lookups, including the top spenders, top order-count customers, unpaid-order status, and usual payment methods."},{"id":"120ec9a3-5150-43ee-9111-4f936f2f833c","tool":"dot","tool_name":"Dot","verdict":"worked","score":null,"score_total":null,"note":"It generated CTE-based SQL that joins customers to paid, non-cancelled orders, aggregates order_count and total_spend, and ranks customers across both measures."},{"id":"4e02ad7d-1469-43d8-a397-ee2961f2a39d","tool":"draxlr","tool_name":"Draxlr","verdict":"worked","score":null,"score_total":null,"note":"Across the three turns it generated production-grade SQL, including ranking logic, unpaid-order checks, and payment-method aggregation with CTEs, ROW_NUMBER, COALESCE, and LEFT JOIN."},{"id":"dd395705-737a-4727-80d8-173b337b5bbf","tool":"futuresmart-nl2sql-agent","tool_name":"FutureSmart NL2SQL Agent","verdict":"worked","score":null,"score_total":null,"note":"Generates a clean JOIN/GROUP BY/ORDER BY/LIMIT query for the top-customers ranking, showing the database-backed answer path is functioning on this turn."},{"id":"a5ea07fa-27b9-4a05-a6f4-ae6b245cefe3","tool":"querio","tool_name":"Querio","verdict":"worked","score":null,"score_total":null,"note":"Generates the ranking SQL and the follow-up SQL correctly across the three-turn customer analysis chain."}]}