{"observation":{"id":"8a630fc2-3d33-4bf5-a7a3-8dd8ce16eea3","tool":"futuresmart-nl2sql-agent","tool_name":"FutureSmart NL2SQL Agent","criterion":"ambiguity-handling","criterion_name":"Ambiguity Handling","criterion_definition":"Does it clarify unclear business terms instead of guessing silently?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"A database assistant must clarify unclear business terms instead of guessing, or it will return wrong answers. (3 of 3 judges)","scenario":"best-customers-with-unpaid-order-and-payment-method-follow-ups","scenario_name":"Best customers with unpaid-order and payment-method follow-ups","group_tag":"live-database-plain-english-queries","scenario_description":"A conversational multi-table customer analysis with follow-up questions. It asks for the best customers by both order volume and spend, then drills into unpaid orders for the top 3 and their usual payment methods. Designed to test ranking logic, join-heavy analysis, and follow-up context retention.","modality":"text","input_text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","input_artifact_refs":[],"stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"],"verdict":"mixed","score":null,"score_total":null,"note":"It states a ranking heuristic in prose, using spend as the primary sort and order count as a secondary sort, but it still leaves the no-status-filter assumption unstated and does not ask the user to clarify the two-dimensional wording.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/9f6a37e2aa8044c5a801b7b0c1fe9ef9.png?v=1","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/1ce00eb6980441aca61128e19b7abd4a.png?v=1","role":"output","alt":null}],"run_id":"af2abc96-3311-484b-a19d-854a2fdd2bf3","study_title":"Query Live Databases Using Plain English with AI","study_kind":"generation","research_task":"86b9y6c99","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","files":[],"modality":"text","stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"]},"tool_page_slug":null,"tool_url":null,"permalink":"https://aidemos.com/evidence/8a630fc2-3d33-4bf5-a7a3-8dd8ce16eea3","api_url":"https://ai.aidemos.com/v1/observations/8a630fc2-3d33-4bf5-a7a3-8dd8ce16eea3"},"peers":[{"id":"104bbb90-a9d0-4ffc-bfae-0e96eac9e7fe","tool":"ai-for-database","tool_name":"AI for Database","verdict":"worked","score":null,"score_total":null,"note":"It resolves the 'order the most AND spend the most' ambiguity by showing both rankings side by side instead of silently collapsing the request into one metric, and it names the customer who tops both lists.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/9e7ad27c49f04104a3656a73cef8902b.png?v=1","evidence_url":"https://aidemos.com/evidence/104bbb90-a9d0-4ffc-bfae-0e96eac9e7fe"},{"id":"77347868-9e9b-44b4-b10b-bd38b78efb24","tool":"anomaly-ai","tool_name":"Anomaly AI","verdict":"worked","score":null,"score_total":null,"note":"When 'best customers' was ambiguous, it did not guess silently; it exposed a two-axis ranking, stated the cancelled-order and total_amount assumptions, and separately ranked repeat business versus spend.","artifact_count":5,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/dd400dc0a3b5474299b4e9edac56b2b5.png?v=1","evidence_url":"https://aidemos.com/evidence/77347868-9e9b-44b4-b10b-bd38b78efb24"},{"id":"1e61c241-7f0d-4a12-977f-2e2b437b2f80","tool":"askyourdatabase","tool_name":"AskYourDatabase","verdict":"worked","score":null,"score_total":null,"note":"It handles the ambiguous 'best customers' request by surfacing both order-count and spend rankings instead of silently choosing one metric.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/ec0ed41fe3dd45b5b13b082d174be6b7.png?v=1","evidence_url":"https://aidemos.com/evidence/1e61c241-7f0d-4a12-977f-2e2b437b2f80"},{"id":"099c140b-33a3-43e3-a1b4-33e60dba5b32","tool":"basedash","tool_name":"Basedash","verdict":"struggled","score":null,"score_total":null,"note":"It did not ask what 'top 3 from that list' meant; instead it silently chose the spend ranking first and then added the order-count check.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1d5ee66912954067955529095f49f9ef.png?v=1","evidence_url":"https://aidemos.com/evidence/099c140b-33a3-43e3-a1b4-33e60dba5b32"},{"id":"90d5508d-c64e-4aac-bd0b-4fabb0ceb785","tool":"blazesql","tool_name":"BlazeSQL","verdict":"worked","score":null,"score_total":null,"note":"It clarifies the ambiguous phrase 'best customers' by detecting that cancelled and failed orders could skew the ranking, then rerunning the analysis on paid, non-cancelled orders instead of silently keeping the first pass.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/060201dc6b174181bc474b4e5563f5d7.png?v=1","evidence_url":"https://aidemos.com/evidence/90d5508d-c64e-4aac-bd0b-4fabb0ceb785"},{"id":"ecf1a8ba-0c5a-4c23-90b9-f6084f0c8b93","tool":"camelai","tool_name":"camelAI","verdict":"worked","score":null,"score_total":null,"note":"It resolved the 'order the most and spend the most' ambiguity by returning separate frequency and spend rankings plus a merged best-overall verdict naming Rahul Sharma as the strongest combination of frequency and value.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1c1df761c0854b5da1ff79f846fd34b9.png?v=1","evidence_url":"https://aidemos.com/evidence/ecf1a8ba-0c5a-4c23-90b9-f6084f0c8b93"},{"id":"0c9c2717-901f-4b0f-a114-233eda2981c1","tool":"definite","tool_name":"Definite","verdict":"failed","score":null,"score_total":null,"note":"It did not clarify the ambiguous 'best customers' wording; it silently chose spend-based ranking and ignored the order-frequency part of the question.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/852010033067408ba7a475364358d97f.png?v=1","evidence_url":"https://aidemos.com/evidence/0c9c2717-901f-4b0f-a114-233eda2981c1"},{"id":"895073b3-49c1-4aad-967e-a214175ae66e","tool":"dot","tool_name":"Dot","verdict":"worked","score":null,"score_total":null,"note":"Faced with 'order the most and spend the most,' it did not silently choose one dimension; it built a combined equal-weight rank and explicitly surfaced the trade-off between all-around value and single-big-order spend.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/c126c58ac82e4d899b11a7006a73d94d.png?v=1","evidence_url":"https://aidemos.com/evidence/895073b3-49c1-4aad-967e-a214175ae66e"},{"id":"c476bbbb-1bf9-4efe-bdc1-3098a868877c","tool":"draxlr","tool_name":"Draxlr","verdict":"worked","score":null,"score_total":null,"note":"It resolved the vague phrase best customers as highest order count plus highest total spend instead of asking for a definition.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/ab41b8340d0a45188a25417725b1cede.png?v=1","evidence_url":"https://aidemos.com/evidence/c476bbbb-1bf9-4efe-bdc1-3098a868877c"},{"id":"471feddc-e3aa-49ae-982a-15b4f964a920","tool":"querio","tool_name":"Querio","verdict":"worked","score":null,"score_total":null,"note":"Resolves the ambiguous phrase 'best customers' by answering with both top-by-orders and top-by-spend views rather than silently choosing one.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/364651fb80d04b7299109e6465d39e8d.png?v=1","evidence_url":"https://aidemos.com/evidence/471feddc-e3aa-49ae-982a-15b4f964a920"}],"other_criteria":[{"id":"55e3e9ab-d8a7-424f-9457-f78a5dbd151f","criterion":"business-insight","criterion_name":"Business Insight","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Explains the ranking in business terms by calling out the high-value customer, the highest-order-count customer, and a loyalty/upsell targeting opportunity.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/55e3e9ab-d8a7-424f-9457-f78a5dbd151f"},{"id":"e91c9860-3478-4496-b5ac-5bbd4f47b93b","criterion":"business-insight","criterion_name":"Business Insight","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"The payment-method follow-up gives a concise dominant-method insight for the top customers, but the result is incomplete as a business answer because it does not surface the full payment-method picture implied by the customer totals.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/e91c9860-3478-4496-b5ac-5bbd4f47b93b"},{"id":"f1273e5e-f18a-444a-8beb-b3b8b303f544","criterion":"follow-up-context","criterion_name":"Follow-Up Context","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"Carries the same top-3 customer trio forward into both follow-ups and reuses those same three customers in the later SQL filters.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/f1273e5e-f18a-444a-8beb-b3b8b303f544"},{"id":"56eb0ed0-bf0d-4077-9f04-bd1a290baeb9","criterion":"plain-english-query-handling","criterion_name":"Plain English Query Handling","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Accepts the best-customers question in plain English and returns a ranked answer without requiring SQL from the user.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/56eb0ed0-bf0d-4077-9f04-bd1a290baeb9"},{"id":"dd395705-737a-4727-80d8-173b337b5bbf","criterion":"sql-generation","criterion_name":"SQL Generation","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Generates a clean JOIN/GROUP BY/ORDER BY/LIMIT query for the top-customers ranking, showing the database-backed answer path is functioning on this turn.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/dd395705-737a-4727-80d8-173b337b5bbf"},{"id":"02427fdf-872a-474c-9905-7412e3124cf0","criterion":"sql-visibility","criterion_name":"SQL Visibility","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"Surfaces the generated SQL in an Executed SQL panel with Copy Code, so the query can be inspected or copied directly.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/02427fdf-872a-474c-9905-7412e3124cf0"}],"appears_in":[{"page_type":"ranking","slug":"ai-database-query-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/ai-database-query-tools","binding":"run"},{"page_type":"ranking","slug":"text-to-sql-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/text-to-sql-tools","binding":"study"},{"page_type":"use-case","slug":"query-live-databases","title":"Query Live Databases Using Plain English with AI","url":"https://aidemos.com/use-cases/query-live-databases","binding":"study"}],"same_scenario":[{"id":"104bbb90-a9d0-4ffc-bfae-0e96eac9e7fe","tool":"ai-for-database","tool_name":"AI for Database","verdict":"worked","score":null,"score_total":null,"note":"It resolves the 'order the most AND spend the most' ambiguity by showing both rankings side by side instead of silently collapsing the request into one metric, and it names the customer who tops both lists."},{"id":"77347868-9e9b-44b4-b10b-bd38b78efb24","tool":"anomaly-ai","tool_name":"Anomaly AI","verdict":"worked","score":null,"score_total":null,"note":"When 'best customers' was ambiguous, it did not guess silently; it exposed a two-axis ranking, stated the cancelled-order and total_amount assumptions, and separately ranked repeat business versus spend."},{"id":"1e61c241-7f0d-4a12-977f-2e2b437b2f80","tool":"askyourdatabase","tool_name":"AskYourDatabase","verdict":"worked","score":null,"score_total":null,"note":"It handles the ambiguous 'best customers' request by surfacing both order-count and spend rankings instead of silently choosing one metric."},{"id":"099c140b-33a3-43e3-a1b4-33e60dba5b32","tool":"basedash","tool_name":"Basedash","verdict":"struggled","score":null,"score_total":null,"note":"It did not ask what 'top 3 from that list' meant; instead it silently chose the spend ranking first and then added the order-count check."},{"id":"90d5508d-c64e-4aac-bd0b-4fabb0ceb785","tool":"blazesql","tool_name":"BlazeSQL","verdict":"worked","score":null,"score_total":null,"note":"It clarifies the ambiguous phrase 'best customers' by detecting that cancelled and failed orders could skew the ranking, then rerunning the analysis on paid, non-cancelled orders instead of silently keeping the first pass."},{"id":"ecf1a8ba-0c5a-4c23-90b9-f6084f0c8b93","tool":"camelai","tool_name":"camelAI","verdict":"worked","score":null,"score_total":null,"note":"It resolved the 'order the most and spend the most' ambiguity by returning separate frequency and spend rankings plus a merged best-overall verdict naming Rahul Sharma as the strongest combination of frequency and value."},{"id":"0c9c2717-901f-4b0f-a114-233eda2981c1","tool":"definite","tool_name":"Definite","verdict":"failed","score":null,"score_total":null,"note":"It did not clarify the ambiguous 'best customers' wording; it silently chose spend-based ranking and ignored the order-frequency part of the question."},{"id":"895073b3-49c1-4aad-967e-a214175ae66e","tool":"dot","tool_name":"Dot","verdict":"worked","score":null,"score_total":null,"note":"Faced with 'order the most and spend the most,' it did not silently choose one dimension; it built a combined equal-weight rank and explicitly surfaced the trade-off between all-around value and single-big-order spend."},{"id":"c476bbbb-1bf9-4efe-bdc1-3098a868877c","tool":"draxlr","tool_name":"Draxlr","verdict":"worked","score":null,"score_total":null,"note":"It resolved the vague phrase best customers as highest order count plus highest total spend instead of asking for a definition."},{"id":"471feddc-e3aa-49ae-982a-15b4f964a920","tool":"querio","tool_name":"Querio","verdict":"worked","score":null,"score_total":null,"note":"Resolves the ambiguous phrase 'best customers' by answering with both top-by-orders and top-by-spend views rather than silently choosing one."}]}