{"observation":{"id":"02427fdf-872a-474c-9905-7412e3124cf0","tool":"futuresmart-nl2sql-agent","tool_name":"FutureSmart NL2SQL Agent","criterion":"sql-visibility","criterion_name":"SQL Visibility","criterion_definition":"Can users inspect or copy the generated SQL?","criterion_evidence_type":"capability","criterion_rank_role":"context","criterion_rank_role_reason":"Being able to inspect or copy SQL is useful for trust and debugging, but it does not determine whether the tool answers well. (3 of 3 judges)","scenario":"best-customers-with-unpaid-order-and-payment-method-follow-ups","scenario_name":"Best customers with unpaid-order and payment-method follow-ups","group_tag":"live-database-plain-english-queries","scenario_description":"A conversational multi-table customer analysis with follow-up questions. It asks for the best customers by both order volume and spend, then drills into unpaid orders for the top 3 and their usual payment methods. Designed to test ranking logic, join-heavy analysis, and follow-up context retention.","modality":"text","input_text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","input_artifact_refs":[],"stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"],"verdict":"worked","score":null,"score_total":null,"note":"Surfaces the generated SQL in an Executed SQL panel with Copy Code, so the query can be inspected or copied directly.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/f70a13085b6b468c83d0816c24d18f84.png?v=1","role":"output","alt":null}],"run_id":"af2abc96-3311-484b-a19d-854a2fdd2bf3","study_title":"Query Live Databases Using Plain English with AI","study_kind":"generation","research_task":"86b9y6c99","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"Who are my best customers — the ones who order the most and spend the most?\n\nFollow-up 1: For the top 3 from that list — do any of them have unpaid orders?\n\nFollow-up 2: What payment methods do these top 3 usually use?","files":[],"modality":"text","stresses":["ambiguous business term interpretation","multi-table joins","aggregation and ranking","follow-up context retention","scoping to a selected subset","payment behavior analysis"]},"tool_page_slug":null,"tool_url":null,"permalink":"https://aidemos.com/evidence/02427fdf-872a-474c-9905-7412e3124cf0","api_url":"https://ai.aidemos.com/v1/observations/02427fdf-872a-474c-9905-7412e3124cf0"},"peers":[],"other_criteria":[{"id":"8a630fc2-3d33-4bf5-a7a3-8dd8ce16eea3","criterion":"ambiguity-handling","criterion_name":"Ambiguity Handling","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"It states a ranking heuristic in prose, using spend as the primary sort and order count as a secondary sort, but it still leaves the no-status-filter assumption unstated and does not ask the user to clarify the two-dimensional wording.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/8a630fc2-3d33-4bf5-a7a3-8dd8ce16eea3"},{"id":"55e3e9ab-d8a7-424f-9457-f78a5dbd151f","criterion":"business-insight","criterion_name":"Business Insight","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Explains the ranking in business terms by calling out the high-value customer, the highest-order-count customer, and a loyalty/upsell targeting opportunity.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/55e3e9ab-d8a7-424f-9457-f78a5dbd151f"},{"id":"e91c9860-3478-4496-b5ac-5bbd4f47b93b","criterion":"business-insight","criterion_name":"Business Insight","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"The payment-method follow-up gives a concise dominant-method insight for the top customers, but the result is incomplete as a business answer because it does not surface the full payment-method picture implied by the customer totals.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/e91c9860-3478-4496-b5ac-5bbd4f47b93b"},{"id":"f1273e5e-f18a-444a-8beb-b3b8b303f544","criterion":"follow-up-context","criterion_name":"Follow-Up Context","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"Carries the same top-3 customer trio forward into both follow-ups and reuses those same three customers in the later SQL filters.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/f1273e5e-f18a-444a-8beb-b3b8b303f544"},{"id":"56eb0ed0-bf0d-4077-9f04-bd1a290baeb9","criterion":"plain-english-query-handling","criterion_name":"Plain English Query Handling","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Accepts the best-customers question in plain English and returns a ranked answer without requiring SQL from the user.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/56eb0ed0-bf0d-4077-9f04-bd1a290baeb9"},{"id":"dd395705-737a-4727-80d8-173b337b5bbf","criterion":"sql-generation","criterion_name":"SQL Generation","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Generates a clean JOIN/GROUP BY/ORDER BY/LIMIT query for the top-customers ranking, showing the database-backed answer path is functioning on this turn.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/dd395705-737a-4727-80d8-173b337b5bbf"}],"appears_in":[{"page_type":"ranking","slug":"ai-database-query-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/ai-database-query-tools","binding":"run"},{"page_type":"ranking","slug":"text-to-sql-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/text-to-sql-tools","binding":"study"},{"page_type":"use-case","slug":"query-live-databases","title":"Query Live Databases Using Plain English with AI","url":"https://aidemos.com/use-cases/query-live-databases","binding":"study"}],"same_scenario":[]}