{"observation":{"id":"b3011f71-6a2d-4dba-a405-75e918d8a684","tool":"askyourdatabase","tool_name":"AskYourDatabase","criterion":"follow-up-context","criterion_name":"Follow-Up Context","criterion_definition":"Does the tool remember previous answers correctly?","criterion_evidence_type":"transformation","criterion_rank_role":null,"criterion_rank_role_reason":null,"scenario":"order-pipeline-breakdown-with-paid-pending-edge-case-and-last-month-comparison","scenario_name":"Order pipeline breakdown with paid-pending edge case and last-month comparison","group_tag":"ecommerce-nl2sql-benchmark","scenario_description":"A deeper operational analysis of current order stages, delivery-versus-cancellation rates, pending-but-paid edge cases, and a month-over-month comparison of the same breakdown.","modality":"text","input_text":"How many orders do we have at each stage right now?\n\nFollow-up 1: What percentage of our orders were successfully delivered vs cancelled?\n\nFollow-up 2: Are there any orders that are pending but already paid?\n\nFollow-up 3: Compare that to last month — same breakdown, I want to see if things have improved or got worse.","input_artifact_refs":["image-2.png"],"stresses":["Order pipeline analysis","Percentage calculation","Edge-case detection","Payment/order status joins","Multi-turn context retention","Month-over-month comparison","Ambiguity handling for 'same breakdown'"],"verdict":"mixed","score":null,"score_total":null,"note":"The tool retained the immediately preceding pending-paid edge-case context, but the report says it used that narrower metric for the follow-up comparison instead of the full original stage breakdown.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/askyourdatabase-image-11-2471cd3c9c12.png","role":null,"alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/askyourdatabase-image-12-71a67b6a625a.png","role":null,"alt":null}],"run_id":"db2bb5d5-0e0e-4cb3-8d76-3555c45c23cd","study_title":"Query Live Databases Using Plain English with AI","study_kind":"generation","research_task":"86b9y6c99","tested_at":null,"completeness":"input-and-output","input":{"state":"text","text":"How many orders do we have at each stage right now?\n\nFollow-up 1: What percentage of our orders were successfully delivered vs cancelled?\n\nFollow-up 2: Are there any orders that are pending but already paid?\n\nFollow-up 3: Compare that to last month — same breakdown, I want to see if things have improved or got worse.","files":[],"modality":"text","stresses":["Order pipeline analysis","Percentage calculation","Edge-case detection","Payment/order status joins","Multi-turn context retention","Month-over-month comparison","Ambiguity handling for 'same breakdown'"]},"tool_page_slug":"askyourdatabase","tool_url":"https://aidemos.com/tools/askyourdatabase","permalink":"https://aidemos.com/evidence/b3011f71-6a2d-4dba-a405-75e918d8a684","api_url":"https://ai.aidemos.com/v1/observations/b3011f71-6a2d-4dba-a405-75e918d8a684"},"peers":[{"id":"bb39a696-0cad-4eff-8a52-fa272573962f","tool":"definite","tool_name":"Definite","verdict":"worked","score":null,"score_total":null,"note":"It remembered the earlier breakdown when asked for 'the same breakdown' last month, reusing the stage categories and the pending-but-paid edge case across turns.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/definite-image-9-ee92993aafc7.png","evidence_url":"https://aidemos.com/evidence/bb39a696-0cad-4eff-8a52-fa272573962f"},{"id":"36b9f3cc-d31a-4e7d-a424-b7ec650d518e","tool":"querio","tool_name":"Querio","verdict":"struggled","score":null,"score_total":null,"note":"Keeps the conversation going across multiple turns, but on the final comparison it reuses the delivered-versus-cancelled thread instead of the pending-paid edge-case context, so context selection breaks on one follow-up.","artifact_count":0,"thumbnail":null,"evidence_url":"https://aidemos.com/evidence/36b9f3cc-d31a-4e7d-a424-b7ec650d518e"}],"other_criteria":[{"id":"99acf947-19e5-4f37-b36f-943b245cd941","criterion":"ambiguity-handling","criterion_name":"Ambiguity Handling","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"Clarified 'last month' upfront as April 20, 2026 before running the comparison, rather than guessing what period the user meant.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/99acf947-19e5-4f37-b36f-943b245cd941"},{"id":"3b7b86d8-e56d-4967-b17c-7b5762d88803","criterion":"business-insight","criterion_name":"Business Insight","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"Turned the comparison into a business judgment by stating that the backlog fell from 13 stuck orders to 2, an 85% reduction that 'improved significantly'.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/3b7b86d8-e56d-4967-b17c-7b5762d88803"},{"id":"11b11634-12da-417a-8c8f-2a69060619ab","criterion":"business-insight","criterion_name":"Business Insight","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"The follow-up summaries explained what the raw counts meant operationally, including delivery/cancellation rates and the backlog reduction story.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/11b11634-12da-417a-8c8f-2a69060619ab"},{"id":"81eca556-a313-4dbb-95f4-66942448a7fc","criterion":"business-insight","criterion_name":"Business Insight","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"The tool added operational context by flagging the two oldest pending-but-paid orders as needing manual intervention.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/81eca556-a313-4dbb-95f4-66942448a7fc"},{"id":"202b6ee0-0279-474b-883d-4d71f83eeea7","criterion":"chart-visualization-support","criterion_name":"Chart / Visualization Support","rank_role":null,"verdict":"failed","score":null,"score_total":null,"note":"Visualization was not automatic; the report says the tool only produced a visual after an explicit follow-up prompt, so charts were not auto-generated on the initial answers.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/202b6ee0-0279-474b-883d-4d71f83eeea7"},{"id":"c49b0bb1-bdab-4c38-adc6-d46ed3b12da0","criterion":"chart-visualization-support","criterion_name":"Chart / Visualization Support","rank_role":null,"verdict":"failed","score":null,"score_total":null,"note":"Visualization did not auto-generate for the order-pipeline flow either; the report says the user had to ask for it separately.","artifact_count":0,"evidence_url":"https://aidemos.com/evidence/c49b0bb1-bdab-4c38-adc6-d46ed3b12da0"},{"id":"5e6617bc-3f2f-4e24-8b8c-377b98c792a9","criterion":"plain-english-query-handling","criterion_name":"Plain English Query Handling","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"Accepted a four-step operational analysis in plain English across the full follow-up chain without needing the user to write SQL.","artifact_count":5,"evidence_url":"https://aidemos.com/evidence/5e6617bc-3f2f-4e24-8b8c-377b98c792a9"},{"id":"4de0a07e-49cb-4ad0-9081-85f2e94fe15b","criterion":"plain-english-query-handling","criterion_name":"Plain English Query Handling","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"The tool correctly handled a multi-step operational conversation without requiring structured SQL from the user at each turn.","artifact_count":5,"evidence_url":"https://aidemos.com/evidence/4de0a07e-49cb-4ad0-9081-85f2e94fe15b"},{"id":"e42bcb50-8973-41a0-a914-e61da29cc1fc","criterion":"result-readability","criterion_name":"Result Readability","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"Returned a clean 7-row status table with 93 total orders and an easy-to-scan delivered-versus-cancelled percentage summary.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/e42bcb50-8973-41a0-a914-e61da29cc1fc"},{"id":"1294531b-cfbb-4ad5-9c62-bc2c47f02496","criterion":"result-readability","criterion_name":"Result Readability","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"The pipeline output was readable because it summarized counts, percentages, and the stuck-versus-resolved comparison in plain tables and short callouts.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/1294531b-cfbb-4ad5-9c62-bc2c47f02496"},{"id":"c62ed979-b402-4f49-a38a-baa062f4997a","criterion":"sql-generation","criterion_name":"SQL Generation","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"Handled the month-over-month follow-up with multiple SQL statements to compare the current snapshot, inspect paid-but-pending orders, and build the prior-month comparison.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/c62ed979-b402-4f49-a38a-baa062f4997a"},{"id":"ea4e0c90-e39a-42db-9b13-d3561f30c240","criterion":"sql-visibility","criterion_name":"SQL Visibility","rank_role":null,"verdict":"worked","score":null,"score_total":null,"note":"Showed the SQL used for each step instead of only presenting the final answer.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/ea4e0c90-e39a-42db-9b13-d3561f30c240"}],"appears_in":[{"page_type":"use-case","slug":"query-live-databases","title":"Query Live Databases Using Plain English with AI","url":"https://aidemos.com/use-cases/query-live-databases","binding":"run"},{"page_type":"ranking","slug":"text-to-sql-tools","title":"Best AI Tools to Query Live Databases Using Plain English","url":"https://aidemos.com/best/text-to-sql-tools","binding":"study"}],"same_scenario":[{"id":"bb39a696-0cad-4eff-8a52-fa272573962f","tool":"definite","tool_name":"Definite","verdict":"worked","score":null,"score_total":null,"note":"It remembered the earlier breakdown when asked for 'the same breakdown' last month, reusing the stage categories and the pending-but-paid edge case across turns."},{"id":"36b9f3cc-d31a-4e7d-a424-b7ec650d518e","tool":"querio","tool_name":"Querio","verdict":"struggled","score":null,"score_total":null,"note":"Keeps the conversation going across multiple turns, but on the final comparison it reuses the delivered-versus-cancelled thread instead of the pending-paid edge-case context, so context selection breaks on one follow-up."}]}