{"observation":{"id":"d7313df2-02ba-4bef-aa11-d9889779422f","tool":"datalab","tool_name":"Datalab","criterion":"structural-clean-output","criterion_name":"Structural Clean Output","criterion_definition":"Is the JSON directly consumable by a downstream AI pipeline or system without requiring a structural transformation layer?","criterion_evidence_type":"transformation","criterion_rank_role":"context","criterion_rank_role_reason":"Being directly consumable by a downstream pipeline is valuable, but it is more about integration convenience than whether the tool actually extracts the data correctly. (3 of 3 judges)","scenario":"invoice-pdf","scenario_name":"Invoice PDF","group_tag":"business-document-extraction","scenario_description":"A two-page broadcast advertising invoice PDF with nested header metadata, billing and remit addresses, and eight line items spanning a page break. It was used to stress hierarchical line-item extraction, amount precision, time/day parsing, code extraction, and summary validation.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/e0e9f678bcde4e29a492a11927fc38f7.pdf?v=1","role":"input","filename":"Invoice PDF.pdf"}],"stresses":["Nested line-item hierarchy extraction","Multi-page continuity across a page break","Precision on large dollar amounts and totals","Parsing complex time slots and day patterns","Extraction of Ad IDs and reconciliation codes","Mapping structured metadata sections correctly","Financial summary validation","Handling political advertising compliance text"],"verdict":"mixed","score":null,"score_total":null,"note":"The invoice JSON is structurally usable but not order-stable: the top-level keys are emitted in a fully shuffled order instead of the requested schema order.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/research-media-invoice-schema-order-85baff3c493d.png","role":"input","alt":"Research media invoice schema order.png"},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-datalab-invoice-output-8c25eb9acd8c.json","role":"output","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-datalab-invoice-schema-order-53439296a014.png","role":"output","alt":null}],"run_id":"a061b9e7-a9c5-443d-a171-b296aaf51b8c","study_title":"Extract and query structured data from documents using natural language","study_kind":"generation","research_task":"86b9y25e5","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/e0e9f678bcde4e29a492a11927fc38f7.pdf?v=1","filename":"Invoice PDF.pdf","alt":"Invoice PDF","role":"input"}],"modality":"pdf","stresses":["Nested line-item hierarchy extraction","Multi-page continuity across a page break","Precision on large dollar amounts and totals","Parsing complex time slots and day patterns","Extraction of Ad IDs and reconciliation codes","Mapping structured metadata sections correctly","Financial summary validation","Handling political advertising compliance text"]},"tool_page_slug":"datalab","tool_url":"https://aidemos.com/tools/datalab","permalink":"https://aidemos.com/evidence/d7313df2-02ba-4bef-aa11-d9889779422f","api_url":"https://ai.aidemos.com/v1/observations/d7313df2-02ba-4bef-aa11-d9889779422f"},"peers":[{"id":"34d850cc-ac0c-4e02-a62f-5f24678223a5","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The invoice output includes per-section OCR confidence metadata alongside the extracted fields while remaining machine-readable JSON that can be consumed without structural transformation.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/416f869ecf584801a343b08e6a2734c6.png?v=1","evidence_url":"https://aidemos.com/evidence/34d850cc-ac0c-4e02-a62f-5f24678223a5"},{"id":"dbf84eab-a8e4-49e1-b7a5-ddf3319de6bc","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Returns directly consumable JSON for the invoice workflow, with the demo reaching extracted results and the report stating the JSON is immediately usable for integration.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/19c0891ec10d462fbb4ae3555f7ed5f3.mp4?v=1","evidence_url":"https://aidemos.com/evidence/dbf84eab-a8e4-49e1-b7a5-ddf3319de6bc"},{"id":"282ed571-0060-4232-a17c-d4a8f9a26056","tool":"retab","tool_name":"Retab","verdict":"struggled","score":null,"score_total":null,"note":"Preserves valid JSON but does not keep the schema's original key order, so downstream consumers that expect schema-consistent ordering need an extra formatting step.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-invoice-schema-order-85baff3c493d.png","evidence_url":"https://aidemos.com/evidence/282ed571-0060-4232-a17c-d4a8f9a26056"}],"other_criteria":[{"id":"327577f6-cc57-4ed2-8146-f26a03913c26","criterion":"extraction-accuracy","criterion_name":"Extraction Accuracy","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Recovers invoice metadata and financial totals accurately: invoice number 4064621-1, invoice date 10/28/12, gross total 29750, agency commission 4462.5, net amount due 25287.5, and 8 aired spots match the source.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/327577f6-cc57-4ed2-8146-f26a03913c26"},{"id":"bdb8a9b6-f2fe-4f93-a1ae-1f0261437067","criterion":"schema-adherence","criterion_name":"Schema Adherence","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The invoice workflow can emit nested schema-shaped JSON with separate invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary objects rather than flat OCR text.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/bdb8a9b6-f2fe-4f93-a1ae-1f0261437067"},{"id":"8f9a73fc-cf51-44d4-8845-b997400cabd1","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Captures derived invoice line-item fields correctly, including time_slot 9a-10a, day_of_week Su, air_time 9:38 AM, and ad_id NRCCWI071005 on line item 2.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/8f9a73fc-cf51-44d4-8845-b997400cabd1"},{"id":"13edb4ed-0752-4b00-afa4-f5afd50a9f3b","criterion":"table-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Extracts all eight invoice line items as separate records without merging, including rows that span the page break.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/13edb4ed-0752-4b00-afa4-f5afd50a9f3b"}],"appears_in":[{"page_type":"ranking","slug":"document-extraction","title":"Best AI Tools for Extracting Structured Data from Business Documents","url":"https://aidemos.com/best/document-extraction","binding":"run"},{"page_type":"tool","slug":"docsumo","title":null,"url":"https://aidemos.com/tools/docsumo","binding":"run"}],"same_scenario":[{"id":"34d850cc-ac0c-4e02-a62f-5f24678223a5","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The invoice output includes per-section OCR confidence metadata alongside the extracted fields while remaining machine-readable JSON that can be consumed without structural transformation."},{"id":"dbf84eab-a8e4-49e1-b7a5-ddf3319de6bc","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Returns directly consumable JSON for the invoice workflow, with the demo reaching extracted results and the report stating the JSON is immediately usable for integration."},{"id":"282ed571-0060-4232-a17c-d4a8f9a26056","tool":"retab","tool_name":"Retab","verdict":"struggled","score":null,"score_total":null,"note":"Preserves valid JSON but does not keep the schema's original key order, so downstream consumers that expect schema-consistent ordering need an extra formatting step."}]}