{"observation":{"id":"43e0f8e1-a013-4335-a8e9-0e66bcff72cd","tool":"llamaparse","tool_name":"LlamaParse","criterion":"schema-adherence","criterion_name":"Schema Adherence","criterion_definition":"Does the output follow the supplied JSON schema hierarchy exactly, with correct nesting, field names, and data types?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"If the tool does not follow the supplied JSON schema exactly, the extracted data cannot be reliably used for structured querying or downstream automation. (3 of 3 judges)","scenario":"invoice-pdf","scenario_name":"Invoice PDF","group_tag":"business-document-extraction","scenario_description":"A two-page broadcast advertising invoice PDF with nested header metadata, billing and remit addresses, and eight line items spanning a page break. It was used to stress hierarchical line-item extraction, amount precision, time/day parsing, code extraction, and summary validation.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/e0e9f678bcde4e29a492a11927fc38f7.pdf?v=1","role":"input","filename":"Invoice PDF.pdf"}],"stresses":["Nested line-item hierarchy extraction","Multi-page continuity across a page break","Precision on large dollar amounts and totals","Parsing complex time slots and day patterns","Extraction of Ad IDs and reconciliation codes","Mapping structured metadata sections correctly","Financial summary validation","Handling political advertising compliance text"],"verdict":"worked","score":null,"score_total":null,"note":"The invoice output follows the supplied hierarchy closely, returning nested invoice_metadata, advertiser, station, line_items, and summary objects instead of a flat text dump.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-invoice-metadata-af804b493598.png","role":"output","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-invoice-extracted-line-items-eb4fd58a2160.png","role":"output","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-invoice-output-016245cc28c0.json","role":"output","alt":null}],"run_id":"a061b9e7-a9c5-443d-a171-b296aaf51b8c","study_title":"Extract and query structured data from documents using natural language","study_kind":"generation","research_task":"86b9y25e5","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/e0e9f678bcde4e29a492a11927fc38f7.pdf?v=1","filename":"Invoice PDF.pdf","alt":"Invoice PDF","role":"input"}],"modality":"pdf","stresses":["Nested line-item hierarchy extraction","Multi-page continuity across a page break","Precision on large dollar amounts and totals","Parsing complex time slots and day patterns","Extraction of Ad IDs and reconciliation codes","Mapping structured metadata sections correctly","Financial summary validation","Handling political advertising compliance text"]},"tool_page_slug":"llamaparse","tool_url":"https://aidemos.com/tools/llamaparse","permalink":"https://aidemos.com/evidence/43e0f8e1-a013-4335-a8e9-0e66bcff72cd","api_url":"https://ai.aidemos.com/v1/observations/43e0f8e1-a013-4335-a8e9-0e66bcff72cd"},"peers":[{"id":"bdb8a9b6-f2fe-4f93-a1ae-1f0261437067","tool":"datalab","tool_name":"Datalab","verdict":"worked","score":null,"score_total":null,"note":"The invoice workflow can emit nested schema-shaped JSON with separate invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary objects rather than flat OCR text.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/e05afc14d51f4f23bae1e9a22929fccd.mp4?v=1","evidence_url":"https://aidemos.com/evidence/bdb8a9b6-f2fe-4f93-a1ae-1f0261437067"},{"id":"d73f718e-a9a9-4284-b33e-de2e4ed6da9b","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The tool reconstructs the invoice hierarchy into nested JSON with invoice_metadata, advertiser, station, summary, and line_items objects rather than returning raw OCR.","artifact_count":5,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-invoice-schema-order-85baff3c493d.png","evidence_url":"https://aidemos.com/evidence/d73f718e-a9a9-4284-b33e-de2e4ed6da9b"},{"id":"a1af2980-0fa9-4aa3-910d-7dcdb0e75178","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the invoice hierarchy into invoice_metadata, advertiser, station, billing_address, remit_address, flight_dates, line_items, and summary objects rather than returning flat OCR text.","artifact_count":4,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/19c0891ec10d462fbb4ae3555f7ed5f3.mp4?v=1","evidence_url":"https://aidemos.com/evidence/a1af2980-0fa9-4aa3-910d-7dcdb0e75178"},{"id":"4a0ab0b3-2f6a-4dc0-8387-09bc3462ac42","tool":"nanonets","tool_name":"Nanonets","verdict":"worked","score":null,"score_total":null,"note":"Preserves the requested invoice hierarchy, populating invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary as separate JSON objects rather than flattening the page.","artifact_count":3,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-docstrange-nanonets-invoice-output-9dc0f476da4c.json","evidence_url":"https://aidemos.com/evidence/4a0ab0b3-2f6a-4dc0-8387-09bc3462ac42"},{"id":"bac4e4b0-48cf-4fdb-8df4-8dac86413cab","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the requested nested invoice JSON rather than flattening the document, populating invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary sections.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1b328999a9744f8782faf8d8632db00c.mp4?v=1","evidence_url":"https://aidemos.com/evidence/bac4e4b0-48cf-4fdb-8df4-8dac86413cab"},{"id":"1aade406-f689-48ea-92b7-1b2089eae31b","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the invoice schema into nested invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary sections rather than returning a flat document dump.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/4819c75428154465a5eff0ce7f23f251.mp4?v=1","evidence_url":"https://aidemos.com/evidence/1aade406-f689-48ea-92b7-1b2089eae31b"},{"id":"87b13324-a0ba-424d-ae9f-8da3844ab650","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Populates the invoice extraction into the requested hierarchy, separating invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary fields.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/cb167e3d358042c28f510cff62220d39.png?v=1","evidence_url":"https://aidemos.com/evidence/87b13324-a0ba-424d-ae9f-8da3844ab650"}],"other_criteria":[{"id":"aa43fa92-f38b-4722-b3e3-4c84e5e0fcc6","criterion":"extraction-accuracy","criterion_name":"Extraction Accuracy","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The invoice totals and headline metadata are extracted consistently: aired_spots is 8, gross_total 29750, agency_commission 4462.5, net_amount_due 25287.5, and payment_terms 30 Days match the source invoice.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/aa43fa92-f38b-4722-b3e3-4c84e5e0fcc6"},{"id":"a896f611-a121-44e4-961b-fc5e78404a9c","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The tool correctly extracts derived scheduling and coding fields for invoice line items, including day_of_week, days_pattern, air_time, ad_id, time_slot, and flight-period references.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/a896f611-a121-44e4-961b-fc5e78404a9c"},{"id":"5c4af38f-4e13-4a9f-a53f-7c44e253ff56","criterion":"table-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"The tool extracts invoice line items as structured rows, but it introduces an incorrect 8th line item with line_number 9, so the table record set is not fully faithful to the source invoice’s eight advertising records.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/5c4af38f-4e13-4a9f-a53f-7c44e253ff56"}],"appears_in":[{"page_type":"ranking","slug":"document-extraction","title":"Best AI Tools for Extracting Structured Data from Business Documents","url":"https://aidemos.com/best/document-extraction","binding":"run"},{"page_type":"tool","slug":"docsumo","title":null,"url":"https://aidemos.com/tools/docsumo","binding":"run"}],"same_scenario":[{"id":"bdb8a9b6-f2fe-4f93-a1ae-1f0261437067","tool":"datalab","tool_name":"Datalab","verdict":"worked","score":null,"score_total":null,"note":"The invoice workflow can emit nested schema-shaped JSON with separate invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary objects rather than flat OCR text."},{"id":"d73f718e-a9a9-4284-b33e-de2e4ed6da9b","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The tool reconstructs the invoice hierarchy into nested JSON with invoice_metadata, advertiser, station, summary, and line_items objects rather than returning raw OCR."},{"id":"a1af2980-0fa9-4aa3-910d-7dcdb0e75178","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the invoice hierarchy into invoice_metadata, advertiser, station, billing_address, remit_address, flight_dates, line_items, and summary objects rather than returning flat OCR text."},{"id":"4a0ab0b3-2f6a-4dc0-8387-09bc3462ac42","tool":"nanonets","tool_name":"Nanonets","verdict":"worked","score":null,"score_total":null,"note":"Preserves the requested invoice hierarchy, populating invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary as separate JSON objects rather than flattening the page."},{"id":"bac4e4b0-48cf-4fdb-8df4-8dac86413cab","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the requested nested invoice JSON rather than flattening the document, populating invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary sections."},{"id":"1aade406-f689-48ea-92b7-1b2089eae31b","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the invoice schema into nested invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary sections rather than returning a flat document dump."},{"id":"87b13324-a0ba-424d-ae9f-8da3844ab650","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Populates the invoice extraction into the requested hierarchy, separating invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary fields."}]}