{"observation":{"id":"b808d6bb-8133-43cb-8a88-7d5e3160cf25","tool":"landing-ai","tool_name":"Landing AI","criterion":"extraction-accuracy","criterion_name":"Extraction Accuracy","criterion_definition":"Are field values correct, complete, and free of OCR or parsing errors, including numerical precision on financial fields?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"Correct field values are the core of the job; wrong or incomplete extraction means the tool failed to retrieve the structured data from the document. (3 of 3 judges)","scenario":"invoice-pdf","scenario_name":"Invoice PDF","group_tag":"business-document-extraction","scenario_description":"A two-page broadcast advertising invoice PDF with nested header metadata, billing and remit addresses, and eight line items spanning a page break. It was used to stress hierarchical line-item extraction, amount precision, time/day parsing, code extraction, and summary validation.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/e0e9f678bcde4e29a492a11927fc38f7.pdf?v=1","role":"input","filename":"Invoice PDF.pdf"}],"stresses":["Nested line-item hierarchy extraction","Multi-page continuity across a page break","Precision on large dollar amounts and totals","Parsing complex time slots and day patterns","Extraction of Ad IDs and reconciliation codes","Mapping structured metadata sections correctly","Financial summary validation","Handling political advertising compliance text"],"verdict":"failed","score":null,"score_total":null,"note":"Misreads at least one alphanumeric identifier: line 1's source Ad-ID NRCCWI071005 is extracted as NRCCW1071005.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/76c0b83ecf0b486897b0b8cea6765956.png?v=1","role":"input","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-invoice-ad-id-line-item-1-24f2be5f6668.png","role":"input","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/90a9cfea70bb42c1bc3838858f1543dd.png?v=1","role":"output","alt":null}],"run_id":"a061b9e7-a9c5-443d-a171-b296aaf51b8c","study_title":"Extract and query structured data from documents using natural language","study_kind":"generation","research_task":"86b9y25e5","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/e0e9f678bcde4e29a492a11927fc38f7.pdf?v=1","filename":"Invoice PDF.pdf","alt":"Invoice PDF","role":"input"}],"modality":"pdf","stresses":["Nested line-item hierarchy extraction","Multi-page continuity across a page break","Precision on large dollar amounts and totals","Parsing complex time slots and day patterns","Extraction of Ad IDs and reconciliation codes","Mapping structured metadata sections correctly","Financial summary validation","Handling political advertising compliance text"]},"tool_page_slug":"landing-ai","tool_url":"https://aidemos.com/tools/landing-ai","permalink":"https://aidemos.com/evidence/b808d6bb-8133-43cb-8a88-7d5e3160cf25","api_url":"https://ai.aidemos.com/v1/observations/b808d6bb-8133-43cb-8a88-7d5e3160cf25"},"peers":[{"id":"327577f6-cc57-4ed2-8146-f26a03913c26","tool":"datalab","tool_name":"Datalab","verdict":"worked","score":null,"score_total":null,"note":"Recovers invoice metadata and financial totals accurately: invoice number 4064621-1, invoice date 10/28/12, gross total 29750, agency commission 4462.5, net amount due 25287.5, and 8 aired spots match the source.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-datalab-invoice-metadat-1c39e0c1c45e.png","evidence_url":"https://aidemos.com/evidence/327577f6-cc57-4ed2-8146-f26a03913c26"},{"id":"b6718c52-9121-4973-a622-4f556cc10d90","tool":"docsumo","tool_name":"Docsumo","verdict":"failed","score":null,"score_total":null,"note":"The Advertiser Code field is cross-contaminated with an unrelated field label/value: it should be 155, but the extracted cell shows Net Amount Due instead.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/bcbadb3747b74bdfb9d02de4b0e8b245.png?v=1","evidence_url":"https://aidemos.com/evidence/b6718c52-9121-4973-a622-4f556cc10d90"},{"id":"cb0c13d5-8296-4248-ad18-7b40db08339e","tool":"extend-ai","tool_name":"Extend AI","verdict":"failed","score":null,"score_total":null,"note":"It over-reads the payment-term field, outputting `Payment Terms 30 Days` instead of the requested `30 Days`.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/6f879e7a7b8049c2b0491fbb731a0cfa.png?v=1","evidence_url":"https://aidemos.com/evidence/cb0c13d5-8296-4248-ad18-7b40db08339e"},{"id":"aa43fa92-f38b-4722-b3e3-4c84e5e0fcc6","tool":"llamaparse","tool_name":"LlamaParse","verdict":"worked","score":null,"score_total":null,"note":"The invoice totals and headline metadata are extracted consistently: aired_spots is 8, gross_total 29750, agency_commission 4462.5, net_amount_due 25287.5, and payment_terms 30 Days match the source invoice.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-invoice-summary-9c05de684d8a.png","evidence_url":"https://aidemos.com/evidence/aa43fa92-f38b-4722-b3e3-4c84e5e0fcc6"},{"id":"768e3232-9212-4371-b4c7-aaf88280dd39","tool":"nanonets","tool_name":"Nanonets","verdict":"worked","score":null,"score_total":null,"note":"Extracts invoice and finance values accurately, including invoice number 4064621-1 and summary totals of 8 aired spots, 29750 gross, 4462.5 commission, and 25287.5 net amount due.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/d4561ca316f744548da52bc2e19a97de.png?v=1","evidence_url":"https://aidemos.com/evidence/768e3232-9212-4371-b4c7-aaf88280dd39"},{"id":"4aab1e32-bad9-4822-9f12-c8f24964cd19","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Matches the invoice's financial values in the summary, including aired_spots, gross_total, agency_commission, and net_amount_due, with the report stating the summary matches the source document.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-reducto-invoice-summary-915cda528c4b.png","evidence_url":"https://aidemos.com/evidence/4aab1e32-bad9-4822-9f12-c8f24964cd19"},{"id":"becc13cd-085e-4764-9236-2a1a4b27286c","tool":"retab","tool_name":"Retab","verdict":"mixed","score":null,"score_total":null,"note":"Retains the source label in payment_terms, returning 'Payment Terms 30 Days' instead of only the requested value, so the field is not fully normalized.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/03e9858033fe4660abc1ffeb342a2d78.png?v=1","evidence_url":"https://aidemos.com/evidence/becc13cd-085e-4764-9236-2a1a4b27286c"},{"id":"970f5805-6ece-482e-8607-5baa9895ea16","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Captures the invoice header fields and financial totals exactly, including invoice_number 4064621-1 and the printed gross_total 29,750.00, agency_commission 4,462.50, and net_amount_due 25,287.50.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/cb167e3d358042c28f510cff62220d39.png?v=1","evidence_url":"https://aidemos.com/evidence/970f5805-6ece-482e-8607-5baa9895ea16"}],"other_criteria":[{"id":"a1af2980-0fa9-4aa3-910d-7dcdb0e75178","criterion":"schema-adherence","criterion_name":"Schema Adherence","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the invoice hierarchy into invoice_metadata, advertiser, station, billing_address, remit_address, flight_dates, line_items, and summary objects rather than returning flat OCR text.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/a1af2980-0fa9-4aa3-910d-7dcdb0e75178"},{"id":"cff21b29-9003-4011-b627-44d5e3ba01dd","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Populates derived scheduling fields such as day_of_week \"M\" and days_pattern \"MTWT\" on invoice line items.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/cff21b29-9003-4011-b627-44d5e3ba01dd"},{"id":"dbf84eab-a8e4-49e1-b7a5-ddf3319de6bc","criterion":"structural-clean-output","criterion_name":"Structural Clean Output","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"Returns directly consumable JSON for the invoice workflow, with the demo reaching extracted results and the report stating the JSON is immediately usable for integration.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/dbf84eab-a8e4-49e1-b7a5-ddf3319de6bc"},{"id":"3638d045-d475-41e7-8277-b423ad99a540","criterion":"table-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Extracts all 8 advertising line items as separate records, preserving the page-break continuation instead of merging or omitting rows.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/3638d045-d475-41e7-8277-b423ad99a540"}],"appears_in":[{"page_type":"ranking","slug":"document-extraction","title":"Best AI Tools for Extracting Structured Data from Business Documents","url":"https://aidemos.com/best/document-extraction","binding":"run"},{"page_type":"tool","slug":"docsumo","title":null,"url":"https://aidemos.com/tools/docsumo","binding":"run"}],"same_scenario":[{"id":"327577f6-cc57-4ed2-8146-f26a03913c26","tool":"datalab","tool_name":"Datalab","verdict":"worked","score":null,"score_total":null,"note":"Recovers invoice metadata and financial totals accurately: invoice number 4064621-1, invoice date 10/28/12, gross total 29750, agency commission 4462.5, net amount due 25287.5, and 8 aired spots match the source."},{"id":"b6718c52-9121-4973-a622-4f556cc10d90","tool":"docsumo","tool_name":"Docsumo","verdict":"failed","score":null,"score_total":null,"note":"The Advertiser Code field is cross-contaminated with an unrelated field label/value: it should be 155, but the extracted cell shows Net Amount Due instead."},{"id":"cb0c13d5-8296-4248-ad18-7b40db08339e","tool":"extend-ai","tool_name":"Extend AI","verdict":"failed","score":null,"score_total":null,"note":"It over-reads the payment-term field, outputting `Payment Terms 30 Days` instead of the requested `30 Days`."},{"id":"aa43fa92-f38b-4722-b3e3-4c84e5e0fcc6","tool":"llamaparse","tool_name":"LlamaParse","verdict":"worked","score":null,"score_total":null,"note":"The invoice totals and headline metadata are extracted consistently: aired_spots is 8, gross_total 29750, agency_commission 4462.5, net_amount_due 25287.5, and payment_terms 30 Days match the source invoice."},{"id":"768e3232-9212-4371-b4c7-aaf88280dd39","tool":"nanonets","tool_name":"Nanonets","verdict":"worked","score":null,"score_total":null,"note":"Extracts invoice and finance values accurately, including invoice number 4064621-1 and summary totals of 8 aired spots, 29750 gross, 4462.5 commission, and 25287.5 net amount due."},{"id":"4aab1e32-bad9-4822-9f12-c8f24964cd19","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Matches the invoice's financial values in the summary, including aired_spots, gross_total, agency_commission, and net_amount_due, with the report stating the summary matches the source document."},{"id":"becc13cd-085e-4764-9236-2a1a4b27286c","tool":"retab","tool_name":"Retab","verdict":"mixed","score":null,"score_total":null,"note":"Retains the source label in payment_terms, returning 'Payment Terms 30 Days' instead of only the requested value, so the field is not fully normalized."},{"id":"970f5805-6ece-482e-8607-5baa9895ea16","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Captures the invoice header fields and financial totals exactly, including invoice_number 4064621-1 and the printed gross_total 29,750.00, agency_commission 4,462.50, and net_amount_due 25,287.50."}]}