{"observation":{"id":"bcc9f94f-bf0e-4b9c-b0e7-15f7884ef516","tool":"unstract","tool_name":"Unstract","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","criterion_definition":"Are derived fields such as transaction_type, transaction_id, cheque_number, day patterns, and ad codes correctly classified or extracted beyond raw OCR?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"For this ranking, correctly identifying derived business fields is part of producing usable structured data, not just a nice extra. (3 of 3 judges)","scenario":"invoice-pdf","scenario_name":"Invoice PDF","group_tag":"business-document-extraction","scenario_description":"A two-page broadcast advertising invoice PDF with nested header metadata, billing and remit addresses, and eight line items spanning a page break. It was used to stress hierarchical line-item extraction, amount precision, time/day parsing, code extraction, and summary validation.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/e0e9f678bcde4e29a492a11927fc38f7.pdf?v=1","role":"input","filename":"Invoice PDF.pdf"}],"stresses":["Nested line-item hierarchy extraction","Multi-page continuity across a page break","Precision on large dollar amounts and totals","Parsing complex time slots and day patterns","Extraction of Ad IDs and reconciliation codes","Mapping structured metadata sections correctly","Financial summary validation","Handling political advertising compliance text"],"verdict":"struggled","score":null,"score_total":null,"note":"Mis-reproduces the fixed-width day_of_week scheduling mask, with examples like MTWT- for source MTWT--- and F- - for source ----F--; the report says days_pattern is the more reliable field.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/fb441b46fb5c428aa23cdc126cf23ff9.png?v=1","role":"input","alt":null}],"run_id":"a061b9e7-a9c5-443d-a171-b296aaf51b8c","study_title":"Extract and query structured data from documents using natural language","study_kind":"generation","research_task":"86b9y25e5","tested_at":null,"completeness":"input-only","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/e0e9f678bcde4e29a492a11927fc38f7.pdf?v=1","filename":"Invoice PDF.pdf","alt":"Invoice PDF","role":"input"}],"modality":"pdf","stresses":["Nested line-item hierarchy extraction","Multi-page continuity across a page break","Precision on large dollar amounts and totals","Parsing complex time slots and day patterns","Extraction of Ad IDs and reconciliation codes","Mapping structured metadata sections correctly","Financial summary validation","Handling political advertising compliance text"]},"tool_page_slug":"unstract","tool_url":"https://aidemos.com/tools/unstract","permalink":"https://aidemos.com/evidence/bcc9f94f-bf0e-4b9c-b0e7-15f7884ef516","api_url":"https://ai.aidemos.com/v1/observations/bcc9f94f-bf0e-4b9c-b0e7-15f7884ef516"},"peers":[{"id":"8f9a73fc-cf51-44d4-8845-b997400cabd1","tool":"datalab","tool_name":"Datalab","verdict":"worked","score":null,"score_total":null,"note":"Captures derived invoice line-item fields correctly, including time_slot 9a-10a, day_of_week Su, air_time 9:38 AM, and ad_id NRCCWI071005 on line item 2.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-datalab-invoice-line-item-2-ab8a8332ed35.png","evidence_url":"https://aidemos.com/evidence/8f9a73fc-cf51-44d4-8845-b997400cabd1"},{"id":"19e65f5c-2d63-4f4b-a846-9ff3728522a5","tool":"docsumo","tool_name":"Docsumo","verdict":"worked","score":null,"score_total":null,"note":"The tool correctly splits compound scheduling data for all 8 line items, with Day of Week and Day Pattern both populated correctly instead of being garbled together.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/9beeb894cf5d41ce8390db3f758c1a98.png?v=1","evidence_url":"https://aidemos.com/evidence/19e65f5c-2d63-4f4b-a846-9ff3728522a5"},{"id":"32fb2961-ac21-4fbb-9f04-80b7d3289d32","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The tool correctly derives higher-level line-item fields such as `day_of_week: Tu` and `days_pattern: -T----` alongside ad ID, reference number, and flight-period dates.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/617ec8adf64f49368ea64d88d5a820cd.png?v=1","evidence_url":"https://aidemos.com/evidence/32fb2961-ac21-4fbb-9f04-80b7d3289d32"},{"id":"cff21b29-9003-4011-b627-44d5e3ba01dd","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Populates derived scheduling fields such as day_of_week \"M\" and days_pattern \"MTWT\" on invoice line items.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/90a9cfea70bb42c1bc3838858f1543dd.png?v=1","evidence_url":"https://aidemos.com/evidence/cff21b29-9003-4011-b627-44d5e3ba01dd"},{"id":"a896f611-a121-44e4-961b-fc5e78404a9c","tool":"llamaparse","tool_name":"LlamaParse","verdict":"worked","score":null,"score_total":null,"note":"The tool correctly extracts derived scheduling and coding fields for invoice line items, including day_of_week, days_pattern, air_time, ad_id, time_slot, and flight-period references.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/b99d7ccd04834af3b17ad475cba5cdf7.png?v=1","evidence_url":"https://aidemos.com/evidence/a896f611-a121-44e4-961b-fc5e78404a9c"},{"id":"96339391-c5f5-446c-8ee6-82afe8577102","tool":"nanonets","tool_name":"Nanonets","verdict":"failed","score":null,"score_total":null,"note":"Partially extracts line 8's schedule metadata: flight_period_start, flight_period_end, frequency, and days_pattern are null even though the source row and other records contain scheduling information.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-invoice-line-item-8-ae95b671f1c5.png","evidence_url":"https://aidemos.com/evidence/96339391-c5f5-446c-8ee6-82afe8577102"},{"id":"56df60a9-b511-4fc1-b010-3e0208830afc","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Correctly derives higher-level invoice metadata such as invoice_month, invoice_period_start, and invoice_period_end alongside invoice_number and order_number, showing semantic field extraction beyond raw OCR text.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1b328999a9744f8782faf8d8632db00c.mp4?v=1","evidence_url":"https://aidemos.com/evidence/56df60a9-b511-4fc1-b010-3e0208830afc"},{"id":"767f2c35-435b-4239-986b-f1f0ae6517a8","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Derives scheduling fields from the line item, including day_of_week 'Su', days_pattern '------S', air_time '09:38:00', and time_slot '9a-10a'.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-retab-invoice-extracted-line-item-ff712d23f334.png","evidence_url":"https://aidemos.com/evidence/767f2c35-435b-4239-986b-f1f0ae6517a8"}],"other_criteria":[{"id":"970f5805-6ece-482e-8607-5baa9895ea16","criterion":"extraction-accuracy","criterion_name":"Extraction Accuracy","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Captures the invoice header fields and financial totals exactly, including invoice_number 4064621-1 and the printed gross_total 29,750.00, agency_commission 4,462.50, and net_amount_due 25,287.50.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/970f5805-6ece-482e-8607-5baa9895ea16"},{"id":"87b13324-a0ba-424d-ae9f-8da3844ab650","criterion":"schema-adherence","criterion_name":"Schema Adherence","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Populates the invoice extraction into the requested hierarchy, separating invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary fields.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/87b13324-a0ba-424d-ae9f-8da3844ab650"},{"id":"d025ae3c-59d0-423e-b723-7bfdbdfd3f34","criterion":"table-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Extracts the line-item table without losing the row that crosses the page break; line 8 is returned in full with no duplication or data loss.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/d025ae3c-59d0-423e-b723-7bfdbdfd3f34"}],"appears_in":[{"page_type":"ranking","slug":"document-extraction","title":"Best AI Tools for Extracting Structured Data from Business Documents","url":"https://aidemos.com/best/document-extraction","binding":"run"},{"page_type":"tool","slug":"docsumo","title":null,"url":"https://aidemos.com/tools/docsumo","binding":"run"}],"same_scenario":[{"id":"8f9a73fc-cf51-44d4-8845-b997400cabd1","tool":"datalab","tool_name":"Datalab","verdict":"worked","score":null,"score_total":null,"note":"Captures derived invoice line-item fields correctly, including time_slot 9a-10a, day_of_week Su, air_time 9:38 AM, and ad_id NRCCWI071005 on line item 2."},{"id":"19e65f5c-2d63-4f4b-a846-9ff3728522a5","tool":"docsumo","tool_name":"Docsumo","verdict":"worked","score":null,"score_total":null,"note":"The tool correctly splits compound scheduling data for all 8 line items, with Day of Week and Day Pattern both populated correctly instead of being garbled together."},{"id":"32fb2961-ac21-4fbb-9f04-80b7d3289d32","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The tool correctly derives higher-level line-item fields such as `day_of_week: Tu` and `days_pattern: -T----` alongside ad ID, reference number, and flight-period dates."},{"id":"cff21b29-9003-4011-b627-44d5e3ba01dd","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Populates derived scheduling fields such as day_of_week \"M\" and days_pattern \"MTWT\" on invoice line items."},{"id":"a896f611-a121-44e4-961b-fc5e78404a9c","tool":"llamaparse","tool_name":"LlamaParse","verdict":"worked","score":null,"score_total":null,"note":"The tool correctly extracts derived scheduling and coding fields for invoice line items, including day_of_week, days_pattern, air_time, ad_id, time_slot, and flight-period references."},{"id":"96339391-c5f5-446c-8ee6-82afe8577102","tool":"nanonets","tool_name":"Nanonets","verdict":"failed","score":null,"score_total":null,"note":"Partially extracts line 8's schedule metadata: flight_period_start, flight_period_end, frequency, and days_pattern are null even though the source row and other records contain scheduling information."},{"id":"56df60a9-b511-4fc1-b010-3e0208830afc","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Correctly derives higher-level invoice metadata such as invoice_month, invoice_period_start, and invoice_period_end alongside invoice_number and order_number, showing semantic field extraction beyond raw OCR text."},{"id":"767f2c35-435b-4239-986b-f1f0ae6517a8","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Derives scheduling fields from the line item, including day_of_week 'Su', days_pattern '------S', air_time '09:38:00', and time_slot '9a-10a'."}]}