{"observation":{"id":"d859ffe7-484f-444c-adcf-a0fe358bcced","tool":"unstract","tool_name":"Unstract","criterion":"extraction-accuracy","criterion_name":"Extraction Accuracy","criterion_definition":"Are field values correct, complete, and free of OCR or parsing errors, including numerical precision on financial fields?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"Correct field values are the core of the job; wrong or incomplete extraction means the tool failed to retrieve the structured data from the document. (3 of 3 judges)","scenario":"bank-statement-pdf","scenario_name":"Bank Statement PDF","group_tag":"business-document-extraction","scenario_description":"A four-page bank statement PDF with dense transaction tables, balance-forward bridges, account metadata, rewards data, and disclaimer text. It was used to stress schema-driven extraction, multi-page continuity, row completeness, and financial numerical accuracy.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/a459a9de9c8a466d8f69b112e04ddad2.pdf?v=1","role":"input","filename":"Bank Statement PDF.pdf"}],"stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with balance-forward bridges","Parsing structured account metadata alongside unstructured transaction descriptions","Numerical accuracy for deposits, withdrawals, running balances, and summaries","Extraction of nested rewards and disclaimer sections"],"verdict":"failed","score":null,"score_total":null,"note":"Understates the derived bank-summary count: summary.total_transactions is reported as 43 even though the PDF contains 51 transactions, a 16% undercount.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","role":"input","alt":"Research media bank statement 2 jul.png"},{"url":"https://cdn.futuresmart.ai/public/aidemos/95d30eaa07b84999934ef7039bd2c57b.png?v=1","role":"context","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-unstract-bank-statement-output-f0bd422ffa9e.json","role":"output","alt":null}],"run_id":"a061b9e7-a9c5-443d-a171-b296aaf51b8c","study_title":"Extract and query structured data from documents using natural language","study_kind":"generation","research_task":"86b9y25e5","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/a459a9de9c8a466d8f69b112e04ddad2.pdf?v=1","filename":"Bank Statement PDF.pdf","alt":"Bank Statement PDF","role":"input"}],"modality":"pdf","stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with balance-forward bridges","Parsing structured account metadata alongside unstructured transaction descriptions","Numerical accuracy for deposits, withdrawals, running balances, and summaries","Extraction of nested rewards and disclaimer sections"]},"tool_page_slug":"unstract","tool_url":"https://aidemos.com/tools/unstract","permalink":"https://aidemos.com/evidence/d859ffe7-484f-444c-adcf-a0fe358bcced","api_url":"https://ai.aidemos.com/v1/observations/d859ffe7-484f-444c-adcf-a0fe358bcced"},"peers":[{"id":"3f57949b-b930-488c-9c76-b9ebc3b73d51","tool":"datalab","tool_name":"Datalab","verdict":"struggled","score":null,"score_total":null,"note":"Misplaces a bank-statement transaction across dates: a row that belongs to 18 Jun is attached to the 19 Jun record after merging, so date association is unreliable.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/80135a758c5a4d2aa2187e8af3f050e8.png?v=1","evidence_url":"https://aidemos.com/evidence/3f57949b-b930-488c-9c76-b9ebc3b73d51"},{"id":"a33acbc5-d5d0-4ccb-a5c1-cbb6138ffa02","tool":"docsumo","tool_name":"Docsumo","verdict":"failed","score":null,"score_total":null,"note":"The bank-statement extractor leaves multiple straightforward metadata fields empty: 1 Bank Name field, both Statement Period dates, all 4 Balances fields, and the Summary Total Transactions value were not populated despite the information being present in the document.","artifact_count":4,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/d492f8a975374cb485db022749070c94.png?v=1","evidence_url":"https://aidemos.com/evidence/a33acbc5-d5d0-4ccb-a5c1-cbb6138ffa02"},{"id":"310a731e-d9ba-4ab5-9782-fb4b8d4dabdb","tool":"extend-ai","tool_name":"Extend AI","verdict":"failed","score":null,"score_total":null,"note":"The summary aggregation is incorrect: `total_transactions` is 49 in the output, while the report says the expected count after exclusions is 40.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/310a731e-d9ba-4ab5-9782-fb4b8d4dabdb"},{"id":"5efe531d-6c34-4551-88f5-d36447fed696","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Preserves statement metadata and balances with exact values, including bank_name \"Standard Chartered\", statement_date \"16 Jul 2019\", currency \"INR\", opening_balance 114453.65, and closing_balance 116149.46.","artifact_count":3,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/5efe531d-6c34-4551-88f5-d36447fed696"},{"id":"a2fd0afa-58a4-4290-92a6-9396e7807f89","tool":"llamaparse","tool_name":"LlamaParse","verdict":"struggled","score":null,"score_total":null,"note":"The tool leaves value_date blank on some bank transactions even though the source statement contains value dates, so transaction metadata is only partially accurate.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-bank-statement-16-jul-value-d-4906fee56b7b.png","evidence_url":"https://aidemos.com/evidence/a2fd0afa-58a4-4290-92a6-9396e7807f89"},{"id":"0102a587-271f-42b7-8426-de7037082e4b","tool":"nanonets","tool_name":"Nanonets","verdict":"worked","score":null,"score_total":null,"note":"Correctly extracts high-level statement values, including account number 42710540422 and total deposits 70986.83, rather than corrupting the header fields.","artifact_count":4,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/0102a587-271f-42b7-8426-de7037082e4b"},{"id":"f4299c5d-6cfa-4e80-8298-ffb066723331","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Correctly extracts the bank statement currency as INR and attaches high-confidence citation metadata, with extract confidence shown as 0.984.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/8bb08fbfe42e417db6ccbeb6262e87f7.png?v=1","evidence_url":"https://aidemos.com/evidence/f4299c5d-6cfa-4e80-8298-ffb066723331"},{"id":"b0b69d62-82c2-408a-ab06-17f4ff60f1c4","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Captures both long bank-statement disclaimer strings as dedicated fields, preserving the insurance_coverage and reporting_period text instead of dropping or flattening it.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/b0b69d62-82c2-408a-ab06-17f4ff60f1c4"}],"other_criteria":[{"id":"7723ce9b-b9fc-4a93-ba5e-6a1782b61721","criterion":"schema-adherence","criterion_name":"Schema Adherence","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Follows the requested nested bank-statement schema instead of flattening the document, populating structured objects such as metadata, account_holder, account, branch, balances, transactions, and summary.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/7723ce9b-b9fc-4a93-ba5e-6a1782b61721"},{"id":"5cf41e1e-0a8d-4000-a164-198ad5e3b579","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Correctly enriches bank rows with transaction_type classification, with the report stating Withdrawal/Deposit was assigned correctly on all 51 transaction rows.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/5cf41e1e-0a8d-4000-a164-198ad5e3b579"},{"id":"661cd077-e5e0-47df-968b-3d030f1ed64d","criterion":"table-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Keeps the bank transaction table complete across all 4 pages, with the report stating the transactions array contains all 51 entries and no rows were dropped or merged.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/661cd077-e5e0-47df-968b-3d030f1ed64d"}],"appears_in":[{"page_type":"ranking","slug":"document-extraction","title":"Best AI Tools for Extracting Structured Data from Business Documents","url":"https://aidemos.com/best/document-extraction","binding":"run"},{"page_type":"tool","slug":"docsumo","title":null,"url":"https://aidemos.com/tools/docsumo","binding":"run"}],"same_scenario":[{"id":"3f57949b-b930-488c-9c76-b9ebc3b73d51","tool":"datalab","tool_name":"Datalab","verdict":"struggled","score":null,"score_total":null,"note":"Misplaces a bank-statement transaction across dates: a row that belongs to 18 Jun is attached to the 19 Jun record after merging, so date association is unreliable."},{"id":"a33acbc5-d5d0-4ccb-a5c1-cbb6138ffa02","tool":"docsumo","tool_name":"Docsumo","verdict":"failed","score":null,"score_total":null,"note":"The bank-statement extractor leaves multiple straightforward metadata fields empty: 1 Bank Name field, both Statement Period dates, all 4 Balances fields, and the Summary Total Transactions value were not populated despite the information being present in the document."},{"id":"310a731e-d9ba-4ab5-9782-fb4b8d4dabdb","tool":"extend-ai","tool_name":"Extend AI","verdict":"failed","score":null,"score_total":null,"note":"The summary aggregation is incorrect: `total_transactions` is 49 in the output, while the report says the expected count after exclusions is 40."},{"id":"5efe531d-6c34-4551-88f5-d36447fed696","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Preserves statement metadata and balances with exact values, including bank_name \"Standard Chartered\", statement_date \"16 Jul 2019\", currency \"INR\", opening_balance 114453.65, and closing_balance 116149.46."},{"id":"a2fd0afa-58a4-4290-92a6-9396e7807f89","tool":"llamaparse","tool_name":"LlamaParse","verdict":"struggled","score":null,"score_total":null,"note":"The tool leaves value_date blank on some bank transactions even though the source statement contains value dates, so transaction metadata is only partially accurate."},{"id":"0102a587-271f-42b7-8426-de7037082e4b","tool":"nanonets","tool_name":"Nanonets","verdict":"worked","score":null,"score_total":null,"note":"Correctly extracts high-level statement values, including account number 42710540422 and total deposits 70986.83, rather than corrupting the header fields."},{"id":"f4299c5d-6cfa-4e80-8298-ffb066723331","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Correctly extracts the bank statement currency as INR and attaches high-confidence citation metadata, with extract confidence shown as 0.984."},{"id":"b0b69d62-82c2-408a-ab06-17f4ff60f1c4","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Captures both long bank-statement disclaimer strings as dedicated fields, preserving the insurance_coverage and reporting_period text instead of dropping or flattening it."}]}