{"observation":{"id":"18323d3b-d8b4-4f0b-81f7-b08fa341a5f1","tool":"llamaparse","tool_name":"LlamaParse","criterion":"table-record-completeness","criterion_name":"Table & Record Completeness","criterion_definition":"Are all tabular rows and line items extracted without merging, duplication, omission, or phantom records?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"Missing, duplicated, or merged rows and line items directly corrupt the structured dataset and break query results. (3 of 3 judges)","scenario":"bank-statement-pdf","scenario_name":"Bank Statement PDF","group_tag":"business-document-extraction","scenario_description":"A four-page bank statement PDF with dense transaction tables, balance-forward bridges, account metadata, rewards data, and disclaimer text. It was used to stress schema-driven extraction, multi-page continuity, row completeness, and financial numerical accuracy.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/a459a9de9c8a466d8f69b112e04ddad2.pdf?v=1","role":"input","filename":"Bank Statement PDF.pdf"}],"stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with balance-forward bridges","Parsing structured account metadata alongside unstructured transaction descriptions","Numerical accuracy for deposits, withdrawals, running balances, and summaries","Extraction of nested rewards and disclaimer sections"],"verdict":"mixed","score":null,"score_total":null,"note":"The tool preserves transaction rows as separate records and maintains order, but the extracted transaction list is inconsistent in size: the report says the array contains 54 records even though the source statement has 51, and the summary shows 44 total_transactions.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-bank-statement-transaction-it-e75317057cb2.png","role":"output","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-bank-statement-transaction-co-ea2cee32bed3.png","role":"output","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-bank-statement-summary-b8a4b9a06733.png","role":"output","alt":null}],"run_id":"a061b9e7-a9c5-443d-a171-b296aaf51b8c","study_title":"Extract and query structured data from documents using natural language","study_kind":"generation","research_task":"86b9y25e5","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/a459a9de9c8a466d8f69b112e04ddad2.pdf?v=1","filename":"Bank Statement PDF.pdf","alt":"Bank Statement PDF","role":"input"}],"modality":"pdf","stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with balance-forward bridges","Parsing structured account metadata alongside unstructured transaction descriptions","Numerical accuracy for deposits, withdrawals, running balances, and summaries","Extraction of nested rewards and disclaimer sections"]},"tool_page_slug":"llamaparse","tool_url":"https://aidemos.com/tools/llamaparse","permalink":"https://aidemos.com/evidence/18323d3b-d8b4-4f0b-81f7-b08fa341a5f1","api_url":"https://ai.aidemos.com/v1/observations/18323d3b-d8b4-4f0b-81f7-b08fa341a5f1"},"peers":[{"id":"9ff398df-39f0-4ff1-b7cd-d485a9d07b0e","tool":"datalab","tool_name":"Datalab","verdict":"struggled","score":null,"score_total":null,"note":"Merges adjacent bank-statement rows into one record; the 21 Jun crop shows two separate transactions, but the extracted description collapses them into one long UPI/merchant string.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-21-jun-transactions-890dc245679d.png","evidence_url":"https://aidemos.com/evidence/9ff398df-39f0-4ff1-b7cd-d485a9d07b0e"},{"id":"b18f6412-ef8f-41f1-ab39-e2924602dfaa","tool":"docsumo","tool_name":"Docsumo","verdict":"failed","score":null,"score_total":null,"note":"It inflates repeated carry-forward content into extra records, returning 55 rows for a statement that actually contains 52 distinct transactions because the BALANCE FORWARD line is duplicated across page breaks.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/53c2c83aadb146fd8751598ad59ff0f0.png?v=1","evidence_url":"https://aidemos.com/evidence/b18f6412-ef8f-41f1-ab39-e2924602dfaa"},{"id":"9efbfd60-4094-4c43-b099-7166bc9de71c","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Returns the full transaction table as separate records; the report says all 51 statement entries were extracted without collapsing rows or dropping records.","artifact_count":3,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-landingai-output-1-d3ff94f60bcb.json","evidence_url":"https://aidemos.com/evidence/9efbfd60-4094-4c43-b099-7166bc9de71c"},{"id":"af7da413-857d-40dc-b4f4-90cf66d97062","tool":"nanonets","tool_name":"Nanonets","verdict":"failed","score":null,"score_total":null,"note":"Leaves the statement summary incomplete: total_transactions is null, and the report says the run extracted 47 transactions when the source contained 51.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/a6afda6d46f44a9892ef7792ed6fb1f4.png?v=1","evidence_url":"https://aidemos.com/evidence/af7da413-857d-40dc-b4f4-90cf66d97062"},{"id":"ff5e7c6c-df96-4b6c-aaf7-ec4140ca5783","tool":"reducto","tool_name":"Reducto","verdict":"failed","score":null,"score_total":null,"note":"Over-produces the repeating transaction table, returning 53 transaction records when the bank statement contains 51, indicating row-count drift in line-item extraction.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-reducto-transactions-5f1507b139e2.png","evidence_url":"https://aidemos.com/evidence/ff5e7c6c-df96-4b6c-aaf7-ec4140ca5783"},{"id":"483fc6b9-1e25-437c-81ac-fd2e1454a401","tool":"retab","tool_name":"Retab","verdict":"mixed","score":null,"score_total":null,"note":"Overcounts the bank-statement transaction table in the summary, reporting total_transactions 43 when the researcher says 40 should be counted after excluding balance-forward, tax, and charge entries.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-retab-bank-statement-total-transactions-c378712469f2.png","evidence_url":"https://aidemos.com/evidence/483fc6b9-1e25-437c-81ac-fd2e1454a401"},{"id":"661cd077-e5e0-47df-968b-3d030f1ed64d","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Keeps the bank transaction table complete across all 4 pages, with the report stating the transactions array contains all 51 entries and no rows were dropped or merged.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-unstract-bank-statement-output-f0bd422ffa9e.json","evidence_url":"https://aidemos.com/evidence/661cd077-e5e0-47df-968b-3d030f1ed64d"}],"other_criteria":[{"id":"a2fd0afa-58a4-4290-92a6-9396e7807f89","criterion":"extraction-accuracy","criterion_name":"Extraction Accuracy","rank_role":"decisive","verdict":"struggled","score":null,"score_total":null,"note":"The tool leaves value_date blank on some bank transactions even though the source statement contains value dates, so transaction metadata is only partially accurate.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/a2fd0afa-58a4-4290-92a6-9396e7807f89"},{"id":"8fb0c5fe-7b66-4d4c-bd2e-f63382a6c431","criterion":"schema-adherence","criterion_name":"Schema Adherence","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The bank-statement output follows the requested nested schema closely, reconstructing metadata, account_holder, account, branch, statement_period, balances, transactions, summary, rewards, and disclaimers as structured objects rather than flat OCR text.","artifact_count":5,"evidence_url":"https://aidemos.com/evidence/8fb0c5fe-7b66-4d4c-bd2e-f63382a6c431"},{"id":"ef93fe5b-83f1-421a-87bf-2923b9c82aae","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"The tool does not reliably enrich bank transactions with derived identifiers or types: transaction_id and transaction_type are left empty across extracted records even when the descriptions contain transaction clues.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/ef93fe5b-83f1-421a-87bf-2923b9c82aae"}],"appears_in":[{"page_type":"ranking","slug":"document-extraction","title":"Best AI Tools for Extracting Structured Data from Business Documents","url":"https://aidemos.com/best/document-extraction","binding":"run"},{"page_type":"tool","slug":"docsumo","title":null,"url":"https://aidemos.com/tools/docsumo","binding":"run"}],"same_scenario":[{"id":"9ff398df-39f0-4ff1-b7cd-d485a9d07b0e","tool":"datalab","tool_name":"Datalab","verdict":"struggled","score":null,"score_total":null,"note":"Merges adjacent bank-statement rows into one record; the 21 Jun crop shows two separate transactions, but the extracted description collapses them into one long UPI/merchant string."},{"id":"b18f6412-ef8f-41f1-ab39-e2924602dfaa","tool":"docsumo","tool_name":"Docsumo","verdict":"failed","score":null,"score_total":null,"note":"It inflates repeated carry-forward content into extra records, returning 55 rows for a statement that actually contains 52 distinct transactions because the BALANCE FORWARD line is duplicated across page breaks."},{"id":"9efbfd60-4094-4c43-b099-7166bc9de71c","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Returns the full transaction table as separate records; the report says all 51 statement entries were extracted without collapsing rows or dropping records."},{"id":"af7da413-857d-40dc-b4f4-90cf66d97062","tool":"nanonets","tool_name":"Nanonets","verdict":"failed","score":null,"score_total":null,"note":"Leaves the statement summary incomplete: total_transactions is null, and the report says the run extracted 47 transactions when the source contained 51."},{"id":"ff5e7c6c-df96-4b6c-aaf7-ec4140ca5783","tool":"reducto","tool_name":"Reducto","verdict":"failed","score":null,"score_total":null,"note":"Over-produces the repeating transaction table, returning 53 transaction records when the bank statement contains 51, indicating row-count drift in line-item extraction."},{"id":"483fc6b9-1e25-437c-81ac-fd2e1454a401","tool":"retab","tool_name":"Retab","verdict":"mixed","score":null,"score_total":null,"note":"Overcounts the bank-statement transaction table in the summary, reporting total_transactions 43 when the researcher says 40 should be counted after excluding balance-forward, tax, and charge entries."},{"id":"661cd077-e5e0-47df-968b-3d030f1ed64d","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Keeps the bank transaction table complete across all 4 pages, with the report stating the transactions array contains all 51 entries and no rows were dropped or merged."}]}