{"observation":{"id":"67ba5c20-1ac1-4cad-9a54-eae062a19854","tool":"datalab","tool_name":"Datalab","criterion":"table-and-record-completeness","criterion_name":"Table & Record Completeness","criterion_definition":"Are all tabular rows (transactions, line items) extracted without merging, duplication, omission, or phantom records?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"For document data extraction, missing, merged, duplicated, or phantom rows directly corrupt the structured dataset and break downstream querying. (3 of 3 judges)","scenario":"bank-statement-pdf","scenario_name":"Bank Statement PDF","group_tag":"financial-document-extraction","scenario_description":"A 4-page bank statement PDF with 51 transactions, balances, rewards, and disclaimer text, used to test schema-driven extraction of dense financial tables and multi-page continuity.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://d3epheqghktydj.cloudfront.net/extract-and-query-structured-data-from-d-bank-statement-pdf-25ec532be6e1.pdf","role":"input","filename":"Bank Statement PDF.pdf"}],"stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with BALANCE FORWARD bridges","Structured metadata vs. free-text transaction descriptions","Numerical accuracy for balances, deposits, withdrawals, and summaries","Nested schema population for account, branch, balances, rewards, and disclaimers"],"verdict":"failed","score":null,"score_total":null,"note":"Can attach a transaction to the wrong date when rows are merged, with an 18 Jun 19 withdrawal appearing under a 19 Jun 19 record in the extracted output.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/80135a758c5a4d2aa2187e8af3f050e8.png?v=1","role":"input","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/cef4de5010ba43ee8f3a32c9127421a6.png?v=1","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/ae80ff9ae7db40e4b27291b0b308083b.png?v=1","role":"input","alt":null}],"run_id":"ec4d736d-95f9-4c88-884c-e280435f7b7b","study_title":"Extract and query structured data from documents using natural language","study_kind":"generation","research_task":"86b9y25e5","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://d3epheqghktydj.cloudfront.net/extract-and-query-structured-data-from-d-bank-statement-pdf-25ec532be6e1.pdf","filename":"Bank Statement PDF.pdf","alt":"Bank Statement PDF","role":"input"}],"modality":"pdf","stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with BALANCE FORWARD bridges","Structured metadata vs. free-text transaction descriptions","Numerical accuracy for balances, deposits, withdrawals, and summaries","Nested schema population for account, branch, balances, rewards, and disclaimers"]},"tool_page_slug":"datalab","tool_url":"https://aidemos.com/tools/datalab","permalink":"https://aidemos.com/evidence/67ba5c20-1ac1-4cad-9a54-eae062a19854","api_url":"https://ai.aidemos.com/v1/observations/67ba5c20-1ac1-4cad-9a54-eae062a19854"},"peers":[{"id":"3b66a1fa-5d29-4130-8e37-0c4cdc033160","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The extractor keeps transaction rows as separate records, and the report says it captured all 51 transactions without merging adjacent rows.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-extend-ai-extracted-transaction-item-4618ec1646be.png","evidence_url":"https://aidemos.com/evidence/3b66a1fa-5d29-4130-8e37-0c4cdc033160"},{"id":"76a18022-79e8-45a6-b3c9-bacd1c8b7330","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Preserves the full transaction table as 51 separate statement records, without merging rows or dropping entries.","artifact_count":5,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-landingai-bank-statement-output-d3ff94f60bcb.json","evidence_url":"https://aidemos.com/evidence/76a18022-79e8-45a6-b3c9-bacd1c8b7330"},{"id":"b9bfb6b9-ede4-4ea0-ab6e-68c2c5b3895b","tool":"llamaparse","tool_name":"LlamaParse","verdict":"failed","score":null,"score_total":null,"note":"Over-segments the bank-statement transaction table, outputting 54 transaction rows for a statement the report says contains 51 transactions and leaving the summary at 44 total_transactions, so row counts are not self-consistent.","artifact_count":3,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-bank-statement-transaction-co-ea2cee32bed3.png","evidence_url":"https://aidemos.com/evidence/b9bfb6b9-ede4-4ea0-ab6e-68c2c5b3895b"},{"id":"4ecae670-4417-4bc0-98d5-6f6fe27654e0","tool":"nanonets","tool_name":"Nanonets","verdict":"failed","score":null,"score_total":null,"note":"It undercounts the transaction table, extracting 47 transactions when the statement actually contains 51, so 4 records are missing.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-docstrange-nanonets-transaction-count-7b5312c3e84a.png","evidence_url":"https://aidemos.com/evidence/4ecae670-4417-4bc0-98d5-6f6fe27654e0"},{"id":"d7084667-c138-440e-ae8a-85055f890e01","tool":"reducto","tool_name":"Reducto","verdict":"failed","score":null,"score_total":null,"note":"Does not preserve the repeating transaction table faithfully, returning 53 transaction records for a statement that contains 51 transactions.","artifact_count":3,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-reducto-transactions-5f1507b139e2.png","evidence_url":"https://aidemos.com/evidence/d7084667-c138-440e-ae8a-85055f890e01"},{"id":"a0e9e32e-85ea-493d-959a-ab26271ebc7f","tool":"retab","tool_name":"Retab","verdict":"mixed","score":null,"score_total":null,"note":"Overcounts the transaction table summary: total_transactions is 43, while the report says the expected count is 40 after excluding Balance Forward, tax, and charge entries.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-retab-bank-statement-total-transactions-c378712469f2.png","evidence_url":"https://aidemos.com/evidence/a0e9e32e-85ea-493d-959a-ab26271ebc7f"},{"id":"1355f0ad-f96e-476c-9a0d-ab7e11352c1e","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Keeps the transaction table intact across all four pages, with the full 51-row record set present and no merged or dropped rows.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/unstract-unstract-bank-statement-output-f0bd422ffa9e.json","evidence_url":"https://aidemos.com/evidence/1355f0ad-f96e-476c-9a0d-ab7e11352c1e"}],"other_criteria":[{"id":"52cee941-b823-4752-81ff-8803d734fa1a","criterion":"extraction-accuracy","criterion_name":"Extraction Accuracy","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Extracts the visible document-level values correctly, including State Bank of India, 16 Jul 2019, INR, account number 42710540422, account type SMART BANKING SAVINGS ACCOUNT, branch Rajaji Salai, MICR 600036005, IFSC SCBL0036078, and phone 25349005.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/52cee941-b823-4752-81ff-8803d734fa1a"},{"id":"3741073d-f776-4beb-b189-4f22c0b5d11e","criterion":"schema-adherence","criterion_name":"Schema Adherence","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Maps the bank statement into the requested nested JSON hierarchy instead of flattening it into OCR text, and preserves field-level citation metadata on the extracted objects.","artifact_count":6,"evidence_url":"https://aidemos.com/evidence/3741073d-f776-4beb-b189-4f22c0b5d11e"},{"id":"6028e5fa-9c96-4f9c-ad9d-5cc900225ace","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Does not populate schema-derived transaction identifiers at all: the 18 Jun withdrawal keeps transaction_id = null even though the identifier is visible in the source row.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/6028e5fa-9c96-4f9c-ad9d-5cc900225ace"},{"id":"263dafe2-bc8e-4694-addc-8205206940ba","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"Transaction typing is inconsistent on merged rows, with one 28 Jun entry labeled Deposit despite showing a 399 withdrawal amount and another 19 Jun merged row labeled Deposit/Withdrawal.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/263dafe2-bc8e-4694-addc-8205206940ba"}],"appears_in":[{"page_type":"ranking","slug":"document-extraction","title":"Best AI Tools for Extracting Structured Data from PDFs and Business Documents","url":"https://aidemos.com/best/document-extraction","binding":"run"},{"page_type":"tool","slug":"unstract","title":null,"url":"https://aidemos.com/tools/unstract","binding":"run"}],"same_scenario":[{"id":"3b66a1fa-5d29-4130-8e37-0c4cdc033160","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The extractor keeps transaction rows as separate records, and the report says it captured all 51 transactions without merging adjacent rows."},{"id":"76a18022-79e8-45a6-b3c9-bacd1c8b7330","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Preserves the full transaction table as 51 separate statement records, without merging rows or dropping entries."},{"id":"b9bfb6b9-ede4-4ea0-ab6e-68c2c5b3895b","tool":"llamaparse","tool_name":"LlamaParse","verdict":"failed","score":null,"score_total":null,"note":"Over-segments the bank-statement transaction table, outputting 54 transaction rows for a statement the report says contains 51 transactions and leaving the summary at 44 total_transactions, so row counts are not self-consistent."},{"id":"4ecae670-4417-4bc0-98d5-6f6fe27654e0","tool":"nanonets","tool_name":"Nanonets","verdict":"failed","score":null,"score_total":null,"note":"It undercounts the transaction table, extracting 47 transactions when the statement actually contains 51, so 4 records are missing."},{"id":"d7084667-c138-440e-ae8a-85055f890e01","tool":"reducto","tool_name":"Reducto","verdict":"failed","score":null,"score_total":null,"note":"Does not preserve the repeating transaction table faithfully, returning 53 transaction records for a statement that contains 51 transactions."},{"id":"a0e9e32e-85ea-493d-959a-ab26271ebc7f","tool":"retab","tool_name":"Retab","verdict":"mixed","score":null,"score_total":null,"note":"Overcounts the transaction table summary: total_transactions is 43, while the report says the expected count is 40 after excluding Balance Forward, tax, and charge entries."},{"id":"1355f0ad-f96e-476c-9a0d-ab7e11352c1e","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Keeps the transaction table intact across all four pages, with the full 51-row record set present and no merged or dropped rows."}]}