{"observation":{"id":"3741073d-f776-4beb-b189-4f22c0b5d11e","tool":"datalab","tool_name":"Datalab","criterion":"schema-adherence","criterion_name":"Schema Adherence","criterion_definition":"Does the output follow the supplied JSON schema hierarchy exactly, with correct nesting, field names, and data types?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"If the output does not match the requested JSON schema exactly, the extracted data cannot be reliably consumed or queried, so this is core to the task. (3 of 3 judges)","scenario":"bank-statement-pdf","scenario_name":"Bank Statement PDF","group_tag":"financial-document-extraction","scenario_description":"A 4-page bank statement PDF with 51 transactions, balances, rewards, and disclaimer text, used to test schema-driven extraction of dense financial tables and multi-page continuity.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://d3epheqghktydj.cloudfront.net/extract-and-query-structured-data-from-d-bank-statement-pdf-25ec532be6e1.pdf","role":"input","filename":"Bank Statement PDF.pdf"}],"stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with BALANCE FORWARD bridges","Structured metadata vs. free-text transaction descriptions","Numerical accuracy for balances, deposits, withdrawals, and summaries","Nested schema population for account, branch, balances, rewards, and disclaimers"],"verdict":"worked","score":null,"score_total":null,"note":"Maps the bank statement into the requested nested JSON hierarchy instead of flattening it into OCR text, and preserves field-level citation metadata on the extracted objects.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","role":"input","alt":"Research media bank statement 2 jul.png"},{"url":"https://cdn.futuresmart.ai/public/aidemos/71f9443de6a54eb6b5500ddc4edef9c5.png?v=1","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/2af752eaee0e4dd39af3196828c236cd.png?v=1","role":"output","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-datalab-bank-statement-extracted-citatio-85bb110cea50.png","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/7d75350e90834327ae4be3ef0137b2a5.mp4?v=1","role":"output","alt":null},{"url":"https://d3epheqghktydj.cloudfront.net/research-media-datalab-bank-statement-output-b9efc30215d3.json","role":"output","alt":null}],"run_id":"ec4d736d-95f9-4c88-884c-e280435f7b7b","study_title":"Extract and query structured data from documents using natural language","study_kind":"generation","research_task":"86b9y25e5","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://d3epheqghktydj.cloudfront.net/extract-and-query-structured-data-from-d-bank-statement-pdf-25ec532be6e1.pdf","filename":"Bank Statement PDF.pdf","alt":"Bank Statement PDF","role":"input"}],"modality":"pdf","stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with BALANCE FORWARD bridges","Structured metadata vs. free-text transaction descriptions","Numerical accuracy for balances, deposits, withdrawals, and summaries","Nested schema population for account, branch, balances, rewards, and disclaimers"]},"tool_page_slug":"datalab","tool_url":"https://aidemos.com/tools/datalab","permalink":"https://aidemos.com/evidence/3741073d-f776-4beb-b189-4f22c0b5d11e","api_url":"https://ai.aidemos.com/v1/observations/3741073d-f776-4beb-b189-4f22c0b5d11e"},"peers":[{"id":"a72ab809-ef71-47eb-8461-d9518c96d28b","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The bank output is rebuilt as nested JSON rather than raw OCR, with branch, account, rewards, balances, summary, and metadata objects populated under the requested statement root.","artifact_count":4,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/a72ab809-ef71-47eb-8461-d9518c96d28b"},{"id":"91f210ef-e7ef-4d23-a3c8-badb7719e663","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the bank statement into the requested nested JSON hierarchy, with distinct statement.metadata, account_holder.address, account, branch, statement_period, and balances objects rather than flat OCR text.","artifact_count":3,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/91f210ef-e7ef-4d23-a3c8-badb7719e663"},{"id":"c7ef413a-1654-4252-a5fe-451fa0f92388","tool":"llamaparse","tool_name":"LlamaParse","verdict":"worked","score":null,"score_total":null,"note":"Keeps a nested statement schema intact, emitting separate metadata, account_holder, account, branch, statement_period, balances, transactions, summary, rewards, and disclaimers objects instead of flattening the document.","artifact_count":3,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/c7ef413a-1654-4252-a5fe-451fa0f92388"},{"id":"86ec5538-27d3-490e-88ab-a30bbd4479e8","tool":"nanonets","tool_name":"Nanonets","verdict":"worked","score":null,"score_total":null,"note":"It preserves the requested nested schema directly in the output, populating structured objects such as statement, account, balances, transactions, summary, rewards, and disclaimers instead of flattening the document into OCR text.","artifact_count":3,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/86ec5538-27d3-490e-88ab-a30bbd4479e8"},{"id":"95071c07-2f7a-463b-b7c7-3738ddbd3651","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the bank statement into a nested JSON structure aligned to the requested schema, with document metadata, account, branch, statement period, transactions, summary, and rewards-style sections instead of flat OCR text.","artifact_count":6,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/95071c07-2f7a-463b-b7c7-3738ddbd3651"},{"id":"18014a53-cc1f-4a5d-8552-fc026341e820","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs dense statement OCR into the requested nested JSON hierarchy, populating separate statement, account_holder, account, branch, balances, transactions, rewards, and disclaimers objects instead of flattening everything into text.","artifact_count":4,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/18014a53-cc1f-4a5d-8552-fc026341e820"},{"id":"db903e38-5c91-4688-ac04-2930dc9e72ab","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Preserves the requested nested statement hierarchy instead of flattening it, with separate metadata, account_holder, account, branch, balances, transactions, summary, and other top-level sections.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/6495c5a868854823bcf3cd94c90bb6aa.pdf?v=1","evidence_url":"https://aidemos.com/evidence/db903e38-5c91-4688-ac04-2930dc9e72ab"}],"other_criteria":[{"id":"52cee941-b823-4752-81ff-8803d734fa1a","criterion":"extraction-accuracy","criterion_name":"Extraction Accuracy","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Extracts the visible document-level values correctly, including State Bank of India, 16 Jul 2019, INR, account number 42710540422, account type SMART BANKING SAVINGS ACCOUNT, branch Rajaji Salai, MICR 600036005, IFSC SCBL0036078, and phone 25349005.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/52cee941-b823-4752-81ff-8803d734fa1a"},{"id":"6028e5fa-9c96-4f9c-ad9d-5cc900225ace","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Does not populate schema-derived transaction identifiers at all: the 18 Jun withdrawal keeps transaction_id = null even though the identifier is visible in the source row.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/6028e5fa-9c96-4f9c-ad9d-5cc900225ace"},{"id":"263dafe2-bc8e-4694-addc-8205206940ba","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"Transaction typing is inconsistent on merged rows, with one 28 Jun entry labeled Deposit despite showing a 399 withdrawal amount and another 19 Jun merged row labeled Deposit/Withdrawal.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/263dafe2-bc8e-4694-addc-8205206940ba"},{"id":"3b62a4f3-f371-4b67-a2bb-ab9a024d1726","criterion":"table-and-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Over-extracts the transaction table, returning 54 transactions where the statement was expected to yield 51, which indicates extra continuation or split records were emitted.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/3b62a4f3-f371-4b67-a2bb-ab9a024d1726"},{"id":"f09df643-ec8c-4318-9173-3099db75a12d","criterion":"table-and-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Merges adjacent statement rows into one overly long transaction description, breaking row boundaries on the 21 Jun example instead of keeping the two transactions separate.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/f09df643-ec8c-4318-9173-3099db75a12d"},{"id":"67ba5c20-1ac1-4cad-9a54-eae062a19854","criterion":"table-and-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Can attach a transaction to the wrong date when rows are merged, with an 18 Jun 19 withdrawal appearing under a 19 Jun 19 record in the extracted output.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/67ba5c20-1ac1-4cad-9a54-eae062a19854"}],"appears_in":[{"page_type":"ranking","slug":"document-extraction","title":"Best AI Tools for Extracting Structured Data from PDFs and Business Documents","url":"https://aidemos.com/best/document-extraction","binding":"run"},{"page_type":"tool","slug":"unstract","title":null,"url":"https://aidemos.com/tools/unstract","binding":"run"}],"same_scenario":[{"id":"a72ab809-ef71-47eb-8461-d9518c96d28b","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"The bank output is rebuilt as nested JSON rather than raw OCR, with branch, account, rewards, balances, summary, and metadata objects populated under the requested statement root."},{"id":"91f210ef-e7ef-4d23-a3c8-badb7719e663","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the bank statement into the requested nested JSON hierarchy, with distinct statement.metadata, account_holder.address, account, branch, statement_period, and balances objects rather than flat OCR text."},{"id":"c7ef413a-1654-4252-a5fe-451fa0f92388","tool":"llamaparse","tool_name":"LlamaParse","verdict":"worked","score":null,"score_total":null,"note":"Keeps a nested statement schema intact, emitting separate metadata, account_holder, account, branch, statement_period, balances, transactions, summary, rewards, and disclaimers objects instead of flattening the document."},{"id":"86ec5538-27d3-490e-88ab-a30bbd4479e8","tool":"nanonets","tool_name":"Nanonets","verdict":"worked","score":null,"score_total":null,"note":"It preserves the requested nested schema directly in the output, populating structured objects such as statement, account, balances, transactions, summary, rewards, and disclaimers instead of flattening the document into OCR text."},{"id":"95071c07-2f7a-463b-b7c7-3738ddbd3651","tool":"reducto","tool_name":"Reducto","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs the bank statement into a nested JSON structure aligned to the requested schema, with document metadata, account, branch, statement period, transactions, summary, and rewards-style sections instead of flat OCR text."},{"id":"18014a53-cc1f-4a5d-8552-fc026341e820","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Reconstructs dense statement OCR into the requested nested JSON hierarchy, populating separate statement, account_holder, account, branch, balances, transactions, rewards, and disclaimers objects instead of flattening everything into text."},{"id":"db903e38-5c91-4688-ac04-2930dc9e72ab","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Preserves the requested nested statement hierarchy instead of flattening it, with separate metadata, account_holder, account, branch, balances, transactions, summary, and other top-level sections."}]}