{"observation":{"id":"263dafe2-bc8e-4694-addc-8205206940ba","tool":"datalab","tool_name":"Datalab","criterion":"semantic-field-enrichment","criterion_name":"Semantic Field Enrichment","criterion_definition":"Are derived fields — transaction_type, transaction_id, cheque_number, day patterns, ad codes — correctly classified or extracted beyond raw OCR?","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"This ranking is not just about copying OCR text; it also depends on whether the tool can correctly infer or classify document-specific fields needed for useful structured output. (3 of 3 judges)","scenario":"bank-statement-pdf","scenario_name":"Bank Statement PDF","group_tag":"financial-document-extraction","scenario_description":"A 4-page bank statement PDF with 51 transactions, balances, rewards, and disclaimer text, used to test schema-driven extraction of dense financial tables and multi-page continuity.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://d3epheqghktydj.cloudfront.net/extract-and-query-structured-data-from-d-bank-statement-pdf-25ec532be6e1.pdf","role":"input","filename":"Bank Statement PDF.pdf"}],"stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with BALANCE FORWARD bridges","Structured metadata vs. free-text transaction descriptions","Numerical accuracy for balances, deposits, withdrawals, and summaries","Nested schema population for account, branch, balances, rewards, and disclaimers"],"verdict":"mixed","score":null,"score_total":null,"note":"Transaction typing is inconsistent on merged rows, with one 28 Jun entry labeled Deposit despite showing a 399 withdrawal amount and another 19 Jun merged row labeled Deposit/Withdrawal.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/research-media-datalab-bank-statement-extracted-28-jun--04644cff3353.png","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/cef4de5010ba43ee8f3a32c9127421a6.png?v=1","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/1bf81505ffe0458dad0fcf54696c9ca6.png?v=1","role":"input","alt":null}],"run_id":"ec4d736d-95f9-4c88-884c-e280435f7b7b","study_title":"Extract and query structured data from documents using natural language","study_kind":"generation","research_task":"86b9y25e5","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://d3epheqghktydj.cloudfront.net/extract-and-query-structured-data-from-d-bank-statement-pdf-25ec532be6e1.pdf","filename":"Bank Statement PDF.pdf","alt":"Bank Statement PDF","role":"input"}],"modality":"pdf","stresses":["Table extraction across 50+ transaction rows","Multi-page continuity with BALANCE FORWARD bridges","Structured metadata vs. free-text transaction descriptions","Numerical accuracy for balances, deposits, withdrawals, and summaries","Nested schema population for account, branch, balances, rewards, and disclaimers"]},"tool_page_slug":"datalab","tool_url":"https://aidemos.com/tools/datalab","permalink":"https://aidemos.com/evidence/263dafe2-bc8e-4694-addc-8205206940ba","api_url":"https://ai.aidemos.com/v1/observations/263dafe2-bc8e-4694-addc-8205206940ba"},"peers":[{"id":"414af8f0-b7ac-40ed-914e-051d339dfa2b","tool":"extend-ai","tool_name":"Extend AI","verdict":"failed","score":null,"score_total":null,"note":"It leaves derived `transaction_id` values as `null` even when reference identifiers are present in the description, so identifier extraction does not generalize.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-extend-ai-extracted-transaction-id-fc2e1cc3385b.png","evidence_url":"https://aidemos.com/evidence/414af8f0-b7ac-40ed-914e-051d339dfa2b"},{"id":"a2a1f6f5-e278-415c-afcd-a43a2ec3e91b","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Adds meaningful transaction_type labels to extracted rows, classifying the sample 18 Jun records as Withdrawal, Withdrawal, and Deposit instead of leaving the field as raw OCR text.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-landing-ai-bank-statement-18-jun-extract-321aa2936cc6.png","evidence_url":"https://aidemos.com/evidence/a2a1f6f5-e278-415c-afcd-a43a2ec3e91b"},{"id":"e10bbd24-0b50-44a1-8902-eb59e2fbaf4f","tool":"llamaparse","tool_name":"LlamaParse","verdict":"failed","score":null,"score_total":null,"note":"Fails to derive transaction-level fields, leaving transaction_id and transaction_type empty even for descriptions that encode ATM, UPI, and CRADJ cues.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-llamaparse-bank-statement-transaction-id-c01a0d30b3be.png","evidence_url":"https://aidemos.com/evidence/e10bbd24-0b50-44a1-8902-eb59e2fbaf4f"},{"id":"db96ee62-b038-4189-9112-9ff53de816f8","tool":"nanonets","tool_name":"Nanonets","verdict":"failed","score":null,"score_total":null,"note":"It does not populate the derived transaction_type field, leaving it null across the statement instead of classifying deposits and withdrawals.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/db96ee62-b038-4189-9112-9ff53de816f8"},{"id":"e2f47c60-3f78-4b14-aa96-2e76d8e2483a","tool":"reducto","tool_name":"Reducto","verdict":"failed","score":null,"score_total":null,"note":"Leaves derived transaction metadata incomplete, with transaction_type and transaction_id missing across transaction rows.","artifact_count":3,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-schema-transactions-00d0c871c403.png","evidence_url":"https://aidemos.com/evidence/e2f47c60-3f78-4b14-aa96-2e76d8e2483a"},{"id":"0067b41d-0138-4fc0-80b1-151bb93e2539","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Derives transaction_type and transaction_id on transaction rows, classifying one record as UPI with transaction_id 917615251879 and cheque_number left empty.","artifact_count":2,"thumbnail":"https://d3epheqghktydj.cloudfront.net/research-media-bank-statement-2-jul-a511cf7bd0d6.png","evidence_url":"https://aidemos.com/evidence/0067b41d-0138-4fc0-80b1-151bb93e2539"},{"id":"9c65076c-303b-4d4c-a3f7-619f2667c82e","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Classifies transaction_type correctly across the transaction array, using Deposit and Withdrawal labels rather than raw OCR text.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/6d9450b3c58b477ea0066e52ceb2f6ca.png?v=1","evidence_url":"https://aidemos.com/evidence/9c65076c-303b-4d4c-a3f7-619f2667c82e"}],"other_criteria":[{"id":"52cee941-b823-4752-81ff-8803d734fa1a","criterion":"extraction-accuracy","criterion_name":"Extraction Accuracy","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Extracts the visible document-level values correctly, including State Bank of India, 16 Jul 2019, INR, account number 42710540422, account type SMART BANKING SAVINGS ACCOUNT, branch Rajaji Salai, MICR 600036005, IFSC SCBL0036078, and phone 25349005.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/52cee941-b823-4752-81ff-8803d734fa1a"},{"id":"3741073d-f776-4beb-b189-4f22c0b5d11e","criterion":"schema-adherence","criterion_name":"Schema Adherence","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Maps the bank statement into the requested nested JSON hierarchy instead of flattening it into OCR text, and preserves field-level citation metadata on the extracted objects.","artifact_count":6,"evidence_url":"https://aidemos.com/evidence/3741073d-f776-4beb-b189-4f22c0b5d11e"},{"id":"3b62a4f3-f371-4b67-a2bb-ab9a024d1726","criterion":"table-and-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Over-extracts the transaction table, returning 54 transactions where the statement was expected to yield 51, which indicates extra continuation or split records were emitted.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/3b62a4f3-f371-4b67-a2bb-ab9a024d1726"},{"id":"f09df643-ec8c-4318-9173-3099db75a12d","criterion":"table-and-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Merges adjacent statement rows into one overly long transaction description, breaking row boundaries on the 21 Jun example instead of keeping the two transactions separate.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/f09df643-ec8c-4318-9173-3099db75a12d"},{"id":"67ba5c20-1ac1-4cad-9a54-eae062a19854","criterion":"table-and-record-completeness","criterion_name":"Table & Record Completeness","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Can attach a transaction to the wrong date when rows are merged, with an 18 Jun 19 withdrawal appearing under a 19 Jun 19 record in the extracted output.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/67ba5c20-1ac1-4cad-9a54-eae062a19854"}],"appears_in":[{"page_type":"ranking","slug":"document-extraction","title":"Best AI Tools for Extracting Structured Data from PDFs and Business Documents","url":"https://aidemos.com/best/document-extraction","binding":"run"},{"page_type":"tool","slug":"unstract","title":null,"url":"https://aidemos.com/tools/unstract","binding":"run"}],"same_scenario":[{"id":"414af8f0-b7ac-40ed-914e-051d339dfa2b","tool":"extend-ai","tool_name":"Extend AI","verdict":"failed","score":null,"score_total":null,"note":"It leaves derived `transaction_id` values as `null` even when reference identifiers are present in the description, so identifier extraction does not generalize."},{"id":"a2a1f6f5-e278-415c-afcd-a43a2ec3e91b","tool":"landing-ai","tool_name":"Landing AI","verdict":"worked","score":null,"score_total":null,"note":"Adds meaningful transaction_type labels to extracted rows, classifying the sample 18 Jun records as Withdrawal, Withdrawal, and Deposit instead of leaving the field as raw OCR text."},{"id":"e10bbd24-0b50-44a1-8902-eb59e2fbaf4f","tool":"llamaparse","tool_name":"LlamaParse","verdict":"failed","score":null,"score_total":null,"note":"Fails to derive transaction-level fields, leaving transaction_id and transaction_type empty even for descriptions that encode ATM, UPI, and CRADJ cues."},{"id":"db96ee62-b038-4189-9112-9ff53de816f8","tool":"nanonets","tool_name":"Nanonets","verdict":"failed","score":null,"score_total":null,"note":"It does not populate the derived transaction_type field, leaving it null across the statement instead of classifying deposits and withdrawals."},{"id":"e2f47c60-3f78-4b14-aa96-2e76d8e2483a","tool":"reducto","tool_name":"Reducto","verdict":"failed","score":null,"score_total":null,"note":"Leaves derived transaction metadata incomplete, with transaction_type and transaction_id missing across transaction rows."},{"id":"0067b41d-0138-4fc0-80b1-151bb93e2539","tool":"retab","tool_name":"Retab","verdict":"worked","score":null,"score_total":null,"note":"Derives transaction_type and transaction_id on transaction rows, classifying one record as UPI with transaction_id 917615251879 and cheque_number left empty."},{"id":"9c65076c-303b-4d4c-a3f7-619f2667c82e","tool":"unstract","tool_name":"Unstract","verdict":"worked","score":null,"score_total":null,"note":"Classifies transaction_type correctly across the transaction array, using Deposit and Withdrawal labels rather than raw OCR text."}]}