{"observation":{"id":"15ea3896-1fbe-49fd-9884-4f76e17b1d03","tool":"llamaparse","tool_name":"LlamaParse","criterion":"table-preservation","criterion_name":"Table Preservation","criterion_definition":"Preserves complex table structures, including rows, columns, multi-row headers, and merged-cell relationships in markdown.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"Accurate Markdown conversion of complex PDFs depends on keeping table structure intact, not flattening it into plain text. (3 of 3 judges)","scenario":"scanned-research-paper","scenario_name":"Scanned Research Paper","group_tag":"scanned-research-paper","scenario_description":"An image-only scanned research paper used to stress OCR and layout recovery in a multi-column academic document with figures, charts, tables, captions, and references.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://d3epheqghktydj.cloudfront.net/convert-a-complex-pdf-into-clean-markdow-scanned-research-pdf-7b86de49784d.pdf","role":"input","filename":"Scanned Research PDF.pdf"}],"stresses":["OCR on scanned pages","Multi-column reading order","Figure and chart handling","Table reconstruction from scans","Caption association","Reference extraction","Overall document structure retention"],"verdict":"failed","score":null,"score_total":null,"note":"Fails to faithfully reconstruct grouped headers in a multi-level stand-effects table, making parent-child column relationships ambiguous.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/b1056870c8314bf6aa3c4d4855e9c0d4.png?v=1","role":"input","alt":null}],"run_id":"6e3160de-fe46-4b45-b071-72560b5c5d0e","study_title":"Convert a Complex PDF into Clean Markdown with an API","study_kind":"generation","research_task":"86b9h7t37","tested_at":null,"completeness":"input-only","input":{"state":"files","text":null,"files":[{"url":"https://d3epheqghktydj.cloudfront.net/convert-a-complex-pdf-into-clean-markdow-scanned-research-pdf-7b86de49784d.pdf","filename":"Scanned Research PDF.pdf","alt":"Scanned Research Paper","role":"input"}],"modality":"pdf","stresses":["OCR on scanned pages","Multi-column reading order","Figure and chart handling","Table reconstruction from scans","Caption association","Reference extraction","Overall document structure retention"]},"tool_page_slug":"llamaparse","tool_url":"https://aidemos.com/tools/llamaparse","permalink":"https://aidemos.com/evidence/15ea3896-1fbe-49fd-9884-4f76e17b1d03","api_url":"https://ai.aidemos.com/v1/observations/15ea3896-1fbe-49fd-9884-4f76e17b1d03"},"peers":[{"id":"1dbde32e-95eb-4904-8433-272d8c5e1fdb","tool":"adobe-api","tool_name":"Adobe API","verdict":"worked","score":null,"score_total":null,"note":"Preserves a multi-column scientific table of tree treatments and before/after diameters, keeping the treatment rows and inch/cm subcolumns readable.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/57a862fbcd21461ab376ee98f35f965b.png?v=1","evidence_url":"https://aidemos.com/evidence/1dbde32e-95eb-4904-8433-272d8c5e1fdb"},{"id":"0e8f2c1a-55ca-4c81-8714-336b3ee4936f","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"Rebuilds multi-row tables with the row/column grid intact, preserving treatment rows and year columns in the scanned table extraction.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/9ee519f5fd3d46b69f1c46f207c7ebc8.png?v=1","evidence_url":"https://aidemos.com/evidence/0e8f2c1a-55ca-4c81-8714-336b3ee4936f"},{"id":"0bfcd599-6c54-476e-94c8-d8773b3595ca","tool":"landing-ai","tool_name":"Landing AI","verdict":"mixed","score":null,"score_total":null,"note":"Partially reflects the text that sits between table columns, but the intervening content still disrupts the table structure in the parsed output.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/fc95e14e58844709b2619a3f3621bc7a.png?v=1","evidence_url":"https://aidemos.com/evidence/0bfcd599-6c54-476e-94c8-d8773b3595ca"},{"id":"de43911e-dc72-4aee-a02a-2145d2d7acfb","tool":"mistral-ai","tool_name":"Mistral AI","verdict":"failed","score":null,"score_total":null,"note":"Broken column boundaries and disrupted value alignment make the reconstructed table significantly less faithful to the source.","artifact_count":4,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/56ad045174884d278b8ec196b932b07f.png?v=1","evidence_url":"https://aidemos.com/evidence/de43911e-dc72-4aee-a02a-2145d2d7acfb"},{"id":"d9350bf4-9574-4634-8293-5ced2ab60d9c","tool":"nutrient-io","tool_name":"Nutrient.io","verdict":"failed","score":null,"score_total":null,"note":"Breaks more complex multi-level tables, with rows and merged cells misaligned or lost once the hierarchy becomes dense.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/aca6328506da49aeb716e311b50a278c.png?v=1","evidence_url":"https://aidemos.com/evidence/d9350bf4-9574-4634-8293-5ced2ab60d9c"},{"id":"fcad1a88-bccf-4eea-8bf5-0d8e5dbc22ee","tool":"reducto","tool_name":"Reducto","verdict":"mixed","score":null,"score_total":null,"note":"Partially reconstructs Table 1: most of the roughly 90 numeric values are exact, but literal 0 values in the 12-inch column become blanks, one mean cell picks up stray digits (33.0 830000), and a row-label-only section header is broadcast across all six columns in one instance.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/463058512d444a0796cf3e61494de04e.png?v=1","evidence_url":"https://aidemos.com/evidence/fcad1a88-bccf-4eea-8bf5-0d8e5dbc22ee"},{"id":"c19621dc-83eb-467f-9210-161a2adcac66","tool":"tensorlake","tool_name":"Tensorlake","verdict":"failed","score":null,"score_total":null,"note":"Struggles with hierarchical scanned tables, misplacing column headers and producing unreliable reconstructions on both the multicolumn table and the denser complex table.","artifact_count":6,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/57a862fbcd21461ab376ee98f35f965b.png?v=1","evidence_url":"https://aidemos.com/evidence/c19621dc-83eb-467f-9210-161a2adcac66"},{"id":"6db07c2d-fbc7-464c-ba3c-822a7495c3d5","tool":"upstage-ai","tool_name":"Upstage AI","verdict":"mixed","score":null,"score_total":null,"note":"Keeps the table values intact but reconstructs the headers incorrectly, leaving a grouped-column table with inconsistent structure.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/57a862fbcd21461ab376ee98f35f965b.png?v=1","evidence_url":"https://aidemos.com/evidence/6db07c2d-fbc7-464c-ba3c-822a7495c3d5"}],"other_criteria":[{"id":"9f0d06be-7ecb-4732-933c-ea6d52a1cc16","criterion":"complex-document-handling","criterion_name":"Complex Document Handling","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Processes a 12-page scanned paper end-to-end and reaches SUCCESS after extracting the page content and figure/table outputs.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/9f0d06be-7ecb-4732-933c-ea6d52a1cc16"},{"id":"fae001a3-d13a-4fa3-8415-862dfb85dca9","criterion":"reading-order-structure","criterion_name":"Reading Order & Structure","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Reflows a two-column scanned page into a single coherent reading order while preserving section-to-body flow.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/fae001a3-d13a-4fa3-8415-862dfb85dca9"},{"id":"04bda6ca-e954-4d0b-93a9-4220eb0b980a","criterion":"text-ocr-completeness","criterion_name":"Text & OCR Completeness","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Recovers dense readable prose from a scanned page-image source, including the section heading and multiple long paragraphs.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/04bda6ca-e954-4d0b-93a9-4220eb0b980a"},{"id":"aef48456-2bfd-45ec-8fb3-91f61fa0605a","criterion":"visual-content-retention","criterion_name":"Visual Content Retention","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"Converts a bar chart into a structured table, preserving the legend/value mapping but not keeping the chart as a visual chart.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/aef48456-2bfd-45ec-8fb3-91f61fa0605a"}],"appears_in":[{"page_type":"ranking","slug":"pdf-to-markdown-apis","title":"Best AI Tools to Convert Complex PDFs into Clean Markdown with an API","url":"https://aidemos.com/best/pdf-to-markdown-apis","binding":"run"}],"same_scenario":[{"id":"1dbde32e-95eb-4904-8433-272d8c5e1fdb","tool":"adobe-api","tool_name":"Adobe API","verdict":"worked","score":null,"score_total":null,"note":"Preserves a multi-column scientific table of tree treatments and before/after diameters, keeping the treatment rows and inch/cm subcolumns readable."},{"id":"0e8f2c1a-55ca-4c81-8714-336b3ee4936f","tool":"extend-ai","tool_name":"Extend AI","verdict":"worked","score":null,"score_total":null,"note":"Rebuilds multi-row tables with the row/column grid intact, preserving treatment rows and year columns in the scanned table extraction."},{"id":"0bfcd599-6c54-476e-94c8-d8773b3595ca","tool":"landing-ai","tool_name":"Landing AI","verdict":"mixed","score":null,"score_total":null,"note":"Partially reflects the text that sits between table columns, but the intervening content still disrupts the table structure in the parsed output."},{"id":"de43911e-dc72-4aee-a02a-2145d2d7acfb","tool":"mistral-ai","tool_name":"Mistral AI","verdict":"failed","score":null,"score_total":null,"note":"Broken column boundaries and disrupted value alignment make the reconstructed table significantly less faithful to the source."},{"id":"d9350bf4-9574-4634-8293-5ced2ab60d9c","tool":"nutrient-io","tool_name":"Nutrient.io","verdict":"failed","score":null,"score_total":null,"note":"Breaks more complex multi-level tables, with rows and merged cells misaligned or lost once the hierarchy becomes dense."},{"id":"fcad1a88-bccf-4eea-8bf5-0d8e5dbc22ee","tool":"reducto","tool_name":"Reducto","verdict":"mixed","score":null,"score_total":null,"note":"Partially reconstructs Table 1: most of the roughly 90 numeric values are exact, but literal 0 values in the 12-inch column become blanks, one mean cell picks up stray digits (33.0 830000), and a row-label-only section header is broadcast across all six columns in one instance."},{"id":"c19621dc-83eb-467f-9210-161a2adcac66","tool":"tensorlake","tool_name":"Tensorlake","verdict":"failed","score":null,"score_total":null,"note":"Struggles with hierarchical scanned tables, misplacing column headers and producing unreliable reconstructions on both the multicolumn table and the denser complex table."},{"id":"6db07c2d-fbc7-464c-ba3c-822a7495c3d5","tool":"upstage-ai","tool_name":"Upstage AI","verdict":"mixed","score":null,"score_total":null,"note":"Keeps the table values intact but reconstructs the headers incorrectly, leaving a grouped-column table with inconsistent structure."}]}