{"observation":{"id":"d6a44781-5816-4bba-9774-fad1fe0cbd76","tool":"reducto","tool_name":"Reducto","criterion":"visual-content-retention","criterion_name":"Visual Content Retention","criterion_definition":"Retains charts, figures, diagrams, and images in the output and places them in the correct reading position.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"For complex PDFs, figures, charts, and images are part of the document content, so preserving them in the right place is part of the core job. (3 of 3 judges)","scenario":"hybrid-earnings-report","scenario_name":"Hybrid Earnings Report","group_tag":"hybrid-earnings-report","scenario_description":"A long, real-world hybrid annual report used to benchmark end-to-end PDF-to-markdown conversion. It combines native text, financial tables, charts/graphics, and scanned signature/stamp regions, stressing preservation of document structure, reading order, and embedded visual content across a multi-section report.","modality":"pdf","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/f63a44b441704753b65d1b89dc9b4bbb.pdf?v=1","role":"input","filename":"Target-2015-Annual-Report.pdf"}],"stresses":["Native digital text extraction","Complex financial table preservation","Chart and graphic retention","Image-based signatures and stamps","Reading order across a long multi-section report","Markdown quality and consistency"],"verdict":"failed","score":null,"score_total":null,"note":"Does not retain handwritten signature graphics as images: a signature crop returns only the <signature> placeholder plus the printed name/title, omitting the cursive mark itself.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/415a2411bd9f4557a37c7c11b13ff186.png?v=1","role":"input","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/f7003a47c4a143728c6c57693d0fb8ed.png?v=1","role":"output","alt":null}],"run_id":"6e3160de-fe46-4b45-b071-72560b5c5d0e","study_title":"Convert a Complex PDF into Clean Markdown with an API","study_kind":"generation","research_task":"86b9h7t37","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/f63a44b441704753b65d1b89dc9b4bbb.pdf?v=1","filename":"Target-2015-Annual-Report.pdf","alt":"Hybrid Earnings Report","role":"input"}],"modality":"pdf","stresses":["Native digital text extraction","Complex financial table preservation","Chart and graphic retention","Image-based signatures and stamps","Reading order across a long multi-section report","Markdown quality and consistency"]},"tool_page_slug":"reducto","tool_url":"https://aidemos.com/tools/reducto","permalink":"https://aidemos.com/evidence/d6a44781-5816-4bba-9774-fad1fe0cbd76","api_url":"https://ai.aidemos.com/v1/observations/d6a44781-5816-4bba-9774-fad1fe0cbd76"},"peers":[{"id":"f9dd710a-724a-40da-a8b5-56464ee25af3","tool":"adobe-api","tool_name":"Adobe API","verdict":"worked","score":null,"score_total":null,"note":"Keeps embedded charts and segment graphics inside the report page rather than dropping them, so the extracted page still shows the financial-highlights block and segment-sales graphics.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/07829e6dcb414445886e618cd047efb2.pdf?v=1","evidence_url":"https://aidemos.com/evidence/f9dd710a-724a-40da-a8b5-56464ee25af3"},{"id":"3c7b525c-4c85-4bab-9ff6-9033a037e77c","tool":"extend-ai","tool_name":"Extend AI","verdict":"mixed","score":null,"score_total":null,"note":"Retains charts and logos as standalone figure blocks with captions, but they are extracted out of page position rather than embedded inline.","artifact_count":5,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/ab0bf9cefafd4414a8fbc135950b69e8.png?v=1","evidence_url":"https://aidemos.com/evidence/3c7b525c-4c85-4bab-9ff6-9033a037e77c"},{"id":"97bf4c42-ab87-4b0a-913c-2823e833dd10","tool":"landing-ai","tool_name":"Landing AI","verdict":"failed","score":null,"score_total":null,"note":"Replaces a waterfall chart with text, preserving the SG&A rate values and year-to-year change labels but not the chart as a visual asset in the output.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/d6cecafea14044a1800a5809e34963a2.png?v=1","evidence_url":"https://aidemos.com/evidence/97bf4c42-ab87-4b0a-913c-2823e833dd10"},{"id":"d43040d7-f125-4699-883b-439cb5b46154","tool":"llamaparse","tool_name":"LlamaParse","verdict":"mixed","score":null,"score_total":null,"note":"Represents logos and signatures as textual placeholders or descriptions rather than retaining them as image assets in the output.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/dda8055aeb4747c3885d30a81a2b993c.png?v=1","evidence_url":"https://aidemos.com/evidence/d43040d7-f125-4699-883b-439cb5b46154"},{"id":"457e867a-acde-40fc-ad65-8f9b5f51378d","tool":"mistral-ai","tool_name":"Mistral AI","verdict":"worked","score":null,"score_total":null,"note":"Visual assets are extracted into page-specific folders, keeping charts, signatures, and other document elements tied to their source pages instead of collapsing them into one consolidated output.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/07829e6dcb414445886e618cd047efb2.pdf?v=1","evidence_url":"https://aidemos.com/evidence/457e867a-acde-40fc-ad65-8f9b5f51378d"},{"id":"83d4d64b-462a-467d-bbd9-d67ac9da8675","tool":"nutrient-io","tool_name":"Nutrient.io","verdict":"mixed","score":null,"score_total":null,"note":"Keeps the surrounding 3-signatory signature-page text, but does not extract the handwritten signatures themselves.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1f45fe222ffc4719ad527014bd6fabbf.png?v=1","evidence_url":"https://aidemos.com/evidence/83d4d64b-462a-467d-bbd9-d67ac9da8675"},{"id":"e528b05b-ad28-4fda-a633-eda580b60c6f","tool":"tensorlake","tool_name":"Tensorlake","verdict":"worked","score":null,"score_total":null,"note":"Retains scanned signature-page visuals as figure blocks plus signer text, including named executives and dates, rather than dropping the signature regions from the output.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1f45fe222ffc4719ad527014bd6fabbf.png?v=1","evidence_url":"https://aidemos.com/evidence/e528b05b-ad28-4fda-a633-eda580b60c6f"},{"id":"417a2325-3f7b-4e6f-a06f-e7653967e43e","tool":"upstage-ai","tool_name":"Upstage AI","verdict":"failed","score":null,"score_total":null,"note":"Does not retain handwritten signature content as visual output on the signatures page, leaving only partial surrounding legal text and no clearly identifiable signatures.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1f45fe222ffc4719ad527014bd6fabbf.png?v=1","evidence_url":"https://aidemos.com/evidence/417a2325-3f7b-4e6f-a06f-e7653967e43e"}],"other_criteria":[{"id":"55895767-e0f8-4be6-9d07-f0998a55a02d","criterion":"advanced-features","criterion_name":"Advanced Features","rank_role":"context","verdict":"failed","score":null,"score_total":null,"note":"On the bar-chart schema, extract.run gets only 3 of 5 Sales values right: 2011 and 2012 are silently pulled from a different table ($69.9B and $73.3B), with no confidence or warning signal.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/55895767-e0f8-4be6-9d07-f0998a55a02d"},{"id":"5b164bff-8ffe-46e2-b493-883b4ed9fed4","criterion":"advanced-features","criterion_name":"Advanced Features","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"The separate schema-driven extract.run endpoint cleanly recovers both a financial table and a donut chart: all requested table rows and all five donut percentages come back exactly.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/5b164bff-8ffe-46e2-b493-883b4ed9fed4"},{"id":"7fb155c1-7886-4bd4-9e2d-1fec0f47b677","criterion":"advanced-features","criterion_name":"Advanced Features","rank_role":"context","verdict":"mixed","score":null,"score_total":null,"note":"The parse pipeline’s confidence field catches all 10 footer-number corruptions and also produces one false alarm on a correct securities table, so the signal is real but not perfectly calibrated.","artifact_count":6,"evidence_url":"https://aidemos.com/evidence/7fb155c1-7886-4bd4-9e2d-1fec0f47b677"},{"id":"1961fa7a-2d5a-4ece-83fe-23b4c0399568","criterion":"complex-document-handling","criterion_name":"Complex Document Handling","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"Processes the full 84-page mixed-content report in 25.5 seconds with no truncation and keeps the hardest nested table’s 27 values exact, but 10 footer numbers are corrupted in the back two-thirds of the file.","artifact_count":9,"evidence_url":"https://aidemos.com/evidence/1961fa7a-2d5a-4ece-83fe-23b4c0399568"},{"id":"277d3d32-8789-468a-b996-53077b678a24","criterion":"markdown-quality","criterion_name":"Markdown Quality","rank_role":"decisive","verdict":"struggled","score":null,"score_total":null,"note":"Produces usable but not clean markdown: the output contains 12 <signature> tags, 2 <empty> placeholders, and literal HTML tags such as <b>, <i>, and <u> instead of pure CommonMark.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/277d3d32-8789-468a-b996-53077b678a24"},{"id":"77c69713-eb97-47e0-8b45-b4d8c8902bc2","criterion":"reading-order-structure","criterion_name":"Reading Order & Structure","rank_role":"decisive","verdict":"struggled","score":null,"score_total":null,"note":"Drops heading structure on most of the filing: the FORM 10-K opening area receives no # or ## markup, and the report says 80 of 84 pages stay as plain paragraphs rather than navigable section headings.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/77c69713-eb97-47e0-8b45-b4d8c8902bc2"},{"id":"88babab8-a43b-4c7e-9f67-8dd32e3984bd","criterion":"reading-order-structure","criterion_name":"Reading Order & Structure","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Linearizes a two-column page correctly, reading the full left column, including all four bullets, before the right-column continuation appears, with no interleaving or sequence inversion.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/88babab8-a43b-4c7e-9f67-8dd32e3984bd"},{"id":"0521f27e-23a0-457d-a653-8d443d84ee9b","criterion":"table-preservation","criterion_name":"Table Preservation","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"Mis-merges two independent side-by-side lists into one table, forcing unrelated cost-of-sales and SG&A bullets into fake row pairs where no row correspondence exists in the source.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/0521f27e-23a0-457d-a653-8d443d84ee9b"},{"id":"95351b6f-5f39-4442-81c6-7e668e79d68f","criterion":"table-preservation","criterion_name":"Table Preservation","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Preserves a genuinely row-paired 52-row state table as a clean grid, keeping the 1,792 stores total and 239,539 retail-square-footage total intact.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/95351b6f-5f39-4442-81c6-7e668e79d68f"},{"id":"4171c2f8-81c8-472e-abf5-a809333282b6","criterion":"text-ocr-completeness","criterion_name":"Text & OCR Completeness","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Converts the full 84-page native-digital annual report with no skipped pages; the output contains all 84 page-marker pairs, and checked high-risk numbers survive exactly, including the $51,550,988,273 market-value figure and 599,982,121 shares outstanding.","artifact_count":7,"evidence_url":"https://aidemos.com/evidence/4171c2f8-81c8-472e-abf5-a809333282b6"}],"appears_in":[{"page_type":"ranking","slug":"pdf-to-markdown-apis","title":"Best AI Tools to Convert Complex PDFs into Clean Markdown with an API","url":"https://aidemos.com/best/pdf-to-markdown-apis","binding":"run"}],"same_scenario":[{"id":"f9dd710a-724a-40da-a8b5-56464ee25af3","tool":"adobe-api","tool_name":"Adobe API","verdict":"worked","score":null,"score_total":null,"note":"Keeps embedded charts and segment graphics inside the report page rather than dropping them, so the extracted page still shows the financial-highlights block and segment-sales graphics."},{"id":"3c7b525c-4c85-4bab-9ff6-9033a037e77c","tool":"extend-ai","tool_name":"Extend AI","verdict":"mixed","score":null,"score_total":null,"note":"Retains charts and logos as standalone figure blocks with captions, but they are extracted out of page position rather than embedded inline."},{"id":"97bf4c42-ab87-4b0a-913c-2823e833dd10","tool":"landing-ai","tool_name":"Landing AI","verdict":"failed","score":null,"score_total":null,"note":"Replaces a waterfall chart with text, preserving the SG&A rate values and year-to-year change labels but not the chart as a visual asset in the output."},{"id":"d43040d7-f125-4699-883b-439cb5b46154","tool":"llamaparse","tool_name":"LlamaParse","verdict":"mixed","score":null,"score_total":null,"note":"Represents logos and signatures as textual placeholders or descriptions rather than retaining them as image assets in the output."},{"id":"457e867a-acde-40fc-ad65-8f9b5f51378d","tool":"mistral-ai","tool_name":"Mistral AI","verdict":"worked","score":null,"score_total":null,"note":"Visual assets are extracted into page-specific folders, keeping charts, signatures, and other document elements tied to their source pages instead of collapsing them into one consolidated output."},{"id":"83d4d64b-462a-467d-bbd9-d67ac9da8675","tool":"nutrient-io","tool_name":"Nutrient.io","verdict":"mixed","score":null,"score_total":null,"note":"Keeps the surrounding 3-signatory signature-page text, but does not extract the handwritten signatures themselves."},{"id":"e528b05b-ad28-4fda-a633-eda580b60c6f","tool":"tensorlake","tool_name":"Tensorlake","verdict":"worked","score":null,"score_total":null,"note":"Retains scanned signature-page visuals as figure blocks plus signer text, including named executives and dates, rather than dropping the signature regions from the output."},{"id":"417a2325-3f7b-4e6f-a06f-e7653967e43e","tool":"upstage-ai","tool_name":"Upstage AI","verdict":"failed","score":null,"score_total":null,"note":"Does not retain handwritten signature content as visual output on the signatures page, leaving only partial surrounding legal text and no clearly identifiable signatures."}]}