{"observation":{"id":"0aa94cc5-85f5-48c0-b3c8-b2d350d5f1fb","tool":"skyvern","tool_name":"Skyvern","criterion":"interaction-stability","criterion_name":"Interaction Stability","criterion_definition":"How reliably the tool executes through dynamic loading and interaction steps without losing sync or failing at runtime.","criterion_evidence_type":"capability","criterion_rank_role":"decisive","criterion_rank_role_reason":"Reliably completing dynamic loading and interaction steps is part of successfully scraping pages, not just convenience. (3 of 3 judges)","scenario":"nike-air-force-1-07-size-options-extraction","scenario_name":"Nike Air Force 1 '07 size options extraction","group_tag":null,"scenario_description":"A Nike product page with client-side JavaScript hydration used to test whether a headless scraper waits for dynamic DOM content before extracting product details and all available shoe sizes.","modality":"mixed","input_text":"https://www.nike.com/t/air-force-1-07-mens-shoes-jBrhbr/CW2288-111 — Wait for the size selection options to fully render. Extract the product name, price, and a list of all available shoe sizes.","input_artifact_refs":[],"stresses":[],"verdict":"failed","score":null,"score_total":null,"note":"The live recording pipeline went out of sync during hydration and froze on the initial page view even though backend extraction still completed, showing unreliable handling of dynamic UI state changes.","evidence_state":"verified","source":"first-party","artifacts":[{"url":"https://d3epheqghktydj.cloudfront.net/skyvern-skyvern-extract-nike-af1-product-data.png","role":"output","alt":"Completed extraction of Nike Air Force 1 product data"}],"run_id":"06e1dbd6-5518-4af8-aa1a-735259a75b4f","study_title":"Scrape Web Pages Into Clean Markdown or Structured Data Using AI","study_kind":"backfill","research_task":"86b9jm3a3","tested_at":"2026-06-23T04:32:57.304000+00:00","completeness":"input-and-output","input":{"state":"text","text":"https://www.nike.com/t/air-force-1-07-mens-shoes-jBrhbr/CW2288-111 — Wait for the size selection options to fully render. Extract the product name, price, and a list of all available shoe sizes.","files":[],"modality":"mixed","stresses":[]},"tool_page_slug":"skyvern","tool_url":"https://aidemos.com/tools/skyvern","permalink":"https://aidemos.com/evidence/0aa94cc5-85f5-48c0-b3c8-b2d350d5f1fb","api_url":"https://ai.aidemos.com/v1/observations/0aa94cc5-85f5-48c0-b3c8-b2d350d5f1fb"},"peers":[{"id":"e41889fa-4519-4c96-9183-067dad4dbeb5","tool":"firecrawl","tool_name":"Firecrawl","verdict":"worked","score":null,"score_total":null,"note":"Reliably waited for client-side hydration and captured the complete size set, from M 5 / W 6.5 through M 18 / W 19.5.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/firecrawl-firecrawl-nike-scrape-markdown-output.png","evidence_url":"https://aidemos.com/evidence/e41889fa-4519-4c96-9183-067dad4dbeb5"},{"id":"813f348c-4de5-462c-b815-5c06a67402cc","tool":"jina-ai-reader","tool_name":"Jina AI Reader","verdict":"failed","score":null,"score_total":null,"note":"It did not wait for the client-side size selector to hydrate; the response shows the product title and $115 price but no concrete size inventory values.","artifact_count":1,"thumbnail":"https://d3epheqghktydj.cloudfront.net/jina-ai-reader-jina-reader-nike-product-extraction.png","evidence_url":"https://aidemos.com/evidence/813f348c-4de5-462c-b815-5c06a67402cc"}],"other_criteria":[{"id":"d49869b0-9f36-4e26-b289-a3f2ab804939","criterion":"js-dom-hydration","criterion_name":"JS DOM Hydration","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The tool can wait for client-side hydration and capture the rendered product state, including a populated size-selection grid; in this run it extracted a structured schema of the Nike Air Force 1 '07 size options and showed multiple size variants rather than an empty shell.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/d49869b0-9f36-4e26-b289-a3f2ab804939"},{"id":"aa8c092a-63a7-467f-ae2c-636e6034ac55","criterion":"js-dom-hydration","criterion_name":"JS DOM Hydration","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"The extraction pipeline recovered the hydrated Nike size data, but the screen-recording/visual trace subsystem was out of sync and froze on an initial page view, so the captured recording did not reflect the final dynamic state.","artifact_count":0,"evidence_url":"https://aidemos.com/evidence/aa8c092a-63a7-467f-ae2c-636e6034ac55"},{"id":"ffd61b21-8c43-4795-bacb-ffa1d87744aa","criterion":"js-dom-hydration","criterion_name":"JS DOM Hydration","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Waits for the client-rendered product page to hydrate and extracts the size grid instead of stopping at the initial shell.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/ffd61b21-8c43-4795-bacb-ffa1d87744aa"},{"id":"bb5b1f48-5766-4e41-b40f-989875cac6cc","criterion":"output-quality","criterion_name":"Output Quality","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Returns accurate product data for the hydrated page, including the correct product name Nike Air Force 1 '07 and a long size list; the visible output shows 14 fully readable size pairs from M 7 / W 8.5 through M 14 / W 15.5.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/bb5b1f48-5766-4e41-b40f-989875cac6cc"},{"id":"ce4fbd1b-c3b9-4ce0-a6a0-fe0bf600238f","criterion":"schema-extraction-integrity","criterion_name":"Schema Extraction Integrity","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Can accurately preserve a hydrated page’s structured output, extracting a complete schema of all 22 shoe sizes without corrupting the requested JSON structure.","artifact_count":0,"evidence_url":"https://aidemos.com/evidence/ce4fbd1b-c3b9-4ce0-a6a0-fe0bf600238f"},{"id":"baefb4cb-3fde-43e0-8f39-e0556ffa208d","criterion":"schema-extraction-integrity","criterion_name":"Schema Extraction Integrity","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Accurately extracts hydrated product-size data into structured output; the visible result enumerates multiple size pairs, including M 7 / W 8.5, M 7.5 / W 9, M 8 / W 9.5, and through M 15 / W 16.5 in the shown excerpt.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/baefb4cb-3fde-43e0-8f39-e0556ffa208d"},{"id":"b903a087-1027-4526-92f2-daf9f93cd3b0","criterion":"schema-extraction-integrity","criterion_name":"Schema Extraction Integrity","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Outputs a structured size list with many men/women pairs, with the visible payload spanning at least M 6 / W 7.5 through M 14 / W 15.5.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/b903a087-1027-4526-92f2-daf9f93cd3b0"}],"appears_in":[{"page_type":"ranking","slug":"ai-web-scraping-tools","title":"Best AI Tools to Scrape Web Pages Into Clean Markdown or Structured Data","url":"https://aidemos.com/best/ai-web-scraping-tools","binding":"run"}],"same_scenario":[{"id":"e41889fa-4519-4c96-9183-067dad4dbeb5","tool":"firecrawl","tool_name":"Firecrawl","verdict":"worked","score":null,"score_total":null,"note":"Reliably waited for client-side hydration and captured the complete size set, from M 5 / W 6.5 through M 18 / W 19.5."},{"id":"813f348c-4de5-462c-b815-5c06a67402cc","tool":"jina-ai-reader","tool_name":"Jina AI Reader","verdict":"failed","score":null,"score_total":null,"note":"It did not wait for the client-side size selector to hydrate; the response shows the product title and $115 price but no concrete size inventory values."}]}