{"observation":{"id":"28c48611-fef5-483c-b57f-183172b3e47b","tool":"aws-transcribe","tool_name":"AWS Transcribe","criterion":"output-quality","criterion_name":"Output quality","criterion_definition":"How accurately the returned transcript matches the human reference transcript, measured by WER.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"WER-based transcript accuracy is the core thing this ranking is about; it directly measures how well the tool transcribes audio. (3 of 3 judges)","scenario":"bilingual-spanish-english-code-switching-speech","scenario_name":"Bilingual Spanish-English code-switching speech","group_tag":"speech-to-text-benchmark","scenario_description":"A Bangor Miami bilingual corpus recording with mid-sentence switches between Spanish and English. It was used to test multilingual recognition, code-switch detection, and preservation of words across language transitions.","modality":"audio","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/8883983a83e848b2a3aef2f1c82c01db.mp3?v=1","role":"input","filename":"mix_language.mp3"}],"stresses":["code-switching detection","multilingual language ID","mid-sentence language transitions","word preservation across language flips","hallucination resistance in bilingual speech"],"verdict":"mixed","score":23.06,"score_total":null,"note":"Transcript quality is middling at 23.06% WER, with 673 substitutions, 665 deletions, and 165 insertions against 6517 reference words; Spanish recall is only 30.0% (24/80 types).","evidence_state":"observed","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/88229f66e1c14211a42b75a2c41db4b8.png?v=1","role":"context","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/dbbf28bc9c45417fbf0a4e69a3dca5dd.png?v=1","role":"context","alt":null}],"run_id":"469de0c2-d727-4f8f-a60e-e3a5bf8e8588","study_title":"Transcribe Audio Accurately — Speech-to-Text Engine Benchmark","study_kind":"generation","research_task":"86baxegpu","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/8883983a83e848b2a3aef2f1c82c01db.mp3?v=1","filename":"mix_language.mp3","alt":"Bilingual Spanish-English code-switching speech","role":"input"}],"modality":"audio","stresses":["code-switching detection","multilingual language ID","mid-sentence language transitions","word preservation across language flips","hallucination resistance in bilingual speech"]},"tool_page_slug":"aws-transcribe","tool_url":"https://aidemos.com/tools/aws-transcribe","permalink":"https://aidemos.com/evidence/28c48611-fef5-483c-b57f-183172b3e47b","api_url":"https://ai.aidemos.com/v1/observations/28c48611-fef5-483c-b57f-183172b3e47b"},"peers":[{"id":"9d4f8145-0a65-45a9-bc6d-0299eef9e293","tool":"assemblyai","tool_name":"AssemblyAI","verdict":"mixed","score":null,"score_total":null,"note":"Accuracy is middling on code-switching speech: 21.04% WER with 589 substitutions, 652 deletions, and 130 insertions against a 6,517-word reference; Spanish token recall is 72.5% (58/80 types), and the run is best of 10 on this input.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/e319813d52304d89a559933409e837ec.png?v=1","evidence_url":"https://aidemos.com/evidence/9d4f8145-0a65-45a9-bc6d-0299eef9e293"},{"id":"061da4b0-2bc7-463e-b526-e5a8492a800e","tool":"deepgram","tool_name":"Deepgram","verdict":"failed","score":38.13,"score_total":null,"note":"Breaks down on code-switching speech: WER is 38.13% with 793 substitutions, 1259 deletions, and 433 insertions against a 6517-word reference, and Spanish token recall is only 3.8% (3/80 types).","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/52600606357c41998634fe876a6f214d.mp3?v=1","evidence_url":"https://aidemos.com/evidence/061da4b0-2bc7-463e-b526-e5a8492a800e"},{"id":"dc0d4b3a-f02b-47bb-9bdb-7fafae6e8e3a","tool":"elevenlabs-scribe","tool_name":"ElevenLabs Scribe","verdict":"struggled","score":28.57,"score_total":null,"note":"Transcript accuracy is weak on code-switching speech, with WER 28.57% and 752 substitutions, 778 deletions, and 332 insertions over a 6,517-word reference.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/1cdb5610e7af4f10abf1e8995c3178f0.png?v=1","evidence_url":"https://aidemos.com/evidence/dc0d4b3a-f02b-47bb-9bdb-7fafae6e8e3a"},{"id":"fcccec3d-9bf1-46cc-8e80-e79e347f3665","tool":"gladia","tool_name":"Gladia","verdict":"failed","score":88.45,"score_total":null,"note":"On bilingual code-switching, the response duplicated across 2 channels, inflating output to 10,765 words against a 6,517-word reference and making the reported 88.45% WER an artefact of duplication rather than a usable accuracy score.","artifact_count":4,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/52600606357c41998634fe876a6f214d.mp3?v=1","evidence_url":"https://aidemos.com/evidence/fcccec3d-9bf1-46cc-8e80-e79e347f3665"},{"id":"a67f900e-85ee-49b2-b23b-49ef57b1f50e","tool":"google-cloud-speech-to-text","tool_name":"Google Cloud Speech-to-Text","verdict":"failed","score":56.13,"score_total":null,"note":"On code-switching speech, the transcript quality is very poor at 56.13% WER, with 716 substitutions, 2884 deletions, and 58 insertions against 6517 reference words, and only 5.0% Spanish token recall (4/80 types).","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/992f66a8f3be4ecb9a38e94bba60576b.png?v=1","evidence_url":"https://aidemos.com/evidence/a67f900e-85ee-49b2-b23b-49ef57b1f50e"},{"id":"3b646019-5692-4034-bc9d-bdced3ff5ff5","tool":"groqcloud","tool_name":"GroqCloud","verdict":"struggled","score":27.94,"score_total":null,"note":"Performs much worse on code-switching speech than on medical narration: WER 27.94% with 636 substitutions, 1052 deletions, and 133 insertions over 6517 reference words, plus only 53.8% Spanish token recall.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/08d9beea3d2349288f4ad5d0f3776231.png?v=1","evidence_url":"https://aidemos.com/evidence/3b646019-5692-4034-bc9d-bdced3ff5ff5"},{"id":"495c8096-6aea-4d43-b87b-1511f11247a8","tool":"openai-speech-to-text","tool_name":"OpenAI Speech-to-Text","verdict":"struggled","score":26.12,"score_total":null,"note":"Transcription quality degrades sharply on code-switching: WER 26.12% with 623 substitutions, 947 deletions, and 132 insertions against 6517 reference words; Spanish token recall is only 46.2% (37/80).","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/af7e0d10892e43b88699378d922e0e08.png?v=1","evidence_url":"https://aidemos.com/evidence/495c8096-6aea-4d43-b87b-1511f11247a8"},{"id":"764c36aa-273f-4766-8408-c3a09834d172","tool":"rev-ai","tool_name":"Rev AI","verdict":"struggled","score":25.16,"score_total":null,"note":"On the code-switching sample, it reached WER 25.16% against a 6517-word reference, with 691 substitutions, 793 deletions and 156 insertions; Spanish token recall was only 8.8% (7/80 types), so it dropped or anglicised most Spanish material.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/52600606357c41998634fe876a6f214d.mp3?v=1","evidence_url":"https://aidemos.com/evidence/764c36aa-273f-4766-8408-c3a09834d172"},{"id":"9ac338da-02d8-4078-91d8-e0d6a5910839","tool":"speechmatics","tool_name":"Speechmatics","verdict":"struggled","score":25.06,"score_total":null,"note":"The transcript struggles on code-switching speech, with 25.06% WER (560 substitutions, 981 deletions, 92 insertions over 6517 reference words) and only 12.5% Spanish token recall (10/80 types).","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/45e5abbf1e184a9890ebb7fb4e05d09f.png?v=1","evidence_url":"https://aidemos.com/evidence/9ac338da-02d8-4078-91d8-e0d6a5910839"}],"other_criteria":[{"id":"00f8839f-963c-4043-8b0a-9f374a3daec5","criterion":"export","criterion_name":"Export","rank_role":"context","verdict":"worked","score":3.0,"score_total":3.0,"note":"Returns a full developer payload with payload depth 3/3, 6311 word-level timed tokens, confidence values, and speaker labels; the raw JSON walk spans 7 levels and 21039 objects.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/00f8839f-963c-4043-8b0a-9f374a3daec5"}],"appears_in":[{"page_type":"ranking","slug":"speech-to-text-apis","title":"Best AI Tools for Accurate Speech-to-Text on Hard Audio","url":"https://aidemos.com/best/speech-to-text-apis","binding":"run"}],"same_scenario":[{"id":"9d4f8145-0a65-45a9-bc6d-0299eef9e293","tool":"assemblyai","tool_name":"AssemblyAI","verdict":"mixed","score":null,"score_total":null,"note":"Accuracy is middling on code-switching speech: 21.04% WER with 589 substitutions, 652 deletions, and 130 insertions against a 6,517-word reference; Spanish token recall is 72.5% (58/80 types), and the run is best of 10 on this input."},{"id":"061da4b0-2bc7-463e-b526-e5a8492a800e","tool":"deepgram","tool_name":"Deepgram","verdict":"failed","score":38.13,"score_total":null,"note":"Breaks down on code-switching speech: WER is 38.13% with 793 substitutions, 1259 deletions, and 433 insertions against a 6517-word reference, and Spanish token recall is only 3.8% (3/80 types)."},{"id":"dc0d4b3a-f02b-47bb-9bdb-7fafae6e8e3a","tool":"elevenlabs-scribe","tool_name":"ElevenLabs Scribe","verdict":"struggled","score":28.57,"score_total":null,"note":"Transcript accuracy is weak on code-switching speech, with WER 28.57% and 752 substitutions, 778 deletions, and 332 insertions over a 6,517-word reference."},{"id":"fcccec3d-9bf1-46cc-8e80-e79e347f3665","tool":"gladia","tool_name":"Gladia","verdict":"failed","score":88.45,"score_total":null,"note":"On bilingual code-switching, the response duplicated across 2 channels, inflating output to 10,765 words against a 6,517-word reference and making the reported 88.45% WER an artefact of duplication rather than a usable accuracy score."},{"id":"a67f900e-85ee-49b2-b23b-49ef57b1f50e","tool":"google-cloud-speech-to-text","tool_name":"Google Cloud Speech-to-Text","verdict":"failed","score":56.13,"score_total":null,"note":"On code-switching speech, the transcript quality is very poor at 56.13% WER, with 716 substitutions, 2884 deletions, and 58 insertions against 6517 reference words, and only 5.0% Spanish token recall (4/80 types)."},{"id":"3b646019-5692-4034-bc9d-bdced3ff5ff5","tool":"groqcloud","tool_name":"GroqCloud","verdict":"struggled","score":27.94,"score_total":null,"note":"Performs much worse on code-switching speech than on medical narration: WER 27.94% with 636 substitutions, 1052 deletions, and 133 insertions over 6517 reference words, plus only 53.8% Spanish token recall."},{"id":"495c8096-6aea-4d43-b87b-1511f11247a8","tool":"openai-speech-to-text","tool_name":"OpenAI Speech-to-Text","verdict":"struggled","score":26.12,"score_total":null,"note":"Transcription quality degrades sharply on code-switching: WER 26.12% with 623 substitutions, 947 deletions, and 132 insertions against 6517 reference words; Spanish token recall is only 46.2% (37/80)."},{"id":"764c36aa-273f-4766-8408-c3a09834d172","tool":"rev-ai","tool_name":"Rev AI","verdict":"struggled","score":25.16,"score_total":null,"note":"On the code-switching sample, it reached WER 25.16% against a 6517-word reference, with 691 substitutions, 793 deletions and 156 insertions; Spanish token recall was only 8.8% (7/80 types), so it dropped or anglicised most Spanish material."},{"id":"9ac338da-02d8-4078-91d8-e0d6a5910839","tool":"speechmatics","tool_name":"Speechmatics","verdict":"struggled","score":25.06,"score_total":null,"note":"The transcript struggles on code-switching speech, with 25.06% WER (560 substitutions, 981 deletions, 92 insertions over 6517 reference words) and only 12.5% Spanish token recall (10/80 types)."}]}