{"observation":{"id":"624ffd2e-c9da-4fb4-acc3-041a5b115a50","tool":"happyscribe","tool_name":"HappyScribe","criterion":"transcription-accuracy","criterion_name":"Transcription Accuracy","criterion_definition":"Word accuracy on the shared call, especially names, tools, numbers, and jargon.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"If the transcript gets names, numbers, and jargon wrong, the note-taker has failed at the core job of capturing the call accurately. (3 of 3 judges)","scenario":"ai-demos-daily-standup-31-july-2026","scenario_name":"AI Demos Daily Standup — 31 July 2026","group_tag":"ai-meeting-notetaker","scenario_description":"A real 25-minute technical engineering daily standup with 14 attendees and about 10 active speakers, used as the single parallel-capture meeting for evaluating AI meeting notetakers on transcription, diarization, summaries, action items, search/chat, and collaboration features.","modality":"image","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/547dd6f13e4a420fa8ad7bf2c88c7612.png?v=1","role":"input","filename":"31-july-meeting-screenshot.png"}],"stresses":["Transcription accuracy for real names, tool names, numbers, and technical jargon","Speaker diarization across multiple active speakers","Robustness to overlapping speech, crosstalk, and rapid turn-taking","Join reliability for bot-based and botless capture","Summary quality on identical source material","Action-item extraction with correct owners and commitments","Topic segmentation of standup updates","Search and chat grounded in the meeting content","Sharing, API, MCP, integrations, plan limits, languages, and privacy feature coverage"],"verdict":"mixed","score":null,"score_total":null,"note":"On this ~25-minute multi-speaker standup, HappyScribe captured the vast majority of names, tools, and jargon correctly, and the report records only 1–2 misheard words.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/768c8f926786411984134803cb5396a5.png?v=1","role":"output","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/64855a79c2564122bac93c704797f365.png?v=1","role":"output","alt":null}],"run_id":"ace58582-3d1e-48ee-996c-9b3cd03f27a2","study_title":"AI Meeting Notetakers — Capture Accurate Transcripts, Summaries & Action Items From Live Calls","study_kind":"generation","research_task":"86baxegnv","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/547dd6f13e4a420fa8ad7bf2c88c7612.png?v=1","filename":"31-july-meeting-screenshot.png","alt":"AI Demos Daily Standup — 31 July 2026","role":"input"}],"modality":"image","stresses":["Transcription accuracy for real names, tool names, numbers, and technical jargon","Speaker diarization across multiple active speakers","Robustness to overlapping speech, crosstalk, and rapid turn-taking","Join reliability for bot-based and botless capture","Summary quality on identical source material","Action-item extraction with correct owners and commitments","Topic segmentation of standup updates","Search and chat grounded in the meeting content","Sharing, API, MCP, integrations, plan limits, languages, and privacy feature coverage"]},"tool_page_slug":"happyscribe","tool_url":"https://aidemos.com/tools/happyscribe","permalink":"https://aidemos.com/evidence/624ffd2e-c9da-4fb4-acc3-041a5b115a50","api_url":"https://ai.aidemos.com/v1/observations/624ffd2e-c9da-4fb4-acc3-041a5b115a50"},"peers":[{"id":"1c4f4fe3-3ed8-490d-992b-a25ddd604669","tool":"fathom","tool_name":"Fathom","verdict":"mixed","score":null,"score_total":null,"note":"Fathom's transcript mostly preserves the meeting's names and technical content, but the report records one confirmed name-level error: \"Mahreen\" was rendered as \"Meryl.\"","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/19b062d60f8e45a6a5fa364948287362.png?v=1","evidence_url":"https://aidemos.com/evidence/1c4f4fe3-3ed8-490d-992b-a25ddd604669"},{"id":"d4b52ad2-b536-4580-99fa-644f4b772a93","tool":"fellow","tool_name":"Fellow","verdict":"worked","score":null,"score_total":null,"note":"The transcript was near-clean: the tool captured nearly all names, technical jargon, and numbers correctly, with no significant misheard terms or hallucinations observed in the tested meeting.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/7476ae6a377e411094940f94f34b21f9.png?v=1","evidence_url":"https://aidemos.com/evidence/d4b52ad2-b536-4580-99fa-644f4b772a93"},{"id":"bac222c7-f7cf-4041-8449-df5aa22c88ac","tool":"fireflies-ai","tool_name":"Fireflies.ai","verdict":"worked","score":null,"score_total":null,"note":"Generated a timestamped transcript view, and the report says the full transcript was very accurate: nearly all names, tools, jargon, and numbers were captured correctly with no significant misheard terms or hallucinations.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/7aecac3bebb2479aa0f18cf6c9f1f0db.png?v=1","evidence_url":"https://aidemos.com/evidence/bac222c7-f7cf-4041-8449-df5aa22c88ac"},{"id":"87d23b55-bc59-4d3e-8547-9b80fc1107f6","tool":"granola","tool_name":"Granola","verdict":"failed","score":null,"score_total":null,"note":"On this 25-minute, multi-speaker standup, Granola’s transcript quality is unreliable: the published excerpt shows garbled phrasing and mistranscribed wording, and the report says the mishearing pattern recurs across early, middle, and late sections rather than being isolated to one moment.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/d3ec039f484d40328057665e51ba34de.png?v=1","evidence_url":"https://aidemos.com/evidence/87d23b55-bc59-4d3e-8547-9b80fc1107f6"},{"id":"5a191ed7-02a9-4979-931d-9219e0b75fce","tool":"meetgeek","tool_name":"MeetGeek","verdict":"worked","score":null,"score_total":null,"note":"It transcribes a normal ~25-minute, ~10-active-speaker engineering standup mostly accurately, with only minor proper-noun/term drift noted in the report; one example given is \"Madin\" being misheard for \"Mahreen\".","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/f3fbc51e628c42c19595b82c5e481a25.png?v=1","evidence_url":"https://aidemos.com/evidence/5a191ed7-02a9-4979-931d-9219e0b75fce"},{"id":"893f5543-bd03-4e2a-ba30-f6426940628b","tool":"notta","tool_name":"Notta","verdict":"worked","score":null,"score_total":null,"note":"Notta’s transcript capture was accurate on the evaluated standup: the report says it correctly captured names, tool names, numbers, and engineering jargon with no significant word-level errors, silent hallucinations, or misheard terms.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/86173a6a33c7494b96de40392d0740b6.png?v=1","evidence_url":"https://aidemos.com/evidence/893f5543-bd03-4e2a-ba30-f6426940628b"},{"id":"0efa5738-c840-44ba-a72a-b35d289e70fb","tool":"otter-ai","tool_name":"Otter.ai","verdict":"worked","score":null,"score_total":null,"note":"Otter generated a full transcript for the standup and, per the report, captured names, tool names, jargon, and numbers correctly with minimal errors, making the transcript reliable for reference.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/0db942cf24dd4c3f9383ac4f5208e840.png?v=1","evidence_url":"https://aidemos.com/evidence/0efa5738-c840-44ba-a72a-b35d289e70fb"}],"other_criteria":[{"id":"a3626a76-5657-440e-9460-f49e46ad880b","criterion":"action-item-extraction","criterion_name":"Action-Item Extraction","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"It extracted the real action item about updating logs, but owner attribution was wrong because the misheard name cascaded into the action item and showed 'Nadine' instead of 'Mahreen.'","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/a3626a76-5657-440e-9460-f49e46ad880b"},{"id":"09e56c77-5be8-4485-90bc-76a453f3197a","criterion":"chat-with-notes-ask-questions","criterion_name":"Chat with Notes / Ask Questions","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"AI chat answers a meeting question with a grounded transcript-backed response, returning that the call was scheduled for '6th August' and explicitly indicating it is reading the transcription.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/09e56c77-5be8-4485-90bc-76a453f3197a"},{"id":"45f5aaac-5c02-4796-8e1c-058a8fce59c8","criterion":"join-method-reliability","criterion_name":"Join Method & Reliability","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The bot joined the Google Meet call and stayed connected through the full meeting capture with no visible dropout or mid-call disconnection.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/45f5aaac-5c02-4796-8e1c-058a8fce59c8"},{"id":"eaee17da-69f8-44d3-8688-a83e219c6972","criterion":"search-across-notes","criterion_name":"Search Across Notes","rank_role":"context","verdict":"struggled","score":null,"score_total":null,"note":"There is no dedicated transcript search UI, so direct keyword lookup across notes is not available from the transcript view and is only routed indirectly through AI Chat.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/eaee17da-69f8-44d3-8688-a83e219c6972"},{"id":"472fb01a-44b0-4713-b649-40ee52da5802","criterion":"speaker-diarization","criterion_name":"Speaker Diarization","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"It identified most speakers, but the report says multiple transcript lines were assigned to the wrong speaker, so speaker-to-statement mapping was not fully reliable across transitions.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/472fb01a-44b0-4713-b649-40ee52da5802"},{"id":"cd60116b-3243-4eea-ab4b-d2f5d7fe58f9","criterion":"summary-quality","criterion_name":"Summary Quality","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The meeting summary is structured into skimmable topic sections rather than one blob, with headings like 'Tool testing & publishing' and 'Access, tracker & expiries.'","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/cd60116b-3243-4eea-ab4b-d2f5d7fe58f9"},{"id":"ab6fc227-dfb2-4524-a6ab-83adb117660f","criterion":"summary-quality","criterion_name":"Summary Quality","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"The summary can hallucinate a person name: the report says it substituted 'Nadine' for 'Mahreen' in a summary bullet, creating a false team-member attribution.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/ab6fc227-dfb2-4524-a6ab-83adb117660f"},{"id":"35b58400-b0b9-4e04-851c-9c0c51b07e1b","criterion":"topic-segmentation","criterion_name":"Topic Segmentation","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"It breaks the standup into logical topic sections that reflect meeting flow, instead of presenting the notes as a single undifferentiated block.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/35b58400-b0b9-4e04-851c-9c0c51b07e1b"}],"appears_in":[{"page_type":"ranking","slug":"ai-meeting-notetakers","title":"Best AI Meeting Notetakers for Accurate Transcripts, Summaries, and Action Items","url":"https://aidemos.com/best/ai-meeting-notetakers","binding":"run"}],"same_scenario":[{"id":"1c4f4fe3-3ed8-490d-992b-a25ddd604669","tool":"fathom","tool_name":"Fathom","verdict":"mixed","score":null,"score_total":null,"note":"Fathom's transcript mostly preserves the meeting's names and technical content, but the report records one confirmed name-level error: \"Mahreen\" was rendered as \"Meryl.\""},{"id":"d4b52ad2-b536-4580-99fa-644f4b772a93","tool":"fellow","tool_name":"Fellow","verdict":"worked","score":null,"score_total":null,"note":"The transcript was near-clean: the tool captured nearly all names, technical jargon, and numbers correctly, with no significant misheard terms or hallucinations observed in the tested meeting."},{"id":"bac222c7-f7cf-4041-8449-df5aa22c88ac","tool":"fireflies-ai","tool_name":"Fireflies.ai","verdict":"worked","score":null,"score_total":null,"note":"Generated a timestamped transcript view, and the report says the full transcript was very accurate: nearly all names, tools, jargon, and numbers were captured correctly with no significant misheard terms or hallucinations."},{"id":"87d23b55-bc59-4d3e-8547-9b80fc1107f6","tool":"granola","tool_name":"Granola","verdict":"failed","score":null,"score_total":null,"note":"On this 25-minute, multi-speaker standup, Granola’s transcript quality is unreliable: the published excerpt shows garbled phrasing and mistranscribed wording, and the report says the mishearing pattern recurs across early, middle, and late sections rather than being isolated to one moment."},{"id":"5a191ed7-02a9-4979-931d-9219e0b75fce","tool":"meetgeek","tool_name":"MeetGeek","verdict":"worked","score":null,"score_total":null,"note":"It transcribes a normal ~25-minute, ~10-active-speaker engineering standup mostly accurately, with only minor proper-noun/term drift noted in the report; one example given is \"Madin\" being misheard for \"Mahreen\"."},{"id":"893f5543-bd03-4e2a-ba30-f6426940628b","tool":"notta","tool_name":"Notta","verdict":"worked","score":null,"score_total":null,"note":"Notta’s transcript capture was accurate on the evaluated standup: the report says it correctly captured names, tool names, numbers, and engineering jargon with no significant word-level errors, silent hallucinations, or misheard terms."},{"id":"0efa5738-c840-44ba-a72a-b35d289e70fb","tool":"otter-ai","tool_name":"Otter.ai","verdict":"worked","score":null,"score_total":null,"note":"Otter generated a full transcript for the standup and, per the report, captured names, tool names, jargon, and numbers correctly with minimal errors, making the transcript reliable for reference."}]}