{"observation":{"id":"73b827a9-4822-4d1e-a8ad-b6853c5eab4d","tool":"granola","tool_name":"Granola","criterion":"speaker-diarization","criterion_name":"Speaker Diarization","criterion_definition":"Correctly attributes who said what across a multi-speaker standup.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"Correctly attributing who said what is part of making the transcript and notes trustworthy in multi-speaker meetings. (3 of 3 judges)","scenario":"ai-demos-daily-standup-31-july-2026","scenario_name":"AI Demos Daily Standup — 31 July 2026","group_tag":"ai-meeting-notetaker","scenario_description":"A real 25-minute technical engineering daily standup with 14 attendees and about 10 active speakers, used as the single parallel-capture meeting for evaluating AI meeting notetakers on transcription, diarization, summaries, action items, search/chat, and collaboration features.","modality":"image","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/547dd6f13e4a420fa8ad7bf2c88c7612.png?v=1","role":"input","filename":"31-july-meeting-screenshot.png"}],"stresses":["Transcription accuracy for real names, tool names, numbers, and technical jargon","Speaker diarization across multiple active speakers","Robustness to overlapping speech, crosstalk, and rapid turn-taking","Join reliability for bot-based and botless capture","Summary quality on identical source material","Action-item extraction with correct owners and commitments","Topic segmentation of standup updates","Search and chat grounded in the meeting content","Sharing, API, MCP, integrations, plan limits, languages, and privacy feature coverage"],"verdict":"failed","score":null,"score_total":null,"note":"Granola’s default capture does not attribute speakers: the settings panel shows Speaker tags switched off, and the transcript excerpt is a plain text wall with no speaker labels, so diarization is absent unless the user manually enables it.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/ecd4eac38c5f4f1f9e0d671349a4f71a.png?v=1","role":"context","alt":null},{"url":"https://cdn.futuresmart.ai/public/aidemos/d3ec039f484d40328057665e51ba34de.png?v=1","role":"output","alt":null}],"run_id":"ace58582-3d1e-48ee-996c-9b3cd03f27a2","study_title":"AI Meeting Notetakers — Capture Accurate Transcripts, Summaries & Action Items From Live Calls","study_kind":"generation","research_task":"86baxegnv","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/547dd6f13e4a420fa8ad7bf2c88c7612.png?v=1","filename":"31-july-meeting-screenshot.png","alt":"AI Demos Daily Standup — 31 July 2026","role":"input"}],"modality":"image","stresses":["Transcription accuracy for real names, tool names, numbers, and technical jargon","Speaker diarization across multiple active speakers","Robustness to overlapping speech, crosstalk, and rapid turn-taking","Join reliability for bot-based and botless capture","Summary quality on identical source material","Action-item extraction with correct owners and commitments","Topic segmentation of standup updates","Search and chat grounded in the meeting content","Sharing, API, MCP, integrations, plan limits, languages, and privacy feature coverage"]},"tool_page_slug":"granola","tool_url":"https://aidemos.com/tools/granola","permalink":"https://aidemos.com/evidence/73b827a9-4822-4d1e-a8ad-b6853c5eab4d","api_url":"https://ai.aidemos.com/v1/observations/73b827a9-4822-4d1e-a8ad-b6853c5eab4d"},"peers":[{"id":"e9695d25-df6e-448b-b185-4ad01e885bdf","tool":"fathom","tool_name":"Fathom","verdict":"mixed","score":null,"score_total":null,"note":"Fathom separates most speakers correctly in a busy multi-speaker standup, but the report observed one rapid-transition segment where two speakers' lines were merged into a single speaker block.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/e6b553ff015d4c309c161b0bb12a1660.png?v=1","evidence_url":"https://aidemos.com/evidence/e9695d25-df6e-448b-b185-4ad01e885bdf"},{"id":"5928b605-59f7-4372-a4b7-914060123f74","tool":"fellow","tool_name":"Fellow","verdict":"worked","score":null,"score_total":null,"note":"The transcript attributed speaker turns correctly across the standup, with all ~10 speakers labeled by name and no attribution errors or generic labels reported.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/f6c095044c7a4498b82f300b683163dd.png?v=1","evidence_url":"https://aidemos.com/evidence/5928b605-59f7-4372-a4b7-914060123f74"},{"id":"a1eae6d9-4eff-41c0-9357-c4b9c8e6f7fb","tool":"fireflies-ai","tool_name":"Fireflies.ai","verdict":"worked","score":null,"score_total":null,"note":"Attributed consecutive turns to distinct speakers in the transcript, and the report says speaker identification was almost complete with only minor attribution errors.","artifact_count":3,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/04c680b706994807aa25f0135512b6ee.png?v=1","evidence_url":"https://aidemos.com/evidence/a1eae6d9-4eff-41c0-9357-c4b9c8e6f7fb"},{"id":"472fb01a-44b0-4713-b649-40ee52da5802","tool":"happyscribe","tool_name":"HappyScribe","verdict":"mixed","score":null,"score_total":null,"note":"It identified most speakers, but the report says multiple transcript lines were assigned to the wrong speaker, so speaker-to-statement mapping was not fully reliable across transitions.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/64855a79c2564122bac93c704797f365.png?v=1","evidence_url":"https://aidemos.com/evidence/472fb01a-44b0-4713-b649-40ee52da5802"},{"id":"4f7e5b21-ff4f-4846-9a07-3219c0659681","tool":"meetgeek","tool_name":"MeetGeek","verdict":"mixed","score":null,"score_total":null,"note":"It identifies most speakers in a multi-speaker standup, but leaves at least one utterance as \"Unknown speaker\" and misattributes some lines to the wrong speaker, so attribution is not fully reliable.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/a909947fdbe64d41b2bc10dca0f44193.png?v=1","evidence_url":"https://aidemos.com/evidence/4f7e5b21-ff4f-4846-9a07-3219c0659681"},{"id":"c1fc0701-9a26-4f73-a1f2-d4dfae6a7109","tool":"notta","tool_name":"Notta","verdict":"struggled","score":null,"score_total":null,"note":"The transcript contained a line labeled with another notetaker’s name (HappyScribe), which indicates cross-tool contamination or labeling error and breaks speaker attribution for that segment.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/665e1b0e39b34e40bb46f742733e7db7.png?v=1","evidence_url":"https://aidemos.com/evidence/c1fc0701-9a26-4f73-a1f2-d4dfae6a7109"},{"id":"9552a2b6-0bc3-4802-8f1b-f9cf98c13b46","tool":"otter-ai","tool_name":"Otter.ai","verdict":"failed","score":null,"score_total":null,"note":"Otter’s diarization was effectively unusable in this multi-speaker standup: only 1 of about 10 active speakers was identified by name, while the other 9 were left as generic labels or unattributed, which the report summarizes as a 90% failure rate.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/c2c822851e2f4e71a1ae3b14247c11d3.png?v=1","evidence_url":"https://aidemos.com/evidence/9552a2b6-0bc3-4802-8f1b-f9cf98c13b46"}],"other_criteria":[{"id":"9af81f24-1ac8-4a6a-b543-1959eafc22b0","criterion":"action-item-extraction","criterion_name":"Action-Item Extraction","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Granola extracts concrete next steps with ownership: the visible action item says to create a subtask and add details, names Mahreen Fathima as owner, and marks the item for same-day follow-up; the report says the extracted list contained no false positives.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/9af81f24-1ac8-4a6a-b543-1959eafc22b0"},{"id":"633e805b-8efa-4b9f-8667-7b79bb93072c","criterion":"chat-with-notes-ask-questions","criterion_name":"Chat with Notes / Ask Questions","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"The chat/Q&A surface gives grounded answers from the meeting record: on the tool-access question it says access was confirmed that day, cites both the notes and transcript, and identifies rerunning testing as the next step.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/633e805b-8efa-4b9f-8667-7b79bb93072c"},{"id":"e181aa66-75ab-4b14-a804-0bb3f6481466","criterion":"editability","criterion_name":"Editability","rank_role":"context","verdict":"mixed","score":null,"score_total":null,"note":"Granola supports editing the summary layer, but transcript text is not editable in-app; the report describes transcript correction as locked by design, so users can fix notes and action items but not the underlying transcript.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/e181aa66-75ab-4b14-a804-0bb3f6481466"},{"id":"c83c6126-1c92-4f50-a102-cbdfe3713f0f","criterion":"join-method-reliability","criterion_name":"Join Method & Reliability","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The botless desktop capture recorded the full ~25-minute meeting end-to-end without visible dropouts; the transcript reaches the call’s closing lines, indicating uninterrupted capture rather than a mid-call failure.","artifact_count":3,"evidence_url":"https://aidemos.com/evidence/c83c6126-1c92-4f50-a102-cbdfe3713f0f"},{"id":"a5c429f5-458e-4d15-807a-a871c71e38be","criterion":"search-across-notes","criterion_name":"Search Across Notes","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"Granola’s note search returns exact-match results with navigation: a query for 'api' produced a 1/1 hit and highlighted the matched word in the transcript, so keyword lookup works directly inside the meeting note.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/a5c429f5-458e-4d15-807a-a871c71e38be"},{"id":"d9b8082d-37a5-4e94-8412-e2a6d59d4aa3","criterion":"summary-quality","criterion_name":"Summary Quality","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Granola produces skimmable summaries with named sections; the meeting output is organized into at least three top-level sections, including Use Case Status and Review Progress, Tool Research and Publishing, and Access Tracker Updates.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/d9b8082d-37a5-4e94-8412-e2a6d59d4aa3"},{"id":"e10b1e9f-289c-4e45-bb24-24d7b3063e1e","criterion":"topic-segmentation","criterion_name":"Topic Segmentation","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"The notes are split into named topic sections instead of one long blob; the published section header 'Diagram Animation and Other Use Cases' and the report’s multi-section summary structure show logical breakpoints for navigation.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/e10b1e9f-289c-4e45-bb24-24d7b3063e1e"},{"id":"87d23b55-bc59-4d3e-8547-9b80fc1107f6","criterion":"transcription-accuracy","criterion_name":"Transcription Accuracy","rank_role":"decisive","verdict":"failed","score":null,"score_total":null,"note":"On this 25-minute, multi-speaker standup, Granola’s transcript quality is unreliable: the published excerpt shows garbled phrasing and mistranscribed wording, and the report says the mishearing pattern recurs across early, middle, and late sections rather than being isolated to one moment.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/87d23b55-bc59-4d3e-8547-9b80fc1107f6"}],"appears_in":[{"page_type":"ranking","slug":"ai-meeting-notetakers","title":"Best AI Meeting Notetakers for Accurate Transcripts, Summaries, and Action Items","url":"https://aidemos.com/best/ai-meeting-notetakers","binding":"run"}],"same_scenario":[{"id":"e9695d25-df6e-448b-b185-4ad01e885bdf","tool":"fathom","tool_name":"Fathom","verdict":"mixed","score":null,"score_total":null,"note":"Fathom separates most speakers correctly in a busy multi-speaker standup, but the report observed one rapid-transition segment where two speakers' lines were merged into a single speaker block."},{"id":"5928b605-59f7-4372-a4b7-914060123f74","tool":"fellow","tool_name":"Fellow","verdict":"worked","score":null,"score_total":null,"note":"The transcript attributed speaker turns correctly across the standup, with all ~10 speakers labeled by name and no attribution errors or generic labels reported."},{"id":"a1eae6d9-4eff-41c0-9357-c4b9c8e6f7fb","tool":"fireflies-ai","tool_name":"Fireflies.ai","verdict":"worked","score":null,"score_total":null,"note":"Attributed consecutive turns to distinct speakers in the transcript, and the report says speaker identification was almost complete with only minor attribution errors."},{"id":"472fb01a-44b0-4713-b649-40ee52da5802","tool":"happyscribe","tool_name":"HappyScribe","verdict":"mixed","score":null,"score_total":null,"note":"It identified most speakers, but the report says multiple transcript lines were assigned to the wrong speaker, so speaker-to-statement mapping was not fully reliable across transitions."},{"id":"4f7e5b21-ff4f-4846-9a07-3219c0659681","tool":"meetgeek","tool_name":"MeetGeek","verdict":"mixed","score":null,"score_total":null,"note":"It identifies most speakers in a multi-speaker standup, but leaves at least one utterance as \"Unknown speaker\" and misattributes some lines to the wrong speaker, so attribution is not fully reliable."},{"id":"c1fc0701-9a26-4f73-a1f2-d4dfae6a7109","tool":"notta","tool_name":"Notta","verdict":"struggled","score":null,"score_total":null,"note":"The transcript contained a line labeled with another notetaker’s name (HappyScribe), which indicates cross-tool contamination or labeling error and breaks speaker attribution for that segment."},{"id":"9552a2b6-0bc3-4802-8f1b-f9cf98c13b46","tool":"otter-ai","tool_name":"Otter.ai","verdict":"failed","score":null,"score_total":null,"note":"Otter’s diarization was effectively unusable in this multi-speaker standup: only 1 of about 10 active speakers was identified by name, while the other 9 were left as generic labels or unattributed, which the report summarizes as a 90% failure rate."}]}