{"observation":{"id":"1b0800ea-3b02-4159-88c1-ed437a0b12a7","tool":"notta","tool_name":"Notta","criterion":"action-item-extraction","criterion_name":"Action-Item Extraction","criterion_definition":"Finds the real action items, ideally with owner and due date, with low false positives.","criterion_evidence_type":"transformation","criterion_rank_role":"decisive","criterion_rank_role_reason":"Finding the real action items is a central outcome readers are hiring the tool for, not just a convenience feature. (3 of 3 judges)","scenario":"ai-demos-daily-standup-31-july-2026","scenario_name":"AI Demos Daily Standup — 31 July 2026","group_tag":"ai-meeting-notetaker","scenario_description":"A real 25-minute technical engineering daily standup with 14 attendees and about 10 active speakers, used as the single parallel-capture meeting for evaluating AI meeting notetakers on transcription, diarization, summaries, action items, search/chat, and collaboration features.","modality":"image","input_text":null,"input_artifact_refs":[{"alt":null,"url":"https://cdn.futuresmart.ai/public/aidemos/547dd6f13e4a420fa8ad7bf2c88c7612.png?v=1","role":"input","filename":"31-july-meeting-screenshot.png"}],"stresses":["Transcription accuracy for real names, tool names, numbers, and technical jargon","Speaker diarization across multiple active speakers","Robustness to overlapping speech, crosstalk, and rapid turn-taking","Join reliability for bot-based and botless capture","Summary quality on identical source material","Action-item extraction with correct owners and commitments","Topic segmentation of standup updates","Search and chat grounded in the meeting content","Sharing, API, MCP, integrations, plan limits, languages, and privacy feature coverage"],"verdict":"worked","score":null,"score_total":null,"note":"The action-item list extracted the real commitments from the call, formatted them as checkbox items with @mentions, and the report says owner assignment was correct with no false positives.","evidence_state":"verified","source":null,"artifacts":[{"url":"https://cdn.futuresmart.ai/public/aidemos/22b60eda9b5e4b05b6ad287c471e5e3c.png?v=1","role":"output","alt":null}],"run_id":"ace58582-3d1e-48ee-996c-9b3cd03f27a2","study_title":"AI Meeting Notetakers — Capture Accurate Transcripts, Summaries & Action Items From Live Calls","study_kind":"generation","research_task":"86baxegnv","tested_at":null,"completeness":"input-and-output","input":{"state":"files","text":null,"files":[{"url":"https://cdn.futuresmart.ai/public/aidemos/547dd6f13e4a420fa8ad7bf2c88c7612.png?v=1","filename":"31-july-meeting-screenshot.png","alt":"AI Demos Daily Standup — 31 July 2026","role":"input"}],"modality":"image","stresses":["Transcription accuracy for real names, tool names, numbers, and technical jargon","Speaker diarization across multiple active speakers","Robustness to overlapping speech, crosstalk, and rapid turn-taking","Join reliability for bot-based and botless capture","Summary quality on identical source material","Action-item extraction with correct owners and commitments","Topic segmentation of standup updates","Search and chat grounded in the meeting content","Sharing, API, MCP, integrations, plan limits, languages, and privacy feature coverage"]},"tool_page_slug":"notta","tool_url":"https://aidemos.com/tools/notta","permalink":"https://aidemos.com/evidence/1b0800ea-3b02-4159-88c1-ed437a0b12a7","api_url":"https://ai.aidemos.com/v1/observations/1b0800ea-3b02-4159-88c1-ed437a0b12a7"},"peers":[{"id":"8340c79d-88de-4721-9c3e-c58e057071fb","tool":"fathom","tool_name":"Fathom","verdict":"worked","score":null,"score_total":null,"note":"Fathom extracts real commitments into an ACTION ITEMS section with owner attribution; the published output shows timestamped tasks and a named owner on the item.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/9e8a3e7fb669448e959de254a7dcec48.png?v=1","evidence_url":"https://aidemos.com/evidence/8340c79d-88de-4721-9c3e-c58e057071fb"},{"id":"21587b7d-dc6e-4aa1-8ee4-4e5f13641a3e","tool":"fellow","tool_name":"Fellow","verdict":"mixed","score":null,"score_total":null,"note":"Action-item extraction was mostly correct, with real commitments and proper owner assignment for most items, but one real action item was misplaced from Mahreen to Anshika; the report states a 95%+ capture rate.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/2f38a356a98d4395b43001c1d7f6c5b3.png?v=1","evidence_url":"https://aidemos.com/evidence/21587b7d-dc6e-4aa1-8ee4-4e5f13641a3e"},{"id":"737cde44-da22-4dd6-b985-62104c4d14a8","tool":"fireflies-ai","tool_name":"Fireflies.ai","verdict":"worked","score":null,"score_total":null,"note":"Grouped action items by owner, attributed them to the correct team member, and exposed a clickable source timestamp (19:13) for at least one item.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/bd68b01019254a5380cf234ec8ab89a1.png?v=1","evidence_url":"https://aidemos.com/evidence/737cde44-da22-4dd6-b985-62104c4d14a8"},{"id":"9af81f24-1ac8-4a6a-b543-1959eafc22b0","tool":"granola","tool_name":"Granola","verdict":"worked","score":null,"score_total":null,"note":"Granola extracts concrete next steps with ownership: the visible action item says to create a subtask and add details, names Mahreen Fathima as owner, and marks the item for same-day follow-up; the report says the extracted list contained no false positives.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/0a21e41b32d648b1b5206dc034a58201.png?v=1","evidence_url":"https://aidemos.com/evidence/9af81f24-1ac8-4a6a-b543-1959eafc22b0"},{"id":"a3626a76-5657-440e-9460-f49e46ad880b","tool":"happyscribe","tool_name":"HappyScribe","verdict":"mixed","score":null,"score_total":null,"note":"It extracted the real action item about updating logs, but owner attribution was wrong because the misheard name cascaded into the action item and showed 'Nadine' instead of 'Mahreen.'","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/672943809b1d40c483bdee07651023c2.png?v=1","evidence_url":"https://aidemos.com/evidence/a3626a76-5657-440e-9460-f49e46ad880b"},{"id":"dfeacb85-baa0-4c8b-a44e-e4a7b005d54f","tool":"meetgeek","tool_name":"MeetGeek","verdict":"worked","score":null,"score_total":null,"note":"It extracts real commitments as action items rather than noise; the report says all extracted items had correct ownership and timing, and the visible note includes an owned action item with timestamp 19:51.","artifact_count":1,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/39691e7c2e9141b3aac5a412f3bb110d.png?v=1","evidence_url":"https://aidemos.com/evidence/dfeacb85-baa0-4c8b-a44e-e4a7b005d54f"},{"id":"914e606f-d39b-4941-8b42-0f3b3653ee40","tool":"otter-ai","tool_name":"Otter.ai","verdict":"struggled","score":null,"score_total":null,"note":"Otter extracted action items, including at least one due-today API-related task with an assignee, but the report says most items were left without an owner and duplicate entries also appeared, so the output needed manual cleanup before delegation.","artifact_count":2,"thumbnail":"https://cdn.futuresmart.ai/public/aidemos/7e0a51fd2a4744aea043309aec25fc20.png?v=1","evidence_url":"https://aidemos.com/evidence/914e606f-d39b-4941-8b42-0f3b3653ee40"}],"other_criteria":[{"id":"b58378a0-5d37-4751-a711-f2d028fa4401","criterion":"chat-with-notes-ask-questions","criterion_name":"Chat with Notes / Ask Questions","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"The Q&A interface answered a natural-language question with a grounded response from the meeting record, including the specific date \"6th August,\" and the report observed no hallucinations.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/b58378a0-5d37-4751-a711-f2d028fa4401"},{"id":"d31ec8ee-e1f2-4d37-8cd5-9eeaf5e99255","criterion":"join-method-reliability","criterion_name":"Join Method & Reliability","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The bot-based Google Meet join was reliable in the tested call: Notta Bot appeared in the meeting list, admitted/managed normally, and the capture ran through the end of the session without disconnects or plan-limit cutoffs.","artifact_count":4,"evidence_url":"https://aidemos.com/evidence/d31ec8ee-e1f2-4d37-8cd5-9eeaf5e99255"},{"id":"3e05ffd9-05fd-44d7-914a-1e1bbb24c13c","criterion":"search-across-notes","criterion_name":"Search Across Notes","rank_role":"context","verdict":"mixed","score":null,"score_total":null,"note":"Search works inside a meeting transcript through AI Chat and returns exact timestamps in plain text, but the timestamps are not clickable, and the report says this was not tested across meetings.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/3e05ffd9-05fd-44d7-914a-1e1bbb24c13c"},{"id":"089aaaff-2da4-4739-8045-46df7a2f1e6b","criterion":"speaker-diarization","criterion_name":"Speaker Diarization","rank_role":"decisive","verdict":"mixed","score":null,"score_total":null,"note":"Speaker attribution was mostly correct, with nearly all speakers identified by name, but the transcript still showed some misattributed lines, so diarization was not fully reliable for every turn.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/089aaaff-2da4-4739-8045-46df7a2f1e6b"},{"id":"c1fc0701-9a26-4f73-a1f2-d4dfae6a7109","criterion":"speaker-diarization","criterion_name":"Speaker Diarization","rank_role":"decisive","verdict":"struggled","score":null,"score_total":null,"note":"The transcript contained a line labeled with another notetaker’s name (HappyScribe), which indicates cross-tool contamination or labeling error and breaks speaker attribution for that segment.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/c1fc0701-9a26-4f73-a1f2-d4dfae6a7109"},{"id":"aa85bfe2-42ee-4df2-b89f-831056fd4fc3","criterion":"summary-quality","criterion_name":"Summary Quality","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"The generated meeting summary was comprehensive and skimmable, with structured sections such as Task & Issue Management and a mindmap-style organization that reflected the meeting flow.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/aa85bfe2-42ee-4df2-b89f-831056fd4fc3"},{"id":"dacc4361-09c8-4791-8895-8a8095e21fe8","criterion":"topic-segmentation","criterion_name":"Topic Segmentation","rank_role":"context","verdict":"worked","score":null,"score_total":null,"note":"The meeting was segmented into useful topic blocks rather than one blob; the report names three sections, including Task & Issue Management, Individual Progress Updates, and API Benchmarking Task.","artifact_count":1,"evidence_url":"https://aidemos.com/evidence/dacc4361-09c8-4791-8895-8a8095e21fe8"},{"id":"893f5543-bd03-4e2a-ba30-f6426940628b","criterion":"transcription-accuracy","criterion_name":"Transcription Accuracy","rank_role":"decisive","verdict":"worked","score":null,"score_total":null,"note":"Notta’s transcript capture was accurate on the evaluated standup: the report says it correctly captured names, tool names, numbers, and engineering jargon with no significant word-level errors, silent hallucinations, or misheard terms.","artifact_count":2,"evidence_url":"https://aidemos.com/evidence/893f5543-bd03-4e2a-ba30-f6426940628b"}],"appears_in":[{"page_type":"ranking","slug":"ai-meeting-notetakers","title":"Best AI Meeting Notetakers for Accurate Transcripts, Summaries, and Action Items","url":"https://aidemos.com/best/ai-meeting-notetakers","binding":"run"}],"same_scenario":[{"id":"8340c79d-88de-4721-9c3e-c58e057071fb","tool":"fathom","tool_name":"Fathom","verdict":"worked","score":null,"score_total":null,"note":"Fathom extracts real commitments into an ACTION ITEMS section with owner attribution; the published output shows timestamped tasks and a named owner on the item."},{"id":"21587b7d-dc6e-4aa1-8ee4-4e5f13641a3e","tool":"fellow","tool_name":"Fellow","verdict":"mixed","score":null,"score_total":null,"note":"Action-item extraction was mostly correct, with real commitments and proper owner assignment for most items, but one real action item was misplaced from Mahreen to Anshika; the report states a 95%+ capture rate."},{"id":"737cde44-da22-4dd6-b985-62104c4d14a8","tool":"fireflies-ai","tool_name":"Fireflies.ai","verdict":"worked","score":null,"score_total":null,"note":"Grouped action items by owner, attributed them to the correct team member, and exposed a clickable source timestamp (19:13) for at least one item."},{"id":"9af81f24-1ac8-4a6a-b543-1959eafc22b0","tool":"granola","tool_name":"Granola","verdict":"worked","score":null,"score_total":null,"note":"Granola extracts concrete next steps with ownership: the visible action item says to create a subtask and add details, names Mahreen Fathima as owner, and marks the item for same-day follow-up; the report says the extracted list contained no false positives."},{"id":"a3626a76-5657-440e-9460-f49e46ad880b","tool":"happyscribe","tool_name":"HappyScribe","verdict":"mixed","score":null,"score_total":null,"note":"It extracted the real action item about updating logs, but owner attribution was wrong because the misheard name cascaded into the action item and showed 'Nadine' instead of 'Mahreen.'"},{"id":"dfeacb85-baa0-4c8b-a44e-e4a7b005d54f","tool":"meetgeek","tool_name":"MeetGeek","verdict":"worked","score":null,"score_total":null,"note":"It extracts real commitments as action items rather than noise; the report says all extracted items had correct ownership and timing, and the visible note includes an owned action item with timestamp 19:51."},{"id":"914e606f-d39b-4941-8b42-0f3b3653ee40","tool":"otter-ai","tool_name":"Otter.ai","verdict":"struggled","score":null,"score_total":null,"note":"Otter extracted action items, including at least one due-today API-related task with an assignee, but the report says most items were left without an owner and duplicate entries also appeared, so the output needed manual cleanup before delegation."}]}