How to Add Speaker Diarization and Word-Level Timestamps to Your AI Workflows
Use IBM Granite Speech 4.1 Plus to add speaker attribution and word-level timestamps to transcription workflows. Better than Whisper X for many use cases.
Why Transcription Alone Isn’t Enough
If you’ve ever tried to use raw audio transcription in a real workflow, you’ve probably run into the same wall: a wall of text with no indication of who said what or exactly when they said it.
That’s fine for simple note-taking. But if you’re building AI workflows that analyze sales calls, generate meeting summaries, process podcast content, or route customer support audio — you need more. You need speaker diarization and word-level timestamps.
This guide covers both: what they are, why they matter for production AI workflows, and how to implement them using IBM Granite Speech 4.1 Plus, which offers a compelling alternative to WhisperX for many use cases.
What Speaker Diarization Actually Means
Speaker diarization is the process of segmenting audio by speaker — answering the question “who spoke when?” rather than just “what was said?”
A standard transcription tool gives you this:
“Hey, can we push the deadline? Sure, no problem, I’ll let the team know.”
A diarization-enabled transcription gives you this:
Speaker 1: “Hey, can we push the deadline?” Speaker 2: “Sure, no problem, I’ll let the team know.”
That distinction is the difference between a transcript that’s searchable and one that’s actually useful for analysis, routing, or agent-driven follow-up.
Where Diarization Gets Hard
The core challenge is that diarization is fundamentally a clustering problem. The model has to:
- Detect voice activity (separate speech from silence and background noise)
- Extract speaker embeddings — a numerical fingerprint of each voice
- Cluster segments by speaker similarity
- Assign consistent labels across the full recording
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Overlapping speech, noisy environments, similar-sounding voices, and variable recording quality all degrade accuracy. This is why diarization quality varies so much between models and why choosing the right tool for your use case matters.
What Word-Level Timestamps Add
Standard transcription gives you a block of text with maybe a start and end time for each segment. Word-level timestamps go further — they tell you the exact start and end time of every individual word in the transcript.
This unlocks a set of capabilities that segment-level timestamps simply can’t support:
- Subtitle generation — Accurate caption sync at the word level, not just the sentence level
- Audio clip extraction — Pull exact quotes from long recordings without manual scrubbing
- Searchable audio — Jump to the precise moment a specific word or phrase was spoken
- Talk-time analytics — Measure exactly how long each speaker talked, down to sub-second precision
- Highlight reels — Automatically clip the moments that matter based on keywords or topics
For AI workflows specifically, word-level timestamps let you pass structured, time-indexed data to downstream models — which is dramatically more useful than a flat string of text.
IBM Granite Speech 4.1 Plus: What You Need to Know
IBM Granite Speech 4.1 Plus is IBM’s production-grade speech model, part of the Granite 4.x model family released in 2025. It’s designed for enterprise speech processing and supports:
- Automatic speech recognition (ASR) with high accuracy across accents and domains
- Speaker diarization built in as a native capability — not bolted on as a post-processing step
- Word-level timestamps for every token in the output
- Multi-speaker support for up to 8+ speakers in a single recording
- Long-form audio handling without the chunking issues that affect some other models
What sets Granite Speech 4.1 Plus apart from many alternatives is that diarization and timestamps aren’t separate pipeline stages — they’re part of the same inference pass. That matters for accuracy (the model has full context during both tasks) and for latency (you’re not chaining two separate API calls).
Output Format
Granite Speech 4.1 Plus returns structured JSON that looks roughly like this:
{
"transcript": "Hey can we push the deadline",
"words": [
{ "word": "Hey", "start": 0.12, "end": 0.34, "speaker": "SPEAKER_01" },
{ "word": "can", "start": 0.36, "end": 0.52, "speaker": "SPEAKER_01" },
{ "word": "we", "start": 0.54, "end": 0.61, "speaker": "SPEAKER_01" },
{ "word": "push", "start": 0.63, "end": 0.89, "speaker": "SPEAKER_01" },
{ "word": "the", "start": 0.91, "end": 0.98, "speaker": "SPEAKER_01" },
{ "word": "deadline", "start": 1.00, "end": 1.45, "speaker": "SPEAKER_01" }
],
"speakers": ["SPEAKER_01", "SPEAKER_02"],
"duration": 14.32
}This structure is immediately useful for downstream processing — you can filter by speaker, group by time window, or pass the full object to an LLM for analysis without any preprocessing.
Granite Speech 4.1 Plus vs. WhisperX
WhisperX has been the go-to option for word-level timestamps and diarization for a while. It’s an open-source wrapper around OpenAI’s Whisper that adds forced alignment (via wav2vec2) and optional diarization (via pyannote.audio). It works well, and for self-hosted setups it’s still widely used.
But it comes with real operational overhead — especially if you’re building production workflows.
| Capability | Granite Speech 4.1 Plus | WhisperX |
|---|---|---|
| Word-level timestamps | Native | Via forced alignment (wav2vec2) |
| Speaker diarization | Native | Via pyannote.audio (separate model) |
| Deployment | API | Self-hosted (Python) |
| Pipeline complexity | Single call | Multi-stage |
| HuggingFace token required | No | Yes (for pyannote) |
| GPU required | No | Strongly recommended |
| Enterprise support | Yes (IBM) | Community |
| License | IBM Granite (open) | Apache 2.0 |
Coding agents automate the 5%. Remy runs the 95%.
The bottleneck was never typing the code. It was knowing what to build.
When WhisperX Still Makes Sense
WhisperX is a strong choice when:
- You need full control over the underlying models and fine-tuning
- You’re already running GPU infrastructure for other workloads
- You want to use a specific Whisper variant (large-v3, turbo, etc.)
- Your use case requires offline/air-gapped processing
When Granite Speech 4.1 Plus Is a Better Fit
Granite Speech 4.1 Plus tends to win when:
- You’re building a workflow or application (not a research pipeline)
- You need reliability without managing infrastructure
- You want diarization and timestamps in a single API call
- You’re integrating with other tools and need consistent structured output
- You’re in an enterprise environment where IBM support matters
For most production AI workflow use cases, the simpler operational model of Granite Speech 4.1 Plus is worth it.
How to Add Diarization and Timestamps to an AI Workflow
Here’s a step-by-step approach to building a workflow that ingests audio, extracts speaker-attributed word-level transcripts, and does something useful with them.
Step 1: Define Your Audio Source
First, decide where your audio is coming from. Common sources:
- Uploaded files — Users upload MP3, WAV, or M4A via a form or API endpoint
- Meeting recordings — Pulled from Zoom, Google Meet, or Microsoft Teams via integration
- Phone calls — Ingested via webhook from Twilio, Vonage, or similar
- Live recordings — Streamed audio captured in real time (more complex — batching is usually better)
Standardize to 16kHz mono WAV or MP3 before processing. Most transcription APIs prefer it, and it reduces file size significantly.
Step 2: Send Audio to Granite Speech 4.1 Plus
The basic API call sends your audio file and requests diarization and word timestamps in the response parameters. You’ll typically pass:
- The audio file (base64 encoded or as a multipart upload)
diarization: trueword_timestamps: true- Optionally: number of speakers if known, language hint, punctuation preference
The response returns the structured JSON object described above.
Step 3: Parse and Structure the Output
Raw word-level JSON needs reshaping before most downstream uses. A common transformation is grouping words by speaker into “turns”:
[
{ speaker: "SPEAKER_01", start: 0.12, end: 1.45, text: "Hey can we push the deadline" },
{ speaker: "SPEAKER_02", start: 1.52, end: 3.20, text: "Sure no problem I'll let the team know" }
]This grouped format is much easier to pass to an LLM for analysis, to render as subtitles, or to store in a database.
Step 4: Feed to Downstream Processing
With a structured, speaker-attributed transcript, you can now build almost any downstream workflow:
- Meeting summaries — Pass the turn-grouped transcript to GPT-4o or Claude with a prompt to summarize by speaker and identify action items
- Sentiment analysis — Analyze each speaker’s turns independently to detect tone shifts
- CRM logging — Extract key points and automatically log them to Salesforce or HubSpot
- Compliance review — Flag specific phrases or patterns in calls with timestamps for easy review
- Content repurposing — Extract quote-worthy moments with their exact timestamps for social clips
Step 5: Handle Edge Cases
A few things to build for before you ship:
- Single-speaker audio — Diarization won’t fail, but it may return one speaker for everything. Handle this gracefully.
- Overlapping speech — Words during cross-talk may have ambiguous speaker labels. Consider a confidence threshold filter.
- Short audio — Files under ~10 seconds may not have enough audio for reliable diarization. Add a minimum duration check.
- Background noise — If your use case involves noisy environments (call centers, in-person events), test with realistic samples, not clean studio audio.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
Practical Use Cases Worth Building
Once you have diarization and word-level timestamps working in a workflow, a few patterns come up repeatedly as genuinely high-value.
Sales Call Analysis
Sales teams generate hundreds of call recordings every week. Most of them go unreviewed. A workflow that automatically transcribes, diarizes, extracts customer objections by speaker, and pushes a summary to the CRM can turn that dead data into a coaching and forecasting resource.
Word-level timestamps mean you can link directly to the exact moment a customer raised a specific concern — not just note that it happened.
Podcast and Video Production
Content teams spend enormous time on transcript cleanup, subtitle generation, and clip selection. A diarized, word-timestamped transcript makes all of these tasks faster:
- Generate SRT subtitle files directly from word timestamps
- Identify the best quote moments by speaker and topic
- Auto-generate show notes with speaker-attributed quotes
Customer Support QA
Quality assurance in support operations traditionally requires manual call sampling. With a diarized transcript workflow, you can analyze every call automatically — measuring talk time ratios, detecting policy keywords, flagging escalations — and only route specific calls for human review.
Interview and Research Processing
Qualitative researchers and journalists deal with hours of interview audio. Word-level diarization lets them search across dozens of interviews for specific speakers’ statements, extract quotes with precise attribution, and build analysis corpora without manual transcription.
Building This Workflow in MindStudio
If you want to put this together without managing infrastructure, MindStudio is a practical option. It’s a no-code workflow builder with access to 200+ AI models out of the box — including speech models — plus direct integrations with tools like HubSpot, Salesforce, Slack, and Notion.
Here’s what a MindStudio workflow for diarized call analysis might look like:
- Trigger — A webhook receives a new call recording from your phone system or meeting tool
- Transcription step — The audio is sent to a speech model that returns diarized, word-timestamped JSON
- Parsing step — A JavaScript function groups the word-level output into speaker turns
- Analysis step — The structured transcript goes to an LLM (Claude, GPT-4o, or whichever fits) with a prompt to extract action items, sentiment, and key topics per speaker
- Output step — Results are pushed to your CRM, sent as a Slack summary, or stored in Airtable
The whole thing can be built in under an hour without writing a backend or managing any model infrastructure. You can also add conditional logic — for example, only trigger a manager alert if the sentiment analysis flags a negative customer experience above a certain threshold.
You can start building for free at mindstudio.ai.
MindStudio is also useful for teams that want to automate audio content workflows or connect transcription pipelines to existing business tools without building custom integrations. If you’re exploring what kinds of agents you can build, the MindStudio use case library has examples across sales, support, content, and operations.
FAQ
What is speaker diarization and how does it work?
One coffee. One working app.
You bring the idea. Remy manages the project.
Speaker diarization is the process of identifying “who spoke when” in an audio recording. The model analyzes voice characteristics — pitch, cadence, spectral features — to create an embedding (a numerical fingerprint) for each speaker. It then clusters segments of audio by embedding similarity and assigns consistent speaker labels across the full recording. The result is a transcript annotated with speaker IDs for each segment or word.
What are word-level timestamps in transcription?
Word-level timestamps give each individual word in a transcript a precise start and end time, measured in seconds from the beginning of the audio. Unlike segment-level timestamps (which only mark the beginning and end of a phrase or sentence), word-level timestamps let you locate any specific word in the audio to the millisecond. This enables accurate subtitle sync, audio clip extraction, and searchable audio archives.
How accurate is IBM Granite Speech 4.1 Plus for speaker diarization?
Granite Speech 4.1 Plus performs well on clean to moderately noisy audio with two to four distinct speakers. Like all diarization models, accuracy degrades with overlapping speech, very similar voices, and poor recording quality. For production use, it’s worth testing with a representative sample of your actual audio before committing. IBM has benchmarked it against enterprise call center and meeting audio, where it generally outperforms multi-stage pipelines in both accuracy and latency.
Is Granite Speech 4.1 Plus better than WhisperX?
It depends on your use case. Granite Speech 4.1 Plus is simpler to deploy in production workflows — diarization and word timestamps are native, not separate pipeline stages, and you don’t need to manage GPU infrastructure or configure multiple models. WhisperX gives you more control and is better for self-hosted, fine-tuned, or air-gapped environments. For most teams building AI workflows or applications, Granite Speech 4.1 Plus is the lower-friction option.
Can I use speaker diarization in real-time audio processing?
Most diarization models, including Granite Speech 4.1 Plus, are optimized for batch processing — sending a complete audio file and receiving a full annotated transcript. Real-time diarization is possible but significantly more complex: it requires streaming audio, a buffering strategy to accumulate enough audio for speaker embedding, and re-labeling as new speakers are detected. For most workflow use cases, batching audio in segments of 30 seconds to a few minutes is a practical middle ground.
How do I handle a recording where I don’t know how many speakers there are?
Most modern diarization models can infer the number of speakers automatically. You can optionally pass a known speaker count (e.g., num_speakers: 2 for a two-party call) to improve accuracy, but leaving it unspecified and letting the model detect speakers dynamically works well for most use cases. If you’re processing content where speaker count varies (podcast interviews, panel discussions, customer calls), leave it dynamic and post-process if needed.
Key Takeaways
- Speaker diarization answers “who spoke when” — it converts a wall of text into speaker-attributed turns that are actually useful for analysis and automation.
- Word-level timestamps go beyond segment timing to give you the precise location of every word in the audio, enabling subtitle generation, audio search, and clip extraction.
- IBM Granite Speech 4.1 Plus provides both capabilities natively in a single API call, without the multi-stage pipeline complexity of WhisperX.
- WhisperX is still a strong choice for self-hosted, fine-tuned, or offline workflows — but for production AI applications, Granite Speech 4.1 Plus typically offers a simpler path.
- With a no-code platform like MindStudio, you can connect diarized transcription output directly to CRMs, messaging tools, and LLM analysis steps without managing any backend infrastructure.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
If you want to build a working diarization workflow without writing a line of backend code, MindStudio is worth a look — it’s free to start, and most workflows like this take under an hour to build.
