Benchmark results

What Voice Isolation does to STT when a background voice is in the room

We ran 265 real recordings through 11 STT configurations, twice: raw, and after Krisp Voice Isolation. It improves word error rate by ~73% on average in the presence of background voices, and perceptual quality rises as well. Below are the numbers, the audios, and the cases where it does not help.

Headline numbers

Corpus-level WER, all 11 STT configurations pooled. Lower is better. "Best rel diff" (Best relative difference) is the largest reduction a Voice Isolation model achieves against raw audio.

Phone calls contain almost no competing speech and already narrowband audio, so there is little to remove; every model lands within about half a point of raw there.

WER by STT engine

While every engine improves, the gains are non-uniform: some see larger boosts than others. Interestingly, the performance gap between engines narrows from 16–37% on raw audio to just 4–8% after applying Voice Isolation. Krisp clearly enables all engines to perform closer to their peak potential on this benchmark. Streaming and batch modes are listed separately. Although our main focus is on streaming, we include batch processing options to show that Voice Isolation’s gains are not constrained by lookahead limitations.

Perceptual quality

DNSMOS-C NISQA mean opinion score, 1–5, higher is better. Scored only on segments where the primary speaker is talking, so this measures residual noise around the target voice rather than silence quality.

Every model raises MOS in every scenario, by up to half a point. VI 2.5 Default and VI 2.5 Lite score almost identically; VI 2.5 Balanced sits slightly lower by design, since it processes the primary voice more lightly.

Listen

Six recordings containing background voices, in raw and after each of the Voice Isolation models. WER shown per file, averaged across all 11 STT configurations.

While Voice Isolation models generally improve WER across a range of STT engines in most cases, there are also examples showing a degradation in WER after Voice Isolation. Such cases may happen in very challenging scenarios, for example, in poor-quality audio or extremely noisy environments where acoustic cues that define the primary speaker are hard to isolate. Below, we share an example demonstrating this.

Methodology

Word error rate

WER is computed with jiwer 4.0.0. Each ground-truth transcript is aligned against the STT hypothesis at the word level with jiwer.process_words(). We report WER at the corpus level: errors are pooled across every file in a scenario and divided by the total number of reference words.

The "All scenarios" row follows the same rule: all 265 files pooled into one count, not an average of the three scenario percentages.

Both reference and hypothesis go through the same three-stage normalization pipeline before alignment. Dependencies: Python 3, jiwer 4.0.0, nemo_text_processing (NVIDIA NeMo Text Processing, WFST-based).

The custom normalization below was built around this particular dataset and is not meant as a general-purpose solution for dates, card numbers or similar entities — it will mishandle inputs the corpus does not contain, such as 1000 1000 dollars. The aim was to keep the rules simple, adding nothing beyond the libraries already listed, and to fit the texts we actually have. They run in sequence, starting with Rule 1 on the input text and ending with Rule 4. The purpose of this benchmark is to show what the Krisp Voice Isolation model family brings, and since the same normalization is applied to raw audio and to every Voice Isolation model alike, that comparison holds either way.

Stage 1 — Structural splitting (regex)

# ── Rule 1: IDs, codes and postcodes ────────────────────────────
# 1–4 capitals, dash / space / nothing, digits, 0–2 trailing capitals.
# A spaced form needs 3+ digits, so "64.99 USD 2 weeks" is left alone.
#   CLM-44820 / CLM44820 / CLM 44820  →  C L M 4 4 8 2 0
#   B13 9AX   / B139AX                →  B 1 3 9 A X
GENERAL_ID_RE = re.compile(
    r"\b([A-Z]{1,4})"            # 1–4 capital letters
    r"(?:-?(\d+)|[ ](\d{3,}))"   # glued/dashed: any digits | spaced: 3+
    r"(?:[ ]?(\d)([A-Z]{2}))?"   # optional postcode incode: 9AX / 3WS
    r"([A-Z]{0,2})\b")           # 0–2 trailing letters

def split_general_id(text: str) -> str:
    def repl(m):
        g = m.groups()
        parts = list(g[0]) + list(g[1] or g[2] or "")
        if g[3]:
            parts += list(g[3]) + list(g[4])
        parts += list(g[5] or "")
        return " ".join(parts)
    return GENERAL_ID_RE.sub(repl, text)

# ── Rule 2: Digit-first codes ───────────────────────────────────
# 4+ digits, optional dash, 1 letter. Decade plurals kept intact.
#   88213A  →  8 8 2 1 3 A
#   1940s   →  1940s (unchanged)
DIGIT_FIRST_RE = re.compile(r"\b(\d{4,})-?([A-Za-z])\b")

def split_digit_first(text: str) -> str:
    def repl(m):
        digits, letter = m.group(1), m.group(2)
        if letter.lower() == 's':
            return m.group(0)
        return " ".join(list(digits) + [letter])
    return DIGIT_FIRST_RE.sub(repl, text)

# ── Rule 3: Standalone 4-digit numbers ──────────────────────────
# Split digit-by-digit, except 1000–2099 (years) and before unit words.
#   3072  →  3 0 7 2
#   1964  →  1964 (unchanged)
UNITS = r"(?:dollars?|pounds?|euros?|percent|miles?|days?|months?|hours?|minutes?|nights?|years?|weeks?|points?|gigabytes?|megabits?)"

def split_standalone_4digits(text: str) -> str:
    def repl(m):
        if 1000 <= int(m.group(1)) <= 2099:
            return m.group(0)
        return " ".join(m.group(1))
    return re.sub(
        rf"(?<![$£€\d,])(?<!\d)\b(\d{{4}})\b(?!\d)(?!\s+{UNITS})",
        repl, text)

# ── Rule 4: Adjacent digit groups ───────────────────────────────
# Flatten runs of space-separated digit groups (phone numbers, sort codes).
#   0800 123 4567     →  0 8 0 0 1 2 3 4 5 6 7
#   07 70 09 00 4 12  →  0 7 7 0 0 9 0 0 4 1 2
def split_adjacent_digit_groups(text: str) -> str:
    def repl(m):
        return " ".join(m.group(0).replace(" ", ""))
    return re.sub(r"\b\d{1,16}(?:\s+\d{1,16}){1,}\b", repl, text)

Stage 2 — Text normalization (NeMo)

After structural splitting, NVIDIA NeMo Text Processing (nemo_text_processing 1.2.0, WFST-based, input_case='cased', lang='en') normalizes the text. NeMo converts remaining numbers to words (210 → two hundred and ten), expands currency ($29.99 → twenty nine dollars and ninety nine cents), verbalizes ordinals (14th → fourteenth), expands abbreviations (Dr. → doctor), and handles time (3:30 → three thirty) and percentages (50% → fifty percent).

nemo = Normalizer(input_case="cased", lang="en")

def preprocess(text: str) -> str:
    text = split_general_id(text)            # Rule 1
    text = split_digit_first(text)           # Rule 2
    text = split_standalone_4digits(text)    # Rule 3
    text = split_adjacent_digit_groups(text) # Rule 4
    text = nemo.normalize(text)              # NeMo WFST
    return text

Stage 3 — jiwer normalization

Finally, both sides go through the same jiwer pipeline:

FILLER_WORDS = ["um", "uh", "umm", "uhh", "hmm", "mm", "mhm", "erm"]

NORMALIZE = jiwer.Compose([
    jiwer.ToLowerCase(),
    jiwer.ExpandCommonEnglishContractions(),
    jiwer.SubstituteRegexes({r"[^\w\s]": " "}),
    jiwer.RemoveSpecificWords(FILLER_WORDS),
    jiwer.RemoveMultipleSpaces(),
    jiwer.Strip(),
    jiwer.ReduceToListOfListOfWords(),
])

def compute_wer(ref_text: str, hyp_text: str):
    ref = preprocess(ref_text)
    hyp = preprocess(hyp_text)
    out = jiwer.process_words(ref, hyp,
              reference_transform=NORMALIZE,
              hypothesis_transform=NORMALIZE)
    D, I, S = out.deletions, out.insertions, out.substitutions
    N = D + out.hits + S
    return (D + I + S) / N * 100 if N > 0 else 0.0

# ── Worked example 1: ID reference + phone number ────────────────
# raw:       "Reference NG-228, call 0800 123 4567"
# rule 1:    "Reference N G 2 2 8, call 0800 123 4567"
# rule 2:    (no digit-first codes)
# rule 3:    "Reference N G 2 2 8, call 0 8 0 0 123 4 5 6 7"
# rule 4:    "Reference N G 2 2 8, call 0 8 0 0 1 2 3 4 5 6 7"
# NeMo:      "Reference N G two two eight, call zero eight zero zero one two three four five six seven"

# ── Worked example 2: ID, postcode, year, currency, fraction ─────
# raw:       "Claim CLM 44820, account 4471-B, postcode B13 9AX: £20,000 since 1964, 3/4 repaid at 4.3%"
# rule 1:    "Claim C L M 4 4 8 2 0, account 4471-B, postcode B 1 3 9 A X: £20,000 since 1964, 3/4 repaid at 4.3%"
# rule 2:    "Claim C L M 4 4 8 2 0, account 4 4 7 1 B, postcode B 1 3 9 A X: £20,000 since 1964, 3/4 repaid at 4.3%"
# rules 3-4: (no further changes — 1964 is inside the 1000–2099 year range)
# NeMo:      "Claim C L M four four eight two zero, account four four seven one B, postcode B one three nine A X:
#             twenty thousand pounds since nineteen sixty four, three quarters repaid at four point three percent"

Stage 1 uses four regex rules to split digit-bearing tokens so that both reference and hypothesis converge regardless of how the STT groups digits — an identifier written glued (CLM44820), dashed (CLM-44820) or spaced (CLM 44820) all reduce to the same tokens. Stage 2 (NeMo) converts remaining numbers to words, expands currencies, ordinals, abbreviations, time, and percentages. Stage 3 lowercases, expands contractions, strips punctuation, drops filler words, and tokenizes. WER then reflects real transcription differences, not formatting conventions.

Perceptual quality

Scored with DNSMOS-C, a reference-free speech quality model trained with contrastive learning on top of the DNSMOS Pro architecture. It predicts a MOS distribution (mean and variance) from the clip alone; no clean reference is needed. We use the NISQA-trained checkpoint (runs/NISQA/model_best.pt, loaded via torch.jit.load), which targets the NISQA corpus MOS scale.

Only segments containing the primary speaker — the mix and primary labels — are scored. Segments under 1 second are dropped before scoring.

Stage 1 — Fixed-window chunking

DNSMOS-C is a fixed-window model, so every segment is broken into overlapping 10 s windows before scoring. Segments shorter than 10 s are looped to fill the window rather than padded with silence, so short segments are never scored on partial or silent audio.

# ── 10s windows, 5s hop → 5s overlap ────────────────────────────
CLIP_SECONDS = 10
SAMPLE_RATE  = 16_000
MIN_SEGMENT_SECONDS = 1.0

def chunk_audio(wav, overlap_seconds=5.0):
    window = CLIP_SECONDS * SAMPLE_RATE
    hop = window - int(overlap_seconds * SAMPLE_RATE)
    assert hop > 0                                # overlap_seconds must be < CLIP_SECONDS
    if len(wav) <= window:
        return [loop_to_length(wav, window)]     # short segment: loop up to 10s
    chunks = [wav[s:s + window] for s in range(0, len(wav) - window + 1, hop)]
    if (len(wav) - window) % hop != 0:
        chunks.append(wav[-window:])                 # tail window, guarantees full coverage
    return chunks

def loop_to_length(wav, length):
    reps = length // len(wav) + 1
    return np.tile(wav, reps)[:length]           # repeat-and-crop, not zero-padded

Stage 2 — Per-segment scoring

Each window is scored independently by the DNSMOS-C model; a segment's score is the mean across its windows.

def score_segment(wav_segment):
    chunks = chunk_audio(wav_segment)
    specs = torch.stack([torch.FloatTensor(utils.stft(c)) for c in chunks])
    out = dns_model(specs[:, None, ...])            # → (MOS, variance) per chunk
    mos_per_chunk = out[:, 0]
    return mos_per_chunk.mean().item()             # segment score = mean across windows

Data

Recordings, transcripts, segment labels, and metadata are on Hugging Face.