Whistle's 86,174-utterance benchmark: test your own clips first

Treat a model card benchmark as a filter, not a verdict. Whistle's Hugging Face card says its word error rates were measured over 86,174 utterances, with comparisons to Whisper and Moonshine models. Your audio has its own accents, music beds and product names, so score 20 clips of your own against a human-checked transcript before you commit.

Whistle facts (16.9 MB, seven languages, Apache 2.0, 16 kHz mono input) are from the card read on 2026-10-13. The Sume steps use video-inspect and remix-media as documented on origin/main b28f3b0.

What should I look at besides the average?

Names, numbers and brand terms: an average can hide a model that gets product names wrong in every clip. Whistle handles one pass up to 30 seconds and windows beyond that, so include a few clips longer than 30 seconds to see whether the seams cost words.

How do I get a draft reference transcript?

Run video-inspect with transcribe: true and segmentation.mode: "sentence" and correct the output by hand. For heavy reference work, the remix media transcribe verb returns a word-timed transcript with per-word confidence; its default engine is WhisperX, and the billed scribe_v2 engine needs allow_billed_stt: true.

# word error rate between a reference and a hypothesis (pure Python)
def wer(ref: str, hyp: str) -> float:
    r, h = ref.lower().split(), hyp.lower().split()
    d = [[0] * (len(h) + 1) for _ in range(len(r) + 1)]
    for i in range(len(r) + 1):
        d[i][0] = i
    for j in range(len(h) + 1):
        d[0][j] = j
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            cost = 0 if r[i - 1] == h[j - 1] else 1
            d[i][j] = min(d[i - 1][j] + 1, d[i][j - 1] + 1, d[i - 1][j - 1] + cost)
    return d[-1][-1] / max(len(r), 1)

print(wer("say hello to sume", "say hello to some"))