{"id":"speech-audio-sequence-objectives","name":"Audio representation, alignment and transcript loss","shortName":"Audio representation, alignment and transcript loss","role":"Relate audio samples, time frames and transcript errors without mixing their denominators.","description":"An audio model receives sampled waveforms or derived features, while a transcript decoder predicts a sequence of text tokens. Sample rate, window length and hop determine the time representation; transcript labels need not align one-to-one with frames. Whisper uses a particular log-Mel representation and sequence-to-sequence task format. That is different from a CTC alignment objective or an audio-generation objective. Evaluation must also declare text normalization: case, punctuation and tokenization can change edit counts. The example below computes geometry and word errors without claiming to execute speech recognition.","mechanism":["Retain waveform provenance, sample rate and task labels.","Compute frame boundaries under a stated padding/window convention.","Distinguish audio frames, text tokens and reference words.","Score substitutions, deletions and insertions against a fixed normalized reference, with speaker/noise slices separate."],"learningObjective":"Relate audio samples, time frames and transcript errors without mixing their denominators.","engineeringExample":"A one-second waveform sampled at 16000 Hz contains 16000 samples. With a 400-sample window, 160-sample hop and no padding, frame count is 1+floor((16000−400)/160)=98. A centered/padded library can differ. Reference “turn left now” and hypothesis “turn right now please” have one substitution and one insertion over three reference words, giving WER=2/3. Insertions can make WER exceed 100%; it is not a bounded accuracy percentage.","evidenceNotes":"Mode: worked example. N at least W, no padding or centering for frame arithmetic; fixed whitespace-normalized reference words for WER. No audio data is uploaded or speech model run by this lesson. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Relate audio samples, time frames and transcript errors without mixing their denominators.","context":"N at least W, no padding or centering for frame arithmetic; fixed whitespace-normalized reference words for WER. No audio data is uploaded or speech model run by this lesson."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Retain waveform provenance, sample rate and task labels."},{"name":"Result and check","direction":"out","signal":"Relate audio samples, time frames and transcript errors without mixing their denominators."}],"tradeoff":"Feature compression reduces sequence length but discards waveform detail. Transcript metrics can miss semantic severity and conceal speaker or noise failures.","whyAI":"Teach modality-specific representation, objective and evaluation choices. Relate audio samples, time frames and transcript errors without mixing their denominators.","misconception":"It is a mistake to conclude: There are three correctly generated audio frames.","check":{"question":"Why is the WER denominator three in the example?","options":["There are three reference words, irrespective of hypothesis length.","There are three correctly generated audio frames.","WER always divides by the number of hypothesis words."],"answer":0,"explanation":"There are three reference words, irrespective of hypothesis length. N at least W, no padding or centering for frame arithmetic; fixed whitespace-normalized reference words for WER. No audio data is uploaded or speech model run by this lesson."},"prerequisites":["tokenizer","attention","evaluation"],"related":["tokenizer","attention","evaluation","image-text-contrastive-learning","visual-language-bridge","denoising-generative-objectives","video-latent-prediction"],"sources":[{"title":"Robust Speech Recognition via Large-Scale Weak Supervision","url":"https://arxiv.org/html/2212.04356v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-multimodal","subbranch":"Audio representation, alignment and transcript loss","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. N at least W, no padding or centering for frame arithmetic; fixed whitespace-normalized reference words for WER. No audio data is uploaded or speech model run by this lesson.","science":[{"title":"Audio representation, alignment and transcript loss · worked relation","equation":"F=1+\\left\\lfloor\\frac{N-W}{H}\\right\\rfloor,\\qquad WER=\\frac{S+D+I}{N_{ref}}","explanation":"A one-second waveform sampled at 16000 Hz contains 16000 samples. With a 400-sample window, 160-sample hop and no padding, frame count is 1+floor((16000−400)/160)=98. A centered/padded library can differ. Reference “turn left now” and hypothesis “turn right now please” have one substitution and one insertion over three reference words, giving WER=2/3. Insertions can make WER exceed 100%; it is not a bounded accuracy percentage.","assumptions":"N at least W, no padding or centering for frame arithmetic; fixed whitespace-normalized reference words for WER. No audio data is uploaded or speech model run by this lesson."}],"history":[]}