{"id":"streaming-speech-turns","name":"Speech is a timed interaction","shortName":"Speech is a timed interaction","role":"Audio frames, transcription revisions, endpointing, interruption and tool confirmation.","description":"Speech interaction has a time structure that a completed text prompt hides. Audio arrives in frames, a recognizer may revise partial transcripts, and endpointing decides when a turn is complete enough for the next stage. A response can also be interrupted after generation or playback has begun. Speech recognition quality and turn-taking behavior therefore need different measurements. Whisper supplies primary context for learned audio-to-transcript modeling; it does not by itself define every streaming, interruption or tool-authorization behavior of a voice assistant. The worked cases here use authored timing and transcription errors. They do not run a recognizer, synthesize speech or execute a real-time audio pipeline.","mechanism":["Audio and transcript events retain timestamps and distinguish partial hypotheses from finalized text.","Endpointing or an explicit user signal determines when the application advances the turn.","Model and tool execution use the intended finalized command while handling later interruption as a separate event.","Recognition error, response delay and external action correctness are evaluated separately."],"learningObjective":"Separate transcription error, turn latency and action authority in a timed speech interaction.","engineeringExample":"On an authored 20-word reference, two substitutions, one deletion and one insertion give word error rate (2+1+1)/20 = 0.20. Separately, after speech ends, a serial pipeline waits 300 ms for endpointing, 100 ms for final recognition, 200 ms for a policy response and 150 ms before playback: 750 ms before audible output. Overlap could change that sum, but it is excluded here. An early partial transcript saying “reserve two” that later changes to “reserve none” cannot be treated as an irrevocable authorization merely because recognition has started.","misconception":"A partial transcript is not necessarily a completed or authorized user command.","check":{"question":"With 2 substitutions, 1 deletion and 1 insertion against 20 reference words, what is WER?","options":["10%","20%","80%"],"answer":1,"explanation":"The three edit counts sum to four, divided by twenty reference words."},"checks":[],"prerequisites":["human-review-cancellation","evaluation"],"related":["tool-loop","evaluation"],"specs":[{"label":"Mechanism boundary","value":"When is a spoken request complete enough to act on?","context":"A fixed reference, alignment convention and tokenization are required. WER is not a probability and does not measure tool correctness, meaning preservation or turn latency."},{"label":"Evidence type","value":"Authored worked example","context":"Worked explanation with a bounded numeric or state example. This lesson does not claim an executing implementation of an external model or platform."}],"interfaces":[{"name":"Declared input and state","direction":"in","signal":"Audio and transcript events retain timestamps and distinguish partial hypotheses from finalized text."},{"name":"Checked outcome","direction":"out","signal":"Separate transcription error, turn latency and action authority in a timed speech interaction."}],"tradeoff":"Short endpoint delays improve responsiveness but can prematurely split turns; longer waits preserve context while increasing perceived latency.","whyAI":"Voice agents must coordinate learned speech representations with temporal control and tool boundaries.","sourceIds":["automation-whisper","automation-ros-actions"],"sources":[{"title":"Robust Speech Recognition via Large-Scale Weak Supervision","url":"https://arxiv.org/abs/2212.04356v1"},{"title":"ROS 2 Jazzy: Understanding actions","url":"https://raw.githubusercontent.com/ros2/ros2_documentation/jazzy/source/Concepts/Basic/About-Actions.rst"}],"science":[{"title":"Speech is a timed interaction · explicit relationship","equation":"Word error rate is substitutions plus deletions plus insertions divided by reference word count.","explanation":"On an authored 20-word reference, two substitutions, one deletion and one insertion give word error rate (2+1+1)/20 = 0.20. Separately, after speech ends, a serial pipeline waits 300 ms for endpointing, 100 ms for final recognition, 200 ms for a policy response and 150 ms before playback: 750 ms before audible output. Overlap could change that sum, but it is excluded here. An early partial transcript saying “reserve two” that later changes to “reserve none” cannot be treated as an irrevocable authorization merely because recognition has started.","assumptions":"A fixed reference, alignment convention and tokenization are required. WER is not a probability and does not measure tool correctness, meaning preservation or turn latency."}],"evidenceNotes":"This is a worked explanation, not an executing external platform, model or robot. The quantities and fixtures are authored. Source scope is recorded in the automation learning review manifest; no source benchmark or private system behavior is claimed. A fixed reference, alignment convention and tokenization are required. WER is not a probability and does not measure tool correctness, meaning preservation or turn latency.","level":"model","branch":"Models become evaluated applications","subbranch":"Speech is a timed interaction","scale":"Logical state, task evidence and declared units; no physical assembly scale","kind":"system","workstream":"W11","visualFamily":"software-flow","labId":"library","labScope":"The worked example explains this mechanism. General numerical labs are a reference; they do not execute this lesson’s full mechanism.","history":[],"workedTrace":{"substitutions":2,"deletions":1,"insertions":1,"referenceWords":20,"wer":0.2,"serialTimingMs":[300,100,200,150],"latencyMs":750}}