Architecture for an on-prem speech system

Authorise and scope the audio

Agree on purpose, applicable legal basis, access and retention with the responsible people. Protect temporary files and define what is removed.

Separate observable stages

Ingestion, voice detection, transcription, speaker attribution and summarisation fail in different ways. Retain intermediate outputs only where necessary and permitted.

Evaluate language, noise and errors

Test expected accents, overlaps, names and audio conditions. A transcription error can contaminate summaries and actions; keep human review where it matters.

A reproducible evaluation

With authorised audio, prepare excerpts representing expected speakers, languages and recording conditions. Create a human-reviewed reference transcript and compare errors relevant to the task, not only one overall number. A misspelt name may matter more than several omitted filler words.

Test speaker changes, silence and overlap too. If the system summarises meetings, evaluate the summary against a corrected transcript and, when needed, the audio. Keep recognition errors, speaker-attribution errors and inferences added by summarisation separate.

Map the stages and their data

A bounded pipeline may include ingestion, format validation, voice activity detection, transcription, speaker attribution, timestamp alignment, review and summary. Decide which stages the task actually needs. Record the input and output of each stage, where it runs, who may read it and when intermediate files are removed. Processing on owned equipment does not itself give permission to record a conversation or retain its transcript.

At ingestion, preserve the original long enough for authorised verification, then use a repeatable conversion procedure and record sample rate, channels and any filtering. A conversion can affect recognition; never compare two transcription runs without noting it. Voice detection may avoid processing silence but can cut quiet speech. Diarisation can misassign overlapping speakers, and a correct transcript can still be attributed to the wrong person. Treat those as separate failure categories.

Decide what a usable output means

Prepare authorised examples with expected languages, accents, microphones, noise, jargon, names, interruptions and overlap. Use a human-corrected reference and record the errors that affect the workflow, such as a negation, amount, person or decision. A single overall word-error figure can hide consequential mistakes. Measure latency and resource use under realistic length and concurrency if the system will serve live work.

If a summary or action list follows transcription, compare its statements with corrected text and, for disputed passages, the recording. Mark uncertain speaker attribution and unsupported inferences for human review. Do not let an automated summary silently become a meeting record or instruction to another system.

Plan access, retention and recovery

Separate permissions for original audio, intermediate text, final transcript and summary. Decide whether temporary files, logs, backups and exports inherit the same deletion rule. Test a deletion and a restricted-role access attempt, rather than relying on a storage diagram. Document how an operator corrects a transcript, reprocesses a recording and rolls back a model change. The speech consulting page frames the service, while the readiness checklist helps define its first permitted workflow.