Abstract
A Dockerized FastAPI service wrapping the ONNX INT4 Nemotron speech model for local ASR. It covers the full lifecycle from model preparation and startup self-tests through REST file transcription and WebSocket streaming audio, exposing health and readiness endpoints for deployment monitoring.
1. What This Is
Nemotron Speech Docker packages the Nemotron ASR model (INT4 quantized, run via ONNX Runtime GenAI) behind a standard HTTP service. The container handles model download and preparation on startup, runs a smoke-test inference to confirm the runtime actually works, then serves two transcription paths: a REST endpoint for uploaded audio files and a WebSocket endpoint for streaming little-endian mono float32 PCM audio with incremental results.
2. How It Works
| # | Stage | Input | Tool | Output |
|---|---|---|---|---|
| 01 | Model preparation | Config (model path / download URL) | ONNX Runtime GenAI | Loaded INT4 Nemotron model in memory |
| 02 | Startup self-test | Synthetic / sample audio | FastAPI startup hook | Inference log; readiness flag set |
| 03 | Health / readiness | HTTP GET | FastAPI endpoints | 200 / 503 status for orchestrators |
| 04 | File transcription | Uploaded audio + language / VAD / chunk params | REST POST → ONNX inference | Full transcription text |
| 05 | Streaming transcription | Mono float32 PCM chunks (WebSocket) | WebSocket → ONNX inference | Incremental transcription output |
3. Implementation Notes
3.1 Startup self-test
Process startup alone does not guarantee the model loaded correctly or that the ONNX runtime can actually run inference. The service runs a smoke-test inference during the startup hook and logs basic runtime information before flipping the readiness flag. An orchestrator or a human checking the container gets a definitive "model is working" signal rather than a bare HTTP 200 from a half-initialized process.
3.2 Dual transcription paths
The REST file endpoint accepts configurable language, VAD, and chunk-related parameters, making it suitable for batch or recorded-audio jobs. The WebSocket path accepts little-endian mono float32 PCM in chunks and returns incremental transcription, covering interactive or live-mic scenarios. Both paths share the same underlying ONNX inference, so model state is consistent regardless of transport.
4. Constraints
-
Single-model, single-container scope
The service wraps one Nemotron INT4 model. There is no model hot-swap, multi-model routing, or GPU fallback logic; changing the model means rebuilding or reconfiguring the container.
-
No authentication or rate limiting
The API is designed for a trusted local network. Exposing it without an auth layer or request throttling would be unsafe on any untrusted network.
-
Streaming assumes a continuous PCM feed
The WebSocket path expects a steady stream of mono float32 PCM. Gaps, format mismatches, or client disconnects mid-stream are not covered by documented recovery semantics.
-
No persistent storage or job queue
Transcription results are returned in the response; nothing is persisted. There is no async job queue, so long files block the request until inference completes.
5. Next
- a. Add a lightweight auth token check (e.g. shared-secret header) so the service can be exposed beyond a single-host LAN.
- b. Introduce an async job queue for long file transcriptions so the REST endpoint can return a job ID immediately instead of blocking.
- c. Add a basic integration test suite that exercises the startup self-test, a short file transcription, and a WebSocket streaming round-trip in CI.
— end of report —