Selected Work
2024 · AI
Octave Audio
Real-time voice cloning API serving 12 languages with 240ms median latency, deployed on a single GPU node.
Octave Audio
Client
Octave
Role
ML Engineer
Year
2024
Discipline
AI
(01)
Problem
Voice cloning APIs were either too slow for real-time use (>2s latency) or too expensive to scale. Octave needed to serve 12 languages at sub-300ms latency without a massive GPU fleet.
(02)
Solution
Distilled a custom model to 1.4B parameters, served it via Triton with dynamic batching, and used ONNX runtime for the audio decoder. Single A100 node handles 80 concurrent streams with headroom.
(03)
Outcome
Median latency: 240ms across 12 languages. 80 concurrent streams on a single GPU. 87% gross margin at $0.004/request.
240ms
Median latency
12
Languages
80
Concurrent streams
87%
Gross margin
Stack
PyTorchONNXTritonFastAPIRedis
Next case