The world of voice AI, with Mati Staniszewski of ElevenLabs

Matty Staniszewski cofounded Eleven Labs in 2022 and has since scaled it to the $11,000,000,000 leader in AI audio. He's credited with capturing the humanness of speech through realistic emotional inflection, inflection, and they're now expanding into everything from agentic workflows to music. Thanks for

doing this.

Thanks for having me. Maybe a place to start is describe to me how like, I know how an LLM works at a high level. Describe to me how an audio model works. Like, if we were Karpathy style looking to, you know, build a toy one from scratch, how does it work?

In the early days, you try to replicate it exactly how you would replicate it with with with the human body. So you try to completely try to reproduce a machine, analog machine, that will create a vocal tract effectively. Then that progressed into trying to create effectively, like, a digital signals for speech. Bell Labs was one of the first to try to, like, create a structured set of signals that will represent the speech, and that is a a first precursor to what we would do today. Then you would try to stitch in phonemes, effectively different sounds of of of how we would speak humans, and t

Yep.

Of of that work. And here, credit to my cofounder, Piotr, who effectively came with that new idea of of how you can now create voice models, which are both reliable, high quality, quick, where you would bring a lot of the ideas from transformer models, from diffusion models into the speech space. Yes. So that prediction of the next token on the phoneme space

Yes.

Wasn't something that was possible. You might be you you always talk briefly about this of, like, how you kind of operate on the text, on the waveform space. There's also a MEL spectrogram space. So, like, usually, you do text, MEL spectrogram, waveform. Sorry. What's the

spectrogram space?