Inside Modulate's Voice AI Stack on AWS

Top-Line Summary

This week is an interview: Mike Pappas and Carter Huffman, the founders of Modulate AI, on why voice is the hardest medium in AI to get right. The through-line is latency — it pushes them off the managed services and down to raw EC2 with GPUs, with some models written in C — and the lesson Brett pulled out is that the value of AWS isn’t moving up the stack, it’s having access to every layer and knowing which one a workload needs.

Show Video

The “Pre-Show” Context

Brett opened with a genuinely unrelated story: a bald eagle doing loops through his Oakville subdivision, backyard to backyard, big enough that he stood in the street staring while neighbors presumably wondered what he was doing. Travers countered with the cabal of eight blue jays his wife (currently deep on birds, last month it was stem cutting and faceting) has trained onto whole peanuts — they form an orderly queue on the feeder and rotate through, and a single jay can cache over a thousand bits of food around a neighborhood.

The Engineering Rundown

• Modulate’s Origin Story: a Missing Minus Sign at MIT (06:17)

  Mike Pappas and Carter Huffman met as MIT undergrads when Mike stood awkwardly behind Carter long enough to spot the missing minus sign in his physics problem. They paired Mike’s breadth (the distractible systems thinker) with Carter’s depth (give him a hard problem and a pizza and he comes back three days later with a solution) and decided to build a company without knowing what it would be. Carter went to NASA’s Jet Propulsion Lab as a machine learning scientist, building low-power models that let spacecraft do autonomous science on the fly: spot something interesting during a 15-minute asteroid flyby and grab an extra measurement before the craft is gone forever. That signal-processing background, plus the mid-2010s wave of adversarial image generation, became the seed of voice AI.

• Why Voice Is the Last Frontier in AI (11:53)

  Carter’s framing is that voice always comes last. People do images first because you can write papers easily, then video, then text, and voice trails everything. Part of the reason is the uncanny valley: a generated face is forgiving (you’ll see a face in anything, extra fingers and all), but if something is off in a voice you notice instantly even when you can’t say why, probably because illness and emotion present through it. Pappas added the nuance that celebrity impersonators rarely sound like their target — it’s the idiolect, the pauses and intonation and cadence, that sells it. Brett connected it to the Echo on his desk: Amazon changed the voice and his wife and parents now hate it while he didn’t notice, which is exactly the all-or-nothing reaction voice provokes.

• The Latency Problem That Breaks Voice Chatbots (18:41)

  The obvious way to build a voice bot — transcribe, send to ChatGPT, run text-to-speech — doesn’t work at all, and the reason is latency. In real conversation the gap between one person stopping and the other starting is roughly zero seconds, so any delay degrades the experience; Modulate’s first voice-skin models ran at 20 milliseconds. To hit that you’re forced onto the flash and lite versions of every model, which makes the bot fast but dumb, and people would rather have the phone tree. Brett noted the human angle too: a pause after you finish talking makes people uncomfortable enough to fill the space, and then you’re talking over the bot and the whole thing degrades further. Voice also carries far more per sentence than text — emotion, tone, cadence — and a transcription pipeline throws all of that away.

• From Call of Duty Moderation to Fraud Detection (24:18)

  Modulate’s first real focus was content moderation for online gaming, powering the AI behind Call of Duty, Grand Theft Auto, and other titles. The hard part is subtlety: telling two 13-year-olds screaming and having a great time apart from genuine harassment, which you can’t do by flagging that someone said a slur. Brett had just broken down and tried DayZ at his nephew’s urging and called it one of the griefiest games he’s played, which made the point — what counts as harassment in DayZ is nothing like a Mario game. That capability extended into fraud: spotting that a crying baby in the background is a recording, or catching a customer pressuring a food-delivery driver into committing fraud over eight dollars of crab rangoon. None of it works from the transcript alone; you have to understand the emotion and intent in the audio.

• Deepfakes Aren’t the Same as Fraud (31:03)

  Modulate starts fraud detection with deepfake detection — and claims a best-in-world detector, thanks to a dataset that captures noise and environment — but the founders were clear that synthetic voice is not the same as fraud. Plenty of legitimate callers use AI agents or text-to-speech, including people who physically can’t speak, so you can’t just hang up on a synthetic voice. The signal is combining that detection with what’s actually being said: a synthetic voice saying “I don’t remember my security questions, let’s do this instead” is suspicious in a way the voice alone isn’t.

• The “OpenClaw Weekly” Tangent (32:16)

  Brett joked the show should be renamed OpenClaw Weekly given how much agent news comes up, and the table riffed on people letting agents act on their behalf — including calling a bank, which is exactly where you’d want fraud detection layered on top. They revisited experiments they’ve covered: a Google DeepMind interviewer who let OpenClaw run a business for two weeks with a credit card, and someone in Europe who let it run a cafe, where it eventually realized it couldn’t physically meet barista candidates and moved interviews to Slack. That spun into Watson’s old cookbook, which trained on protein-centric recipes and so generated cocktails built around a protein — there’s a literal shrimp cocktail and a bourbon steak, none of which Mike recommends after trying them in his youth.

• An Ensemble of 100 Models, Not One Big LLM (35:56)

  Asked how this differs from the LLMs the audience uses daily, Carter drew both a technical and a philosophical line. A big foundation model is general and impressive but tends to hyper-focus on one aspect and run with it, so you get an answer to your question but not a holistic read of the input. Modulate instead runs up to 100 models in parallel and in real time on a single conversation — more than a dozen transcription models alone, some general, some tuned for medical or gaming terminology — and picks the best for the situation. The outputs combine into one dense, information-rich representation of the whole conversation that you layer analyses on top of, which Carter likened to the pre-LLM unsupervised-learning idea of an autoencoder bottleneck. Starting from that rich base gives far more deterministic, reliable results than dropping an audio file into a model and hoping.

• The AWS Stack: EC2, GPUs, and Going Down to C (41:48)

  It all comes back to latency, and not just average latency — jitter and predictability matter just as much for a real-time stream. So Modulate strips out layers of abstraction, even valuable ones, and runs a lot of work directly on Linux boxes with GPUs on EC2, with auto-scaling groups and load balancing. Non-latency-critical models run in Python; the critical ones go down to C, because memory allocation can add a variable delay that almost never matters except when you’re doing real-time audio and the 0.1% cases pile up. They lean on AWS scale and redundancy with intelligent auto-scaling that pre-warms instances — loading models into VRAM and warming queues and caches before traffic hits — and run multi-region, multi-AZ so they can route to the closest box, because adding 100 milliseconds to a voice call is instantly noticeable.

• Access to Every Layer of Abstraction (46:47)

  This is where Brett drew the lesson of the episode. He and Travers spend a lot of time talking about moving up the stack and joke that nobody cares about another batch of EC2 announcements — yet here is an advanced product built on plain EC2, auto-scaling, load balancing, and multi-region, because the latency requirement forces them down to the lowest level. Carter’s correction reframed it: the value isn’t moving up the layers of abstraction, it’s having access to all of them, staying high when you can and going deep when you must. Their training stack proves the point — SageMaker and a lot of back-and-forth with Bedrock for data throughput and agility, where a little latency is fine, then all the way down to the metal when it isn’t.

• FFmpeg, Claude, and Taking Over the World (49:58)

  Travers asked how much time the team spends fighting FFmpeg, and Mike’s answer was that as the commercial lead he hasn’t shipped code in five and a half years but still uses FFmpeg weekly. His half-joking case for being an AI believer: Claude can use FFmpeg and write JavaScript, the two things he can’t do himself, and between him and the AI there’s nothing they can’t do. A light moment that landed a real point about where these tools slot into an experienced engineer’s workflow.

• The Whiteboard Physics Problem (51:10)

  For a video showing off the voice-changing tech, the founders wanted something fascinating on the whiteboard behind them, and with no obvious ML equations they recreated the MIT physics problem from their origin story. They nearly failed to re-solve it after years away from physics: two spinning drums, sand pouring from one into the other, and the question of how much the spilling sand rotates the receiving drum — an angular-momentum problem where at time infinity you work out the outer drum’s velocity. Travers immediately clocked the Good Will Hunting parallel, which is a fitting origin story for a company.

• Closing: Voice AI in 12 Months (53:55)

  The founders’ send-off was optimism grounded in the pattern: images, video, and text all crossed the threshold, and voice is next, with the richness and latency constraints making the payoff feel even bigger once they’re solved. Brett closed on his recurring Ethan Mollick line — the model you’re using right now is the worst one you’ll ever use — and Carter one-upped it: at this pace, the model you’re using is already outdated, go get the new one.

Off-the-Clock Recommendations

• The Backrooms (2026) — Travers’s pick. An ARG-style horror series about falling into a parallel liminal space; he’s excited for it dropping this Friday.
• The Mandalorian and Grogu (2026) — Travers saw it and can’t recommend it, though Brett noted that having a movie theater to yourself can salvage almost anything.
• Red Rising (2014, Pierce Brown) — Brett’s pick. A sci-fi space opera, Mars versus everybody, with enough nuance to pull you away from the keyboard; he’s losing sleep to the second book.
• Invasion of the Body Snatchers (1978) — Brett’s pick. The Donald Sutherland version; a classic that holds up and isn’t actually that scary.
• Amnesia: The Dark Descent (2010) — Carter’s pick, with a catch. Its sanity mechanic is terrifying at 2 a.m. with a group of eight (which is how he met Mike’s now-wife Kirsten) and laughably goofy played alone in the afternoon — at one point the underwater monster is literally just a sphere.
• The Cabin in the Woods (2011) — Mike’s pick. He argues it doesn’t deserve to be lumped in with crappy horror.
• A web-novel Pokémon rewrite — Mike’s deep-nerdom pick. He singled out one genuinely unsettling chapter that reframes the discovery of Ditto as a body-snatcher horror.