Loading article
November 3, 2026
Voice traffic spikes without warning, and that’s exactly when concurrency limits bite. Concurrency limits are bottlenecked by your speed to start new containers. To support absurdly high concurrency limits, we cut model initialization from 5.6 minutes to 26 seconds.
A month ago, one of our customers, a home security company, had a service outage late at night. Their support line usually runs a handful of calls at that hour. Within ten minutes, call volume was up more than tenfold, and it stayed there for most of an hour. About four in ten callers brought up the outage.
This is just natural for most voice traffic: as the customer’s traffic changes, voice should be resilient and adaptable. Most voice traffic will look flat for hours, then jumps tenfold when something breaks, often at an hour nobody planned for. That's also where most voice AI platforms enforce concurrency limits, and calls past the limit get rejected, queued, or billed at a premium.
Why do limits exist at all? A model provider's concurrency is roughly number of containers × calls per container. Calls per container is a question of model efficiency, and the subject of a future post. This one is about the number of containers. In a spike, what matters isn't how many containers a provider could run; it's how many they can bring online before callers give up. When a new GPU container takes five minutes to start, voice model providers have to enforce lower caps so they don’t pay for an idle warm GPU pool (the classic peak-to-average problem in system design!)
To support absurdly high concurrency limits, we cut model initialization, the slowest part of a cold start, from 5.6 minutes to 26 seconds.
To understand how cold start time affects the ability to scale concurrency, we built a quick toy model from production traffic. It replays the home security outage above as it happened, minute by minute, on a shared platform alongside other customers’ steady traffic. An autoscaler keeps a little spare capacity and adds containers as calls climb, but each new container can’t take calls until its cold start finishes. The only thing we change is how long that cold start takes.
Drag the slider to try your own cold start.
From looking at this plot, it’s quite clear that fast cold starts make generous concurrency limits affordable.
Cold starts determine how much spare capacity a provider needs to keep running. Every provider keeps some containers warm to handle spikes, but it pays for those GPUs regardless of spiking traffic or not. The longer it takes to start a new container, the larger that buffer needs to be. For this outage, a 26-second cold start needs less than 3% spare capacity to handle every call. If the cold start time is 5.6 minutes, then we need about 18% spare capacity.
To make matters worse, with the same 10% buffer, a 26-second cold start never drops a single call. Under this toy model, the 5.6-minute cold start runs out of capacity for more than four minutes, leaving nearly 800 calls with nowhere to go at its worst.
This is why cold starts matter for concurrency limits. If we can provision containers faster, we can handle a spike without wasting money on a warm pool of GPUs or charging customers for bursty traffic. Phonic supports several thousand concurrent calls per customer, with no burst surcharge.
So how did we make that possible?
We started by profiling what a fresh container spends its time on, from the moment it starts to the moment it can take its first call.
Model initialization alone took 336 seconds, 61% of startup, and almost all of it was compiling the model.
Compiling turns the model into GPU code specialized for the exact shapes of the data it will process, such as how many conversations are in a batch and how long each input is. Specialized code runs much faster than generic code, which is why serious serving stacks compile.
Almost everyone compiles just in time, on the machine that will serve the model, with torch.compile. The reason is generality. Engines like vLLM and SGLang support hundreds of model architectures on many kinds of GPUs, and torch.compile works on almost any PyTorch code.
Traffic shapes still vary, so they compile once at startup with the batch size left flexible, prepare a fixed set of batch sizes in advance, and pad every batch up to the nearest one. After that, they rarely compile again, but every new server pays for it at startup.
Most of the design decisions in voice come down to quality, latency, and price. Compilation touches the last two. If our goal is to pack as many conversations as possible into each container and start containers faster, we can trade some of the generality of just-in-time compilation for a more specialized approach that’s faster.
First, what if we just eliminate redundant work? Every container generated the same GPU code on boot, so there was no reason to regenerate it. We moved the CUDA, Triton, and PyTorch Inductor caches onto shared storage, so the first container to compile a kernel saves it for every container after it.
# Point PyTorch's compile caches at a volume every container shares.
# Compiled kernels only work on the hardware and software that built them,
# so the cache key covers all three.
key = f"{gpu_type}_{cpu_march}_torch{torch_version}"
os.environ["TORCHINDUCTOR_FX_GRAPH_CACHE"] = "1"
os.environ["TORCHINDUCTOR_CACHE_DIR"] = f"/data/torchinductor/{key}"
os.environ["TRITON_CACHE_DIR"] = f"/data/triton/{key}"
os.environ["CUDA_CACHE_PATH"] = f"/data/nv/{key}"Model initialization dropped from 5.6 minutes to 2.5.
Caching got us to 2.5 minutes, but much of what remained was the same functions compiling again. Those are recompiles. When torch.compile compiles a function, it assumes things about the inputs, like each tensor’s size, and checks those assumptions with guards on every call. When a guard fails, PyTorch compiles the function again.
Setting TORCH_LOGS=recompiles makes PyTorch log each recompile and the guard that caused it. A typical entry from our logs:
torch/_dynamo/guards.py:2791] [0/1] [__recompiles] Recompiling function _embed_and_mask_input in /root/echo/models/models.py:180
torch/_dynamo/guards.py:2791] [0/1] [__recompiles] triggered by the following guard failure(s):
torch/_dynamo/guards.py:2791] [0/1] [__recompiles] - 0/0: tensor 'L['tokens']' size mismatch at index 1. expected 8, actual 64Here, _embed_and_mask_input had been compiled for the first size it saw, a sequence length of 8. Then an input arrived with 64 tokens, the guard on that dimension failed, and PyTorch compiled the whole function again.
At the time, the fix was a one-line hint telling PyTorch which dimensions vary, for example torch._dynamo.decorators.mark_unbacked(tokens, 1). Hints on every tensor that tripped a guard brought model initialization to about 2 minutes.
Recompiles also disrupted live calls. Whenever traffic introduced an input shape the model hadn’t seen before, PyTorch would compile it on the running server. Because PyTorch shares compiler tracing state across the process, compilation held a lock that blocked audio decoding too. A burst of incoming calls could therefore stall the audio on calls already in progress.
At 2 minutes, we could see the curve flattening: caching saved 186 seconds, fixing recompiles only 30. Every fix made compilation faster, but the curve was approaching a floor, not zero. As long as a container compiles anything at startup, cold start time has a floor set by that compilation.
By caching and giving hints to the compiler, we were able to make it faster, but we were still approaching the limit of compiling at all. So we asked a different question: what if a new container could get away without compiling anything?
That’s what ahead-of-time compilation does. We compile the model once at build time with torch.export and AOTInductor, ship the result as a binary, and load it when a container starts.
The just-in-time path, torch.compile, watches the model as it runs. The first time it sees a new shape, it traces the Python code, compiles what it traced, and falls back to ordinary Python for anything it can't trace. torch.export does the tracing once, ahead of time, with no fallback. It runs the model's Python code symbolically and records every tensor operation into one complete graph with no Python left in it. Anything it can't capture, like a branch that depends on a tensor's values, is an error at export time rather than a slowdown in production. AOTInductor then compiles that graph into a self-contained package: the GPU kernels plus the CPU code that launches them, which a server loads like any other library.
The hard part is that a just-in-time compiler gets to see the traffic before it commits to anything. Compiling ahead of time means making every one of those decisions in advance:
The exported model has to accept any number of active calls, any cache size, and any input length, without ever recompiling. We declare all three as dynamic dimensions at export time, and each varies independently of the others. Active calls are packed into compact rows, and each row points to its conversation's slot in the cache, so the same compiled program serves one call or a full container. Each call keeps its slot from the moment it starts until it ends.
# Three sizes vary independently, so one program accepts any of them.
active_batch = Dim("active_batch", min=1, max=max_batch_size) # calls this step
cache_batch = Dim("cache_batch", min=1, max=max_batch_size) # cache slots
seq_len = Dim("seq_len", min=1, max=max_seq_len) # input length
exported = torch.export.export(
model,
args=(h, input_pos, cache_batch_idx, k_caches, v_caches),
dynamic_shapes={
"h": {0: active_batch, 1: seq_len},
"input_pos": {0: active_batch, 1: seq_len},
"cache_batch_idx": {0: active_batch}, # each row's cache slot
"k_caches": [{0: cache_batch}] * num_layers,
"v_caches": [{0: cache_batch}] * num_layers,
},
)An exported model is a pure function: inputs go in, outputs come out, and nothing persists in between. A conversation's memory, the KV cache, normally lives inside the model, so we moved it out and made it an input. Attention writes new entries into that cache in place, and we had to declare that write explicitly (the mutates_args in the next example). Without the declaration, the compiler is free to assume the cache never changes, and the model produces wrong output instead of an error.
class ExportableModel(nn.Module):
# The KV cache is passed in rather than stored on the module,
# so the exported program stays a pure function.
def forward(self, h, input_pos, cache_batch_idx, k_caches, v_caches):
for layer, k_cache, v_cache in zip(self.model.layers, k_caches, v_caches):
h = layer(h, input_pos=input_pos, cache_batch_idx=cache_batch_idx,
k_cache=k_cache, v_cache=v_cache)
return self.model.norm(h)Exporting also meant removing every point where the model handed control back to the CPU mid-inference, which made each call faster on its own.
Our attention runs on FlashAttention, and our activations use a fused kernel. Both are hand-written GPU code the compiler can't look inside. We wrapped each one as a custom operator with a stand-in that tells the compiler what shape the kernel returns, without running it. That keeps our fastest kernels inside the compiled program.
# FlashAttention as a custom op. It writes the cache in place, and says so.
@torch.library.custom_op("phonic::flash_attn_with_kvcache_aoti_compiled",
mutates_args=("k_cache", "v_cache"))
def flash_attn(q, k_cache, v_cache, k, v, cache_seqlens, cache_batch_idx):
return flash_attn_with_kvcache(q=q, k_cache=k_cache, v_cache=v_cache, k=k, v=v,
cache_seqlens=cache_seqlens,
cache_batch_idx=cache_batch_idx, causal=True)
# The stand-in: tells the compiler what comes back, without running the kernel.
@flash_attn.register_fake
def _(q, *args):
return torch.empty_like(q)The compiled package isn't only GPU code. It's a zip file, and here's what's inside ours:
| File | What it is |
|---|---|
.wrapper.so | The generated C++ compiled into a shared library, with the model’s weights embedded in it |
.wrapper.cpp | The generated C++ source that replaced the Python forward pass. It checks input shapes and launches the GPU kernels in order |
.cubin | The GPU kernels Inductor generated and compiled |
.wrapper.json | Metadata, such as the input and output specs |
.kernel.cpp | A small kernel that runs on the CPU |
Everything except the .cubin files runs on the server's CPU, and the shared library is built for that CPU's instruction set. So every release is built twice, once for AMD Zen and once for Intel Sapphire Rapids. Each server detects its CPU when it boots and loads the matching build. If it doesn't recognize the CPU, it refuses to start rather than run code built for different hardware.
# Inductor emits CPU code tuned for the machine it compiles on,
# so every release is built once per CPU family.
CPU_TARGETS = {
"znver": "AMD Zen",
"sapphirerapids": "Intel Sapphire Rapids",
}
for cpu_march in CPU_TARGETS:
export_model(checkpoint, output=f"{release}/{cpu_march}/model.pt2",
expected_cpu_march=cpu_march)
# At boot, each server loads the build for its own CPU,
# and refuses to start on one it doesn't recognize.
model = aoti_load_package(f"{release}/{get_cpu_march()}/model.pt2")Production has no just-in-time path to fall back on. The build and the server pin the same PyTorch version, and a release is published only after both CPU builds succeed. We traded flexibility for predictability: if something is wrong, it fails at build time, not on a live call.
Model initialization went from 5.6 minutes to 26 seconds.
Along the same theme, we took the redundant computation out of startup and moved it to build time. That came with a new advantage: we can autotune kernels at build time too, searching for the fastest configuration of each GPU kernel, a search that would take far too long at startup. In production, the model runs through a blazing fast C++ implementation, so the process-wide lock is gone and audio decoding runs alongside it without stalls.
Because we only serve our own models, we can make the opinionated trade-offs that ahead-of-time compilation requires. That focus is why our model initialization is about 4× faster than vLLM or SGLang.
Memory snapshotting is the other well-known route to fast cold starts: let the container initialize once, save its memory, and restore that state on new machines. It works well, and for many workloads it's the right tool.
We went a different way because the two solve different problems. A snapshot skips startup work by restoring memory that has already been initialized. Ahead-of-time compilation removes the work altogether, and the changes it forced on us made every call faster, not just the first one. The output is also a versioned binary we build and test with every release, rather than a captured process that has to match the host it's restored on.
The two aren't mutually exclusive. With compilation off the critical path, snapshotting is one of the tools we're looking at for what remains.
Unfortunately, we do trade off this speed for a bit of generality. Now, every architecture change has to stay export-compatible: fixed control flow, no state on the module, nothing handed back to Python mid-inference.
It also adds work to every release. Each new checkpoint means building, autotuning, and validating a binary for both CPU families.
And cold starts are only half the equation. They decide how many containers we can add in a spike. How many calls fit in each one is a separate problem, and the subject of a future post.
This is how we keep our customers’ agents answering calls when traffic spikes, and it’s the same engineering that goes into every part of Phonic.
If you’re building voice agents, sign up and try it on your own traffic. And if you want to build the models behind those calls, from GPU kernels to large-scale pretraining and reinforcement learning, we’re hiring.
