Python Concurrency: Serving ML Without Dying in the Process

One July, a launch of one of our models (called Celestino) fanned out to 189 pods. On a normal day it scaled somewhere between a handful and 60; this was three times its usual ceiling, and the CPUs on every one of those 189 pods sat close to idle. Our autoscaler was doing exactly what we’d asked it to do: watch request latency, and add pods when requests slow down. It kept adding them because requests kept slowing down. What it never told us was why: not one of those pods was short on compute.

We are part of the team behind Lykeion, Cabify’s internal ML platform. model-api is the library that wraps a trained model and exposes it as an HTTP service: validate the input, run inference, serialize the output, expose metrics. Getting to the bottom of those 189 pods meant finally getting precise about a distinction we’d been fuzzy about for years:

• Concurrency is making progress on several things over time without necessarily doing them at the same instant: one thread juggling many waits.
• Parallelism is genuinely doing more than one thing at the same instant, which requires more than one execution unit actually running code.

Celestino’s launch is the story of what happens when your serving layer can’t tell those two apart, and how three different architectures (Flask, then MLServer, then a custom FastAPI service) got us to a place where it finally could.

One process, one job (until it’s two)

The earliest model-api served models packaged as an MLflow PyFunc behind a plain Flask app. The first real change was putting gunicorn in front of it with several workers, each a separate OS process. That bought something specific: in CPython (the interpreter you’re running unless you went out of your way not to), only one thread per process can execute Python bytecode at any instant, the Global Interpreter Lock. It exists because CPython counts references to every object as it runs, to know when to free one, and letting multiple threads increment and decrement that count at once, unguarded, corrupts it. Threading arrived years after refcounting, and one lock around the whole interpreter, the GIL, was the smallest change that let the existing interpreter survive contact with threads.

The consequence: for code actually running in the Python interpreter, threads inside one process don’t get you real parallel computing; separate processes, each with their own interpreter and their own GIL, do. (Two exceptions to that, both real and both coming up later: a thread that’s genuinely waiting rather than computing, and compute that’s secretly happening in compiled C rather than in the interpreter at all.) Several gunicorn workers meant several real processes, which is exactly what a genuinely CPU-bound model needs, and it did help the first time we hit one.

It also came with a cost we hadn’t fully priced: each worker wasn’t just another copy of the model, it was another full copy of the entire API server, HTTP stack included. And it solved a problem we were about to solve again, one layer up: once this runs inside Kubernetes, replicas already give you more OS processes across more pods. Gunicorn workers inside a pod and pod replicas across the cluster are the same lever, pulled twice. Gunicorn’s multi-worker model makes sense on a VM with nothing else scaling it horizontally, and starts fighting your own autoscaler the moment an orchestrator is already doing that job for you.

What MLServer couldn’t tell apart

We wanted to improve on that, and went for MLServer, the framework behind Seldon Core, mainly for one feature: a real, already-built process pool for inference, on top of a REST/gRPC front door we didn’t have to write ourselves. For models that only ever computed on the input they were given, it worked well.

Celestino was the first model that needed to fetch something before it could predict, at any real scale. Cabify has an internal library, the Lykeion Data Connector, that lets a model call out to a feature store, Bigtable, ClickHouse, or even another model, all genuinely I/O-bound work: waiting on a network round trip rather than on the CPU. Celestino was the first model mixing that kind of waiting with the CPU-bound work of the model itself, in the same request, under production traffic.

Figure 1: Request time breakdown (p95) during Celestino’s launch.

That panel isn’t a trace with spans laid end to end; it’s four separate p95s (request total, external-data fetch, model prediction, response formatting), each its own Prometheus histogram, plotted as overlapping bands rather than stacked ones. They will never add up to the total, and they’re not supposed to: a p95 doesn’t distribute over a sum, the request that lands in the slowest 5% for the external-data fetch isn’t necessarily the same request that lands in the slowest 5% overall. So don’t read it as arithmetic; read it as coverage. If the request-total band is almost entirely covered by the other three, none of the request’s time is unaccounted for, it’s all fetch, prediction, or formatting; a visible sliver of the total band left uncovered is time spent somewhere the panel isn’t measuring. Here, the external-data fetch band alone came close to covering the whole request-total band on its own, which is what actually pointed at where most of every request’s time was going.

MLServer had nowhere to put that distinction. Each worker handled one request start to finish, inside one process: the external-data call and the model’s own computation both just ran, sequentially, on the same worker. A worker blocked on a network call wasn’t computing anything; it was simply occupied, unavailable to pick up the next request, for exactly as long as the network took to answer, indistinguishable, from the pool’s point of view, from a worker that was genuinely CPU-bound and busy.

That’s how we got to 189 pods. Under Celestino’s launch traffic, most of what was arriving was exactly this: calls that needed to wait on external data. Every worker doing that wait was a full OS process: its own copy of the model loaded, its own memory footprint. CPU utilization stayed low, because nothing was actually running on the CPU; MLServer’s own request queue, sitting in front of the worker pool, kept growing anyway, because every worker was occupied regardless of what it was doing. Growing queues mean growing latency, and our autoscaler was watching latency: it kept doing the only correct thing it could do with that signal, and added more pods.

It wasn’t wrong to: more full-process workers genuinely does mean more concurrent network waits absorbed. It’s just an absurdly expensive way to buy wait-capacity: every one of those pods carried its own full copy of the model in memory, to do nothing more than sit and wait on a socket, for work that was never CPU-bound in the first place.

Some of those pods didn’t even get to sit there quietly. Piling up occupied workers, each holding its own copy of the model plus whatever state the stalled request was carrying, pushed memory usage up in irregular spikes until Kubernetes killed the pod outright; a fresh one took its place and started the same climb over again.

Figure 2: Per-pod memory usage before the fix, spiking until Kubernetes killed the pod.

This is where we learned, the expensive way, the rule the rest of this post is built on:

The rule: I/O-bound waiting belongs on an event loop, where one thread can juggle hundreds of concurrent waits without ever needing a process each. CPU-bound computing belongs in a separate process, because that’s the only thing in CPython that gets you two things executing at the same literal instant.

An event loop doesn’t remove the wait; it just means the wait doesn’t cost you an entire OS process’s worth of capacity while it happens.

Why this had to live in the framework

The immediate fire got put out fast: we retuned KEDA, the autoscaler, to scale Celestino off the pool’s own queue depth instead of request latency, and unblocked the launch itself by reusing the DataFrame-attribute escape hatch (external data stashed on a custom attribute, ahead of predict()) as a one-off, hand-built Docker image pushed straight to the registry, bypassing the platform’s normal deploy pipeline entirely. This was not a platform-wide solution. Both only reached Celestino, and the next model that mixed I/O waiting with CPU computing would hit the identical wall, because the actual defect wasn’t in what we measured: MLServer’s worker contract gave us exactly one hook per request, with nowhere to put an I/O-bound phase except inside that same synchronous predict() call, on that same worker. Fixing that for good meant fixing it in our serving layer, for every model, not per incident.

That had to be the framework’s job, not each model’s, for a reason that isn’t purely technical: we’re Data Engineering Platform, and the model’s own code is Data Science’s. That line is deliberate. Optimizing how a request gets served (threads, processes, event loops, pickling) isn’t something we want a data scientist shipping a model to ever have to think about; it’s a platform concern, and platform concerns belong in the platform, as an abstraction the model owner gets for free, not a skill they’re expected to pick up.

It’s also why the fix wasn’t “stop using Python”: a data scientist’s whole workflow, training included, already lives in Python, and asking them to hand off a model to be rewritten in Go or C++ to serve it just moves the same organizational line somewhere worse, adding a translation step and a second codebase to keep in sync with the first. The problem was never the language; it was which half of the request path had to think about serving mechanics at all.

Building the split ourselves

Figure 3: MLServer’s single worker vs. the FastAPI event loop/process pool split.

So we rewrote model-api a second time, this time on top of FastAPI, and built the split into the framework itself instead of leaving it to each model’s own code. Getting there wasn’t a straight line. Before Celestino ever existed, the first way we let any model fetch external data was an optional external_data() function whose result got stashed as a custom attribute bolted onto the pandas DataFrame, riding along until it reached predict(). It’s the kind of workaround every engineer recognizes: technical debt taken out on purpose, to buy time on a problem nobody had hit yet, with every intention of paying it back the moment it actually mattered.

Once Celestino forced the real question, two actual false starts followed. We tried making MLflow’s synchronous predict() behave asynchronously by spawning a thread to run the model’s own inference inside it. That didn’t help, because the thread was hosting CPU-bound compute, not a wait. It’s the same GIL constraint from the very start of this post, just rediscovered from the inside: a thread inside an already-busy process doesn’t add capacity for CPU-bound code, it only adds bookkeeping. We tried spawning a process pool from inside the model’s own code next. That also didn’t hold up, because the model’s code was exactly the layer that shouldn’t have to know about any of this.

What we built instead is model-api’s adapter pattern, and it exists specifically to give the framework, not each model’s code, control over which half of a request goes where:

Figure 4: Request flow through the adapter pattern.

From a model owner’s side, that split is nearly invisible. This is the entire adapter for a model that needs external data:

class MyModelAdapter(Adapter):
    async def preprocess(self, request):
        features = await fetch_features(request.user_id)
        return build_dataframe(features)

    def predict(self, dataframe):
        return self.model.predict(dataframe)

preprocess() is async def; predict() is a plain method. No threads, no processes, no pickling in sight, and that’s the point: the framework decides where each of those runs, not the person who wrote them.

preprocess() and postprocess() are coroutines: async def functions scheduled on FastAPI’s single-threaded event loop. Calling one doesn’t run it; awaiting it does, and the moment it hits a point where it would have to wait (a network call, in our case), it suspends and hands control straight back to the loop, which is free to resume some other suspended request in the meantime. That’s the entire difference from what MLServer did: a request awaiting a feature-store call still takes however long it takes, but for the whole duration of that wait it isn’t holding an OS process hostage; the loop is free to progress every other request in flight, all on the one thread, no extra process per waiting request.

This only works because the Lykeion Data Connector’s client is itself built on asyncio: an async def wrapped around a call to a regular, synchronous client would still block the one thread the whole loop depends on for however long that call takes, and every other request in flight would stall behind it, same as it would under MLServer.

The GIL isn’t in tension with any of this: a thread genuinely blocked on I/O releases it, which is exactly why threads were the original way to do concurrent I/O in Python, before asyncio offered a lighter-weight version of the same trick: one thread instead of many. It’s also why FastAPI itself can run plain, non-async route functions in a thread pool behind the scenes, instead of refusing to serve them. (FastAPI’s own docs on this distinction are worth the ten minutes if any of this felt rushed.)

Predict, in its own process

For models where predict() is expensive enough to need it, model-api never runs it inline on that loop; it goes to a dedicated process pool:

_model = None  # populated once per worker process, never passed as an argument

def _create_model(config):
    global _model
    _model = load_model(config.model_uri)

def _model_predict(model_input):
    return _model.predict(model_input)

class PredictorProcessPool(Predictor):
    async def initialize(self) -> None:
        self._pool = ProcessPoolExecutor(
            max_workers=self._config.parallel_workers,
            mp_context=multiprocessing.get_context("spawn"),  # start a fresh interpreter per worker
            initializer=_create_model,
            initargs=(self._config,),
        )

    async def _predict_core(self, model_input):
        loop = asyncio.get_running_loop()
        predict_future = loop.run_in_executor(self._pool, _model_predict, model_input)
        return await asyncio.wait_for(predict_future, timeout=self._config.predict_timeout_seconds)

That await is doing the same job it did in preprocess(): the moment the pool takes the work, this coroutine suspends and hands control back to the loop, exactly as if it were awaiting a network call. The loop doesn’t know or care that what it’s waiting on is a whole other process computing rather than a socket; either way, it’s free to serve every other request in flight for as long as this one takes, instead of sitting there blocked until predict() returns.

We ask for spawn explicitly rather than multiprocessing’s own default (fork, on Linux), because our own process already runs an event loop and, depending on configuration, other threads; forking that is a known way to inherit half-held locks and a copy of state that was never meant to be duplicated mid-flight. spawn costs more at startup (a genuinely fresh Python interpreter, re-importing the module from scratch, inheriting nothing from the parent) in exchange for never having to reason about what state survived the fork. Nothing the parent process holds, including the loaded model, exists in a worker unless the worker creates it itself, which is why _create_model runs once, via initializer, at worker startup.

That’s also why the model lives in a module-level global instead of being passed into _model_predict as an argument. Two separate processes have separate address spaces (a pointer that means something in one is meaningless garbage in the other), so nothing gets shared directly; anything that crosses has to be pickled first, serialized to plain bytes and rebuilt from scratch on the other side. Not everything can make that trip. (Even initargs has to clear that bar: config above is a plain, picklable settings object, not anything holding a live connection or a lock.) Loading the model inside the worker, once, sidesteps the question entirely: it’s created there and never crosses the boundary in either direction.

run_in_executor submitting _model_predict by its plain module path, rather than a bound method or a closure, is the same constraint from the other side: ProcessPoolExecutor pickles a function by reference, so it has to be importable at that exact path from inside the worker. The model and the function get to dodge pickling entirely; model_input and the returned prediction don’t: those genuinely cross the boundary on every single request, which is real, ongoing serialization cost, not a one-time setup tax.

Trade-offs we made on purpose

One limitation this same code has, worth naming plainly: asyncio.wait_for timing out here will end our wait, but the worker keeps running. The process pool has no way to interrupt a worker mid-computation, so a timed-out request frees the caller while the worker keeps computing to completion, occupied, producing an answer nobody will read: a smaller-scale version of the exact failure mode this whole post is about. We accept that trade-off for a bounded response to the caller; it isn’t a way to reclaim the worker’s capacity.

A separate process is also a stricter rule than physics strictly demands: if a model’s compute is mostly NumPy or another C extension that releases the GIL while it runs, threads inside one process can genuinely parallelize that specific workload, no separate process required. We still don’t offer that as an option, and it’s a deliberate trade, not an oversight: “always use a process pool” costs some memory we didn’t strictly need on those models, but it’s a mistake that shows up immediately, in the bill.

Key design decision: The alternative (a per-model flag saying “safe to thread here”) fails silently instead: get it wrong once, because a library update quietly stopped releasing the GIL, and you’re back to the exact CPU contention this whole post is about, with no test suite around to catch it, only a symptom to trace back to a cause nobody had reason to suspect. We’d rather overpay in memory, visibly, than carry a flag that can go stale without telling anyone.

The full system, three years later

We later added a second, separate pool just for preprocess()’s own CPU-heavy moments, because some adapters mixed the I/O-bound external-data fetch with genuinely CPU-bound feature transformation on the fetched data (a pandas DataFrame transformation heavy enough to matter, say), and the CPU-bound half of that, left inside the same coroutine, blocks the loop for every other request in flight, not just its own. Same rule as before, applied one level deeper.

Put all of that together and here’s what actually runs in production, two rewrites after the naive Flask approach we started with:

Figure 5: The full model-api architecture running in production today.

The gains weren’t only architectural. Because each model-api worker loads only the model, its telemetry, and its logging (nothing else), the memory overhead dropped sharply compared to MLServer’s equivalent workers: running the identical model on the identical pod sizing, six model-api worker processes settled at roughly 1.45 GiB of total RSS (actual RAM in use, not just reserved), against 3.2 GiB for six equivalent MLServer workers.

That’s not just a tidier number, it’s a cost one: the 189 pods weren’t only an operational headache, they were 189 pods’ worth of memory billed for workers that were never computing anything, sitting idle on a network wait. Cutting per-worker memory roughly in half compounds across however many replicas an autoscaler decides to add, which is exactly the multiplier that made Celestino’s launch expensive in the first place.

Owning the framework also finally let us make sense of a bug that, under MLServer, had only ever shown up as an intermittent crash we couldn’t explain: an adapter calling asyncio.run() from inside predict(). It’s the same instinct as the reverted thread-in-predict() attempt from earlier, just one layer more subtle: trying to run I/O-flavored async logic inside the CPU-bound half of the pipeline, instead of moving it to the half we’d already built for exactly that.

asyncio.run() starts a brand-new event loop and refuses outright if one is already running on that thread, so whether the call crashes comes down entirely to whether the thread it lands on already has a loop live on it. Under MLServer it crashed unpredictably, because we couldn’t tell which thread that code path would run on, or what else was already running there. Under our own process pool the question has a clean answer: a spawned worker starts clean by design, so there’s no loop to conflict with — asyncio.run() would never crash there.

The lesson: “Wouldn’t crash” isn’t the same as “belongs there.”

The real fix wasn’t a smarter runtime check; it was pulling that logic out of predict() entirely and letting preprocess() do it, on the event loop, the way everything else I/O-bound already did. Left where it was, asyncio.run() would have kept quietly working: it gives a worker its own throwaway loop, cheaply, and nothing crashes. What it costs is that the worker sits occupied for the length of a network round trip while it runs, the same waste as the 189 idle pods, just at the scale of one process instead of a fleet.

Stepping back: N processes inside K replicated pods is, on paper, the exact same lever we said not to pull twice back with gunicorn. It’s not repeating that mistake, and the difference is what each pull is actually buying. The process pool still copies the model into every worker; that cost never went away, and the memory numbers above are proof of it. What changed is what else gets copied along with it. Gunicorn’s workers each duplicated the entire API server, fighting Kubernetes’ own pod replicas for the same job; these pool workers duplicate only the thing that actually needs a core each to compute in parallel, while exactly one event loop handles HTTP and I/O for the whole pod. Concurrency and parallelism finally have separate dials, instead of one process type standing in for both.

We’re not going to dress that up as a single headline number (a launch retrospective isn’t the place for a vanity metric), but the position is categorically different. A launch like Celestino’s, today, produces exactly as many processes as its compute actually needs, and one event loop absorbing however many I/O waits show up alongside it, instead of one more idle-CPU pod for every request that happened to be reaching for the network.

What we’d tell our past selves

• A worker that’s waiting is a worker that’s occupied, whether or not it’s using the CPU. If your autoscaler watches a downstream symptom (latency, CPU) instead of the resource that’s actually full, it will keep adding pods for as long as that symptom keeps getting worse, at a cost the problem never actually justified.
• I/O-bound work goes on an event loop; CPU-bound work goes to a separate process; nothing else in CPython gives you real parallel compute. Threads only help the CPU-bound side when there’s compiled code underneath releasing the GIL for you, the way NumPy or a Cython extension does around its own C loops; otherwise they buy nothing over doing the work in-line, because the thread is still just taking turns executing Python bytecode.
• Crossing a process boundary means pickling, and not everything survives the trip. Load the model inside the worker via an initializer, not as an argument, and only ever offload functions importable by their module path.
• Owning the request path means owning its failure modes too. A bug that was a guessing game inside someone else’s framework became obvious the moment the code, and the event loop it ran on, were ours.

The GIL isn’t gone yet, though it’s on its way out: free-threaded Python, first proposed in PEP 703, is now officially supported as of 3.14 under PEP 779’s phased rollout, just still opt-in rather than the default build. Until it is the default, this exact question, “is this waiting or computing,” will keep deciding whether a Python service scales or quietly fills up with idle CPUs and a growing queue behind them. FastAPI’s own docs on concurrency and parallelism are the clearest short version of the same idea, if you want it without the 189 pods.