Client
By early 2023, Kyros Insights was running its actuarial pipelines as multi-task jobs on Databricks. Each job was a graph of tasks — clean, prepare, train, predict — and each task declared the kind of cluster it needed. The jobs were correct. They just cost more, and ran longer, than the work actually needed.
The cause was how tasks were mapped onto clusters. Every task that asked for the same kind of cluster was put on one shared cluster. That wasted compute two ways. Independent tasks of the same type ran on that one cluster at the same time and split its resources, so each ran starved and slow. And when a sequence stepped onto a different cluster mid-way — a CPU task handing off to a GPU task — the first cluster sat idle through the detour, billing the whole time, waiting for the work to come back.
The job was right. The way it used clusters wasn't.
We let the dependency graph decide the clusters. Walking it once, each task makes a single choice: reuse its parent's cluster whenever it can — when it has a single parent and wants the same kind of cluster. That keeps one cluster of a given type alive across a whole sequence of tasks, so the work flows through with no gap and Databricks pays the cluster's spin-up time once, not once per task. Anywhere the graph branches, merges, or switches cluster type, the task gets its own cluster instead:
Parallel tasks each get a full cluster, instead of several splitting one.
A task waiting on several others gets its own cluster, so none of its inputs' clusters sit idle waiting for the rest.
A change of cluster type gets the cluster it asked for.
Then we share what's genuinely identical: a cluster definition used in more than one place is spun up once and reused wherever that's safe. The result is the fewest clusters the graph allows — parallel work runs in parallel, chains run without a gap, and nothing sits idle. It's a single pass over the graph, so it costs nothing to run.
Two problems only surfaced once real jobs ran through it.
The first was a job with no chains at all — every task independent, all meant to run side by side. Our graph walk handed out a new cluster as it stepped from one task to the next down a chain; with no chains to step through, it never handed out a second one, and every parallel task landed back on a single cluster — the exact resource-splitting we'd set out to fix. The fix was small: give each independent task its own cluster up front, so a flat job spreads out the same way a deep one does.
The second showed up only on the biggest jobs, and it wasn't about clusters at all — they simply wouldn't submit to Databricks. The problem was the size of the request we built. Every task listed its dependencies, but each dependency was the full task object, which listed its dependencies, and so on; on a large job that nesting snowballed until the request blew past Databricks' size limit. We changed it to send each dependency as just its name — a short reference instead of the whole task — and the request fit again.
| Before | After | |
|---|---|---|
| Tasks of the same type | All on one shared cluster | Spread across just enough clusters |
| Independent tasks | Split one cluster's resources | Each gets a full cluster |
| A detour onto another cluster type | First cluster idles through it | Each part runs on its own cluster |
| Identical cluster definitions | Shared | Still shared |
"The team at Qwertee has helped us tackle highly specific and complex problems with impressive technical depth, the kind most teams wouldn't know where to start with. They've proven themselves as dependable partners we're glad to bring our hardest problems to."— Rob Chin, CTO, Kyros Insights
If your Databricks bill keeps growing while your jobs queue up on clusters that are somehow both overloaded and idle, we've untangled that before. Happy to compare notes — even if you're not looking to bring anyone in.
Get in Touch
Subscribe To Our Newsletter. We’ll Send Email Notification Everytime We Release A New Article Or Upgrade Our Services.
2017-2026 Qwertee Technology. All Rights Reserved.