The craft-era tax on your data team
Imagine a world where every car manufacturer starts by digging up iron ore and smelting it into steel. Before a single vehicle rolls off the line, you're mining, refining, and forging raw materials from scratch. No supply chain. No specialization. Just ore to steel to car, every time. No car company operates this way. The competition wouldn't allow it. A manufacturer that smelts its own steel doesn't survive long enough to build its first car. The company down the road that buys sheet steel and spends its energy on engineering is already three model years ahead.
I love cars. Growing up, a car was your ticket to freedom. Turning 16 meant independence. My weakness is the two-seater convertible roadster. The Mercedes-Benz SL, the Mazda MX-5, the BMW Z4, the Porsche 718 Spyder. Beautiful works of art, every one of them. And I would never have experienced a single one if they weren't built in a factory, on a production line, from materials someone else refined. The factory didn't kill the craft. The factory is what made the craft possible.
Now consider data engineering. I used to work at a pediatric healthcare company. Their mission was treating children. They employed a team of infrastructure engineers to build data pipelines from scratch: wiring up containers, writing deployment scripts, managing state databases, debugging scheduler failures at 2am. A company that exists to help sick kids, smelting its own steel so it could have basic analytics.
This is not an edge case. This is industry-standard. Every company that wants a data pipeline starts by building the factory that builds the pipeline. No one sells the factory. So your engineers build it, from raw materials, every time. The tax shows up in three ways.
Artisanal by default. No standard process, no reusable architecture, no way to go faster the twentieth time than the first. You're paying senior engineers to solve the same infrastructure problems they solved last quarter, and the quarter before that. Every new project is a greenfield rebuild of the same foundation.
The backlog never shrinks. Your engineers aren't slow. They're buried in infrastructure work, environment issues, and deployment toil. The bottleneck isn't talent. It's the absence of a production system around the talent. The data work sits in a queue while the plumbing work consumes the sprint.
Production breaks require a hero. When a pipeline fails at 3am, someone has to diagnose it manually, fix it manually, and hope the fix holds. There's no runbook, no automated recovery, no standard playbook. You're running production workloads on craft-era tooling, hoping the person who built it is still reachable.
This is not a SQLMesh problem or a dbt problem. Those tools are excellent. The engine and the transmission are production-grade. What's missing is the factory that gets them into the hands of every team that needs them.
What industrialized looks like
The industrial model isn't about removing engineers from the loop. It's about removing the wrong work from their days.
When manufacturing industrialized, craftsmen didn't disappear. They moved up the stack. They designed better products instead of hand-filing parts. They solved harder problems instead of repeating solved ones. The factory handled the repeatable work. The people handled the work that required judgment.
Sequoia Capital laid this out in "Services: The New Software": for every dollar spent on software, six dollars are spent on services, on the human labor to deploy, operate, and maintain it. The next generation of consequential companies won't sell tools. They'll sell outcomes. Every other professional function has already made this shift. Accounting, legal, payroll. You don't buy ledger software and hire a team to run the books from scratch. You buy the outcome, the same way a car manufacturer buys sheet steel instead of mining ore. Data engineering is one of the last functions still buying the tool and assembling the factory yourself. That's the gap.
Here's what the outcome looks like when a team has crossed it:
- The data engineer thinks about the model, not the container. Their day is data modeling, business logic, analytical problems. Not Dockerfiles, scheduler configs, or deployment pipelines. That work doesn't exist in their world.
- They merge and move on. The pipeline runs. They don't monitor a deploy. They don't check a dashboard to see if it made it. They're already working on the next problem.
- When something breaks, they get a description of the data problem, not a page about infrastructure. They diagnose a logic error or a schema change. Not a dead pod. Not a failed container restart.
- Starting a new project feels like starting a new project. They open a repository, write their first model, and it runs. They don't spend the first two weeks building the scaffolding to run anything at all.
This is where the industry is heading. Teams that get there first don't just work faster. They work on different problems entirely.
The economics of assembling it yourself
Sequoia's 1:6 ratio is a useful lens here. One dollar of software, six dollars of labor. Data teams generally follow one of two paths, and both end up in the same place.
The first path is self-hosted open source. You pick your transformation framework, containerize it yourself, build your own CI/CD, manage your own state databases, wire up your own scheduling and monitoring. The tools are free. The labor to operate them is the entire bill.
The second path is commercial. You buy Snowflake, Databricks, Fivetran, dbt Cloud. Five or six different products. And you still do all the integration labor yourself. You're still wiring them together, managing the handoffs between them, building the deployment pipelines, debugging the failures at the seams. The tools are not the expensive part. The labor to make them work together is.
That labor cost is not producing any competitive differentiation. It is producing infrastructure that every other company is also producing, independently, from scratch, right now.
The car manufacturer who buys sheet steel doesn't just save money. They redirect the saved labor toward building a better car. Their competitor, who is still mining ore, cannot compete on the car. They are too busy on the ore.
This is the pressure that closes craft eras. It isn't a philosophical shift. It's a competitive one. The teams that stop assembling their own production systems will redirect that engineering capacity toward the work that actually moves their business. The teams that keep assembling will fall further behind, not because they are less talented, but because they are spending that talent on the wrong layer.
The steel has already been refined. The question is whether you're still running the smelter.