All posts Engineering

Running SQLMesh on Kubernetes: Jobs, CronJobs, and the State Database

Scotty Pate 10 min read

What SQLMesh Needs From a Kubernetes Cluster

SQLMesh doesn't care what it runs on. What it needs is four things: a state database it can reach over the network, something that invokes sqlmesh run regularly, warehouse credentials, and network egress to the warehouse.

The schedule itself lives in your models. Every SQLMesh model declares its own cron:

MODEL (
  name analytics.orders_hourly,
  kind INCREMENTAL_BY_TIME_RANGE (time_column order_ts),
  cron '@hourly'
);

This means your sqlmesh project has everything needed to run on generic compute without the need for the typical database orchestration layer, i.e. Airflow or Dagster.

Each time sqlmesh run starts, it works out which models are due and evaluates only those. An hourly model runs once an hour and a daily model runs once a day, from the same command. That is the whole scheduler. All you need is something that calls sqlmesh run on a regular tick.

Kubernetes is a good way to provide all of this. It's a good fit if you already run a cluster for other workloads, since you get the tick, secrets, and resource isolation without standing up a separate box just for this. It isn't a requirement. This post assumes you've already decided a cluster makes sense and walks through what running SQLMesh on Kubernetes actually involves.

Plan as a Job, Run as a CronJob

SQLMesh has two commands that matter here, and they map to two different Kubernetes objects. sqlmesh run evaluates every model whose cron is due, then exits. That's a recurring, unattended job with no human in the loop, which is what a CronJob is for. sqlmesh plan computes and applies a diff against an environment. That's something CI triggers deliberately, once, when a change merges, which is a plain Job. Don't reuse one manifest for both.

Here's a working sqlmesh run CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: sqlmesh-run
  namespace: data-platform
spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 1200
      template:
        spec:
          restartPolicy: Never
          automountServiceAccountToken: false
          containers:
          - name: sqlmesh-run
            image: your-registry/your-sqlmesh-project:1.4.2
            command: ["sqlmesh", "--log-to-stdout", "run"]
            envFrom:
            - secretRef:
                name: sqlmesh-warehouse-credentials
            resources:
              requests:
                cpu: "500m"
                memory: "1Gi"
              limits:
                cpu: "1"
                memory: "3Gi"

A few of these fields matter more than they look:

The plan side is a Job your CI creates per deploy. Give it a unique name (a commit SHA suffix works), the same image and Secret as above, and command: ["sqlmesh", "--log-to-stdout", "plan", "prod", "--no-prompts", "--auto-apply"]. Set backoffLimit: 0: a failed auto-apply should fail once and wait for a human. Part 1 covers the CI wiring around it.

Pin what the container runs. Build the image from a tagged release commit, or bake the project files in at build time. Part 1 has a working Dockerfile. config.yaml and the model files come from the image; only the secrets are injected at runtime. plan reads your model files directly, so a half-updated checkout produces a plan for a project that never existed. run doesn't load model files, but it still reads the config file from disk. Pin the image and neither can happen.

Check that it actually worked

  1. Trigger one manually: kubectl create job --from=cronjob/sqlmesh-run sqlmesh-run-manual-1 -n data-platform. This runs the same image, command, and env as the scheduled job.
  2. Watch it: kubectl get jobs -n data-platform, then kubectl logs -n data-platform -l job-name=sqlmesh-run-manual-1 --prefix --tail=-1 to see every attempt, including retries.
  3. Confirm the interval landed, not just that the pod exited 0. Check the model's target table for a row at today's timestamp, or compare row counts before and after.

Keep the State Database Outside the Cluster

Don't run the state database as a StatefulSet, even though Kubernetes makes it easy to. A Postgres pod backed by a PVC has to survive node drains, cluster upgrades, and out-of-disk events, and that's ongoing work you don't need to take on. Use a managed instance instead: RDS, Cloud SQL, whatever your cloud already offers. Point every gateway's state_connection at it, the same way you would outside Kubernetes.

Turn on automated backups on whatever managed service you use, and run a restore at least once so you know it actually works. Part 2 of this series covers what the state database tracks and why losing it without a backup is a real incident.

Credentials and Connectivity

Warehouse credentials go in a Kubernetes Secret. Create it however your pipeline manages secrets:

kubectl create secret generic sqlmesh-warehouse-credentials \
  -n data-platform \
  --from-literal=SNOWFLAKE_PASSWORD=... \
  --from-literal=SQLMESH_STATE_DB_PASSWORD=...

SQLMesh reads connections from config.yaml, so each key here has to match an {{ env_var('NAME') }} reference in your gateway config. Part 1 shows how those are named; use the same names in the command, or use the SQLMESH__GATEWAYS__<GATEWAY>__CONNECTION__<FIELD> override form instead.

BigQuery wants a service account JSON file instead of env vars. Mount it as a file:

volumes:
- name: gcp-credentials
  secret:
    secretName: sqlmesh-gcp-credentials
containers:
- name: sqlmesh-run
  volumeMounts:
  - name: gcp-credentials
    mountPath: /var/secrets/gcp
    readOnly: true
  env:
  - name: GOOGLE_APPLICATION_CREDENTIALS
    value: /var/secrets/gcp/key.json

GOOGLE_APPLICATION_CREDENTIALS works with the default oauth method. If the gateway sets method: service-account, set keyfile to the mounted path instead.

Snowflake key-pair auth works the same way. Mount the private key as a file and set private_key_path in the gateway config to that path.

The other half of credentials is reachability. The pod running sqlmesh run needs a network path to both the state database and the warehouse, through a NAT gateway, a private endpoint, or a peered VPC, depending on where each one lives. If your cluster runs default-deny network policies, reasonable on a multi-tenant cluster, add an explicit egress rule from the SQLMesh namespace to both destinations. Otherwise the job fails on its first connection attempt with nothing more informative than a timeout.

Four Things That Will Bite You

The detailed logs aren't in kubectl logs. By default, SQLMesh writes its log output to a file in the project directory, not to stdout. What lands on stdout without any flags is console output: plan summaries, progress bars, high-level status. That's usually enough, until a run fails for a reason that only shows up in the file. The CronJob manifest above already runs sqlmesh --log-to-stdout run, so kubectl logs gets the detail too. That flag is a group option: it has to come before the subcommand, or SQLMesh errors. --log-file-dir points the file at a mounted volume instead, if you'd rather ship it that way. The file logs at INFO by default; --debug adds more.

backoffLimit retries hit your warehouse. A failed sqlmesh run retrying under backoffLimit usually isn't a correctness risk. run only processes intervals that haven't completed, so a retry recomputes a missing interval instead of duplicating it. The exception is append-only kinds like INCREMENTAL_UNMANAGED, where a retry can insert the same rows twice. What a retry always does, everywhere, is hit your warehouse again for a query that was never going to succeed. That happens N times in a row, on Kubernetes' schedule instead of yours. Keep backoffLimit at 0 or 1, fix the model, and let the next scheduled run pick up the missing interval.

OOMKilled doesn't look like a SQLMesh error. The kernel enforces limits.memory; requests only affects scheduling. A container that crosses the limit is killed with SIGKILL, and SQLMesh logs nothing because the process never gets the chance to. With restartPolicy: Never, the pod doesn't just vanish. It stays around in a Failed state until history limits or node churn clean it up, and kubectl describe pod shows OOMKilled and exit code 137 in the meantime. What's missing is any explanation from SQLMesh itself; from its side, an interval simply didn't get processed. Set limits.memory above the worst run you've seen, and alert on data freshness and missing intervals instead of pod status.

Logs disappear with the pod. successfulJobsHistoryLimit and failedJobsHistoryLimit delete old Jobs and their pods once you're past the limit, which keeps the namespace from filling up with dead objects. Node churn works faster than that limit: autoscaling down, a spot interruption, or routine draining can take a pod's logs before the history limit ever applies. Once the pod is gone, so is anything kubectl logs could have shown you; there's no separate copy unless you made one. Ship container stdout to whatever log aggregation your cluster already runs, a node-level collector like Fluent Bit or Vector, or your cloud's container log service, before you need it.

When One Runner Stops Being Enough

concurrencyPolicy: Forbid means one sqlmesh run at a time. For a single project that is the right call. It starts to hurt as the project grows: every model shares that one run, so a slow daily model can make the hourly models behind it late. Adding a second runner only works if something outside SQLMesh decides which models each runner picks up, and that coordinator is real software to build and maintain. The Parts of SQLMesh That Aren't Production Ready covers why splitting sqlmesh run by hand doesn't get you there.

What It Costs to Keep Running

None of the above is a one-time setup cost. Someone owns it on an ongoing basis. These are rough planning numbers, not measured data, but they're the categories worth budgeting for:

Call it 10-20 hours a quarter, plus on-call.

Skip Running It Yourself

The CronJob and Job manifests, the state database, the credential wiring, and the alerting are what running SQLMesh on Kubernetes yourself actually involves. If you'd rather not own the version upgrades, the backup drills, and the on-call for missed runs, dagctl runs SQLMesh as a hosted service. The state database, scheduled runs, and plan deployment are managed; you connect a git repo and deploy. Pricing is a flat rate by model count, not per seat.

Have feedback or corrections? Reach out directly — we update these posts when we find things that are wrong or outdated.