denis256 notes and projects

Recent activity in Prefect, the open source workflow orchestration project, is mostly about fewer surprises in production. The headline change is clearer Kubernetes health behavior, backed by worker changes that move generated environment values into Secrets when operators opt in.

Kubernetes probes got less ambiguous

The maintainers clarified /api/health and /api/ready for self hosted server deployments. This matters because a probe that only checks that the process is alive is not the same as a probe that checks whether the API can use Postgres. When those two are mixed up, Kubernetes keeps routing traffic to a pod that is technically alive but useless for real requests.

The docs change updates both the server Helm guide and the self hosted guide. The useful distinction is simple:

• /api/health checks that the HTTP server is running
• /api/ready checks database connectivity
• readiness should use /api/ready when Postgres reachability matters
• liveness can stay on /api/health if a database outage should not restart the pod

That last point is the sharp edge. If the server does not recover cleanly after Postgres returns, operators may still need a manual restart. Pointing liveness at /api/ready can automate that, but it can also restart pods during short database blips. That is not a Prefect specific problem. It is just Kubernetes doing exactly what it was told.

Worker env vars can move into Secrets

The Kubernetes worker change to store generated env vars in per job Kubernetes Secrets is the most operator relevant code change in the window. When enabled, plain environment values are moved out of the Job manifest and referenced through envFrom.secretRef.

The setting is opt in. The companion docs commit documents the RBAC needed for the setting: the worker needs Secret permissions in the namespace where it creates Jobs. The code creates a Secret, rewrites the Job manifest, and then patches owner references so Kubernetes can garbage collect the Secret with the Job.

env:
  PREFECT_INTEGRATIONS_KUBERNETES_WORKER_STORE_ENV_AS_SECRET: "true"

rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "patch", "delete"]

This does not make environment values magically safe. Anyone with Secret read access can still see them. But it avoids putting generated values directly in Job manifests, which are often easier to inspect through broad Job permissions, events, or debugging habits.

Flow run triggers can target deployments

Automation work also landed. The UI now supports deployment filters for flow run state triggers through the deployment filter change. That means an automation can be scoped to a deployment, not only to flows or tags.

The implementation is not just another select box. The trigger shape now accepts related resource specs for flow, tag, and deployment resources, while still preserving the older single object form when possible. That helps existing automations keep their meaning while the UI gains a more precise target.

For users, the practical result is cleaner alerting and less noisy action wiring:

• page on failures for one deployment without cloning the flow
• keep tag based automations broad when that is the right model
• show deployment targeting in trigger details instead of hiding it in raw JSON

There were several smaller ui-v2 fixes in the same window too. Search on the deployments page no longer gets blocked by an empty state, date inputs allow manual typing and month or year selection, and run logs can link URLs. None of that is glamorous. It is the kind of polish that saves time during an incident, which is when UI paper cuts become expensive.

Smaller fixes with real operator value

The storage fix is worth reading if you use remote file systems for deployment or result storage. Prefect hardened RemoteFileSystem path containment so sibling prefixes, encoded traversal, mixed separators, and mixed case schemes are handled more carefully. The tests added around this are a good sign because path handling bugs tend to hide in boring looking string code.

Deployment creation also got a small but useful CLI fix. The maintainer change to handle module path entrypoints avoids treating every entrypoint as a file plus function pair. If your entrypoint is a module path, Prefect now falls back to the working directory instead of splitting on a colon that is not there.

On the worker side, 401 and 403 worker channel setup failures now keep different meanings. That is small, but it helps separate bad authentication from bad authorization. The same practical theme shows up in the event client change that reconnects from a time based checkpoint when a connection dies with unconfirmed events still buffered.

How to prepare

For self hosted Kubernetes installs, review probe settings before copying the new docs into production. Use /api/ready for readiness. Only point liveness at it if restarts after database failure are preferable to waiting for manual action.

For Kubernetes workers, test PREFECT_INTEGRATIONS_KUBERNETES_WORKER_STORE_ENV_AS_SECRET in a non production namespace first. The RBAC failure mode is predictable: missing Secret permissions will block Job creation, and that is better found before a busy work pool hits it.

Also check recent docs if you use variables. The variables docs now match JSON values and dash support after the variables documentation update. It is a documentation fix, but it can prevent a silly naming argument during configuration work. Those are the worst arguments because everyone is technically right and still wasting time.