Back to Blog
Aerial view of a container port where stacked shipping containers move through staged handling by gantry cranes

AI Workflow Versioning: Upgrade Without Breaking In-Flight Runs

10 min read

AI workflow versioning becomes a production problem the moment a run survives longer than one deployment. New code may work for every new request while an old approval, retry, or checkpoint resumes into a graph it never started with. The result can be a nondeterminism error, an unreadable state object, a repeated side effect, or a case that simply stops moving.

A safe deployment needs more than prompt versions and canary traffic. Treat workflow code, tool contracts, and persisted state as one compatibility boundary. Classify each change, route active runs deliberately, test old histories and checkpoints, and preserve a rollback target. The result is a release sequence your team can operate and verify.

Why ordinary deployments fail for durable workflows

A stateless API request normally starts and finishes on one deployed version. A durable workflow is different. It may pause for a manager, sleep until a deadline, wait for a webhook, retry a vendor call, or retain an agent thread for weeks. Its next step depends on facts written by older code.

That creates three independent compatibility questions:

  1. Can the new code replay or interpret the old execution history?
  2. Can it deserialize and validate the old persisted state?
  3. Can it safely continue the business process under changed prompts, tools, and policies?

Passing one does not prove the others. A checkpoint can deserialize while its old tool name no longer exists. A replay can succeed while a new prompt changes an approval recommendation. A database migration can complete while an active worker still writes the previous shape.

Temporal's safe deployment guidance explains the first problem clearly: workflow code must remain deterministic against recorded history, and incompatible changes need version pinning or explicit patching. LangGraph persistence establishes the second boundary by storing thread state as checkpoints for resumption, human review, time travel, and recovery. Your application owns the third boundary because only it understands whether a changed decision is acceptable for an in-flight case.

A deployment plan that says only "roll out the new container" has no answer to those questions.

Define the version contract before changing code

Give every durable run enough metadata to identify the contracts that created it. Do not rely on a Git commit alone. One release can contain several workflow types, state migrations, prompts, and tool schemas.

A practical run record can look like this: 

{
  "run_id": "case-8421",
  "workflow_type": "invoice-review",
  "workflow_version": 7,
  "state_schema_version": 4,
  "prompt_bundle_version": 12,
  "tool_contract_version": 5,
  "worker_build": "invoice-worker-2026-08-17",
  "status": "waiting_for_approval"
}

Store these values with the workflow or checkpoint, not only in logs. A resuming worker must be able to inspect them before it reads state or chooses the next node.

Keep the versions separate. A prompt edit is not automatically a state migration. A tool adds an optional argument without changing the graph. A state field can be renamed while the business sequence stays the same. Separate versions let the deployment gate choose the smallest safe response.

The contract also needs an ownership rule: which worker builds may process each workflow version, and which code may read and write each state schema version? If the answer is "the latest worker handles everything," the system has no compatibility policy.

Classify each change before choosing a rollout

Use a change review that forces the author to identify the affected boundary. The following categories are useful starting points.

Change Main risk Default treatment
Logging, metrics, or internal refactor Replay behavior changes accidentally Replay old histories, then deploy normally
New optional state field with a default Older checkpoints omit the field Read old and new shapes, write the new shape
Renamed or retyped state field Old checkpoints cannot satisfy the new schema Add an explicit state migration
Added, removed, or reordered workflow branch Old history no longer matches code Pin old runs or add a version gate
Tool argument or result contract change A resumed run calls an incompatible tool Support both contracts or migrate at a safe boundary
Prompt or model policy change In-flight decisions use mixed behavior Pin the decision stage or record the policy version
Business rule change Old cases may require a different legal or operational path Get an explicit product decision before migration

Microsoft describes the core issue in its Durable Functions versioning guidance: orchestration changes can break running instances, so deployments need a strategy for old and new versions. The exact mechanisms differ by platform, but the decision comes first. Determine what changed before selecting pinning, patching, migration, restart, or compensation.

A code review checklist should require the workflow version impact, state schema impact, oldest supported version, migration function, replay fixtures, rollout mode, and rollback limit. This turns compatibility from release folklore into an inspectable artifact.

Choose pin, patch, migrate, restart, or compensate

Select an upgrade strategy for each workflow type and change class. Different runs may require different treatment in the same release.

Pin short-lived runs to their starting build

Pinning is the cleanest choice when executions finish before retaining old workers becomes expensive. New runs go to the new build. Existing runs finish on the build where they started.

Temporal Worker Versioning documents this model directly, including pinned workflows, traffic ramping, rollback, and different choices for workflows that span deployments. Its decision guide specifically includes AI agents and chatbots that can remain active for weeks.

Pinning avoids cross-version replay and mixed business behavior, but it creates an operational obligation. The old build must remain available until its assigned runs drain. Track active runs by build, age of the oldest run, and reasons runs are not finishing. Do not delete an old deployment because the new one looks healthy.

Patch auto-upgrading runs at recorded boundaries

Use an explicit version gate when a run must move to new code while preserving its old path through a changed branch. The workflow records which branch it took, and replay follows that recorded choice.

A patch is appropriate for durable execution logic, not as a general feature flag. Keep the old branch until every history that can reference it has completed or continued into a new run. Removing the branch early turns a successful rollout into a delayed failure.

Migrate persisted state independently

State migration should be a deterministic function from one known schema to the next. It must not call a model, read a mutable external service, or depend on the current clock. Those inputs make the same checkpoint produce different migration results. 

def migrate_state(raw_state, from_version, target_version):
    state = raw_state.copy()
    version = from_version

    while version < target_version:
        if version == 2:
            state["approval"] = {
                "status": state.pop("approved", False) and "approved" or "pending",
                "actor_id": None,
            }
        elif version == 3:
            state.setdefault("tool_contract_version", 5)
        else:
            raise UnsupportedStateVersion(version)
        version += 1

    validate_state(state, target_version)
    return state

Make the write atomic: read the old version, migrate, validate, and update only if the stored version has not changed. Keep the original checkpoint or an immutable migration record until the rollback window closes.

LangGraph issue 6902 reports a mixed checkpoint migration state where a version row exists but required schema objects are absent. Setup can appear successful and defer failure until reads or writes. Verify schema invariants rather than trusting a migration number alone.

Restart only when replay is unnecessary and side effects are known

Some runs can be restarted from normalized business data instead of migrated. That is safe only if you can identify completed side effects and prevent them from running again. Create a replacement run with a link to the original, carry forward approved facts, and explicitly mark which steps must not repeat.

Do not delete the old run. Keep it as evidence for support, audit, and compensation.

Compensate when the old action cannot be continued safely

A breaking business rule may make continuation invalid. In that case, stop the run, reverse reversible side effects, route irreversible ones to an operator, and start a new process under the new rule. Compensation is a business decision, not a generic retry.

Build the deployment sequence

Once each change has a treatment, use a release sequence that separates proof from exposure.

  1. Inventory active runs by workflow version, state schema, status, age, and assigned worker build.
  2. Add readers that accept the oldest supported state before any writer emits a new shape.
  3. Deploy migration code and verify it against copied checkpoints without changing production records.
  4. Replay representative histories with the candidate workflow code.
  5. Start the new worker build with no production traffic or only designated test runs.
  6. Route a small share of new starts to the candidate. Keep old runs on their declared path.
  7. Compare completion, replay, migration, tool, approval, and compensation outcomes by version.
  8. Increase new-start traffic only while the compatibility checks remain healthy.
  9. Drain, migrate, or explicitly terminate runs assigned to old builds.
  10. Remove old code only after negative verification shows no history, checkpoint, or active run still requires it.

Replay testing deserves its own release gate. Temporal recommends replaying existing event histories against candidate code before and during deployment. Sample histories by workflow type and branch, not only by recency. Include rare approval paths, retries, timeout handling, compensation, and runs created by the oldest supported version.

Checkpoint tests need the same diversity. Maintain fixtures for every supported schema version. Load each fixture, migrate it, validate it, resume at the next node, and confirm the expected tool and business action. A migration unit test that only checks field names is not enough.

Handle failures without creating a second incident

Stop rollout when the candidate cannot replay an old history, rejects a supported checkpoint, changes a completed side effect, or sends an active run to an unsupported tool contract. Those are compatibility failures, even if aggregate error rates remain low.

Rollback has two separate meanings. Traffic rollback sends new starts to the previous build. State rollback restores data changed by migrations. The first is fast. The second may be impossible after the new code writes fields that old code cannot interpret. Define the state rollback point before migration and block old workers from consuming new state unless they are forward compatible.

Keep migrations monotonic where possible. New code should read both old and new shapes during the rollout, while only new code writes the new shape. After old workers drain, remove old-shape writes. Remove old-shape reads in a later release after an inventory query proves none remain.

Do not let automatic retries hammer a checkpoint that fails deterministic migration. Quarantine the run with its original state, migration error, source version, target version, and worker build. An operator can then choose repair, restart, or compensation without losing evidence.

Verify the upgrade contract

The absence of alerts does not prove compatibility. Check these properties directly:

  • Every active run maps to an available worker or an approved migration plan.
  • Every supported history replays on its assigned target build.
  • Every supported checkpoint migrates deterministically and validates afterward.
  • Replaying or resuming does not repeat completed side effects.
  • New starts use only the intended workflow, prompt, and tool versions.
  • Old workers cannot consume state they do not understand.
  • Rollback routing works before production traffic is ramped.
  • No old build is removed while a run, history, or checkpoint still requires it.

Run the negative cases too. Remove a required checkpoint table in a disposable environment and confirm startup fails closed. Present an unknown schema version and confirm the worker quarantines it. Break a replay fixture and confirm deployment stops. Simulate rollback after some records have migrated and verify the old worker refuses incompatible state rather than guessing.

Platform documentation usually treats code replay, checkpoint migration, prompt behavior, tool contracts, rollout, and rollback as separate concerns. The inventory, classification table, strategy choice, and release sequence above combine them into one deployment contract for an AI workflow.

Take the next action

Choose one persistent workflow before the next release. Export its active-run inventory, list every code and state version still present, and add one old history plus one old checkpoint to CI. Classify the next planned change with the table above. Do not deploy it until the team can name the path for every in-flight run: pin, patch, migrate, restart, or compensate.

References

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the version contracts, checkpoint migrations, and replay gates described above, at published fixed prices. Schedule a call to discuss your next project.