Back to Blog
A bridge across water representing migrating AI workflows between LLM providers

LLM Provider Migration: Moving Production AI Workflows

10 min read

A new model endpoint can return sensible text and still be unsafe for production. That shallow test misses ignored request fields, changed system-instruction semantics, malformed tool calls, and stored conversations the new API cannot replay. A proper LLM provider migration lets you move traffic without repeating business actions, corrupting workflow state, or leaving customers to find the incompatibilities.

Put the provider behind a contract your application owns. Test its adapter against real workflow history. Keep side effects disabled until tool behavior is proven, move traffic through a canary, and preserve a fast rollback path. The rest of this guide shows how those pieces fit together.

Why an endpoint swap is not a migration

Compatibility at the HTTP or SDK layer does not mean behavioral compatibility. Anthropic's OpenAI SDK compatibility guide says its layer is mainly for testing and comparison, not a long-term production solution for most use cases. It also documents important semantic differences. The compatibility layer ignores strict for function calling, silently ignores several unsupported fields, and combines system and developer messages into one initial system message.

Google exposes an OpenAI-compatible Gemini endpoint, but its documentation also lists current limitations and parameters that the compatibility layer silently ignores. A client library can accept the request while the provider drops a setting your workflow relied on. A successful response only proves that the transport worked.

Abstraction libraries help, but they do not erase provider behavior. LangChain's model interface gives applications a common way to initialize and call models from several providers. The same page notes that supported parameters vary by model and provider, and that integrations expose provider-specific options. Use that common interface to reduce application rewrites, not as evidence that two models will interpret every request identically.

The breakage usually hides in details that a text-response test never exercises:

  • A response schema accepted by one provider is treated as a suggestion by another.
  • A system instruction changes position or gets merged with later developer instructions.
  • A tool with no arguments is represented as an omitted value instead of an empty object.
  • Parallel tool calls arrive in a different order.
  • Stored message history contains provider-specific blocks, identifiers, or metadata.
  • Safety behavior, stop conditions, token accounting, and retry signals change.

One concrete Agno issue about cross-provider history reports a zero-argument Anthropic tool call being persisted without the required empty arguments object. Replaying that history through OpenAI then fails before the model can answer. The incident is narrow, but it exposes the general problem: old conversations are part of the migration surface.

Define the contract your workflow owns

Write down the semantics that must survive the move before cataloging provider request fields. The contract should describe what the workflow promises its callers and tools.

For a support-triage workflow, that contract might require:

  • Every accepted ticket gets exactly one classification result.
  • The result conforms to a locally validated schema.
  • A tool call cannot change a ticket until policy checks pass.
  • The workflow records the model, adapter version, prompt version, and tool outcome.
  • Stored conversations can be replayed or deliberately migrated.
  • A provider timeout cannot cause a completed action to run twice.

Represent messages, tool requests, and results in a canonical internal envelope. Keep raw provider payloads for diagnostics, but do not use them as the durable business record. 

{
  "conversation_id": "case-1842",
  "turn_id": "turn-7",
  "message": {
    "role": "assistant",
    "text": null,
    "tool_requests": [
      {
        "call_id": "call-3",
        "tool_name": "route_ticket",
        "arguments": {},
        "schema_version": "2"
      }
    ]
  },
  "execution": {
    "provider": "candidate",
    "model": "configured-model",
    "adapter_version": "4",
    "side_effect_mode": "dry_run"
  }
}

The canonical record always includes the empty arguments object, local call identifier, and schema version. An adapter may translate them into a provider's native shape, but the persisted record stays stable.

The OpenAI function-calling guide separates tools supplied by the application, tool calls requested by the model, and tool-call outputs returned by the application. Keep those stages separate in your contract. Never persist only a provider's combined response and assume another adapter can reconstruct the boundaries later.

Inventory provider-bound behavior

Build the capability inventory from the production workflow itself. Search code, configuration, prompts, saved traces, and persisted messages for every behavior that crosses the adapter boundary.

The inventory needs to cover these items:

  1. Message roles and ordering rules, including system and developer instructions.
  2. Structured-output mode, schema dialect, strictness, and local validation.
  3. Tool definitions, tool-choice controls, parallel calls, call identifiers, and error responses.
  4. Streaming events and the point at which partial output becomes visible to users.
  5. Token limits, truncation policy, stop reasons, timeout behavior, and rate-limit signals.
  6. Safety settings, refusal handling, data-retention settings, and regional endpoints.
  7. Provider-specific features such as hosted tools, prompt caching, reasoning controls, or file handles.
  8. Durable provider data stored in your database, queue, cache, or object store.

Label each dependency as required, optional, or removable. A required feature blocks migration until the candidate adapter implements an equivalent or the application changes its contract. An optional feature can fall back explicitly. A removable feature should disappear before the migration so it does not complicate two systems at once.

If a required field has no equivalent, reject startup or the affected request with a clear compatibility error. Silently dropping the field can produce plausible output while violating the workflow contract.

Build explicit adapters instead of one leaky wrapper

Each provider adapter translates requests, normalizes responses, validates capabilities, and classifies errors. Business decisions stay outside it.

Request translation maps the canonical message and tool schema to native API fields. Response normalization converts native content blocks, tool calls, usage, stop reasons, and errors back into your internal envelope. Capability validation checks required behavior before traffic reaches the adapter. Error classification maps provider failures into workflow categories such as retryable throttling, retryable outage, invalid request, policy denial, or permanent incompatibility.

A lowest-common-denominator wrapper often removes the features that made the workflow reliable. Keep a portable core and declare optional provider extensions instead. For example, the core can require locally validated JSON and application-executed tools, while one adapter supports hosted retrieval behind a separate capability flag.

Version adapters independently from prompts and workflow code. A migration changes at least three things that engineers often bundle together: provider adapter, model, and provider-side behavior. Pin the model version where the provider allows it. Deploy the adapter change separately. If results degrade, this separation tells you what to roll back.

Test the migration without repeating side effects

A useful test corpus includes normal cases, old conversations, and failures. Build it from sampled production traces after removing sensitive data. Add cases for every tool, each structured-output variant, refusals, long inputs, malformed tool output, timeouts, empty arguments, and conversations created by every active adapter version.

Run deterministic checks before comparing model behavior.

Validate JSON locally. Require known tool names and schema versions. Check that required arguments are present. Confirm that every normalized stop reason and error maps to an allowed internal value. Reject unknown content blocks instead of discarding them.

After those checks pass, compare behavior. The candidate does not need to produce identical prose, but it must preserve the business outcome. For a triage flow, compare the selected queue, urgency band, approval requirement, and tool trajectory. Record disagreements as well as cases where the candidate reaches the same outcome through a different path.

Tool tests must use dry-run implementations. Return realistic results, including errors, but do not send email, update tickets, issue refunds, or write customer records. Log the proposed action and compare it with the baseline. Only deterministic application code should cross from a model request into a real side effect.

Historical replay deserves its own gate. Load stored conversations from each supported schema version, normalize them through the old adapter, then serialize them through the candidate adapter. The Agno failure described earlier would surface here even if every new conversation passed.

Migrate traffic in controlled steps

Once offline replay passes, deploy the candidate adapter with no user traffic and verify startup capability checks. Then mirror eligible requests to it while the current provider remains authoritative. Discard candidate tool actions after recording them. Compare latency, errors, structured-output validity, tool selection, and business outcomes.

Move to a small canary only after the shadow results meet written thresholds. Choose a cohort you can identify and stop. Keep one provider assignment for the life of a conversation unless cross-provider replay has passed, because switching halfway through a thread exercises the hardest compatibility path.

Define pause and rollback rules before the canary starts. Useful hard stops include:

  • Any unauthorized or duplicated tool action.
  • A rise in local schema-validation failures.
  • Unknown response blocks or unmapped stop reasons.
  • Stored conversations that cannot be resumed.
  • Error or latency increases beyond the workflow's agreed budget.

Rollback should change routing, not redeploy the whole application. Keep the old adapter, credentials, configuration, and compatible prompt version available through the observation period. If the candidate wrote any provider-specific durable state, document whether rollback can read it. If not, pin affected conversations to the candidate until completion or migrate that state explicitly.

Example: move a ticket workflow safely

Assume the current workflow classifies a ticket, retrieves account context, and proposes a routing action. A human approves high-risk moves. The team wants a second provider for resilience and commercial leverage.

A risky migration changes the base URL and model name, runs ten example prompts, then sends all production traffic to the new endpoint.

The tested plan starts with the contract. The application owns a TicketDecision schema and validates it locally. It also owns tool authorization and approval. The provider can request get_account_context or route_ticket, but only the tool gateway can execute them.

Engineers export a redacted corpus with routine tickets, empty optional fields, long histories, policy refusals, and past tool errors. They add stored conversations containing zero-argument tool calls. Both adapters process the same corpus in dry-run mode. The test compares the validated decision, proposed tool, normalized arguments, and approval requirement.

During shadow traffic, the candidate's proposed actions remain inert. If it omits an empty arguments object, the adapter inserts one only when the canonical tool schema permits no arguments. It does not invent missing required values. Once disagreement and error thresholds pass, the team canaries new conversations from one internal group. Old conversations remain on their original provider until replay tests prove they can move.

That sequence closes the gap a compatibility endpoint leaves open. It tests the full workflow lifecycle rather than the ability to produce one response.

Verify the migration before removing the old path

Keep the old provider available until the candidate has handled a representative production period and completed in-flight conversations. Then run a final verification:

  • Every required capability has an implemented, tested mapping.
  • Unsupported required fields fail visibly rather than disappearing.
  • Local schema validation runs after every structured response.
  • All tool requests pass authorization and idempotency controls outside the model.
  • Stored conversations from every active version can resume or have an explicit pinning policy.
  • Shadow and canary comparisons meet the written outcome, error, and latency thresholds.
  • Operators can route traffic back without an application deployment.
  • Logs identify provider, model, adapter, prompt, conversation, and tool outcome.
  • Provider-specific durable state has a retention and rollback plan.

Fluent canary answers are not a retirement criterion for the old adapter. Remove it only after the new path has satisfied the same contract under live conditions and the rollback window has closed.

Keep portability as a tested property

Portability drifts as APIs add fields, models change behavior, and applications adopt provider-specific features. Keep the golden corpus in CI, run a smaller replay set when an adapter or model changes, and schedule a full fallback-provider exercise. Until the fallback has replayed current production history, it remains unproven.

Start by exporting twenty redacted traces that include your most consequential tools and oldest supported conversation format. Write the canonical envelope for those traces, then make both the current and candidate adapters pass the same deterministic checks before you compare model quality.

References

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the provider-neutral contracts, replay testing, and controlled canary rollout described above, at published fixed prices. Schedule a call to discuss your next project.