Back to Blog
A network of connected nodes representing isolated tenants in a shared AI platform

Multi-Tenant AI Architecture: Isolation by Default

11 min read

A multi tenant AI architecture can authenticate the correct user and still return another customer's document. The leak might happen after authentication, inside a vector query, shared cache, agent checkpoint, tool call, or trace viewer. Fixing the API gateway while leaving those layers tenant-blind does not solve the problem. The practical goal is stricter: one server-verified tenant identity must constrain every read, write, retrieval, action, and diagnostic record for the request. This guide defines that contract, shows where to enforce it, and gives you negative tests that prove the boundary instead of assuming it works.

Why authentication at the edge is not enough

Tenant isolation is a system property, not a login feature. AWS calls tenant isolation foundational to SaaS design because shared infrastructure must still prevent one tenant from reaching another tenant's resources. An AI workflow adds more stateful layers to that ordinary SaaS problem.

Consider a support agent serving two workspaces. The request begins with a valid session for Workspace A. The application then:

  1. loads customer records from PostgreSQL;
  2. retrieves policy documents from a vector store;
  3. restores an agent checkpoint;
  4. checks a response cache;
  5. lets the model choose a CRM tool;
  6. records prompts and tool results in a trace platform.

Each step can lose, replace, or ignore the tenant identity. A database query can omit its predicate. A retrieval client can default to a shared namespace. A cache key can include the prompt but not the tenant. A checkpoint can use a conversation ID that is only unique inside one workspace. A tool can accept tenant_id from model-generated arguments. A trace viewer can expose raw prompts to an operator who should only see one account.

The failure is architectural because no single layer knows whether every other layer applied the boundary. Adding more prompt instructions cannot repair it. The model must not control the identity that limits the model.

Define one isolation contract

Write the isolation contract before choosing pooled or dedicated infrastructure. A useful contract for an AI workflow is:

For every operation, the service derives one tenant identity from authenticated server-side context. Every stateful dependency receives that identity through a typed interface and enforces it independently. User text, retrieved content, model output, and tool arguments cannot create or replace it.

The contract has four consequences.

Tenant context is immutable during one operation. A user may switch workspaces through a new authorized request, but an agent run cannot switch because a prompt or tool argument asks it to.

Enforce the boundary at more than one layer. The application chooses the tenant partition, while the data system rejects an operation outside it. This limits the damage from one missing filter.

A resource identifier is not authorization. Knowing a document ID, thread ID, or namespace name does not prove the caller may use it. Resolve each identifier inside the verified tenant boundary.

Every asynchronous handoff must carry the same context. Queue messages, retries, scheduled continuations, and human approval callbacks need a signed or server-resolved tenant reference. Do not reconstruct identity from free-form payloads.

Microsoft's multitenant AI guidance separates training and inference concerns and requires protection against unauthorized access to other tenants' data or models. That distinction matters when a shared model is acceptable but shared customer context is not.

Carry tenant context outside model control

Represent tenant context as a server-owned type, not another string passed through every function. Construct it after authentication and authorization. Make lower-level clients require it. 

type TenantContext = Readonly<{
  tenantId: string;
  actorId: string;
  requestId: string;
  allowedToolScopes: readonly string[];
}>;

async function runAgent(input: UserInput, ctx: TenantContext) {
  const records = await tenantDb(ctx).loadCase(input.caseId);
  const evidence = await tenantVectorStore(ctx).search(input.question);
  const checkpoint = await tenantState(ctx).load(input.threadId);

  return agent.execute({
    input,
    records,
    evidence,
    checkpoint,
    tools: tenantTools(ctx),
  });
}

The model receives the business context it needs, but it does not receive authority to select another tenant client. Tool schemas should omit tenantId unless the value is informational. The server wrapper injects the verified tenant after validating the requested action.

For background jobs, pass an opaque run reference. The worker resolves that reference to tenant context from a trusted control table. If the job payload contains both a run ID and a tenant ID, reject any mismatch rather than choosing one value.

Enforce relational isolation below the query builder

A pooled database is reasonable for many SaaS workloads, but a mandatory WHERE tenant_id = ? convention is fragile. One missed predicate can expose data. PostgreSQL row-level security can add a database-enforced boundary. The PostgreSQL row security documentation explains that policies can restrict visible and modifiable rows per user, and that enabling row security without a policy creates default-deny behavior.

A simplified policy might read the tenant from a transaction-local setting: 

ALTER TABLE agent_threads ENABLE ROW LEVEL SECURITY;
ALTER TABLE agent_threads FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_threads ON agent_threads
USING (tenant_id = current_setting('app.tenant_id', true)::uuid)
WITH CHECK (tenant_id = current_setting('app.tenant_id', true)::uuid);

At transaction start, the service sets app.tenant_id from TenantContext. Use a database role that cannot bypass row security. PostgreSQL documents that superusers, roles with BYPASSRLS, and usually table owners can bypass policies, so an application connection should not run as any of them.

Row security is not the only option. Separate schemas, databases, or accounts can provide stronger isolation for regulated or unusually large tenants. Pick the boundary per data class and customer requirement. Keep the same application contract so moving one tenant to a silo does not change authorization semantics.

Partition vector retrieval before running the query

Multi-tenant RAG architecture needs a hard retrieval boundary. Filtering by tenant_id inside a shared vector search can work, but it makes isolation depend on every query supplying the correct filter. Prefer a store feature that requires the tenant partition before search when your provider supports it.

Pinecone recommends one namespace per tenant and states that reads and writes target one namespace. Its guidance also notes that separate namespaces reduce the risk of an application bug querying the wrong tenant's data. Weaviate documents multitenancy as a data-isolation feature and requires operations to address a tenant partition.

Wrap the vector client so application code cannot issue an unscoped query. Derive the namespace from trusted tenant context, not a request field. Reject an unknown partition. Avoid automatic tenant creation on read paths because a misspelled tenant name can create a separate, empty partition and hide a routing defect. Weaviate's documentation warns that tenant names are case sensitive, so normalize identifiers before provisioning and store the provider's partition name in a trusted mapping.

Test ingestion and deletion as well as retrieval. A secure query path does not help if the indexing worker writes a document to the wrong namespace or offboarding leaves embeddings behind.

Isolate caches, checkpoints, and model context

Caches often leak because their keys describe content but not ownership. A semantic cache keyed only by an embedding or normalized prompt can return an answer built from another customer's records. Prefix every cache key with tenant identity and the policy version that controls sharing. If any cached value contains customer data, disable cross-tenant deduplication even when prompts appear identical.

Agent checkpoints need composite identity. Use (tenant_id, thread_id) or a tenant-scoped opaque key. Apply the same rule to memory records, pending approvals, retry state, uploaded files, evaluation datasets, and conversation summaries. Durable-memory controls still apply inside each tenant; this article adds the outer boundary that prevents one tenant from reaching another tenant's state.

Do not place another tenant's content in the model context and ask the model to ignore it. Isolation must happen before prompt assembly. The prompt can include a tenant label for observability, but that label is not an access control and should not drive storage selection.

Shared model inference can still be valid. Microsoft describes shared, tenant-specific, and tuned shared model approaches. The decision depends on data sensitivity and whether learning across tenants is acceptable. Document whether customer data trains or tunes a model, and separate that decision from ordinary per-request retrieval.

Bind tools to tenant-scoped authority

An agent tool is a privileged server operation. Its wrapper should authorize the action with tenant context before calling the downstream system. The model may select update_ticket and provide a ticket ID, but the server resolves that ticket inside the current tenant and rejects a foreign object.

Create tool clients from the tenant context. Where possible, mint short-lived downstream credentials scoped to the tenant, resource, and action. If a shared service credential is unavoidable, enforce tenant ownership in the adapter before using it. Log the verified tenant, actor, tool, target, policy decision, and outcome without copying secrets or unnecessary prompt content.

The OWASP agentic threats and mitigations guide frames risks through threat modeling for generative AI connected to autonomous systems and business tools. Apply that approach at each tool boundary: identify what the tool can read or change, then test whether model-controlled input can escape the current tenant.

Human approval does not replace this check. An approver can be shown a misleading label. Resolve the target again inside tenant context when approval is recorded and immediately before execution.

Keep telemetry useful without making it a side channel

Traces, logs, replay screens, and evaluation stores contain some of the most sensitive workflow data. Include a verified tenant identifier as structured metadata, but restrict who can query it. Partition storage or apply tenant-aware access policies to diagnostic systems. Redact secrets and fields that operators do not need.

Do not trust trace filtering in the user interface alone. The API behind the viewer must enforce tenant access. Administrative cross-tenant views should require a separate support role and leave an audit record. If engineers can export traces, apply the same authorization and redaction rules to exports.

Use correlation IDs that are globally unique and tenant-scoped lookups that require both values. A random trace ID makes guessing difficult, but it is not permission to view the trace.

Implement the boundary in a fixed sequence

Start with an inventory. List every component that stores, retrieves, transmits, or displays workflow data. Include queues, object storage, caches, vector indexes, checkpoints, model gateways, tool adapters, traces, analytics, backups, and support consoles.

Apply the controls in this order:

  1. Define the canonical tenant identifier and the trusted mapping from authenticated actor to tenant.
  2. Create an immutable TenantContext and require it in every data, state, retrieval, tool, and telemetry client.
  3. Remove tenant identifiers from model-controlled tool arguments wherever possible.
  4. Add independent enforcement in each dependency, such as row policies, namespaces, scoped credentials, or separate accounts.
  5. Make missing context fail closed. A client without a tenant must reject the operation rather than choosing a default partition.
  6. Add structured audit fields for tenant, actor, run, policy decision, target, and outcome.
  7. Run cross-tenant negative tests before enabling real customer data.

Roll out one workflow at a time. A complete boundary around one path is safer than partial conventions spread across every integration.

Prove isolation with negative tests

Create two test tenants with deliberately recognizable records. For example, Tenant Red owns case RED-ONLY-741, while Tenant Blue owns BLUE-ONLY-286. Use synthetic data, not production copies.

Your test suite should attempt all of these failures:

  • ask the Red agent to retrieve the exact Blue marker through natural language;
  • call the relational repository with Blue's object ID under Red context;
  • query the vector client with a Blue document identifier under Red context;
  • replay a Red prompt that would collide with a Blue semantic-cache entry;
  • load the same thread ID under both tenants and verify different state;
  • submit a tool argument containing Blue's tenant or object identifier;
  • resume a queued job after changing the tenant field in its payload;
  • open Blue's trace or export URL with a Red operator session;
  • delete Red and verify that Red relational rows, vectors, files, caches, and checkpoints are gone without affecting Blue.

Assert denials and empty results, not only the absence of Blue text in the final answer. Check logs to confirm the attempt was attributed to the correct tenant and blocked at the expected layer. Run these tests against every storage mode you support, including any dedicated customer silo.

Mistakes that weaken the boundary

Do not accept tenant_id from the request body just because the user is authenticated. Authentication identifies the actor; authorization still has to resolve which tenant the actor may use.

Mixing scoped and unscoped clients creates another bypass. One raw database or vector client in application code gives future changes a bypass. Keep raw clients inside adapters and expose only methods that require TenantContext.

Random IDs are not an isolation control. UUIDs reduce guessing but do not stop a disclosed ID, a log leak, or an incorrect join. Enforce ownership every time.

Do not test only the happy path. Cross-tenant isolation is proven by operations that should fail. Add those failures to CI and to release checks whenever you change a database policy, cache key, namespace strategy, agent framework, tool schema, or trace provider.

Make one tenant boundary visible this week

Choose one production workflow and draw its tenant path from authentication through storage, retrieval, agent state, tools, and telemetry. Mark every point where the service enforces tenant identity and every point that merely trusts a caller-supplied value. Replace the first trust-only boundary with a typed tenant client and one cross-tenant denial test. Repeat until no data-bearing operation can run without verified tenant context.

References

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the typed tenant context, per-dependency enforcement, and cross-tenant testing described above, at published fixed prices. Schedule a call to discuss your next project.