Back to Blog
A curved library gallery lined with densely packed bookshelves

Preventing RAG Data Poisoning in Internal Knowledge Bases

10 min read

RAG data poisoning starts before the model call. A malicious or compromised document enters a shared drive, wiki, mailbox, or upload queue, gets chunked and embedded, then appears as trusted evidence for an ordinary employee question. The pipeline may be working exactly as designed while it spreads a false fact or hidden instruction. Another system prompt will not fix that. The ingestion path must control who can add evidence, record where every chunk came from, quarantine risky changes, constrain retrieval, and remove poisoned content quickly. The same controls need tests against the real ingestion and retrieval path.

Why a normal RAG pipeline preserves the attack

A typical ingestion job rewards availability and freshness. It discovers files, extracts text, splits the text into chunks, writes embeddings, and advances a sync cursor. Those steps answer whether content can be retrieved. They do not answer whether the content deserves authority.

That distinction matters because document text is an input controlled by somebody outside the RAG runtime. Microsoft classifies malicious instructions inside third-party documents and emails as document attacks, a form of indirect prompt injection that can alter model behavior or cause unintended actions. Its Prompt Shields documentation also makes clear that user prompts and document content are separate attack surfaces.

Poisoning is broader than a visible instruction such as "ignore the policy." A document can falsify a policy date, replace an account number, introduce biased examples, or add language engineered to rank for a target query. OWASP LLM04:2025 includes manipulated embedding data in its definition of data poisoning. The PoisonedRAG paper demonstrates the underlying risk: crafted knowledge documents can corrupt answers produced by a retrieval-augmented system.

Freshness checks do not solve this. A fast incremental index can distribute a poisoned version faster. Parsing checks do not solve it either. A layout-aware parser can preserve the hostile instruction perfectly. Quality evaluation may catch a known attack, but it will not control who can change the corpus tomorrow.

Define the ingestion trust contract

Treat the knowledge base as a security-sensitive data product. Every retrievable chunk should satisfy a small contract before it reaches the active index:

  1. An authorized connector or uploader introduced the source.
  2. The system retained immutable source identity and version provenance.
  3. Extraction and security checks produced a recorded decision.
  4. The source has an explicit trust class and sensitivity class.
  5. Retrieval policy can enforce those classes for the current user and use case.
  6. The source can be revoked without waiting for a full rebuild.

This contract closes the gap left by controls that only scan text. A detector can miss a novel instruction. Provenance, authorization, retrieval constraints, and revocation still limit the blast radius when detection fails.

OWASP LLM08:2025 covers risks around vector and embedding systems, including unauthorized access, cross-context leakage, and manipulated content. Use that guidance as the threat model, then make the six statements above enforceable in your own data path.

Build the control sequence before embedding

Authorize corpus writes

Start with the connector, not the document. Maintain an allowlist of approved source systems, collections, owners, and service identities. A SharePoint site approved for published policies should not imply that every personal folder in the tenant is approved. A support mailbox may be valid evidence for ticket context but invalid authority for HR or finance policy.

For direct uploads, authenticate the uploader and bind the file to an approved collection. Record the uploader, business owner, intended audience, retention rule, and reason for inclusion. Do not let the model choose these values from document text.

Use separate ingestion identities for separate collections. Limit each identity to the folders it needs. If a connector is compromised, collection boundaries should prevent it from rewriting every knowledge domain.

Preserve immutable provenance

Give every source and version a stable identity before parsing. Hash the original bytes, but do not use the hash as the only identifier because legitimate edits change it. Keep both a stable source ID and an immutable version ID.

A minimal record can look like this: 

source_id: policy-handbook
version_id: policy-handbook-2026-08-20-01
source_uri: sharepoint://published-hr/policy-handbook.pdf
connector_id: sharepoint-published-hr
owner: people-operations
content_hash: sha256-of-original-bytes
trust_class: approved-policy
sensitivity: internal
review_state: quarantined
supersedes: policy-handbook-2026-07-10-02

Carry source_id, version_id, content_hash, trust_class, and review_state onto every chunk. Store the original file in an evidence location with tighter write access than the source collection. If an answer becomes suspicious, an investigator must be able to identify the exact bytes, extraction result, policy decision, and index mutation that produced each citation.

NIST's current adversarial machine learning taxonomy provides common language for poisoning and other attacks. Use that vocabulary in security events and runbooks so ingestion incidents are not filed as vague "bad answer" bugs.

Extract into quarantine

New or changed versions should enter a quarantine index, not the active retrieval index. Run ordinary file checks first: type validation, size limits, malware scanning, archive expansion limits, and extraction confidence. Then inspect the extracted representation, including hidden text, document metadata, links, annotations, OCR layers, and unusually repeated query terms.

A prompt-attack classifier can be one signal. Microsoft's document-attack control is a maintained example, but no detector should be your sole admission rule. Combine detector output with source trust, magnitude of change, uploader history, collection sensitivity, and whether the document can influence consequential actions.

Decision rules should be explicit:

  • Auto-admit a small edit from an approved publishing workflow when extraction checks pass and no high-risk signal fires.
  • Require review for a new source, a changed owner, a large semantic change, hidden text, conflicting policy statements, or a detector alert.
  • Reject executable files, unsupported containers, missing provenance, and content from an unauthorized collection.
  • Keep the previous approved version active until its replacement passes admission.

This design prevents a failed review from becoming an outage. It also gives reviewers a bounded diff rather than asking them to reread a whole corpus.

Enforce trust during retrieval

Admission is not the last control. Retrieval queries should filter by the user's authorization, the workflow's purpose, source sensitivity, review state, and trust class before vector or lexical ranking. Do not retrieve an unreviewed upload and ask the model to ignore it. Once content enters context, it can influence the answer even if the final citation is hidden.

Rank trust separately from semantic similarity. A highly similar comment in a writable team folder should not outrank an approved policy for a policy question merely because it repeats the query terms. One practical policy is to select eligible trust classes first, then rank within that set. If no authoritative source is eligible, return an evidence gap instead of silently falling back to lower-trust content.

Keep citations tied to version IDs. The user-facing link may resolve to the current source, but the audit record should preserve the exact version used for the answer.

Separate evidence from action authority

Retrieved text can provide facts. It must not grant permission. A document that says "send this report to an external address" is still document content, not an authorization decision.

Tool calls should pass through policy based on the authenticated user, workflow identity, approved action schema, and current business state. The policy layer may use retrieved facts as parameters after validation, but it should never accept a permission, destination, or credential merely because a retrieved chunk requested it. This boundary limits indirect prompt injection even when a poisoned chunk passes admission.

Work through a realistic example

Consider an internal HR assistant backed by an approved policy library and a broad shared drive. An employee can edit a team document but cannot edit published HR policies. They add a paragraph claiming that expense limits changed and include a hidden instruction telling the assistant to treat that file as the newest policy.

A weak pipeline indexes the edit, retrieves it for "What is the travel expense limit?", and gives the model both the legitimate policy and the poisoned team document. Similarity determines which one wins.

Under the trust contract, the shared-drive connector assigns the document a team-working-copy trust class. The retrieval policy for policy questions permits only approved-policy sources. The modified document stays eligible for team search but cannot answer an authoritative HR question. If the document also triggers hidden-text or instruction signals, its new version remains quarantined while the previous approved version, if any, stays active.

Perfect detection is unlikely. Several independent boundaries should have to fail before this document becomes authoritative.

Plan revocation before an incident

When poisoning is confirmed, deleting the source file is not enough. Copies may remain in chunk tables, vector indexes, keyword indexes, response caches, evaluation datasets, and generated summaries.

Build a revocation operation around version_id. It should:

  1. Mark the version denied for retrieval immediately.
  2. Remove or tombstone all chunks and embeddings derived from it.
  3. Invalidate response caches and generated artifacts that cite it.
  4. Identify answers, users, and tool proposals influenced by the version.
  5. Restore the last approved version when appropriate.
  6. Replay representative queries and confirm the poisoned evidence is absent.

Advance an ingestion cursor only after the active index and provenance record agree. If deletion from one store fails, keep the version denied at query time and retry cleanup. This is safer than allowing partial cleanup to make the source appear healthy.

Verify RAG poisoning prevention with fixtures

Security testing should operate on the complete ingestion and retrieval path, not only on a model prompt. Create a small corpus of poison fixtures and expected decisions. Include:

  • A visible instruction in an otherwise valid policy document.
  • Hidden text or an OCR layer that conflicts with visible content.
  • A writable team document that repeats a target policy query many times.
  • A forged "new policy" uploaded by an unauthorized user.
  • A legitimate large policy revision that requires review but must not be lost.
  • A poisoned version followed by deletion and restoration of the prior version.
  • Two users with different source permissions issuing the same query.
  • A retrieved document that asks the workflow to call a consequential tool.

For each fixture, assert the admission decision, active version, eligible trust classes, retrieved chunk IDs, citations, tool-policy result, security events, and cleanup result. The test passes only when the poisoned version is absent from active retrieval or safely constrained to its intended low-trust use.

Also measure false positives. If reviewers must approve every routine edit, they will create shortcuts or disable the gate. Tune auto-admission around well-controlled publishing paths, not around a promise that one classifier will detect every attack.

Common implementation mistakes

OAuth on a connector proves which workload fetched the file. It does not prove that every person who can edit the source is allowed to publish model authority, so authentication alone is not a corpus admission rule.

Document-level provenance is also insufficient. Retrieval returns chunks, and each chunk needs enough identity to reconstruct its source and version without a fragile join to mutable metadata.

Scan the extracted representation rather than only the raw file text. Hidden layers, OCR output, annotations, and extraction artifacts can differ from what a reviewer sees. The security check should inspect what the chunker will receive.

Trust filters belong before candidate generation where the storage system permits it. Filtering an already selected result can leave too few candidates and may expose restricted text to logging or reranking components.

Citations help a reader inspect evidence, but they are not a security control. They cannot prevent the model from following a malicious instruction or a tool layer from accepting an unauthorized action.

Put the boundary into production

Start with one sensitive collection. Inventory who can write to it, define two or three trust classes, and add immutable source and version metadata before changing model prompts. Route new versions through quarantine, enforce review_state=approved during retrieval, and implement one version-level revocation command.

Then run the fixture suite against the real parser, index, retriever, model, and tool policy. The operating sequence must cover corpus authorization, provenance, quarantine, trust-aware retrieval, action separation, revocation, and proof of cleanup. With those controls in place, the team can handle poisoning as a data incident with a defined recovery path instead of debugging it as an unexplained model answer.

References

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the ingestion trust contracts, quarantine gates, and trust-aware retrieval described above, at published fixed prices. Schedule a call to discuss your next project.