RAG Query Rewriting Without Losing User Intent

RAG query rewriting can rescue a vague follow-up such as “what about contractors?” by restoring the policy, country, and date discussed earlier. It can also quietly corrupt the request by inventing a location, collapsing two questions into one, or issuing so many variants that irrelevant documents crowd out the right evidence. Use a bounded query-planning layer that keeps the original request, chooses a strategy deliberately, validates every generated constraint, and measures retrieval gain separately from intent drift.

This guide defines that layer as a typed, testable contract for internal-document retrieval.

Why raw conversational queries fail

A retrieval index does not receive the context a person assumes. A message such as “Does that apply to contractors?” may depend on the previous turn's subject, jurisdiction, business unit, and effective date. Embedding only the latest message produces a semantically weak vector. Lexical search has even less to work with because the nouns that appear in the source policy are absent.

Compound questions create a different failure. “What is the approval limit, and who can override it for an emergency purchase?” contains at least two retrieval jobs. A single nearest-neighbor search may return the approval table but miss the exception procedure. Increasing top_k often adds loosely related passages rather than ensuring coverage of both clauses.

Vocabulary mismatch causes a third problem. A user asks about “parental leave,” while the policy uses “family leave.” Multiple controlled variants may bridge that gap. The LlamaIndex retriever catalog documents query fusion, decomposition, routing, and recursive retrieval as separate techniques. That separation matters: each technique solves a different failure and carries a different cost.

Asking a model to “improve this search query” for every request creates a new failure path. A fluent rewrite can introduce a country, product, date, or policy interpretation that was not in trusted context. Retrieval then works perfectly for the wrong question.

Define a query plan, not a rewritten string

Treat pre-retrieval processing as a compiler stage. Its input is the current message plus a bounded set of trusted context fields. Its output is a plan that can be inspected before any search runs.

{
  "original_question": "Does that apply to contractors?",
  "resolved_question": "Does the 2026 UK equipment reimbursement policy apply to contractors?",
  "strategy": "contextualize",
  "queries": [
    {
      "text": "2026 UK equipment reimbursement policy contractors eligibility",
      "purpose": "eligibility rule"
    }
  ],
  "constraints": {
    "country": "UK",
    "effective_year": 2026,
    "document_family": "equipment-reimbursement"
  },
  "context_evidence": ["turn:12", "turn:13"],
  "max_results_per_query": 8
}

The plan keeps the raw question because later evaluation needs to compare the generated search intent with what the user actually asked. context_evidence identifies where each restored detail came from. Constraints are structured rather than buried in prose, so policy code can reject unsupported values.

Do not expose the whole conversation to the planner. Pass only the current message, a short conversation summary with provenance, and explicit state such as selected workspace, jurisdiction, document collection, or date. Previous assistant claims are not trusted facts. If a location came only from an earlier generated answer, the planner must not turn it into a retrieval filter.

Choose the least powerful strategy that works

Start with a deterministic decision order. More query generation is not automatically better.

Use the original query when it is already complete

Skip rewriting when the message names the subject and asks one self-contained question. Every rewrite adds latency and another opportunity for semantic drift. The no-rewrite path should be the default, not an optimization added later.

Contextualize references from trusted state

Use contextualization for pronouns, ellipsis, and follow-ups. Restore only entities found in user messages or trusted application state. If two earlier subjects could satisfy “that,” ask for clarification instead of guessing.

A useful validator compares every named entity, date, location, account, and document family in the resolved question against an allowlist extracted from trusted inputs. New values fail the plan. General connective words and harmless synonyms do not need exact matching, but business constraints do.

Decompose questions with independent answers

Split a request when separate clauses can be answered from different passages. Preserve a parent identifier so evidence from each branch can be reassembled without pretending that one source answers the whole request.

For example, divide an emergency-purchase question into:

1. What is the standard approval limit?
2. Who may approve an emergency exception?
3. What record must be retained after the exception?

Cap decomposition at a small number, usually three or four branches. If the planner produces eight subquestions from one sentence, it is likely elaborating rather than decomposing.

Microsoft's agentic retrieval overview describes planning focused subqueries, running them in parallel, and returning structured grounding data. Use that architecture without handing the planner unlimited fan-out. A production contract still needs explicit limits for branches, results, tokens, and elapsed time.

Generate variants only for vocabulary mismatch

Multi-query retrieval is useful when the corpus may express the same concept with different terminology. Generate two or three variants, not a page of paraphrases. Require each variant to preserve the same entities and constraints as the original.

Lexical variants should target corpus vocabulary. “Family leave policy,” “parental leave benefit,” and “new parent time off” may improve recall. Variants such as “best parental leave companies” change intent and should be rejected.

Reserve HyDE for sparse conceptual queries

Hypothetical Document Embeddings, or HyDE, generate a possible answer-like passage and embed that passage for retrieval. The primary HyDE paper reports a zero-shot dense retrieval method based on hypothetical documents. This can help short conceptual questions whose wording differs sharply from the corpus.

HyDE output is a retrieval probe, not evidence. Never show it as an answer, cite it, or let generated facts become metadata filters. Keep the hypothetical text in traces so a bad retrieval can be diagnosed. Avoid HyDE for exact identifiers, legal clauses, numeric limits, or queries where invented specifics could narrow search incorrectly.

Execute branches with hard budgets

Once a plan passes validation, run its branches under one shared budget. A simple controller should enforce:

• at most four generated queries;
• a fixed result count per query;
• one total retrieval deadline;
• one collection and permission scope inherited from trusted identity;
• deduplication by canonical document and passage identity;
• a maximum number of passages passed to reranking;
• provenance from each result back to its originating query.

Use both lexical and vector retrieval when the corpus contains policy terms, identifiers, and natural-language explanations. Fuse rankings rather than comparing incomparable raw scores. Rank fusion belongs after each bounded branch returns. It must not erase which query found a passage.

Permission filters are never planner output. Derive tenant, workspace, role, and document access from the authenticated request. A rewrite may narrow an already allowed collection, but it cannot widen access.

If one branch times out, return a typed partial-plan result. The answer layer should know which subquestion lacks evidence. Silently answering the successful half makes a compound answer look complete when it is not.

Reject plans that change the request

Validation needs more than JSON schema checks. Add semantic invariants that are cheap enough to run on every request.

First, extract protected constraints from trusted input: names, dates, regions, product versions, departments, document families, and negation. Compare them with every planned query. Reject introduced constraints and missing required constraints.

Second, preserve polarity. “Which expenses are not reimbursable?” must not become “reimbursable expenses.” Negation loss is a small text change with a large business effect.

Third, check decomposition coverage. Every branch must map to a clause in the original request, and every material clause must map to at least one branch. This can be represented as character spans or clause IDs instead of relying on another free-form explanation.

Fourth, make uncertainty explicit. If context resolution has two plausible antecedents, the correct plan is needs_clarification. Retrieval should not decide which subject the user meant.

Fifth, compare answers against the original question, not only the resolved query. A system can retrieve passages relevant to its rewrite and still fail the user's task.

Test retrieval gain and intent preservation separately

Build an evaluation set from real question patterns, not polished standalone prompts. Include follow-ups, pronouns, compound requests, internal abbreviations, negation, stale conversational topics, exact policy numbers, and questions that genuinely require clarification.

For each case, store:

• trusted conversational context;
• the original question;
• allowed and forbidden constraints;
• expected strategy;
• relevant passage IDs for each clause;
• whether clarification is required;
• latency and branch budgets.

Run the raw query as a baseline. Then run the planned queries. Measure recall at a fixed result budget, reciprocal rank of the first relevant passage, clause coverage, and whether the final cited answer is supported. Also score intent preservation: added constraints, lost constraints, polarity changes, unsupported decomposition branches, and incorrect resolution of references.

Do not approve a planner because retrieval recall rises in aggregate. A system that broadens every question may retrieve more relevant passages while also adding more distractors and changing high-risk requests. Set release gates by slice. Exact-identifier queries should usually remain untouched. Ambiguous follow-ups should improve without guessed entities. Compound questions should cover every clause within the same total context budget.

Track operational cost as part of quality. Record planner tokens, query count, retrieval latency, reranker load, and passages sent to generation. A two-point recall gain that triples tail latency may not be worth deploying for every request. Route only the classes that benefit.

Handle failures without hiding them

If the planner fails schema validation, fall back to the original query only when it is self-contained. If the message depends on missing context, ask for clarification. Do not turn a planner failure into an ungrounded answer.

If generated branches return no evidence, report which interpretation was searched and offer a narrower clarification. If branches disagree, preserve both evidence sets for reranking or human inspection instead of merging them into one synthetic claim.

Log the plan version, input context references, generated queries, validation outcome, branch timing, retrieved passage IDs, and final citations. Avoid logging sensitive conversation text by default. Stable IDs and redacted fixtures are usually enough for regression analysis.

Roll out by request class. Start with compound questions in shadow mode, compare plans with raw retrieval, then enable execution for a small traffic slice. Add contextual follow-ups only after entity provenance and clarification behavior pass. Keep a kill switch that routes all requests to raw retrieval.

Common implementation mistakes

• Rewriting every query makes complete questions pay latency and drift costs for little gain.
• Evaluating only the final answer hides whether improvement came from better retrieval, a lucky model guess, or answer generation ignoring the evidence. Persist plan and passage-level measurements.
• Treating generated queries as authorization filters lets model text influence access scope. Derive scope from authenticated application state.
• Allowing unlimited fan-out can reduce precision, overload reranking, and create unstable latency. Enforce one shared budget.
• Letting hypothetical text become evidence confuses a retrieval probe with a source. Only retrieved passages may support the answer.
• Using a model-based similarity score as the sole intent check misses high-impact changes that deterministic protected-constraint and polarity checks can catch.

Verify the contract before enabling it

Start with 50 to 100 representative queries and label the expected strategy. Add adversarial cases where a rewrite is tempted to invent a region, effective date, policy owner, or exception. Require zero unauthorized scope changes and zero accepted forbidden constraints.

Then compare raw and planned retrieval under the same passage budget. Inspect wins and losses by strategy. Set a maximum branch count and latency budget from observed tail behavior, not averages. Verify that every answer citation points to a retrieved passage and that each compound clause has evidence or an explicit unresolved state.

Finally, replay the set whenever the planner prompt, model, conversation summarizer, embedding model, corpus vocabulary, or ranking logic changes. Query planning is part of the retrieval system, so it needs the same versioning and regression discipline as chunking and reranking.

The next action is concrete: define the query-plan schema, label 50 real conversational cases, and run them once with raw retrieval and once with the bounded planner. Do not ship rewriting until the planner improves fixed-budget retrieval while passing every intent-preservation check.

References

1. Microsoft Learn agentic retrieval overview: query planning, parallel subqueries, hybrid retrieval, and structured grounding output.
2. LlamaIndex retriever modules: maintained retrieval patterns including fusion, decomposition, routing, and recursive retrieval.
3. Precise Zero-Shot Dense Retrieval without Relevance Labels: primary research introducing Hypothetical Document Embeddings for zero-shot dense retrieval.

---