Building Multilingual RAG for Internal Knowledge Bases

A multilingual RAG system can return fluent nonsense. The failure starts in retrieval: a user asks in Japanese, the strongest policy document is in English, lexical search misses it, and the vector model retrieves a weaker Japanese page. The generator then writes a polished answer from poor evidence. To prevent that, treat language as part of the retrieval contract. Record it, route on it, preserve permissions through every branch, and test each query-language and document-language direction separately. This guide gives AI and search engineers an implementation sequence, routing schema, failure rules, and rollout gate for multilingual RAG.

Why multilingual retrieval fails quietly

Monolingual RAG usually assumes that query terms, document terms, analyzers, and embeddings share one language. Multilingual systems break that assumption in several places at once.

Lexical retrieval depends on tokenization, stemming, stop words, and normalization. Azure AI Search exposes different analyzers for different languages because one tokenizer cannot handle every writing system and grammar correctly. Its language support documentation lists language-specific analyzers and capabilities. If French, Japanese, and Arabic content all enter one field with one English analyzer, lexical scores become hard to interpret before an embedding model is involved.

Dense retrieval has a different weakness. A model marketed as multilingual does not guarantee equal retrieval quality for every language or direction. English query to German document may work better than German query to English document. Technical vocabulary, product names, abbreviations, and low-resource languages can widen the gap. The MIRACL benchmark covers 18 languages, which is useful evidence that multilingual retrieval needs multilingual evaluation. It does not prove that a particular internal corpus will inherit the benchmark results.

Translation can recover lexical matches, but it can also alter identifiers, legal terms, quoted phrases, names, or negation. Direct multilingual embeddings avoid a translation hop, yet may underperform for one pair. There is no universal winner. Measure both routes on your own documents and questions.

A final failure occurs after retrieval. The model may answer in the requested language while citing a source that does not support the translated claim. Fluency is not evidence quality. Keep the original query, every transformed query, retrieved spans, source language, and final claim mapping available for inspection.

Define the cross-language contract first

Before choosing models, write down what one retrieval request must preserve. A useful contract contains these fields:

{
  "request_id": "req_123",
  "query_original": "日本の請負業者にもこの規則は適用されますか",
  "query_language": "ja",
  "answer_language": "ja",
  "allowed_source_languages": ["ja", "en"],
  "retrieval_routes": ["direct_dense", "translated_lexical"],
  "principal_id": "employee_42",
  "acl_filter": {"department": "legal", "region": "jp"},
  "entity_constraints": ["contractor", "Japan"],
  "max_candidates_per_route": 20,
  "required_citations": true
}

The language detector may return a ranked set rather than one hard label, especially for short queries containing product names or code. Keep the user-selected interface language as a strong signal. Let users override a wrong detection instead of silently translating the request several times.

The source record also needs explicit language metadata. Store the detected language, declared language from the source system, document version, chunk identifier, parent identifier, ACL attributes, and provenance. Do not infer language again at query time from a short chunk if the document already supplied a reliable locale.

Choose one canonical representation for locale tags, such as BCP 47 tags normalized by your application. Decide how regional variants behave. A team may allow a pt-BR query to search both pt-BR and pt sources while keeping legal documents from Portugal in a separate policy collection. That is a business rule, not a model setting.

Build language-aware indexing

Keep the original text as the evidence of record. Translated copies can improve retrieval, but they should remain derived artifacts linked to the original chunk. Otherwise an answer may cite machine-translated wording as though it came from the source document.

For lexical search, place language variants in fields with appropriate analyzers or in language-specific indexes. The right design depends on corpus size and search infrastructure. A shared index with separate fields makes cross-language fusion easier. Separate indexes can simplify analyzer configuration and residency controls. In both cases, apply the same stable document and ACL identifiers to every variant.

For vector search, select an embedding model that supports the required languages and query-document use case. Cohere's Embed documentation distinguishes search queries from search documents and documents multilingual model options. That distinction belongs in your indexing contract. Using the document input mode for both sides may produce vectors, but it ignores the model's intended asymmetric retrieval behavior.

Open-source teams can evaluate multilingual sentence encoders rather than assuming an English model will transfer. Sentence Transformers describes multilingual knowledge distillation, where translated sentence pairs train a multilingual student to align with a source model. This explains how cross-language alignment can be built. It still leaves your team responsible for testing domain vocabulary and required language pairs.

Version these fields together:

• embedding provider, model, dimension, and input mode
• analyzer and normalization configuration by language
• chunking and parent-expansion rules
• translation provider, prompt or settings, and glossary version
• language detector and confidence policy
• ACL and provenance schema

A change to any one can alter results. Write the version bundle into each indexed record and the retrieval trace so mixed generations are visible during migration.

Route queries using measured rules

Start with two candidate routes for each required language pair: direct multilingual retrieval and controlled query translation. Add more routes only when tests show a gap.

Direct retrieval embeds the original query and searches original-language documents. It avoids translation drift and often handles cross-language semantic matching. Translated lexical retrieval translates the query into one or more source languages, protects named entities through placeholders or a glossary, then runs language-specific lexical search. A third route can translate the query before dense retrieval when direct cross-language embedding quality is weak.

Use a fixed routing matrix rather than asking a generator to improvise. For example:

if query_language == source_language:
    use language_specific_lexical + direct_dense
elif direct_dense_recall_at_10 >= acceptance_threshold:
    use direct_dense + translated_lexical
else:
    use translated_dense + translated_lexical

always:
    apply the same ACL filter before candidate text reaches generation
    keep original and transformed queries in the trace
    reject transformations that alter protected entities or constraints

The threshold must come from evaluation data for that direction. A score measured on English queries against English documents says nothing about Japanese queries against English documents.

Fuse route results by rank unless the scores are calibrated onto a shared scale. Raw BM25 scores, cosine similarity, and reranker scores are not interchangeable. Reciprocal rank fusion is a reasonable baseline because it combines order without pretending that the scores mean the same thing. Deduplicate candidates by stable chunk identity before reranking.

Apply access control inside every retrieval route. Filtering only after fusion creates two risks: unauthorized content can influence reranking, and a route can return too few permitted results after forbidden candidates are removed. The principal and ACL filter should be immutable request inputs passed to lexical, dense, translated, and fallback searches alike.

Preserve entities and meaning during translation

Translation deserves its own validation step. Extract terms that must remain stable, including product names, ticket IDs, policy codes, dates, amounts, quoted phrases, and people. Use an approved glossary for domain terms. Compare protected entities before and after translation, and reject a transformed query when a required item disappears or changes.

Store transformation provenance with each branch. An engineer investigating a bad answer should see that the original Japanese query became a specific English query under glossary version 14, then retrieved three English chunks. Without this trace, the team cannot distinguish translation drift from embedding failure.

Keep answer translation separate from evidence selection. The generator can answer in Japanese from an English source, but citations should point to the original English passage. If your product shows a translated excerpt, label it as a translation and retain a way to inspect the source text. Never let a generated translation replace the canonical citation.

For mixed-language questions, preserve meaningful language switches. A German sentence containing an English API name should not trigger an English-only route. Language detection at document, paragraph, and query level may disagree, so the router should accept mixed or unknown states and fall back to broader retrieval with stricter evidence checks.

Evaluate every language direction

Build the evaluation set as a matrix, not one multilingual bucket. Rows are query languages. Columns are source-document languages. Include same-language retrieval, cross-language retrieval in both directions, and mixed-language queries. Weight the cells by business traffic and risk, but set a minimum floor for every supported pair.

Each example should include the original question, expected supporting document or passage, principal and ACL context, protected entities, acceptable answer language, and an explicit unanswerable label when appropriate. Write questions from real internal vocabulary rather than translating one English test set mechanically. A translated test set can miss the abbreviations and phrasing employees use locally.

Measure retrieval before answer quality. Useful checks include recall at a fixed candidate count, mean reciprocal rank, evidence precision after reranking, ACL violations, and protected-entity preservation. Then measure whether the final answer is supported by the retrieved passages, cites the right source, uses the requested language, and abstains when evidence is absent.

The MIRACL benchmark is a useful external comparison, but internal acceptance depends on your corpus, model version, and query mix. A multilingual model can score well on public passages while missing an internal acronym that appears only in one regional policy set.

Use slice-level release rules. A weighted global score can pass while a low-volume language fails completely. Require every supported high-risk pair to clear its own floor. For lower-risk pairs, define an explicit fallback such as translated search, English-only evidence with a notice, or human escalation.

Handle predictable failures

Language detection will fail on short queries. Use interface locale, conversation history, and user override as additional signals. If confidence remains low, run a bounded set of plausible routes instead of forcing one language.

Translation may be unavailable or slow. Set a route-specific deadline. Direct retrieval can continue if it has passed the acceptance floor for that pair. Otherwise return a clear temporary failure or narrow-language fallback rather than generating from weak evidence.

One retrieval branch may return no permitted documents. Do not drop the ACL filter to recover recall. Record the empty permitted result and try another language route under the same policy. If all routes fail, abstain.

A model or analyzer upgrade can improve common languages and damage a smaller one. Keep the prior version available, replay the full matrix, and canary the new version by language pair. The local multilingual implementation in the DocOps pull request is practitioner evidence that language detection, multilingual embeddings, retrieval, and answer-language handling must be joined in code. Treat it as an implementation example, not a product guarantee.

Watch for index skew. New English documents may appear quickly while translated artifacts lag. Track freshness by source language and derived language. A translated branch should report the source version it represents so stale translations cannot outrank a current original without warning.

Verify the production path

Before rollout, trace one request through detection, routing, filtering, retrieval, fusion, reranking, generation, and citation rendering. Confirm that every stage carries the same principal, source identity, language metadata, and version bundle.

Run negative tests as well as happy paths:

• a Japanese query whose answer exists only in English
• the reverse direction for the same concept
• a query with a product name that translation must preserve
• two users with different ACLs asking the same translated question
• a mixed-language query with an ambiguous detector result
• a source update while a translated artifact is stale
• translation timeout with weak direct retrieval
• an unanswerable question that resembles a known policy

Log route-level latency, candidate counts before and after ACL filtering, retrieval metrics by pair, translation rejection reasons, answer-support results, and abstention rates. Avoid logging raw sensitive queries unless policy allows it. Stable request and source identifiers are enough for many operational joins.

Release one language pair at a time. Compare it with the previous route or a human-supported process, inspect failed queries, and add those failures to the evaluation set. Roll back a pair independently when its support score or ACL tests fail. Do not disable multilingual retrieval for every team because one direction regressed.

Put the contract into practice

Start with the two highest-value language directions in your organization. Collect at least a small set of native questions for each direction, label the supporting passages and ACL context, and benchmark direct multilingual retrieval against controlled translation. Choose the route from those results, then persist the language, transformation, evidence, and version trace for every request.

Your next action is concrete: take 20 representative questions for one query-language and source-language pair, run both routes, and inspect every miss. Do not add that pair to production until retrieval, permission, citation, and abstention checks all pass independently.

References

1. Microsoft Azure AI Search language support supports the use of language-specific analyzers and documents supported language behavior.
2. Cohere Embed documentation supports multilingual embedding selection and separate query and document input types.
3. Sentence Transformers multilingual models explains multilingual knowledge distillation and aligned sentence embeddings.
4. MIRACL multilingual retrieval benchmark supplies primary benchmark evidence across 18 languages.
5. DocOps multilingual semantic RAG pull request provides current practitioner implementation evidence for joining language detection, multilingual embeddings, retrieval, and answer-language control.

---