How to Evaluate Korean-English RAG Search Across Mixed-Language Documents

Evaluate Korean and English queries against Korean, English and mixed-language passages with a language-pair test set, compare relevance at top-k, and trace misses to ingestion or retrieval settings.

How to Evaluate Korean-English RAG Search Across Mixed-Language Documents

Key takeaways

  • Separate the routes: Score Korean-to-Korean, English-to-English, Korean-to-English, English-to-Korean, mixed-query, and mixed-language-passage cases as distinct slices.
  • Check the source before ranking: Verify that the relevant Korean or English text survived parsing and appears with enough context in the indexed passage.
  • Compare settings fairly: Keep the documents, query set, relevance labels, cutoff, and other conditions fixed when testing retrieval settings.
  • Measure language performance: Use representative Korean-English questions and documents, and report the route-specific results.

The short answer: test language directions separately

Evaluate same-language and cross-language retrieval separately, then test code-switched queries and passages that contain both languages. ChemCLIR-Bench separates same-language from cross-language relevance and notes that aggregate metrics can mask cross-lingual failures (ChemCLIR-Bench).

There are at least two different tasks hidden inside “mixed-language search.” A cross-language query asks in one language and needs evidence in another. A code-switched query combines Korean and English, while a mixed-language document or passage itself contains both. These are not interchangeable test cases. Research on mixed-language search reports that results vary across native-language, English, and mixed-language queries; the study is a reason to separate these test slices, not a result for Korean-English Seahorse Cloud search (MiLQ benchmark).

A practical evaluation therefore begins with a labeled set of actual user questions and source passages, and finishes with a route-by-route report. Seahorse Cloud’s official product page describes storage, parsing, semantic chunking, vector synchronization, and managed agents (Seahorse). The rest of this guide turns that distinction into a repeatable test.


Build a Korean-English query and document test matrix

Start by labeling two independent properties for every test: the language or languages used in the query, and the language or languages in the gold passage that contains the answer. This prevents a mixed-language question from being mislabeled as cross-language merely because it contains English terminology, and prevents a bilingual passage from disappearing inside a broad “Korean” or “English” category. Cross-lingual retrieval is specifically a query-document language mismatch; bilingual retrieval datasets define and evaluate such directions separately (CLIRMatrix).

Illustrative examples only; replace these questions and passages with examples from your own corpus.

Query sliceEvidence sliceExample test questionWhat a pass must retrieve
KoreanKorean“문서 보존 기간은 얼마인가요?”The Korean passage that states the retention period.
EnglishEnglish“How long must the records be retained?”The corresponding English passage, if one exists.
KoreanEnglish“이 retention period가 얼마야?”The English source passage that answers the question.
EnglishKorean“What is the 보존 기간 in the policy?”The Korean source passage that answers the question.
Korean-English mixedKorean, English, or both“계약의 renewal 조건이 뭐야?”The passage that answers the renewal question, regardless of its source language.
Korean or EnglishMixed Korean-English passage“Which approval is required before 배포?”The bilingual passage with the needed condition and surrounding context.

Build questions from real search tasks rather than translating every prompt word-for-word. Include native Korean and English phrasing, common code-switching patterns, acronyms, product or team names, numbers, and questions where a technical English term appears in a Korean sentence. Have bilingual reviewers confirm that each mixed-language question preserves the same intent as its monolingual counterpart. The MiLQ study reports that multilingual IR models perform inconsistently across native-language, English, and mixed-language queries; use that finding as a reason to include mixed queries, not as proof about Korean-English performance (MiLQ).

For every query, record one or more known-relevant source passages, the passage language, document version, and whether the source is monolingual or bilingual. Include both directions of cross-language search and query-language mixtures. Where a document has parallel Korean and English versions, keep the relationship explicit rather than treating duplicate versions as unrelated evidence. Keep an unanswerable set too, so a plausible-looking but irrelevant passage is not mistakenly counted as success.


Judge evidence and score retrieval at the intended cutoff

Create a relevance judgment for each query before comparing configurations: name the source document and passage that actually support an answer, and let bilingual reviewers judge borderline translations, paraphrases, and partial matches. A query may have multiple valid passages. Record whether a retrieved passage is sufficient to answer, merely related, or irrelevant; keep the judgment rule consistent across Korean, English, and mixed-language cases.

Choose the cutoff that matches the application’s evidence window, then report metrics separately for every language route. Recall@k is the fraction of known relevant passages that appear in the top k results (Microsoft Learn). nDCG@k discounts relevance scores by result position, applying a logarithmic discount and normalizing against the ideal ranking; higher-ranked relevant results contribute more to the score (scikit-learn; Microsoft Learn). BEIR describes nDCG@k as suitable for binary and graded relevance judgments and emphasizes that no one retrieval approach consistently wins across datasets (BEIR evaluation paper). A domain-specific cross-language study separates same-language from cross-language relevance and reports Recall@10 by route (ChemCLIR-Bench).

A useful result table has one row per query slice and includes query count, number of judged-relevant passages, Recall@k, nDCG@k, and a short list of representative misses. Show Korean-to-Korean and English-to-English alongside Korean-to-English, English-to-Korean, mixed-query, and mixed-passage results. If the sample sizes differ, show the counts and avoid letting a larger easy slice dominate the overall average. Set acceptance thresholds with the owners of the workflow, based on the risk and cost of a missed passage; do not borrow a threshold from an unrelated benchmark.

Public benchmarks can help frame the test but cannot replace it. For example, MIRACL explicitly targets monolingual retrieval, where query and corpus share a language, so its language coverage alone does not establish performance on a cross-language direction or on code-switched queries (MIRACL dataset description). Keep your organization’s own labeled questions and documents as the decision set, and report whether each metric measures retrieval evidence or the quality of the final generated answer.


When a query misses, first locate the failure stage instead of immediately changing the embedding model or ranking settings. Use one source document and one known-relevant passage as a diagnostic case. Check the original file, the text produced by ingestion, the passage or chunk submitted for indexing, and the retrieved results. If the key sentence is absent from extracted or indexed text, record that as an ingestion-stage issue: search indexers extract text and populate the index with it (Microsoft Learn). If the sentence appears in a chunk but is separated from a heading, table label, condition, or neighboring explanation needed to interpret it, check whether that loss of context affects retrieval; Microsoft’s RAG guidance notes that chunks without sufficient context can lead to poor outcomes (Microsoft Learn).

For each example, capture the exact original passage and its page or section, the extracted text, the chunk or indexed record where it appears, and the result list for the query. Check both Korean Hangul and English terms in the source; where the file mixes languages, inspect whether the transition between scripts, punctuation, numbers, or tables changes what remains together.

Seahorse Cloud’s official product description documents an AI-powered Storage pipeline using VLM, OCR, and an LLM-based document parser, along with semantic chunking and automatic vector database synchronization (Seahorse). Its Storage manual describes file upload, automatic embedding, and vector Table creation as part of the Storage workflow (Storage creation guide). Do not turn the presence of a parser or semantic chunking into a language-performance guarantee.

Classify each failure in a short log: source text missing or corrupted; relevant content split from needed context; correct content indexed but absent from results; relevant passage retrieved but ranked too low; or relevant evidence retrieved but mishandled by answer generation. Use the classification to choose what to inspect or change, then rerun the same query and relevance judgment. Rechecking the evidence path is a useful diagnostic procedure for distinguishing ingestion-stage issues from retrieval-stage issues.


Compare retrieval settings without changing the test

Once you have confirmed the relevant text was indexed, compare retrieval settings against the same corpus snapshot, query set, relevance labels, user access conditions, and result cutoff. Start with a baseline and change one setting at a time. Include Korean, English, cross-language, mixed-query, and mixed-passage slices in every run, and report the results separately by language route; ChemCLIR-Bench, for example, reports Recall@10 split by language route (ChemCLIR-Bench).

Seahorse Cloud’s Semantic Search documentation lists Hybrid, Dense, and Sparse as search modes, with Hybrid as the default. It describes Dense as meaning-based search and Sparse as keyword-oriented matching (Semantic Search guide). Test the modes with your own labeled queries. For example, compare an English technical term embedded in a Korean question against relevant passages in Korean, English, and mixed-language form; verify the actual result and rank instead of assuming that semantic similarity or literal term matching will handle it.

The manual documents Top K as the maximum number of returned results for Hybrid, Dense, and Sparse; its Dense description lists Top K and EF Search as related parameters (documented search parameters). Keep the controls recorded in each run, and do not alter multiple controls together if you need to know which change moved a result. Also record the selected Dense and Sparse embedding endpoints: the Storage creation guide lists endpoint selection and says Storage settings cannot be changed after creation (Storage endpoint setup). For any comparison requiring a different endpoint, plan equivalent test corpora and configuration carefully rather than treating an already-created Storage as if its creation settings had changed.

Avoid tuning only to a handful of misses. Use a development set for iteration and keep a separate holdout set for the final comparison. If a new setting is chosen, rerun the entire holdout matrix and retain the previous run’s exact settings, query text, corpus version, and judgments. This makes gains, regressions, and trade-offs auditable.


Decide what the results actually establish

A good result supports a bounded statement: “On this corpus, with this documented configuration and test set, retrieval reached these scores for these query and evidence-language routes.” It does not establish a universal capability for all Korean-English documents, every file type, every domain, or every Seahorse Cloud deployment. A 2026 report should include the test date, corpus and document versions, query counts by route, relevance-review method, result cutoff, retrieval mode, endpoint and search controls, and the route-by-route metrics. Add representative successful and failed examples so readers can see what each score means in practice.

Dnotitia describes Seahorse Cloud as a managed RAG platform with Storage, parsing, semantic chunking, a vector database, and managed agents (Seahorse). These are documented platform capabilities; present Korean-English retrieval outcomes only when they come from a measured evaluation of the relevant workflow.

The practical decision is simple: document what the product materials establish, measure what your own workflow requires, and keep those two forms of evidence separate. If Korean-English performance is a purchase or launch criterion, make the route-specific results part of acceptance testing rather than inferring them from the platform description.


Does hybrid search guarantee better Korean-English retrieval?

No. Seahorse Cloud documents Dense, Sparse, and Hybrid search options, but the documentation cited here does not establish which performs best for Korean-English queries or passages. Compare them on the same labeled test set.

Is a mixed Korean-English query the same test as a Korean query finding an English passage?

No. A mixed query uses both languages in the query; cross-language retrieval has a query and relevant passage in different languages. Include both, and test bilingual passages separately.

Can a multilingual benchmark replace our own test?

Not by itself. Check whether the benchmark tests the same language directions, mixed queries, document types, and domain. For example, MIRACL is designed for same-language query and corpus retrieval, so it does not alone validate Korean-to-English retrieval.

지금 체험

See how these criteria hold up on the real thing.

지금 체험