Robust Multilingual RAG under Query Perturbations: An English-Persian Benchmark

Retrieval-augmented generation (RAG) is commonly evaluated on clean inputs that underrepresent realistic multilingual variation. We present an English-Persian movie-domain robustness benchmark built from a corpus of ۳۱,۵۶۴ records, ۱۲۰ clean queries, and ۷۲۰ aligned perturbations. The benchmark covers six deterministic query types and ۱۴ operational perturbation labels grouped into four families. We compare BM۲۵, multilingual dense retrieval, character n-gram TF-IDF, and hybrid retrieval, and evaluate top-۱ deterministic answer extraction against a field-specific top-۵ RAG system using Qwen۲-۷B-Instruct. Hybrid retrieval achieves ۸۱.۵۰ MRR@۱۰ on clean queries and ۶۷.۷۶ under perturbation; field-specific RAG reaches ۸۴.۱۷% and ۷۲.۰۸% accuracy, respectively. Clustered paired-bootstrap ۹۵% confidence intervals exclude zero for all principal system differences. English-title noise is the most damaging family, whereas query-form and punctuation variation is comparatively well tolerated. A ۴۳-case consistency audit verifies implementation of the rule-based failure categories, and full-output analysis shows that retrieval-coverage errors dominate the difficult English-title family. These results support component-level evaluation of multilingual RAG robustness.

کلیدواژه ها:

retrieval-augmented generation ، multilingual information retrieval ، robustness evaluation ، English-Persian retrieval ، hybrid retrieval