Check how your AI product performs in every language you support
A system that works well in English can fail in other languages in ways your team cannot see. I test each language and show you where the problems are.
The problem
Quality is not the same across languages. Some are fine. Others give wrong meanings, miss information, or sound wrong to native readers. Most teams do not have speakers of every language on staff, so these problems reach users first.
Translation decisions are also hard to make without data. Should you keep your current machine translation vendor or move to an LLM? Which one for which language and content type?
Who this is for
- Companies launching an AI assistant or feature in several languages.
- Localization teams choosing between machine translation engines, LLMs and human translation.
- Teams comparing vendors and needing a fair test on their own content.
What I do
- Scope. Agree on the languages and the content types that matter most.
- Build test sets per language. Using your real content, so results reflect your use case.
- Compare systems. Run your current setup and the alternatives side by side, with human review and automatic measures.
- Classify errors by severity. A wrong meaning is not the same as awkward wording or a terminology slip. The report separates them.
- Recommend. Which system to use for which language and content, and where human review should stay.
Languages and reviewers
I have evaluated machine translation and LLM output across more than 35 languages. I read and speak English, Farsi and Azerbaijani, and have some Turkish. For other languages I work with native-speaking reviewers, either yours or ones I bring in, and I design the review so their ratings can be compared.
What you get
- A per-language quality report with examples of the failures that matter.
- A comparison of the systems you are considering.
- A recommendation you can take to your team or vendor.
- Optionally, the evaluation setup so you can repeat it when models or vendors change.
Illustrative examples
Invented to show how the work goes. Not real clients. More on the examples page.
Assistant in 10 languages. I test each language and find that three are clearly weaker than the rest. The report shows which failures change the meaning and which are only style, so you know where to spend effort first.
Vendor or LLM? A localization team asks whether to switch from its machine translation vendor to an LLM. I run both on a sample of its own content and report the results by language and content type.
Why me
I worked for eight years on machine translation as an individual contributor, and my PhD is in machine translation and its quality estimation. At Oracle I designed benchmarking frameworks for internal and external translation engines and LLMs.
Common questions
Do you translate? No. I measure and compare translation systems.
Do we need reference translations? Not always. There are methods that judge output without references.
Next step
Tell me your languages, your content and what you are deciding.