Your Translation Quality Metrics Aren't Working. Here's Why
A lot of clients try to track translation quality using metrics like SAE J2450 or MQM. That's fine, but it isn't enough.
That's because all of these traditional quality metrics focus on linguistic errors. They try to find instances where something is linguistically wrong with the translation.
And if there's nothing wrong, that means it's a good translation, right? Well... here's where I beg to differ.
Formal Metrics vs. Real-World Objectives
You might be old enough to remember the time when the ISO 900x standards for quality management were introduced. A common joke back then was that it was perfectly possible to have an ISO 9001-certified facility that manufactured lifebelts made of concrete.
What's true about that joke is that you can have a perfectly managed and controlled process that outputs garbage, all the time, every time. And similarly, it's possible to create translations that will pass any common metric, but completely miss the point. (I've seen too many of them.)
I'm not saying that you shouldn't use traditional linguistic quality metrics. On the contrary, they're fine for weeding out really bad translators and adding a certain reliability to your process. In practice, though, they are insufficient for two reasons.
Traditional Metrics Often Aren't Used Right
First, in a lot of business contexts, the usefulness of a translation heavily depends on things like audience appropriateness, tone, clarity, meaning and subtext. While traditional metrics do consider those aspects, it's time-consuming for reviewers to rate them objectively. What's more, discussions on those issues tend to be rather lengthy, which is something that neither reviewers nor translation vendors want.
Thus, reviewers often pay less attention to those areas and focus on “easier” aspects instead, like spelling, grammar, or style guide adherence. But those often aren't the things that make or break your business outcome.
Your Business Objectives Go Beyond Linguistic Quality
The main reason, though, is this: When you buy translations, those translations must contribute towards your business objectives. And those business objectives usually don't include “passing linguistic quality metrics.” You're not buying translations just because you want error-free texts. You buy them to enhance customer satisfaction, to get more people to use your services, to drive sales on your website, to raise awareness for your cause in foreign countries... you name it. You're trying to achieve a business outcome that likely goes beyond publishing error-free texts.
Don't Focus on Errors, Focus on Success
You buy translations because you assume they will help you achieve your business objectives. And I very much suggest that you find a way to verify that assumption. That is: to measure how well your translations support your business objectives. You could call this “translation success.”
In theory, translation success is easy to measure. A translation is good when it supports your business objectives at least as well as the original source-language content. To verify this, you define your business objectives and associated measures, and then you compare the outcome across different languages. And if your metrics come up worse for the translation than for the source language, you know that there's a problem with the translation.
Reality Is Messy
The tricky part is: This only holds true if all other things are equal. In practice, they often aren't. For example, local competition in a foreign market might be so strong that winning customers will be much harder than in your home market. In that case, your performance there might be weaker even though translation quality is perfectly fine.
Scientists have similar problems all the time. They set up experiments to observe cause-and-effect relations—you do something, and something happens as a result. But often, other factors will intervene and distort their results. That's why they design their experiments very carefully. They want to be sure that any outcomes they observe are really due to what they're checking in the experiment, and not to hidden outside factors or pure chance.
Think Like a Scientist
Try to follow that example when you set up measures for translation success. Make sure that external conditions are as similar as possible between languages, and that content quality is the only remaining factor to explain any variance. If that isn't possible, consider adjusting your objectives. For example, if you can't reach the same market share as in your home market due to an overwhelmingly strong local competitor, set your targets lower.
Sample Metrics
Here are some examples of metrics you could consider, along with caveats for each:
| Line of Business | Metric | Caveats |
|---|---|---|
| Sales & Marketing | Click-through rates for ads | Are your audiences similar enough? Is there enough demand in the target market? |
| Sales & Marketing | Conversion rates | Do you offer the same service levels across markets? Is your product as attractive to foreign customers as it is to your domestic customers? |
| Sales & Marketing | Market share (percentage) | Are there overwhelmingly strong local competitors? Are your sales efforts similar enough across markets? |
| Online Training | Test pass rates | Are participants similar enough in terms of previous qualification/experience? |
| Service & Support | Customer satisfaction with documentation | Are your audiences similar enough in terms of industry, application requirements, previous qualification/experience etc.? |
Get Started With Your Own Success Metrics
With 25+ years of hands-on experience, our consultants can help you define success metrics that work for you. Send us a message or request a free consultation. By the way, our consultations come with our unique $100 guarantee: If we're wasting your time, we'll give you a $100 gift card or make a $100 donation to a cause that matters to you. (Terms and conditions apply.) Thanks for reading, and see you again next time!
Get Your Free Consultation
