To compare machine translations, give each system the same source text, then judge the results against the same audience and purpose. Do not choose a winner by smoothness alone. A polished sentence can omit a key condition; a less elegant one may preserve the meaning more accurately. The comparison should ask what the reader needs to understand or do.
Set the task before comparing
First identify what the text is for. A product instruction, a public notice, a poem, and a casual message call for different kinds of success. For a safety instruction, preserving the action and its limits matters most. For a poem, sound and image may matter alongside literal meaning. There is no useful score without a defined task.
The United Nations describes translation as rendering source content clearly and accurately into another language. Its discussion of language professionals also emphasizes that translation sits within broader work that supports clarity in public communication. That is a useful baseline: compare what the translation communicates, not only how natural it sounds. [1]
Prepare a fair comparison
Use the same source sentence, language direction, and context for each system. Include nearby sentences when pronouns, references, or tone depend on context. If the text contains a term with a defined meaning, note it before you compare. Otherwise, one output may appear stronger simply because it guessed a different context.
Decide on a short checklist in advance: meaning, completeness, terminology, tone, and readability for the intended reader. Treat each item separately. A translation can be readable but incomplete; it can preserve facts yet sound too formal for a friendly invitation.
Worked example: a small condition with a large effect
Suppose a fictional community notice in English reads: “Bring a reusable cup if you have one; cups will be available.” You are comparing two translations into a language you understand, or with help from a qualified reviewer. Output A reads naturally but its back-translation suggests that everyone must bring a cup. Output B sounds slightly stiff, yet preserves the optional condition and says cups are available.
The first question is not which output sounds more polished in isolation. It is whether either version changes the practical instruction. Mark the source into units: bring a cup; reusable; only if available; cups supplied otherwise. Check that each unit survives in both candidates. Then ask a fluent reviewer to assess naturalness and register. A back-translation can help reveal a possible change, but it cannot prove accuracy; the review must return to the source and target languages.
This example is hypothetical. Its method is reusable: identify the sentence’s important distinctions, then test whether each one remains visible in the translation.
Look for predictable failure points
Check negation, quantities, dates, names, conditions, and who does what to whom. A missing “not” or a changed “unless” can matter more than several awkward phrases. In longer text, look for inconsistent terms and shifts in formality. In a poem, inspect image and rhythm as well as dictionary meaning; a word-for-word result may flatten a deliberate ambiguity.
Also consider the target locale. W3C notes that localization can include adapting dates, time, symbols, and culturally specific references, not only translating text. A technically accurate translation may still need changes to suit the reader’s conventions or context. [2]
Use a decision record, not a hunch
For each candidate, record concrete observations: “keeps the invitation optional,” “omits the alternative,” or “uses an inconsistent term.” Avoid a single unexplained score. If the text is important, have a qualified human reviewer check the source and translation, especially where a mistake could affect rights, safety, money, or access.
A practical next step is to create a two-column checklist for the next comparison: source meaning to preserve, and evidence in each output. Add a note for tone and audience. This makes disagreement specific: reviewers can point to the condition or phrase at issue instead of arguing over which result merely feels better.
