EXPLANATIONS


Engines

I spent some time with a two web search engines and was able to locate 17 translation websites in addition to 4 I already knew. In fact, I found many more, but I discarded those that would not translate into Estonian or displayed a notice like "powered by Google Translate" or "translations provided by Microsoft", which obviously means they are mere clones of Google or Bing, repsectively.

Three of the sites I found were what I call meta-translators. It means they accessed several translation engines. I have listed them as Gemini (via AiTranslator), Libre (via AiTranslator) etc., AiTranslator being the site into which you enter your text and Gemini and Libre being engines accessed by it. That's why the rankings include, for example, four instances of Google. Google means Google Translate accessed from its own site, Google (via Sider) means Google Translate accessed through the Sider metatranslator's site, etc. Bing and Deepl are represented in the test only through metatranslators, because Bing's own website wouldn't let me copy the translated text and Deepl's site didn't work properly with my browser.

So, all in all the test included 42 translation-producing units which I shall refer to as "translation engines" or "engines" for short.

I included only free translators in the test. I don't mind paying for machine translations, but not before I've even had a chance to find out if they're any good.

I discarded Yandex Translator. It has always had the annoying habit of pestering the user with captchas. This time, it hit me with a captcha the very first time I wanted to use it after a break of half a year, possibly longer. On top of that, the captcha was impossible to solve. So I left Yandex out of the test together. Who do they think they are? (And their translator is crap anyway.)


Texts

I chose 16 texts, each between 600 and 800 characters long (because some translation sites have a 1000-character limit), 8 in English, 8 in German, for each language 4 non-fiction texts and 4 fiction texts. Some texts were from the works that I had previously translated for money. The choice of other texts was arbitrary and indeed quite random.


Test

I let every engine translate all 16 texts into Estonian and copied the translations into a Word file. That is with the exception of one engine that began demanding money after 12 translations, and one that did not offer German-to-Estonian translation at all.

Then I assigned each of the 600 (42 * 16 – 4 – 8) translations a random 6-digit code.

Then I printed all the translations and cut the paper so that each slip contained one translation along with the 6-digit code and the number of the text translated, but not the name of the engine.

Then I went through all the translations of the text #1 and graded each one from 0 to 5 without knowing which engine had made which translation. Then I did the same with the text #2, etc. Being a native Estonian speaker and a professional translator, I am perfectly qualified to tell how good or bad one or another translation is.

Having graded all 600 translations, I used the 6-digit codes to connect each translation with its translator and typed the scores into an Excel table.


Meaning of scores

The scores are relative rather than absolute. A score 5/5 means the translation was remarkably good compared to most of the others, not that it was 100% correct. Usually the best translations were about 60–70% correct, rarely more, sometimes less. Correspondingly, 0/5 sometimes means incomprehensible gibberish, but sometimes it merely indicates that the translation stuck out as remarkably bad in comparison to the others.

In other words, I did not define objective scoring rules beforehand. That's because I had no idea how good or bad translations I should expect. What I did was to compare the translations to each other and divide them into six heaps (not equally sized) with the best ones getting 5 points, the next best ones getting 4 points etc.


Multiple instance of the same translator

I was somewhat curious to find out whether or not it makes any difference if you access the same translator from different sites. I found out it does.

Several translators are featured in both AiTranslator and Machine. Sometimes they returned exactly the same translations, letter by letter, but sometimes there were slight differences (with the notable exception of ChatGPT that seems to be thinking completely anew each time). Curiously enough, the translations by Machine versions tended to be worse than translations by AiTranslator versions.

Most remarkably, Google when accessed through Sider returned results far worse compared to Google accessed through its own site or the other two metatranslators. Let us look at but two examples that make the discrepancy strikingly clear.

The English sentence
The thief didn't expect to be interrupted.
is correctly translated as
Varas ei oodanud segamist.

Google (via itself) translated it as
Varas ei oodanud, et teda segatakse.
(The thief didn't expect that he will be interrupted.)
which is correct.

Google (via Sider) translated it as
Varas ei lootnud katkestada.
(The thief didn't hope to interrupt.)
which is obviously wrong.

The German phrase
vermummte Wingerts-Wächter
(masked vineyard-guards)
is correctly translated as
maskeeritud viinamäevahid (or viinamarjaistanduse- or -valvurid)

Google (via itself) translated it as
maskides viinamarjaistanduste valvurid
(guards of masked vineyards)
Even though separate writing makes it look like the vineyards were masked and not the guards, I was most impressed that Google was able to guess that Wingert meant "vineyard" which I had a tough time finding out.

Google (via Sider) translated it as
kapuutsiga ääriste valvurid
(guards of hooded edges)
which makes no sense whatsoever.

What caused such differences? Your guess is as good as mine.


Impressions

I saw several translations where the engine seemed to really have an intuitive understanding of the Estonian language. By this I mean mistakes that didn't look like they stemmed from online texts the engine may have seen. Rather they looked as if the translation engine was actually thinking in Estonian, making a mistake a child would make – one that makes sense to our language instinct, even though it's not considered grammatical. It was eerie.



back