Last chance (3 days!) to participate in the refreshed WMT evaluation shared task. Pick any subtask and submit an automatic metric that's aligned with human annotations of translation quality across many languages. www2.statmt.org/wmt26/mteval...

Not many people know this but this button was not invented to complain that reviewers didn't give you a higher score.

We're already accepted more than 500 examples from 118 contributors across many many languages (dialects are more than welcome!). We'll be analyzing these, putting them in a rolling-release dataset, and writing a paper about it.

In the Last Translation Benchmark we solve both by: - collecting hard-to-translate inputs (texts, images, audios) - requiring human-readable "verification rules", which enable provable evaluation of future translations Here are some cool examples:

There are many things machine translation still can't do. Help us steer the next direction by contributing hard-to-translate inputs (and be on a cool paper).

The best rebuttal I read is just 4 sentences in 3 bullet points clarifying my listed weaknesses.

they patented η but with the monthyl canonne subscription (25% off this month!) you can use it in your papers too

All my homies left for ACL/ICML and left me home managing a project *for which we're looking for coauthors* 🥸. Join us (also talk to @sethjsa.bsky.social @onadegibert.bsky.social @niyatibafna.bsky.social @patuchen.bsky.social @michellewastl.bsky.social @ayukh.bsky.social)

Be the reviewer you want (or your AC wants) to have. (The 1 goes to the AI written paper. No thank you for wasting 2 hours of my life. Pleasure reviewing the rest.)

github.com/coli-saar/tracl/

Beginning to think that the reviewing disaster in comptuer science is caused by us not even liking to read papers. Some parents pay kids 1$ for each finished book. I propose we give each researcher +0.1 citations for successfully reading a paper.

We are at @eamt2026.bsky.social with a large-scale stealth project called "[redacted] Translation [redacted]". Looking for contributors across all languages to challenge MTs. Talk to @patuchen.bsky.social, @sethjsa.bsky.social and me to be a coauthor!

..and learn some other things as well. Slides (in Typst!) and other materials: github.com/zouharvi/hum...

Learn how to setup human evaluation to classify.. bugs? 🐞

We'll be hosting a tutorial at EAMT (already next week!), KONVENS and MT Marathon on human evaluation. Come learn with us! With support of @maikezufle.bsky.social and @patuchen.bsky.social

is this you?

environmental storytelling

Deadline extension! - The task is simple: get audio + its translation and estimate how good it is. - Mark your name as the winner of the first Speech Translation Metrics Shared Task at IWSLT 2026 🏆 Predictions submission: May 7, 2026 Description paper: May 10, 2026

Come to La Palmaraie (EACL) for the first Multilingual Multicultural Evaluation workshop! 🧐 now. Organized by @pinzhen.bsky.social @hanxuhu.bsky.social @simi97k.bsky.social Wenhao Zhu @bazril.bsky.social Alexandra Birch @afaji.bsky.social Rico Sennrich @sarahooker.bsky.social

Machine translation is tough to evaluate, partly because most of what you throw at is too easy. That doesn't at all mean that translation is solved; we're just not doing a good job finding interesting inputs.

Where do we need to use so much QE? In search. Generating 1000 hypotheses is cheap. Finding the best one isn't. We treat this as a multi-armed bandit where pulling the arm corresponds to getting a more accurate estimate. UCB works much better than pruning with LogProb and running full QE always.

Sometimes the metric already knows the right answer at layer 4. Why compute the rest? We attach the two regressors to every layer and if the self-confidence* is high, we exit early. Faster evaluation. *In this case e^ is the difference from its own final prediction.

We train COMET with additional regressor that predicts its own expected error, simply: L2​(y, y^​)+β⋅L2​(∣y−y^​∣, e^). When it's uncertain, it predicts high absolute error e^. This costs almost nothing and is much faster than MC Dropout.

Quality estimation (automated metrics) are amazing. Truly. We would like to use them everywhere. That gets compute-expensive very quickly. We also don't know when they don't know. In "Early-Exit and Instant Confidence Translation Quality Estimation" (at EACL26) we fix that.

Thanks! Experience reports should be more common. Pearmut was created out of the frustration to set up humeval using existing tools with good defaults. In the paper we have 5 researchers trying to set up humeval using 5 different platforms and reporting on time and ease of use and customizability.

Get started with the following commands: pip install pearmut # Download example campaign wget raw.githubusercontent.com/zouharvi/pea... # Load and start pearmut add esa.json pearmut run

The tool supports multiple annotation protocols of translation and multilingual tasks out of the box: - direct assessment (with custom sliders), - ESA, MQM, - contrastive evaluation, video/audio/image, attention checks, tutorials, statistically sound model comparison, etc.

How often is human evaluation skipped in papers/workflows just because it's too difficult to set up? Yet even small humeval can give so much more signal than automatic metrics. Introducing Pearmut, Human Evaluation of Translation Made Trivial🍐 arxiv.org/pdf/2601.02933

Have you ever wondered how speech translation gets evaluated? Sadly, most speech evaluation downgrades to text-based metrics. Let’s do better! At IWSLT 2026, we’re launching the first-ever ✨Speech Translation Metrics Shared Task ✨!

Dissatisfied with EACL paper decisions? Fret not and submit your paper with ARR reviews to Multilingual Multicultural Evaluation workshop at EACL (both archival or nonarchival) until January 5th. 🔍🙂 multilingual-multicultural-evaluation.github.io