Welcome to OCELoT!

OCELoT stands for Open, Competitive Evaluation Leaderboard of Translations. This project started as part of the Fifth Machine Translation Marathon in the Americas, hosted at UMD, College Park, MD, from May 28–June 1, 2019. Project OCELoT aims to create an open platform for competitive evaluation of machine translation output, based on both automatic metrics and human evalation. Code is available from GitHub and shared under an open license.

From June 22nd to June 29th, OCELoT will be used to collect submissions to the Shared Task: Machine Translation of News which is part of the EMNLP 2020 Fifth Conference on Machine Translation (WMT20), replacing the previously used matrix which had grown stale over time. You can read more about this year's shared task and changes compared to previous years in the competition updates section. We're looking forward to your participation in WMT20!

From July 10th to July 17th, OCELoT will collect submissions to the Shared Task: Machine Translation Robustness.

Download test sets Register your team Create submission Competition updates

Leaderboard

robustness20-set1 test set (de-en)

# Name SacreBLEU score chrF score Date
1 Anonymous submission #1683 43.9 0.667 July 21, 2020, 7:56 a.m.
2 Anonymous submission #1707 43.5 0.667 July 21, 2020, 9:45 a.m.
3 Anonymous submission #1693 43.4 0.666 July 21, 2020, 8:22 a.m.
4 Anonymous submission #1689 43.3 0.667 July 21, 2020, 8:05 a.m.
5 Anonymous submission #1666 42.8 0.662 July 17, 2020, 1:01 p.m.
6 Anonymous submission #1730 42.7 0.668 July 21, 2020, 11:39 a.m.
7 Anonymous submission #1701 42.1 0.656 July 21, 2020, 8:43 a.m.
8 Anonymous submission #1708 41.2 0.652 July 21, 2020, 9:45 a.m.
9 Anonymous submission #1670 41.1 0.646 July 18, 2020, 5:10 a.m.
10 Anonymous submission #1671 40.9 0.644 July 20, 2020, 2:13 a.m.
Systems in bold face are your submissions. We only display the top-10 submissions per language pair. SGML validation errors denoted by -1.0 score.

robustness20-set1 test set (en-de)

# Name SacreBLEU score chrF score Date
1 Anonymous submission #1682 48.0 0.684 July 21, 2020, 7:54 a.m.
2 Anonymous submission #1669 42.9 0.654 July 17, 2020, 1:26 p.m.
3 Anonymous submission #1692 42.2 0.632 July 21, 2020, 8:21 a.m.
4 Anonymous submission #1709 42.1 0.633 July 21, 2020, 9:46 a.m.
5 Anonymous submission #1660 41.9 0.633 July 17, 2020, 10:28 a.m.
6 Anonymous submission #1699 41.9 0.632 July 21, 2020, 8:28 a.m.
7 Anonymous submission #1688 41.4 0.636 July 21, 2020, 8:05 a.m.
8 Anonymous submission #1705 41.4 0.626 July 21, 2020, 9:43 a.m.
9 Anonymous submission #1700 40.7 0.622 July 21, 2020, 8:42 a.m.
10 Anonymous submission #1664 40.2 0.633 July 17, 2020, 12:16 p.m.
Systems in bold face are your submissions. We only display the top-10 submissions per language pair. SGML validation errors denoted by -1.0 score.

robustness20-set1 test set (en-ja)

# Name SacreBLEU score chrF score Date
1 Anonymous submission #1684 37.6 0.335 July 21, 2020, 7:58 a.m.
2 Anonymous submission #1710 35.3 0.317 July 21, 2020, 9:47 a.m.
3 Anonymous submission #1696 33.3 0.305 July 21, 2020, 8:25 a.m.
4 Anonymous submission #1731 31.9 0.276 July 21, 2020, 11:39 a.m.
5 Anonymous submission #1674 30.5 0.267 July 20, 2020, 6:49 a.m.
6 Anonymous submission #1681 27.8 0.249 July 21, 2020, 7:09 a.m.
Systems in bold face are your submissions. We only display the top-10 submissions per language pair. SGML validation errors denoted by -1.0 score.

robustness20-set1 test set (ja-en)

# Name SacreBLEU score chrF score Date
1 Anonymous submission #1676 26.6 0.513 July 20, 2020, 2:03 p.m.
2 Anonymous submission #1672 25.6 0.525 July 20, 2020, 2:19 a.m.
3 Anonymous submission #1715 25.4 0.526 July 21, 2020, 10:07 a.m.
4 Anonymous submission #1706 25.2 0.504 July 21, 2020, 9:44 a.m.
5 Anonymous submission #1661 24.7 0.493 July 17, 2020, 11:31 a.m.
6 Anonymous submission #1695 24.5 0.506 July 21, 2020, 8:24 a.m.
7 Anonymous submission #1729 23.5 0.487 July 21, 2020, 11:39 a.m.
Systems in bold face are your submissions. We only display the top-10 submissions per language pair. SGML validation errors denoted by -1.0 score.

robustness20-set2 test set (en-ja)

# Name SacreBLEU score chrF score Date
1 Anonymous submission #1686 29.2 0.254 July 21, 2020, 8 a.m.
2 Anonymous submission #1675 28.7 0.249 July 20, 2020, 7:52 a.m.
3 Anonymous submission #1716 28.4 0.244 July 21, 2020, 10:20 a.m.
4 Anonymous submission #1717 28.4 0.244 July 21, 2020, 10:20 a.m.
5 Anonymous submission #1713 28.3 0.250 July 21, 2020, 9:53 a.m.
6 Anonymous submission #1698 25.6 0.228 July 21, 2020, 8:26 a.m.
7 Anonymous submission #1733 23.4 0.207 July 21, 2020, 11:40 a.m.
Systems in bold face are your submissions. We only display the top-10 submissions per language pair. SGML validation errors denoted by -1.0 score.

robustness20-set2 test set (ja-en)

# Name SacreBLEU score chrF score Date
1 Anonymous submission #1703 15.2 0.401 July 21, 2020, 9:38 a.m.
2 Anonymous submission #1685 14.3 0.393 July 21, 2020, 7:59 a.m.
3 Anonymous submission #1663 13.9 0.388 July 17, 2020, 11:34 a.m.
4 Anonymous submission #1712 13.6 0.386 July 21, 2020, 9:49 a.m.
5 Anonymous submission #1662 13.6 0.388 July 17, 2020, 11:33 a.m.
6 Anonymous submission #1697 13.3 0.379 July 21, 2020, 8:26 a.m.
7 Anonymous submission #1732 9.4 0.311 July 21, 2020, 11:40 a.m.
Systems in bold face are your submissions. We only display the top-10 submissions per language pair. SGML validation errors denoted by -1.0 score.

robustness20-set3 test set (de-en)

# Name SacreBLEU score chrF score Date
1 Anonymous submission #1711 44.7 0.653 July 21, 2020, 9:47 a.m.
2 Anonymous submission #1687 44.3 0.653 July 21, 2020, 8:02 a.m.
3 Anonymous submission #1668 44.0 0.649 July 17, 2020, 1:04 p.m.
4 Anonymous submission #1694 44.0 0.649 July 21, 2020, 8:23 a.m.
5 Anonymous submission #1724 43.8 0.649 July 21, 2020, 11:17 a.m.
6 Anonymous submission #1723 43.8 0.648 July 21, 2020, 11:16 a.m.
7 Anonymous submission #1702 43.4 0.645 July 21, 2020, 8:44 a.m.
8 Anonymous submission #1734 43.3 0.650 July 21, 2020, 11:40 a.m.
9 Anonymous submission #1659 43.3 0.655 July 17, 2020, 3:59 a.m.
10 Anonymous submission #1673 43.3 0.653 July 20, 2020, 2:37 a.m.
Systems in bold face are your submissions. We only display the top-10 submissions per language pair. SGML validation errors denoted by -1.0 score.