champollion 0.3.3

package/shared/catalogue/metric-reliability.json

@@ -0,0 +1 @@
+{"_comment":"MACHINE-DERIVED metric meta-evaluation index (master plan B3): correlations between automatic MT metrics and WMT human judgments (DA/MQM/ESA, wmt19-wmt25), per (test set, language pair, human lane, level, metric), rolled up per TARGET-language family. All correlation values are champollion-derived; the underlying human judgments, metric scores and system outputs belong to the WMT Metrics shared tasks and are NEVER tracked here — fetch-from-source per the usage block in arena/scripts/import_wmt_metaeval.py (which regenerates this file). RELIABILITY evidence, not quality scores: a cell says how well a metric tracked humans on that pair, never how good any system is. NOT bundled into the npm package or the PyPI wheel.","version":1,"provider":"wmt-metrics-metaeval","sources":[{"name":"mt-metrics-eval v2 data (WMT Metrics shared-task archive)","repo":"https://github.com/google-research/mt-metrics-eval","commit":"68a481aea1a787392d55af8f76ac8ef40c5f4664","data_url":"https://data.statmt.org/wmt26/mt-metrics-eval-v2.tgz","data_sha256":"6708eec9aaa8a9deca5e370bdd0e23db4881aeca129f29d5281575eaa66c7e10","data_etag":"36579a46-64ff8bb1d3080","data_last_modified":"Tue, 21 Apr 2026 13:52:50 GMT","data_bytes":911710790,"retrieved":"2026-07-07","license":"Apache-2.0 (mt-metrics-eval toolkit code). The bundled human-judgment + test-set DATA carries no explicit upstream license statement — founder review pending; non-commercial hold (see license_lane).","credit":"WMT Metrics shared-task organizers and annotators (Markus Freitag, Nitika Mathur, Tom Kocmi et al.) and the google-research/mt-metrics-eval maintainers. Human judgments were produced by the WMT campaigns; all we add is correlation arithmetic.","note":"The tarball is NOT immutable (no versioned releases): data_sha256/data_etag/data_last_modified pin the copy these numbers were computed from. The upstream README's storage.googleapis.com URL is stale (403); data.statmt.org is the live host."},{"name":"google/wmt-mqm-human-evaluation","repo":"https://github.com/google/wmt-mqm-human-evaluation","commit":"7fadea281aff8db61cc72fc3d469a4dca78b82a6","retrieved":"2026-07-07","license":"Apache-2.0","credit":"Google MQM annotation releases — Freitag et al., \"Experts, Errors, and Context\" (TACL 2021) and the successive WMT MQM releases.","note":"Upstream origin of the MQM expert ratings redistributed inside mt-metrics-eval; every cell with human='mqm' traces to this project (high-resource pairs only). Pinned for provenance + license lineage of the mqm lanes."}],"provenance":"champollion-derived [derived from google-research/mt-metrics-eval + google/wmt-mqm-human-evaluation] — every correlation, roll-up and selection here is computed by Champollion from the pinned upstream data; no upstream text, judgment or score is redistributed","license_lane":{"commercial_ok":false,"note":"The mt-metrics-eval DATA license is not explicitly stated upstream (the Apache-2.0 grant covers the toolkit code). Until the founder license review clears it, this artifact stays out of commercial/prize lanes; the mqm lanes' underlying annotations are Apache-2.0 via google/wmt-mqm-human-evaluation."},"score_orientation":"All scores are used exactly as distributed by mt-metrics-eval, which stores every human and metric score as higher-is-better (MQM and MetricX are stored negated upstream). No sign flipping is applied; verified empirically (MetricX-23 vs COMET sys-level Pearson +0.97 on wmt23 en-de).","correlation_definitions":{"sys":"Per-system scores, aligned on systems that have both a human and a metric score, references/human translations excluded (names matching the pair's reference set or prefixed ref/human/synthetic). pearson/spearman/kendall_tau_b are the standard coefficients; pairwise_accuracy is Kocmi-style system-ranking accuracy (share of strictly-human-ordered system pairs the metric orders identically; metric ties count as disagreement). Cells need >= 3 aligned systems.","seg":"Flattened (system, segment) vector: every segment with both a human and a metric segment score, all systems pooled, no grouping (mt-metrics-eval 'none' averaging). Cells need >= 10 aligned points across >= 2 systems. kendall_tau_b is the tie-adjusted tau-b on the flattened vector."},"human_lanes":{"preference":["mqm","esa-merged","esa","da-sqm","wmt-z","wmt-appraise-z","wmt-appraise","wmt","wmt-raw"],"note":"Per (testset, pair, level) every available allowlisted human lane gets cells; exactly the best-available lane is flagged preferred=true, and ONLY preferred cells enter the family roll-up (no double counting). Per-rater files, span-level *.rating files and doc/domain levels are out of scope."},"metrics":{"bleu":{"registry_id":"bleu","upstream_candidates":["BLEU","sentBLEU"],"upstream_names_used":["BLEU","sentBLEU"],"note":"Scores verbatim from the WMT metric submissions/baselines named in upstream_names_used (version-pin doctrine: the exact upstream name is on every cell)."},"spbleu":{"registry_id":"spbleu","upstream_candidates":["f200spBLEU","spBLEU"],"upstream_names_used":["f200spBLEU","spBLEU"],"note":"Scores verbatim from the WMT metric submissions/baselines named in upstream_names_used (version-pin doctrine: the exact upstream name is on every cell)."},"chrf_plain":{"registry_id":"chrf_plain","upstream_candidates":["chrF"],"upstream_names_used":["chrF"],"note":"Scores verbatim from the WMT metric submissions/baselines named in upstream_names_used (version-pin doctrine: the exact upstream name is on every cell)."},"chrf_plus_plus":{"registry_id":"chrf_plus_plus","upstream_candidates":["chrF++"],"upstream_names_used":["chrF++ (computed, sacrebleu word_order=2)","chrF++"],"note":"Computed lane: chrF++ is only bundled upstream for wmt20; cells with computed=true were scored by Champollion with sacrebleu CHRF(word_order=2) against the recorded reference stream."},"comet_score":{"registry_id":"comet_score","upstream_candidates":["COMET-22","COMET22","COMET-DA_2021","COMET-DA_2020","COMET"],"upstream_names_used":["COMET","COMET-DA_2021","COMET-22","COMET22"],"note":"Scores verbatim from the WMT metric submissions/baselines named in upstream_names_used (version-pin doctrine: the exact upstream name is on every cell)."},"metricx_score":{"registry_id":"metricx_score","upstream_candidates":["MetricX-24-Hybrid","MetricX-24","MetricX-23","MetricX-25-Ref","MetricX-25","metricx_xxl_MQM_2020"],"upstream_names_used":["metricx_xxl_MQM_2020","MetricX-23","MetricX-24-Hybrid","MetricX-25-Ref","MetricX-25"],"note":"Scores verbatim from the WMT metric submissions/baselines named in upstream_names_used (version-pin doctrine: the exact upstream name is on every cell)."}},"languages":{"ar":{"iso639_3":"arb","family":"Afro-Asiatic","genus":"Arabic"},"bho":{"iso639_3":"bho","family":"Indo-European","genus":"Bhojpuric"},"bn":{"iso639_3":"ben","family":"Indo-European","genus":"Gauda-Banga"},"cs":{"iso639_3":"ces","family":"Indo-European","genus":"Czech-Slovak"},"de":{"iso639_3":"deu","family":"Indo-European","genus":"Global German"},"en":{"iso639_3":"eng","family":"Indo-European","genus":"Germanic"},"es":{"iso639_3":"spa","family":"Indo-European","genus":"Castilic"},"et":{"iso639_3":"est","family":"Uralic","genus":"Finnic"},"fi":{"iso639_3":"fin","family":"Uralic","genus":"Nuclear Finnish"},"fr":{"iso639_3":"fra","family":"Indo-European","genus":"Global French"},"gu":{"iso639_3":"guj","family":"Indo-European","genus":"Indic"},"ha":{"iso639_3":"hau","family":"Afro-Asiatic","genus":"West Chadic"},"he":{"iso639_3":"heb","family":"Afro-Asiatic","genus":"Hebrewic"},"hi":{"iso639_3":"hin","family":"Indo-European","genus":"Hindustani"},"hr":{"iso639_3":"hrv","family":"Indo-European","genus":"Western South Slavic"},"is":{"iso639_3":"isl","family":"Indo-European","genus":"Icelandic"},"it":{"iso639_3":"ita","family":"Indo-European","genus":"Central Italian Romance"},"iu":{"iso639_3":"iku","family":"Eskimo-Aleut","genus":"Inuit"},"ja":{"iso639_3":"jpn","family":"Japonic","genus":"Japan-Taiwan Japanese"},"kk":{"iso639_3":"kaz","family":"Turkic","genus":"Kipchak"},"km":{"iso639_3":"khm","family":"Austroasiatic","genus":"Khmeric"},"ko":{"iso639_3":"kor","family":"Koreanic","genus":null},"liv":{"iso639_3":"liv","family":"Uralic","genus":"Coastal Finnic"},"lt":{"iso639_3":"lit","family":"Indo-European","genus":"Baltic"},"mas":{"iso639_3":"mas","family":"Nilotic","genus":"Nuclear Maa"},"pl":{"iso639_3":"pol","family":"Indo-European","genus":"Polish-Silesian"},"ps":{"iso639_3":"pus","family":"Indo-European","genus":"Nuclear Pashto"},"ru":{"iso639_3":"rus","family":"Indo-European","genus":"East Slavic"},"sah":{"iso639_3":"sah","family":"Turkic","genus":"Sakha-Dolgan"},"sr":{"iso639_3":"srp","family":"Indo-European","genus":"Western South Slavic"},"ta":{"iso639_3":"tam","family":"Dravidian","genus":"Tamil-Kannada"},"uk":{"iso639_3":"ukr","family":"Indo-European","genus":"Ukrainian-Rusyn"},"xh":{"iso639_3":"xho","family":"Niger-Congo","genus":"Bantu"},"zh":{"iso639_3":"cmn","family":"Sino-Tibetan","genus":"Mandarinic"},"zu":{"iso639_3":"zul","family":"Niger-Congo","genus":"Bantu"}},"family_rollup_note":"families is keyed by the TARGET language's family (cli/data/champollion.db `languages.family`) — metric reliability is about scoring output in the target language. Means are weighted by n_sys (sys level) / n_points (seg level). Beware: all xx->en pairs land in Indo-European; per-pair cells carry src/tgt for finer slicing (genus is in languages).","families":{"Afro-Asiatic":{"n_pairs":2,"metrics":{"bleu":{"seg":{"n_cells":2,"n_pairs":2,"weight":14394,"pairs":["wmt21.news:en-ha","wmt25:en-ar"],"pearson_weighted_mean":0.2459,"spearman_weighted_mean":0.2582,"kendall_tau_b_weighted_mean":0.1819},"sys":{"n_cells":2,"n_pairs":2,"weight":32,"pairs":["wmt21.news:en-ha","wmt25:en-ar"],"pearson_weighted_mean":0.8815,"spearman_weighted_mean":0.8094,"kendall_tau_b_weighted_mean":0.6239,"pairwise_accuracy_weighted_mean":0.8119}},"chrf_plain":{"seg":{"n_cells":2,"n_pairs":2,"weight":14394,"pairs":["wmt21.news:en-ha","wmt25:en-ar"],"pearson_weighted_mean":0.3062,"spearman_weighted_mean":0.2888,"kendall_tau_b_weighted_mean":0.2024},"sys":{"n_cells":2,"n_pairs":2,"weight":32,"pairs":["wmt21.news:en-ha","wmt25:en-ar"],"pearson_weighted_mean":0.8464,"spearman_weighted_mean":0.8187,"kendall_tau_b_weighted_mean":0.6221,"pairwise_accuracy_weighted_mean":0.7986}},"chrf_plus_plus":{"seg":{"n_cells":2,"n_pairs":2,"weight":14394,"pairs":["wmt21.news:en-ha","wmt25:en-ar"],"pearson_weighted_mean":0.3198,"spearman_weighted_mean":0.3027,"kendall_tau_b_weighted_mean":0.2131},"sys":{"n_cells":2,"n_pairs":2,"weight":32,"pairs":["wmt21.news:en-ha","wmt25:en-ar"],"pearson_weighted_mean":0.8652,"spearman_weighted_mean":0.8322,"kendall_tau_b_weighted_mean":0.6578,"pairwise_accuracy_weighted_mean":0.8289}},"comet_score":{"seg":{"n_cells":2,"n_pairs":2,"weight":14394,"pairs":["wmt21.news:en-ha","wmt25:en-ar"],"pearson_weighted_mean":0.3533,"spearman_weighted_mean":0.334,"kendall_tau_b_weighted_mean":0.2325},"sys":{"n_cells":2,"n_pairs":2,"weight":32,"pairs":["wmt21.news:en-ha","wmt25:en-ar"],"pearson_weighted_mean":0.669,"spearman_weighted_mean":0.6799,"kendall_tau_b_weighted_mean":0.517,"pairwise_accuracy_weighted_mean":0.7585}},"metricx_score":{"seg":{"n_cells":1,"n_pairs":1,"weight":3582,"pairs":["wmt25:en-ar"],"pearson_weighted_mean":-0.318,"spearman_weighted_mean":-0.3545,"kendall_tau_b_weighted_mean":-0.2543},"sys":{"n_cells":1,"n_pairs":1,"weight":18,"pairs":["wmt25:en-ar"],"pearson_weighted_mean":-0.6242,"spearman_weighted_mean":-0.612,"kendall_tau_b_weighted_mean":-0.4248,"pairwise_accuracy_weighted_mean":0.2876}},"spbleu":{"seg":{"n_cells":1,"n_pairs":1,"weight":3582,"pairs":["wmt25:en-ar"],"pearson_weighted_mean":0.524,"spearman_weighted_mean":0.5318,"kendall_tau_b_weighted_mean":0.3971},"sys":{"n_cells":1,"n_pairs":1,"weight":18,"pairs":["wmt25:en-ar"],"pearson_weighted_mean":0.9508,"spearman_weighted_mean":0.8452,"kendall_tau_b_weighted_mean":0.6863,"pairwise_accuracy_weighted_mean":0.8431}}}},"Dravidian":{"n_pairs":1,"metrics":{"bleu":{"seg":{"n_cells":1,"n_pairs":1,"weight":7891,"pairs":["wmt20:en-ta"],"pearson_weighted_mean":0.2685,"spearman_weighted_mean":0.3479,"kendall_tau_b_weighted_mean":0.2361},"sys":{"n_cells":1,"n_pairs":1,"weight":15,"pairs":["wmt20:en-ta"],"pearson_weighted_mean":0.8801,"spearman_weighted_mean":0.9286,"kendall_tau_b_weighted_mean":0.8286,"pairwise_accuracy_weighted_mean":0.9143}},"chrf_plain":{"seg":{"n_cells":1,"n_pairs":1,"weight":7891,"pairs":["wmt20:en-ta"],"pearson_weighted_mean":0.4883,"spearman_weighted_mean":0.458,"kendall_tau_b_weighted_mean":0.3192},"sys":{"n_cells":1,"n_pairs":1,"weight":15,"pairs":["wmt20:en-ta"],"pearson_weighted_mean":0.9367,"spearman_weighted_mean":0.9357,"kendall_tau_b_weighted_mean":0.8095,"pairwise_accuracy_weighted_mean":0.9048}},"chrf_plus_plus":{"seg":{"n_cells":1,"n_pairs":1,"weight":7891,"pairs":["wmt20:en-ta"],"pearson_weighted_mean":0.4791,"spearman_weighted_mean":0.4539,"kendall_tau_b_weighted_mean":0.316},"sys":{"n_cells":1,"n_pairs":1,"weight":15,"pairs":["wmt20:en-ta"],"pearson_weighted_mean":0.9288,"spearman_weighted_mean":0.9393,"kendall_tau_b_weighted_mean":0.8286,"pairwise_accuracy_weighted_mean":0.9143}},"comet_score":{"seg":{"n_cells":1,"n_pairs":1,"weight":7891,"pairs":["wmt20:en-ta"],"pearson_weighted_mean":0.6601,"spearman_weighted_mean":0.6338,"kendall_tau_b_weighted_mean":0.4531},"sys":{"n_cells":1,"n_pairs":1,"weight":15,"pairs":["wmt20:en-ta"],"pearson_weighted_mean":0.9443,"spearman_weighted_mean":0.9036,"kendall_tau_b_weighted_mean":0.7714,"pairwise_accuracy_weighted_mean":0.8857}}}},"Eskimo-Aleut":{"n_pairs":1,"metrics":{"bleu":{"seg":{"n_cells":1,"n_pairs":1,"weight":9981,"pairs":["wmt20:en-iu"],"pearson_weighted_mean":0.1833,"spearman_weighted_mean":0.238,"kendall_tau_b_weighted_mean":0.1615},"sys":{"n_cells":1,"n_pairs":1,"weight":11,"pairs":["wmt20:en-iu"],"pearson_weighted_mean":0.1629,"spearman_weighted_mean":0.3636,"kendall_tau_b_weighted_mean":0.2364,"pairwise_accuracy_weighted_mean":0.6182}},"chrf_plain":{"seg":{"n_cells":1,"n_pairs":1,"weight":9981,"pairs":["wmt20:en-iu"],"pearson_weighted_mean":0.3631,"spearman_weighted_mean":0.382,"kendall_tau_b_weighted_mean":0.2651},"sys":{"n_cells":1,"n_pairs":1,"weight":11,"pairs":["wmt20:en-iu"],"pearson_weighted_mean":0.3499,"spearman_weighted_mean":0.4182,"kendall_tau_b_weighted_mean":0.3455,"pairwise_accuracy_weighted_mean":0.6727}},"chrf_plus_plus":{"seg":{"n_cells":1,"n_pairs":1,"weight":9981,"pairs":["wmt20:en-iu"],"pearson_weighted_mean":0.3383,"spearman_weighted_mean":0.3747,"kendall_tau_b_weighted_mean":0.2588},"sys":{"n_cells":1,"n_pairs":1,"weight":11,"pairs":["wmt20:en-iu"],"pearson_weighted_mean":0.328,"spearman_weighted_mean":0.4,"kendall_tau_b_weighted_mean":0.3091,"pairwise_accuracy_weighted_mean":0.6545}},"comet_score":{"seg":{"n_cells":1,"n_pairs":1,"weight":9981,"pairs":["wmt20:en-iu"],"pearson_weighted_mean":0.3082,"spearman_weighted_mean":0.3043,"kendall_tau_b_weighted_mean":0.2111},"sys":{"n_cells":1,"n_pairs":1,"weight":11,"pairs":["wmt20:en-iu"],"pearson_weighted_mean":0.8598,"spearman_weighted_mean":0.4636,"kendall_tau_b_weighted_mean":0.3818,"pairwise_accuracy_weighted_mean":0.6909}}}},"Indo-European":{"n_pairs":42,"metrics":{"bleu":{"seg":{"n_cells":84,"n_pairs":84,"weight":947636,"pairs":["wmt19:de-cs","wmt19:de-en","wmt19:de-fr","wmt19:en-cs","wmt19:en-de","wmt19:en-gu","wmt19:en-lt","wmt19:en-ru","wmt19:fi-en","wmt19:fr-de","wmt19:gu-en","wmt19:kk-en","wmt19:lt-en","wmt19:ru-en","wmt19:zh-en","wmt20:cs-en","wmt20:de-en","wmt20:en-cs","wmt20:en-de","wmt20:en-pl","wmt20:en-ru","wmt20:iu-en","wmt20:ja-en","wmt20:km-en","wmt20:pl-en","wmt20:ps-en","wmt20:ru-en","wmt20:ta-en","wmt20:zh-en","wmt21.flores:bn-hi","wmt21.flores:hi-bn","wmt21.news:cs-en","wmt21.news:de-en","wmt21.news:de-fr","wmt21.news:en-cs","wmt21.news:en-de","wmt21.news:en-is","wmt21.news:en-ru","wmt21.news:fr-de","wmt21.news:ha-en","wmt21.news:is-en","wmt21.news:ja-en","wmt21.news:ru-en","wmt21.news:zh-en","wmt21.tedtalks:en-de","wmt21.tedtalks:en-ru","wmt21.tedtalks:zh-en","wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:ru-en","wmt22:sah-ru","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.1604,"spearman_weighted_mean":0.1633,"kendall_tau_b_weighted_mean":0.1135},"sys":{"n_cells":84,"n_pairs":84,"weight":1042,"pairs":["wmt19:de-cs","wmt19:de-en","wmt19:de-fr","wmt19:en-cs","wmt19:en-de","wmt19:en-gu","wmt19:en-lt","wmt19:en-ru","wmt19:fi-en","wmt19:fr-de","wmt19:gu-en","wmt19:kk-en","wmt19:lt-en","wmt19:ru-en","wmt19:zh-en","wmt20:cs-en","wmt20:de-en","wmt20:en-cs","wmt20:en-de","wmt20:en-pl","wmt20:en-ru","wmt20:iu-en","wmt20:ja-en","wmt20:km-en","wmt20:pl-en","wmt20:ps-en","wmt20:ru-en","wmt20:ta-en","wmt20:zh-en","wmt21.flores:bn-hi","wmt21.flores:hi-bn","wmt21.news:cs-en","wmt21.news:de-en","wmt21.news:de-fr","wmt21.news:en-cs","wmt21.news:en-de","wmt21.news:en-is","wmt21.news:en-ru","wmt21.news:fr-de","wmt21.news:ha-en","wmt21.news:is-en","wmt21.news:ja-en","wmt21.news:ru-en","wmt21.news:zh-en","wmt21.tedtalks:en-de","wmt21.tedtalks:en-ru","wmt21.tedtalks:zh-en","wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:liv-en","wmt22:ru-en","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.753,"spearman_weighted_mean":0.6977,"kendall_tau_b_weighted_mean":0.5713,"pairwise_accuracy_weighted_mean":0.7854}},"chrf_plain":{"seg":{"n_cells":84,"n_pairs":84,"weight":947636,"pairs":["wmt19:de-cs","wmt19:de-en","wmt19:de-fr","wmt19:en-cs","wmt19:en-de","wmt19:en-gu","wmt19:en-lt","wmt19:en-ru","wmt19:fi-en","wmt19:fr-de","wmt19:gu-en","wmt19:kk-en","wmt19:lt-en","wmt19:ru-en","wmt19:zh-en","wmt20:cs-en","wmt20:de-en","wmt20:en-cs","wmt20:en-de","wmt20:en-pl","wmt20:en-ru","wmt20:iu-en","wmt20:ja-en","wmt20:km-en","wmt20:pl-en","wmt20:ps-en","wmt20:ru-en","wmt20:ta-en","wmt20:zh-en","wmt21.flores:bn-hi","wmt21.flores:hi-bn","wmt21.news:cs-en","wmt21.news:de-en","wmt21.news:de-fr","wmt21.news:en-cs","wmt21.news:en-de","wmt21.news:en-is","wmt21.news:en-ru","wmt21.news:fr-de","wmt21.news:ha-en","wmt21.news:is-en","wmt21.news:ja-en","wmt21.news:ru-en","wmt21.news:zh-en","wmt21.tedtalks:en-de","wmt21.tedtalks:en-ru","wmt21.tedtalks:zh-en","wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:ru-en","wmt22:sah-ru","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.2559,"spearman_weighted_mean":0.2466,"kendall_tau_b_weighted_mean":0.172},"sys":{"n_cells":84,"n_pairs":84,"weight":1042,"pairs":["wmt19:de-cs","wmt19:de-en","wmt19:de-fr","wmt19:en-cs","wmt19:en-de","wmt19:en-gu","wmt19:en-lt","wmt19:en-ru","wmt19:fi-en","wmt19:fr-de","wmt19:gu-en","wmt19:kk-en","wmt19:lt-en","wmt19:ru-en","wmt19:zh-en","wmt20:cs-en","wmt20:de-en","wmt20:en-cs","wmt20:en-de","wmt20:en-pl","wmt20:en-ru","wmt20:iu-en","wmt20:ja-en","wmt20:km-en","wmt20:pl-en","wmt20:ps-en","wmt20:ru-en","wmt20:ta-en","wmt20:zh-en","wmt21.flores:bn-hi","wmt21.flores:hi-bn","wmt21.news:cs-en","wmt21.news:de-en","wmt21.news:de-fr","wmt21.news:en-cs","wmt21.news:en-de","wmt21.news:en-is","wmt21.news:en-ru","wmt21.news:fr-de","wmt21.news:ha-en","wmt21.news:is-en","wmt21.news:ja-en","wmt21.news:ru-en","wmt21.news:zh-en","wmt21.tedtalks:en-de","wmt21.tedtalks:en-ru","wmt21.tedtalks:zh-en","wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:liv-en","wmt22:ru-en","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.7859,"spearman_weighted_mean":0.727,"kendall_tau_b_weighted_mean":0.6033,"pairwise_accuracy_weighted_mean":0.7997}},"chrf_plus_plus":{"seg":{"n_cells":86,"n_pairs":86,"weight":965555,"pairs":["wmt19:de-cs","wmt19:de-en","wmt19:de-fr","wmt19:en-cs","wmt19:en-de","wmt19:en-gu","wmt19:en-lt","wmt19:en-ru","wmt19:fi-en","wmt19:fr-de","wmt19:gu-en","wmt19:kk-en","wmt19:lt-en","wmt19:ru-en","wmt19:zh-en","wmt20:cs-en","wmt20:de-en","wmt20:en-cs","wmt20:en-de","wmt20:en-pl","wmt20:en-ru","wmt20:iu-en","wmt20:ja-en","wmt20:km-en","wmt20:pl-en","wmt20:ps-en","wmt20:ru-en","wmt20:ta-en","wmt20:zh-en","wmt21.flores:bn-hi","wmt21.flores:hi-bn","wmt21.news:cs-en","wmt21.news:de-en","wmt21.news:de-fr","wmt21.news:en-cs","wmt21.news:en-de","wmt21.news:en-is","wmt21.news:en-ru","wmt21.news:fr-de","wmt21.news:ha-en","wmt21.news:is-en","wmt21.news:ja-en","wmt21.news:ru-en","wmt21.news:zh-en","wmt21.tedtalks:en-de","wmt21.tedtalks:en-ru","wmt21.tedtalks:zh-en","wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:ru-en","wmt22:sah-ru","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23.sent:en-de","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-it","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.2529,"spearman_weighted_mean":0.2457,"kendall_tau_b_weighted_mean":0.1715},"sys":{"n_cells":86,"n_pairs":86,"weight":1069,"pairs":["wmt19:de-cs","wmt19:de-en","wmt19:de-fr","wmt19:en-cs","wmt19:en-de","wmt19:en-gu","wmt19:en-lt","wmt19:en-ru","wmt19:fi-en","wmt19:fr-de","wmt19:gu-en","wmt19:kk-en","wmt19:lt-en","wmt19:ru-en","wmt19:zh-en","wmt20:cs-en","wmt20:de-en","wmt20:en-cs","wmt20:en-de","wmt20:en-pl","wmt20:en-ru","wmt20:iu-en","wmt20:ja-en","wmt20:km-en","wmt20:pl-en","wmt20:ps-en","wmt20:ru-en","wmt20:ta-en","wmt20:zh-en","wmt21.flores:bn-hi","wmt21.flores:hi-bn","wmt21.news:cs-en","wmt21.news:de-en","wmt21.news:de-fr","wmt21.news:en-cs","wmt21.news:en-de","wmt21.news:en-is","wmt21.news:en-ru","wmt21.news:fr-de","wmt21.news:ha-en","wmt21.news:is-en","wmt21.news:ja-en","wmt21.news:ru-en","wmt21.news:zh-en","wmt21.tedtalks:en-de","wmt21.tedtalks:en-ru","wmt21.tedtalks:zh-en","wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:liv-en","wmt22:ru-en","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23.sent:en-de","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-it","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.7632,"spearman_weighted_mean":0.7102,"kendall_tau_b_weighted_mean":0.5893,"pairwise_accuracy_weighted_mean":0.7946}},"comet_score":{"seg":{"n_cells":69,"n_pairs":69,"weight":702340,"pairs":["wmt20:cs-en","wmt20:de-en","wmt20:en-cs","wmt20:en-de","wmt20:en-pl","wmt20:en-ru","wmt20:iu-en","wmt20:ja-en","wmt20:km-en","wmt20:pl-en","wmt20:ps-en","wmt20:ru-en","wmt20:ta-en","wmt20:zh-en","wmt21.flores:bn-hi","wmt21.flores:hi-bn","wmt21.news:cs-en","wmt21.news:de-en","wmt21.news:de-fr","wmt21.news:en-cs","wmt21.news:en-de","wmt21.news:en-is","wmt21.news:en-ru","wmt21.news:fr-de","wmt21.news:ha-en","wmt21.news:is-en","wmt21.news:ja-en","wmt21.news:ru-en","wmt21.news:zh-en","wmt21.tedtalks:en-de","wmt21.tedtalks:en-ru","wmt21.tedtalks:zh-en","wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:ru-en","wmt22:sah-ru","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.3627,"spearman_weighted_mean":0.3488,"kendall_tau_b_weighted_mean":0.2498},"sys":{"n_cells":69,"n_pairs":69,"weight":852,"pairs":["wmt20:cs-en","wmt20:de-en","wmt20:en-cs","wmt20:en-de","wmt20:en-pl","wmt20:en-ru","wmt20:iu-en","wmt20:ja-en","wmt20:km-en","wmt20:pl-en","wmt20:ps-en","wmt20:ru-en","wmt20:ta-en","wmt20:zh-en","wmt21.flores:bn-hi","wmt21.flores:hi-bn","wmt21.news:cs-en","wmt21.news:de-en","wmt21.news:de-fr","wmt21.news:en-cs","wmt21.news:en-de","wmt21.news:en-is","wmt21.news:en-ru","wmt21.news:fr-de","wmt21.news:ha-en","wmt21.news:is-en","wmt21.news:ja-en","wmt21.news:ru-en","wmt21.news:zh-en","wmt21.tedtalks:en-de","wmt21.tedtalks:en-ru","wmt21.tedtalks:zh-en","wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:liv-en","wmt22:ru-en","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.8057,"spearman_weighted_mean":0.7722,"kendall_tau_b_weighted_mean":0.6433,"pairwise_accuracy_weighted_mean":0.8215}},"metricx_score":{"seg":{"n_cells":38,"n_pairs":38,"weight":403619,"pairs":["wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:ru-en","wmt22:sah-ru","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-it","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.4061,"spearman_weighted_mean":0.3786,"kendall_tau_b_weighted_mean":0.2737},"sys":{"n_cells":38,"n_pairs":38,"weight":509,"pairs":["wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:liv-en","wmt22:ru-en","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-it","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.8383,"spearman_weighted_mean":0.8157,"kendall_tau_b_weighted_mean":0.6766,"pairwise_accuracy_weighted_mean":0.8383}},"spbleu":{"seg":{"n_cells":37,"n_pairs":37,"weight":399749,"pairs":["wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:ru-en","wmt22:sah-ru","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.1889,"spearman_weighted_mean":0.1924,"kendall_tau_b_weighted_mean":0.1359},"sys":{"n_cells":37,"n_pairs":37,"weight":491,"pairs":["wmt22:cs-en","wmt22:cs-uk","wmt22:de-en","wmt22:en-cs","wmt22:en-de","wmt22:en-hr","wmt22:en-ru","wmt22:en-uk","wmt22:ja-en","wmt22:liv-en","wmt22:ru-en","wmt22:uk-cs","wmt22:uk-en","wmt22:zh-en","wmt23:cs-uk","wmt23:de-en","wmt23:en-cs","wmt23:en-de","wmt23:he-en","wmt23:ja-en","wmt23:zh-en","wmt24:cs-uk","wmt24:en-cs","wmt24:en-de","wmt24:en-es","wmt24:en-hi","wmt24:en-is","wmt24:en-ru","wmt24:en-uk","wmt25:cs-de","wmt25:cs-uk","wmt25:en-bho","wmt25:en-cs","wmt25:en-is","wmt25:en-ru","wmt25:en-sr","wmt25:en-uk"],"pearson_weighted_mean":0.7634,"spearman_weighted_mean":0.7138,"kendall_tau_b_weighted_mean":0.5782,"pairwise_accuracy_weighted_mean":0.7891}}}},"Japonic":{"n_pairs":1,"metrics":{"bleu":{"seg":{"n_cells":6,"n_pairs":6,"weight":68377,"pairs":["wmt20:en-ja","wmt21.news:en-ja","wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.0799,"spearman_weighted_mean":0.0616,"kendall_tau_b_weighted_mean":0.0402},"sys":{"n_cells":6,"n_pairs":6,"weight":85,"pairs":["wmt20:en-ja","wmt21.news:en-ja","wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.5234,"spearman_weighted_mean":0.4755,"kendall_tau_b_weighted_mean":0.3677,"pairwise_accuracy_weighted_mean":0.6838}},"chrf_plain":{"seg":{"n_cells":6,"n_pairs":6,"weight":68377,"pairs":["wmt20:en-ja","wmt21.news:en-ja","wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.2557,"spearman_weighted_mean":0.2401,"kendall_tau_b_weighted_mean":0.1641},"sys":{"n_cells":6,"n_pairs":6,"weight":85,"pairs":["wmt20:en-ja","wmt21.news:en-ja","wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.9289,"spearman_weighted_mean":0.8834,"kendall_tau_b_weighted_mean":0.759,"pairwise_accuracy_weighted_mean":0.8795}},"chrf_plus_plus":{"seg":{"n_cells":6,"n_pairs":6,"weight":68377,"pairs":["wmt20:en-ja","wmt21.news:en-ja","wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.2526,"spearman_weighted_mean":0.2383,"kendall_tau_b_weighted_mean":0.1629},"sys":{"n_cells":6,"n_pairs":6,"weight":85,"pairs":["wmt20:en-ja","wmt21.news:en-ja","wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.8407,"spearman_weighted_mean":0.8517,"kendall_tau_b_weighted_mean":0.6959,"pairwise_accuracy_weighted_mean":0.848}},"comet_score":{"seg":{"n_cells":6,"n_pairs":6,"weight":68377,"pairs":["wmt20:en-ja","wmt21.news:en-ja","wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.4805,"spearman_weighted_mean":0.4406,"kendall_tau_b_weighted_mean":0.3072},"sys":{"n_cells":6,"n_pairs":6,"weight":85,"pairs":["wmt20:en-ja","wmt21.news:en-ja","wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.7317,"spearman_weighted_mean":0.6575,"kendall_tau_b_weighted_mean":0.5618,"pairwise_accuracy_weighted_mean":0.7809}},"metricx_score":{"seg":{"n_cells":4,"n_pairs":4,"weight":43943,"pairs":["wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.3984,"spearman_weighted_mean":0.396,"kendall_tau_b_weighted_mean":0.2726},"sys":{"n_cells":4,"n_pairs":4,"weight":59,"pairs":["wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.7433,"spearman_weighted_mean":0.6605,"kendall_tau_b_weighted_mean":0.532,"pairwise_accuracy_weighted_mean":0.766}},"spbleu":{"seg":{"n_cells":4,"n_pairs":4,"weight":43943,"pairs":["wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.1891,"spearman_weighted_mean":0.1985,"kendall_tau_b_weighted_mean":0.1346},"sys":{"n_cells":4,"n_pairs":4,"weight":59,"pairs":["wmt22:en-ja","wmt23:en-ja","wmt24:en-ja","wmt25:en-ja"],"pearson_weighted_mean":0.8937,"spearman_weighted_mean":0.8057,"kendall_tau_b_weighted_mean":0.6494,"pairwise_accuracy_weighted_mean":0.8247}}}},"Koreanic":{"n_pairs":1,"metrics":{"bleu":{"seg":{"n_cells":1,"n_pairs":1,"weight":2128,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.0982,"spearman_weighted_mean":0.0258,"kendall_tau_b_weighted_mean":0.0196},"sys":{"n_cells":1,"n_pairs":1,"weight":19,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.8864,"spearman_weighted_mean":0.8737,"kendall_tau_b_weighted_mean":0.6725,"pairwise_accuracy_weighted_mean":0.8363}},"chrf_plain":{"seg":{"n_cells":1,"n_pairs":1,"weight":2128,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.0977,"spearman_weighted_mean":0.0285,"kendall_tau_b_weighted_mean":0.0223},"sys":{"n_cells":1,"n_pairs":1,"weight":19,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.8745,"spearman_weighted_mean":0.8877,"kendall_tau_b_weighted_mean":0.7076,"pairwise_accuracy_weighted_mean":0.8538}},"chrf_plus_plus":{"seg":{"n_cells":1,"n_pairs":1,"weight":2128,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.0957,"spearman_weighted_mean":0.032,"kendall_tau_b_weighted_mean":0.0248},"sys":{"n_cells":1,"n_pairs":1,"weight":19,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.8813,"spearman_weighted_mean":0.8807,"kendall_tau_b_weighted_mean":0.6959,"pairwise_accuracy_weighted_mean":0.848}},"comet_score":{"seg":{"n_cells":1,"n_pairs":1,"weight":2128,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.2643,"spearman_weighted_mean":0.3502,"kendall_tau_b_weighted_mean":0.2539},"sys":{"n_cells":1,"n_pairs":1,"weight":19,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.5511,"spearman_weighted_mean":0.5632,"kendall_tau_b_weighted_mean":0.462,"pairwise_accuracy_weighted_mean":0.731}},"metricx_score":{"seg":{"n_cells":1,"n_pairs":1,"weight":2128,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.2311,"spearman_weighted_mean":0.0331,"kendall_tau_b_weighted_mean":0.0265},"sys":{"n_cells":1,"n_pairs":1,"weight":19,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.7689,"spearman_weighted_mean":0.8105,"kendall_tau_b_weighted_mean":0.6374,"pairwise_accuracy_weighted_mean":0.8187}},"spbleu":{"seg":{"n_cells":1,"n_pairs":1,"weight":2128,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.0409,"spearman_weighted_mean":-0.0945,"kendall_tau_b_weighted_mean":-0.0673},"sys":{"n_cells":1,"n_pairs":1,"weight":19,"pairs":["wmt25:en-ko"],"pearson_weighted_mean":0.8711,"spearman_weighted_mean":0.8772,"kendall_tau_b_weighted_mean":0.6959,"pairwise_accuracy_weighted_mean":0.848}}}},"Niger-Congo":{"n_pairs":2,"metrics":{"bleu":{"seg":{"n_cells":2,"n_pairs":2,"weight":5470,"pairs":["wmt21.flores:xh-zu","wmt21.flores:zu-xh"],"pearson_weighted_mean":0.236,"spearman_weighted_mean":0.2747,"kendall_tau_b_weighted_mean":0.1894},"sys":{"n_cells":2,"n_pairs":2,"weight":11,"pairs":["wmt21.flores:xh-zu","wmt21.flores:zu-xh"],"pearson_weighted_mean":0.9445,"spearman_weighted_mean":0.8325,"kendall_tau_b_weighted_mean":0.7455,"pairwise_accuracy_weighted_mean":0.8727}},"chrf_plain":{"seg":{"n_cells":2,"n_pairs":2,"weight":5470,"pairs":["wmt21.flores:xh-zu","wmt21.flores:zu-xh"],"pearson_weighted_mean":0.3687,"spearman_weighted_mean":0.359,"kendall_tau_b_weighted_mean":0.25},"sys":{"n_cells":2,"n_pairs":2,"weight":11,"pairs":["wmt21.flores:xh-zu","wmt21.flores:zu-xh"],"pearson_weighted_mean":0.9981,"spearman_weighted_mean":0.9689,"kendall_tau_b_weighted_mean":0.9273,"pairwise_accuracy_weighted_mean":0.9636}},"chrf_plus_plus":{"seg":{"n_cells":2,"n_pairs":2,"weight":5470,"pairs":["wmt21.flores:xh-zu","wmt21.flores:zu-xh"],"pearson_weighted_mean":0.3579,"spearman_weighted_mean":0.3545,"kendall_tau_b_weighted_mean":0.2471},"sys":{"n_cells":2,"n_pairs":2,"weight":11,"pairs":["wmt21.flores:xh-zu","wmt21.flores:zu-xh"],"pearson_weighted_mean":0.9958,"spearman_weighted_mean":0.9234,"kendall_tau_b_weighted_mean":0.8364,"pairwise_accuracy_weighted_mean":0.9182}},"comet_score":{"seg":{"n_cells":2,"n_pairs":2,"weight":5470,"pairs":["wmt21.flores:xh-zu","wmt21.flores:zu-xh"],"pearson_weighted_mean":0.4304,"spearman_weighted_mean":0.4169,"kendall_tau_b_weighted_mean":0.2954},"sys":{"n_cells":2,"n_pairs":2,"weight":11,"pairs":["wmt21.flores:xh-zu","wmt21.flores:zu-xh"],"pearson_weighted_mean":0.9974,"spearman_weighted_mean":0.9377,"kendall_tau_b_weighted_mean":0.8545,"pairwise_accuracy_weighted_mean":0.9273}}}},"Nilotic":{"n_pairs":1,"metrics":{"chrf_plus_plus":{"seg":{"n_cells":1,"n_pairs":1,"weight":3330,"pairs":["wmt25:en-mas"],"pearson_weighted_mean":0.0561,"spearman_weighted_mean":0.0878,"kendall_tau_b_weighted_mean":0.0665},"sys":{"n_cells":1,"n_pairs":1,"weight":18,"pairs":["wmt25:en-mas"],"pearson_weighted_mean":0.4991,"spearman_weighted_mean":0.4737,"kendall_tau_b_weighted_mean":0.3333,"pairwise_accuracy_weighted_mean":0.6667}},"metricx_score":{"seg":{"n_cells":1,"n_pairs":1,"weight":2960,"pairs":["wmt25:en-mas"],"pearson_weighted_mean":-0.0578,"spearman_weighted_mean":-0.0852,"kendall_tau_b_weighted_mean":-0.0653},"sys":{"n_cells":1,"n_pairs":1,"weight":16,"pairs":["wmt25:en-mas"],"pearson_weighted_mean":-0.0944,"spearman_weighted_mean":-0.1647,"kendall_tau_b_weighted_mean":-0.1167,"pairwise_accuracy_weighted_mean":0.4417}}}},"Sino-Tibetan":{"n_pairs":2,"metrics":{"bleu":{"seg":{"n_cells":9,"n_pairs":9,"weight":114008,"pairs":["wmt19:en-zh","wmt20:en-zh","wmt21.news:en-zh","wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.143,"spearman_weighted_mean":0.1195,"kendall_tau_b_weighted_mean":0.0827},"sys":{"n_cells":9,"n_pairs":9,"weight":142,"pairs":["wmt19:en-zh","wmt20:en-zh","wmt21.news:en-zh","wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.4876,"spearman_weighted_mean":0.4688,"kendall_tau_b_weighted_mean":0.3715,"pairwise_accuracy_weighted_mean":0.6857}},"chrf_plain":{"seg":{"n_cells":9,"n_pairs":9,"weight":114008,"pairs":["wmt19:en-zh","wmt20:en-zh","wmt21.news:en-zh","wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.2353,"spearman_weighted_mean":0.2245,"kendall_tau_b_weighted_mean":0.155},"sys":{"n_cells":9,"n_pairs":9,"weight":142,"pairs":["wmt19:en-zh","wmt20:en-zh","wmt21.news:en-zh","wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.6811,"spearman_weighted_mean":0.6135,"kendall_tau_b_weighted_mean":0.482,"pairwise_accuracy_weighted_mean":0.74}},"chrf_plus_plus":{"seg":{"n_cells":9,"n_pairs":9,"weight":114008,"pairs":["wmt19:en-zh","wmt20:en-zh","wmt21.news:en-zh","wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.2238,"spearman_weighted_mean":0.2186,"kendall_tau_b_weighted_mean":0.1508},"sys":{"n_cells":9,"n_pairs":9,"weight":142,"pairs":["wmt19:en-zh","wmt20:en-zh","wmt21.news:en-zh","wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.616,"spearman_weighted_mean":0.5806,"kendall_tau_b_weighted_mean":0.461,"pairwise_accuracy_weighted_mean":0.7305}},"comet_score":{"seg":{"n_cells":8,"n_pairs":8,"weight":94500,"pairs":["wmt20:en-zh","wmt21.news:en-zh","wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.4112,"spearman_weighted_mean":0.3972,"kendall_tau_b_weighted_mean":0.2778},"sys":{"n_cells":8,"n_pairs":8,"weight":130,"pairs":["wmt20:en-zh","wmt21.news:en-zh","wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.7232,"spearman_weighted_mean":0.7495,"kendall_tau_b_weighted_mean":0.6233,"pairwise_accuracy_weighted_mean":0.8116}},"metricx_score":{"seg":{"n_cells":6,"n_pairs":6,"weight":52510,"pairs":["wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.3835,"spearman_weighted_mean":0.4046,"kendall_tau_b_weighted_mean":0.2843},"sys":{"n_cells":6,"n_pairs":6,"weight":89,"pairs":["wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.8164,"spearman_weighted_mean":0.7598,"kendall_tau_b_weighted_mean":0.6178,"pairwise_accuracy_weighted_mean":0.8089}},"spbleu":{"seg":{"n_cells":6,"n_pairs":6,"weight":52510,"pairs":["wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.2272,"spearman_weighted_mean":0.2175,"kendall_tau_b_weighted_mean":0.1532},"sys":{"n_cells":6,"n_pairs":6,"weight":89,"pairs":["wmt22:en-zh","wmt23:en-zh","wmt24:en-zh","wmt24:ja-zh","wmt25:en-zh","wmt25:ja-zh"],"pearson_weighted_mean":0.677,"spearman_weighted_mean":0.5291,"kendall_tau_b_weighted_mean":0.4129,"pairwise_accuracy_weighted_mean":0.7065}}}},"Turkic":{"n_pairs":1,"metrics":{"bleu":{"seg":{"n_cells":1,"n_pairs":1,"weight":9012,"pairs":["wmt19:en-kk"],"pearson_weighted_mean":0.3401,"spearman_weighted_mean":0.3888,"kendall_tau_b_weighted_mean":0.2675},"sys":{"n_cells":1,"n_pairs":1,"weight":11,"pairs":["wmt19:en-kk"],"pearson_weighted_mean":0.8515,"spearman_weighted_mean":0.6182,"kendall_tau_b_weighted_mean":0.4909,"pairwise_accuracy_weighted_mean":0.7455}},"chrf_plain":{"seg":{"n_cells":1,"n_pairs":1,"weight":9012,"pairs":["wmt19:en-kk"],"pearson_weighted_mean":0.4863,"spearman_weighted_mean":0.4701,"kendall_tau_b_weighted_mean":0.3273},"sys":{"n_cells":1,"n_pairs":1,"weight":11,"pairs":["wmt19:en-kk"],"pearson_weighted_mean":0.972,"spearman_weighted_mean":0.8091,"kendall_tau_b_weighted_mean":0.6727,"pairwise_accuracy_weighted_mean":0.8364}},"chrf_plus_plus":{"seg":{"n_cells":1,"n_pairs":1,"weight":9012,"pairs":["wmt19:en-kk"],"pearson_weighted_mean":0.5019,"spearman_weighted_mean":0.4988,"kendall_tau_b_weighted_mean":0.3499},"sys":{"n_cells":1,"n_pairs":1,"weight":11,"pairs":["wmt19:en-kk"],"pearson_weighted_mean":0.966,"spearman_weighted_mean":0.7818,"kendall_tau_b_weighted_mean":0.6364,"pairwise_accuracy_weighted_mean":0.8182}}}},"Uralic":{"n_pairs":3,"metrics":{"bleu":{"seg":{"n_cells":2,"n_pairs":2,"weight":19499,"pairs":["wmt19:en-fi","wmt25:en-et"],"pearson_weighted_mean":0.1303,"spearman_weighted_mean":0.1355,"kendall_tau_b_weighted_mean":0.0909},"sys":{"n_cells":3,"n_pairs":3,"weight":35,"pairs":["wmt19:en-fi","wmt22:en-liv","wmt25:en-et"],"pearson_weighted_mean":0.8506,"spearman_weighted_mean":0.8457,"kendall_tau_b_weighted_mean":0.7265,"pairwise_accuracy_weighted_mean":0.8633}},"chrf_plain":{"seg":{"n_cells":2,"n_pairs":2,"weight":19499,"pairs":["wmt19:en-fi","wmt25:en-et"],"pearson_weighted_mean":0.4645,"spearman_weighted_mean":0.4659,"kendall_tau_b_weighted_mean":0.3259},"sys":{"n_cells":3,"n_pairs":3,"weight":35,"pairs":["wmt19:en-fi","wmt22:en-liv","wmt25:en-et"],"pearson_weighted_mean":0.9057,"spearman_weighted_mean":0.8701,"kendall_tau_b_weighted_mean":0.7279,"pairwise_accuracy_weighted_mean":0.8639}},"chrf_plus_plus":{"seg":{"n_cells":2,"n_pairs":2,"weight":19499,"pairs":["wmt19:en-fi","wmt25:en-et"],"pearson_weighted_mean":0.4675,"spearman_weighted_mean":0.4691,"kendall_tau_b_weighted_mean":0.3283},"sys":{"n_cells":3,"n_pairs":3,"weight":35,"pairs":["wmt19:en-fi","wmt22:en-liv","wmt25:en-et"],"pearson_weighted_mean":0.9093,"spearman_weighted_mean":0.8733,"kendall_tau_b_weighted_mean":0.7413,"pairwise_accuracy_weighted_mean":0.8707}},"comet_score":{"seg":{"n_cells":1,"n_pairs":1,"weight":3420,"pairs":["wmt25:en-et"],"pearson_weighted_mean":0.351,"spearman_weighted_mean":0.3526,"kendall_tau_b_weighted_mean":0.2402},"sys":{"n_cells":2,"n_pairs":2,"weight":23,"pairs":["wmt22:en-liv","wmt25:en-et"],"pearson_weighted_mean":0.7515,"spearman_weighted_mean":0.6207,"kendall_tau_b_weighted_mean":0.4399,"pairwise_accuracy_weighted_mean":0.72}},"metricx_score":{"seg":{"n_cells":1,"n_pairs":1,"weight":3420,"pairs":["wmt25:en-et"],"pearson_weighted_mean":0.4368,"spearman_weighted_mean":0.4526,"kendall_tau_b_weighted_mean":0.3124},"sys":{"n_cells":2,"n_pairs":2,"weight":23,"pairs":["wmt22:en-liv","wmt25:en-et"],"pearson_weighted_mean":0.8059,"spearman_weighted_mean":0.6805,"kendall_tau_b_weighted_mean":0.5013,"pairwise_accuracy_weighted_mean":0.7506}},"spbleu":{"seg":{"n_cells":1,"n_pairs":1,"weight":3420,"pairs":["wmt25:en-et"],"pearson_weighted_mean":0.3255,"spearman_weighted_mean":0.3021,"kendall_tau_b_weighted_mean":0.206},"sys":{"n_cells":2,"n_pairs":2,"weight":23,"pairs":["wmt22:en-liv","wmt25:en-et"],"pearson_weighted_mean":0.8835,"spearman_weighted_mean":0.8555,"kendall_tau_b_weighted_mean":0.7212,"pairwise_accuracy_weighted_mean":0.8606}}}}},"cells":[{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.941,"spearman":0.9545,"kendall_tau_b":0.8545,"pairwise_accuracy":0.9273},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.974,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9709,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9481,"spearman":0.9545,"kendall_tau_b":0.8545,"pairwise_accuracy":0.9273},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9736,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9699,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":16900,"pearson":0.0643,"spearman":0.0821,"kendall_tau_b":0.055},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":16900,"pearson":0.4434,"spearman":0.4246,"kendall_tau_b":0.2959},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":16900,"pearson":0.4453,"spearman":0.4262,"kendall_tau_b":0.2967},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":16900,"pearson":0.0571,"spearman":0.0713,"kendall_tau_b":0.0481},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":16900,"pearson":0.3798,"spearman":0.3728,"kendall_tau_b":0.2603},{"testset":"wmt19","pair":"de-cs","pair_verbatim":"de-cs","src":"de","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":16900,"pearson":0.3804,"spearman":0.3736,"kendall_tau_b":0.2605},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.849,"spearman":0.7629,"kendall_tau_b":0.5714,"pairwise_accuracy":0.7881},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.9165,"spearman":0.8218,"kendall_tau_b":0.6387,"pairwise_accuracy":0.822},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":16,"pearson":0.9064,"spearman":0.813,"kendall_tau_b":0.6387,"pairwise_accuracy":0.822},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.8119,"spearman":0.7941,"kendall_tau_b":0.6167,"pairwise_accuracy":0.8083},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.8849,"spearman":0.8265,"kendall_tau_b":0.6833,"pairwise_accuracy":0.8417},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":16,"pearson":0.879,"spearman":0.8059,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":31948,"pearson":0.0182,"spearman":0.017,"kendall_tau_b":0.0114},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":31948,"pearson":0.3065,"spearman":0.3206,"kendall_tau_b":0.2192},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":16,"n_points":31948,"pearson":0.3105,"spearman":0.3247,"kendall_tau_b":0.2221},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":31948,"pearson":0.0243,"spearman":0.0172,"kendall_tau_b":0.0118},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":31948,"pearson":0.285,"spearman":0.2993,"kendall_tau_b":0.2081},{"testset":"wmt19","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":16,"n_points":31948,"pearson":0.2882,"spearman":0.3028,"kendall_tau_b":0.2106},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8909,"spearman":0.9091,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9312,"spearman":0.9545,"kendall_tau_b":0.8545,"pairwise_accuracy":0.9273},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9572,"spearman":0.9455,"kendall_tau_b":0.8545,"pairwise_accuracy":0.9273},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8733,"spearman":0.9273,"kendall_tau_b":0.8182,"pairwise_accuracy":0.9091},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9246,"spearman":0.9727,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9506,"spearman":0.9636,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":6700,"pearson":0.364,"spearman":0.3638,"kendall_tau_b":0.2494},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":6700,"pearson":0.3963,"spearman":0.3849,"kendall_tau_b":0.2658},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":6700,"pearson":0.4047,"spearman":0.3933,"kendall_tau_b":0.2717},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":6700,"pearson":0.3343,"spearman":0.3425,"kendall_tau_b":0.2371},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":6700,"pearson":0.3618,"spearman":0.3633,"kendall_tau_b":0.2531},{"testset":"wmt19","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":6700,"pearson":0.3697,"spearman":0.3702,"kendall_tau_b":0.258},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8971,"spearman":0.8273,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9896,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9848,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8971,"spearman":0.8727,"kendall_tau_b":0.7455,"pairwise_accuracy":0.8727},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9868,"spearman":0.9727,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9835,"spearman":0.9727,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":18206,"pearson":0.0032,"spearman":0.0001,"kendall_tau_b":0.0001},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":18206,"pearson":0.3463,"spearman":0.3428,"kendall_tau_b":0.2375},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":18206,"pearson":0.3443,"spearman":0.3404,"kendall_tau_b":0.2355},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":18206,"pearson":0.005,"spearman":-0.0054,"kendall_tau_b":-0.0035},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":18206,"pearson":0.3307,"spearman":0.3206,"kendall_tau_b":0.223},{"testset":"wmt19","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":18206,"pearson":0.3272,"spearman":0.317,"kendall_tau_b":0.2201},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":22,"pearson":0.9208,"spearman":0.6632,"kendall_tau_b":0.5403,"pairwise_accuracy":0.7652},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":22,"pearson":0.9793,"spearman":0.9054,"kendall_tau_b":0.7636,"pairwise_accuracy":0.8826},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":22,"pearson":0.9765,"spearman":0.9156,"kendall_tau_b":0.7722,"pairwise_accuracy":0.887},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":22,"pearson":0.8983,"spearman":0.6215,"kendall_tau_b":0.4826,"pairwise_accuracy":0.7359},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":22,"pearson":0.9699,"spearman":0.9209,"kendall_tau_b":0.7922,"pairwise_accuracy":0.8961},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":22,"pearson":0.9655,"spearman":0.9322,"kendall_tau_b":0.8009,"pairwise_accuracy":0.9004},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":22,"n_points":38136,"pearson":0.0247,"spearman":0.0034,"kendall_tau_b":0.0023},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":22,"n_points":38136,"pearson":0.3031,"spearman":0.2576,"kendall_tau_b":0.1757},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":22,"n_points":38136,"pearson":0.3007,"spearman":0.2555,"kendall_tau_b":0.1741},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":22,"n_points":38136,"pearson":0.0292,"spearman":-0.0,"kendall_tau_b":0.0},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":22,"n_points":38136,"pearson":0.2818,"spearman":0.236,"kendall_tau_b":0.1641},{"testset":"wmt19","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":22,"n_points":38136,"pearson":0.2789,"spearman":0.2336,"kendall_tau_b":0.1623},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9688,"spearman":0.972,"kendall_tau_b":0.8788,"pairwise_accuracy":0.9394},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.986,"spearman":0.958,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.9843,"spearman":0.958,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9438,"spearman":0.958,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9802,"spearman":0.965,"kendall_tau_b":0.8788,"pairwise_accuracy":0.9394},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.9757,"spearman":0.965,"kendall_tau_b":0.8788,"pairwise_accuracy":0.9394},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":16079,"pearson":0.1013,"spearman":0.1141,"kendall_tau_b":0.0762},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":16079,"pearson":0.4932,"spearman":0.5015,"kendall_tau_b":0.3518},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":16079,"pearson":0.4972,"spearman":0.5059,"kendall_tau_b":0.355},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":16079,"pearson":0.1038,"spearman":0.1099,"kendall_tau_b":0.074},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":16079,"pearson":0.4581,"spearman":0.4698,"kendall_tau_b":0.3306},{"testset":"wmt19","pair":"en-fi","pair_verbatim":"en-fi","src":"en","tgt":"fi","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":16079,"pearson":0.4616,"spearman":0.4731,"kendall_tau_b":0.3329},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.7366,"spearman":0.8636,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8406,"spearman":0.9,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.8313,"spearman":0.9091,"kendall_tau_b":0.7455,"pairwise_accuracy":0.8727},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.7835,"spearman":0.8455,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8838,"spearman":0.9545,"kendall_tau_b":0.8545,"pairwise_accuracy":0.9273},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.8791,"spearman":0.9545,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":6925,"pearson":0.4453,"spearman":0.4498,"kendall_tau_b":0.3099},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":6925,"pearson":0.4693,"spearman":0.4988,"kendall_tau_b":0.3511},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":6925,"pearson":0.492,"spearman":0.5091,"kendall_tau_b":0.358},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":6925,"pearson":0.4312,"spearman":0.4266,"kendall_tau_b":0.2931},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":6925,"pearson":0.4395,"spearman":0.4592,"kendall_tau_b":0.3202},{"testset":"wmt19","pair":"en-gu","pair_verbatim":"en-gu","src":"en","tgt":"gu","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":6925,"pearson":0.4756,"spearman":0.48,"kendall_tau_b":0.3347},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8515,"spearman":0.6182,"kendall_tau_b":0.4909,"pairwise_accuracy":0.7455},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.972,"spearman":0.8091,"kendall_tau_b":0.6727,"pairwise_accuracy":0.8364},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.966,"spearman":0.7818,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8332,"spearman":0.6091,"kendall_tau_b":0.4545,"pairwise_accuracy":0.7273},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9648,"spearman":0.7636,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9563,"spearman":0.7545,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9012,"pearson":0.3401,"spearman":0.3888,"kendall_tau_b":0.2675},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9012,"pearson":0.4863,"spearman":0.4701,"kendall_tau_b":0.3273},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":9012,"pearson":0.5019,"spearman":0.4988,"kendall_tau_b":0.3499},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9012,"pearson":0.3258,"spearman":0.3772,"kendall_tau_b":0.26},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9012,"pearson":0.4833,"spearman":0.4618,"kendall_tau_b":0.3223},{"testset":"wmt19","pair":"en-kk","pair_verbatim":"en-kk","src":"en","tgt":"kk","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":9012,"pearson":0.4967,"spearman":0.4894,"kendall_tau_b":0.3441},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9888,"spearman":0.972,"kendall_tau_b":0.8788,"pairwise_accuracy":0.9394},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.981,"spearman":0.979,"kendall_tau_b":0.9091,"pairwise_accuracy":0.9545},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.9754,"spearman":0.9301,"kendall_tau_b":0.8182,"pairwise_accuracy":0.9091},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9881,"spearman":0.986,"kendall_tau_b":0.9394,"pairwise_accuracy":0.9697},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9805,"spearman":0.958,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.9781,"spearman":0.9091,"kendall_tau_b":0.7576,"pairwise_accuracy":0.8788},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8961,"pearson":0.0814,"spearman":0.1096,"kendall_tau_b":0.0731},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8961,"pearson":0.4098,"spearman":0.4032,"kendall_tau_b":0.2791},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":8961,"pearson":0.4118,"spearman":0.403,"kendall_tau_b":0.2785},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8961,"pearson":0.0965,"spearman":0.1228,"kendall_tau_b":0.0823},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8961,"pearson":0.3975,"spearman":0.3906,"kendall_tau_b":0.2711},{"testset":"wmt19","pair":"en-lt","pair_verbatim":"en-lt","src":"en","tgt":"lt","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":8961,"pearson":0.3988,"spearman":0.3903,"kendall_tau_b":0.2709},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.986,"spearman":0.9632,"kendall_tau_b":0.8703,"pairwise_accuracy":0.9242},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9429,"spearman":0.9371,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.9887,"spearman":0.979,"kendall_tau_b":0.9394,"pairwise_accuracy":0.9697},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9726,"spearman":0.9562,"kendall_tau_b":0.8703,"pairwise_accuracy":0.9242},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9563,"spearman":0.9091,"kendall_tau_b":0.7879,"pairwise_accuracy":0.8939},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.9896,"spearman":0.951,"kendall_tau_b":0.8788,"pairwise_accuracy":0.9394},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":17420,"pearson":0.0814,"spearman":0.0734,"kendall_tau_b":0.0494},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":17420,"pearson":0.5654,"spearman":0.41,"kendall_tau_b":0.2872},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":17420,"pearson":0.5376,"spearman":0.4372,"kendall_tau_b":0.3066},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":17420,"pearson":0.0942,"spearman":0.0746,"kendall_tau_b":0.0509},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":17420,"pearson":0.5776,"spearman":0.4062,"kendall_tau_b":0.2866},{"testset":"wmt19","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":17420,"pearson":0.5465,"spearman":0.43,"kendall_tau_b":0.304},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9005,"spearman":0.7552,"kendall_tau_b":0.6061,"pairwise_accuracy":0.803},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.88,"spearman":0.7063,"kendall_tau_b":0.5758,"pairwise_accuracy":0.7879},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.8074,"spearman":0.6503,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8201,"spearman":0.7343,"kendall_tau_b":0.6061,"pairwise_accuracy":0.803},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.7866,"spearman":0.6853,"kendall_tau_b":0.5758,"pairwise_accuracy":0.7879},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.752,"spearman":0.6364,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":19508,"pearson":0.2974,"spearman":0.2841,"kendall_tau_b":0.1928},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":19508,"pearson":0.2976,"spearman":0.2792,"kendall_tau_b":0.1908},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":19508,"pearson":0.2397,"spearman":0.233,"kendall_tau_b":0.1583},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":19508,"pearson":0.2407,"spearman":0.2152,"kendall_tau_b":0.1464},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":19508,"pearson":0.2387,"spearman":0.2099,"kendall_tau_b":0.1434},{"testset":"wmt19","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":19508,"pearson":0.1954,"spearman":0.177,"kendall_tau_b":0.1206},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9823,"spearman":0.9231,"kendall_tau_b":0.7879,"pairwise_accuracy":0.8939},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9918,"spearman":0.951,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.9913,"spearman":0.958,"kendall_tau_b":0.8788,"pairwise_accuracy":0.9394},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9745,"spearman":0.9161,"kendall_tau_b":0.7576,"pairwise_accuracy":0.8788},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9883,"spearman":0.958,"kendall_tau_b":0.8788,"pairwise_accuracy":0.9394},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"pearson":0.986,"spearman":0.951,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":18973,"pearson":0.0489,"spearman":0.0197,"kendall_tau_b":0.0133},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":18973,"pearson":0.3226,"spearman":0.3312,"kendall_tau_b":0.2271},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":18973,"pearson":0.3269,"spearman":0.3356,"kendall_tau_b":0.2301},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":18973,"pearson":0.0509,"spearman":0.0209,"kendall_tau_b":0.0142},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":18973,"pearson":0.3038,"spearman":0.3134,"kendall_tau_b":0.2161},{"testset":"wmt19","pair":"fi-en","pair_verbatim":"fi-en","src":"fi","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":12,"n_points":18973,"pearson":0.309,"spearman":0.318,"kendall_tau_b":0.2194},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.8637,"spearman":0.9152,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.8644,"spearman":0.8424,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":10,"pearson":0.8661,"spearman":0.8909,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.8614,"spearman":0.9515,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.859,"spearman":0.8788,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":10,"pearson":0.8607,"spearman":0.9152,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":4000,"pearson":-0.0086,"spearman":-0.0157,"kendall_tau_b":-0.0107},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":4000,"pearson":0.401,"spearman":0.4013,"kendall_tau_b":0.2783},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":10,"n_points":4000,"pearson":0.3923,"spearman":0.3932,"kendall_tau_b":0.272},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":4000,"pearson":-0.0076,"spearman":-0.014,"kendall_tau_b":-0.0098},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":4000,"pearson":0.3791,"spearman":0.3927,"kendall_tau_b":0.2755},{"testset":"wmt19","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":10,"n_points":4000,"pearson":0.3715,"spearman":0.3846,"kendall_tau_b":0.2693},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8338,"spearman":0.8909,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9548,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.946,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8879,"spearman":0.8909,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9691,"spearman":0.9727,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9622,"spearman":0.9545,"kendall_tau_b":0.8545,"pairwise_accuracy":0.9273},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11176,"pearson":0.2667,"spearman":0.2546,"kendall_tau_b":0.1726},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11176,"pearson":0.3427,"spearman":0.324,"kendall_tau_b":0.2222},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":11176,"pearson":0.3373,"spearman":0.3186,"kendall_tau_b":0.2182},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11176,"pearson":0.2518,"spearman":0.2346,"kendall_tau_b":0.1594},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11176,"pearson":0.3144,"spearman":0.2992,"kendall_tau_b":0.2053},{"testset":"wmt19","pair":"gu-en","pair_verbatim":"gu-en","src":"gu","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":11176,"pearson":0.3114,"spearman":0.2946,"kendall_tau_b":0.202},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9464,"spearman":0.9455,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9779,"spearman":0.8636,"kendall_tau_b":0.7455,"pairwise_accuracy":0.8727},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9744,"spearman":0.9455,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9461,"spearman":0.9455,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9775,"spearman":0.8636,"kendall_tau_b":0.7455,"pairwise_accuracy":0.8727},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9746,"spearman":0.9455,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11000,"pearson":0.3998,"spearman":0.4341,"kendall_tau_b":0.3004},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11000,"pearson":0.4876,"spearman":0.461,"kendall_tau_b":0.3223},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":11000,"pearson":0.4927,"spearman":0.4701,"kendall_tau_b":0.3292},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11000,"pearson":0.3717,"spearman":0.4066,"kendall_tau_b":0.2817},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11000,"pearson":0.4555,"spearman":0.437,"kendall_tau_b":0.3054},{"testset":"wmt19","pair":"kk-en","pair_verbatim":"kk-en","src":"kk","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":11000,"pearson":0.4605,"spearman":0.445,"kendall_tau_b":0.3114},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9614,"spearman":0.9364,"kendall_tau_b":0.8182,"pairwise_accuracy":0.9091},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9397,"spearman":0.9545,"kendall_tau_b":0.8545,"pairwise_accuracy":0.9273},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.947,"spearman":0.9545,"kendall_tau_b":0.8545,"pairwise_accuracy":0.9273},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9485,"spearman":0.8818,"kendall_tau_b":0.7455,"pairwise_accuracy":0.8727},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9216,"spearman":0.9,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"pearson":0.9306,"spearman":0.9,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11000,"pearson":0.0663,"spearman":0.0674,"kendall_tau_b":0.0449},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11000,"pearson":0.3923,"spearman":0.3567,"kendall_tau_b":0.2457},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":11000,"pearson":0.397,"spearman":0.3631,"kendall_tau_b":0.2502},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11000,"pearson":0.0623,"spearman":0.0663,"kendall_tau_b":0.0446},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":11000,"pearson":0.3437,"spearman":0.3246,"kendall_tau_b":0.2233},{"testset":"wmt19","pair":"lt-en","pair_verbatim":"lt-en","src":"lt","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":11,"n_points":11000,"pearson":0.3486,"spearman":0.3305,"kendall_tau_b":0.2274},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.8788,"spearman":0.8549,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9449,"spearman":0.8549,"kendall_tau_b":0.7363,"pairwise_accuracy":0.8681},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":14,"pearson":0.9147,"spearman":0.8593,"kendall_tau_b":0.7582,"pairwise_accuracy":0.8791},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.8566,"spearman":0.8725,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9364,"spearman":0.8901,"kendall_tau_b":0.7582,"pairwise_accuracy":0.8791},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":14,"pearson":0.9051,"spearman":0.8945,"kendall_tau_b":0.7802,"pairwise_accuracy":0.8901},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":23752,"pearson":0.0205,"spearman":0.0146,"kendall_tau_b":0.0098},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":23752,"pearson":0.3161,"spearman":0.3152,"kendall_tau_b":0.2156},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":14,"n_points":23752,"pearson":0.3156,"spearman":0.316,"kendall_tau_b":0.2161},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":23752,"pearson":0.0228,"spearman":0.0166,"kendall_tau_b":0.0113},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":23752,"pearson":0.2893,"spearman":0.2877,"kendall_tau_b":0.1986},{"testset":"wmt19","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":14,"n_points":23752,"pearson":0.2881,"spearman":0.2882,"kendall_tau_b":0.199},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.8987,"spearman":0.8571,"kendall_tau_b":0.6952,"pairwise_accuracy":0.8476},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.9555,"spearman":0.8679,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":15,"pearson":0.9467,"spearman":0.8679,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.907,"spearman":0.8857,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.9602,"spearman":0.8964,"kendall_tau_b":0.7524,"pairwise_accuracy":0.8762},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":15,"pearson":0.9522,"spearman":0.8964,"kendall_tau_b":0.7524,"pairwise_accuracy":0.8762},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":20199,"pearson":0.3982,"spearman":0.4154,"kendall_tau_b":0.2869},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":20199,"pearson":0.4743,"spearman":0.4497,"kendall_tau_b":0.3129},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":15,"n_points":20199,"pearson":0.4696,"spearman":0.4496,"kendall_tau_b":0.3129},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":20199,"pearson":0.3562,"spearman":0.3822,"kendall_tau_b":0.2662},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":20199,"pearson":0.4237,"spearman":0.415,"kendall_tau_b":0.2905},{"testset":"wmt19","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"ref","computed":true,"n_sys":15,"n_points":20199,"pearson":0.4201,"spearman":0.415,"kendall_tau_b":0.2906},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.851,"spearman":0.951,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8724,"spearman":0.9371,"kendall_tau_b":0.8182,"pairwise_accuracy":0.9091},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8672,"spearman":0.9371,"kendall_tau_b":0.8182,"pairwise_accuracy":0.9091},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.7826,"spearman":0.8601,"kendall_tau_b":0.7273,"pairwise_accuracy":0.8636},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8305,"spearman":0.8881,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8436,"spearman":0.8531,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8426,"spearman":0.8531,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.7365,"spearman":0.8392,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":7532,"pearson":0.1636,"spearman":0.1524,"kendall_tau_b":0.1031},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":7532,"pearson":0.1991,"spearman":0.1941,"kendall_tau_b":0.1314},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":7532,"pearson":0.2009,"spearman":0.1928,"kendall_tau_b":0.1305},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":7532,"pearson":0.2234,"spearman":0.2285,"kendall_tau_b":0.1549},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":7532,"pearson":0.1446,"spearman":0.1309,"kendall_tau_b":0.0886},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":7532,"pearson":0.1637,"spearman":0.1591,"kendall_tau_b":0.1078},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":7532,"pearson":0.168,"spearman":0.1603,"kendall_tau_b":0.1087},{"testset":"wmt20","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":7532,"pearson":0.1871,"spearman":0.1861,"kendall_tau_b":0.1263},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9847,"spearman":0.8601,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9975,"spearman":0.8741,"kendall_tau_b":0.7273,"pairwise_accuracy":0.8636},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9972,"spearman":0.8601,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9982,"spearman":0.8741,"kendall_tau_b":0.7576,"pairwise_accuracy":0.8788},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9821,"spearman":0.8042,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9955,"spearman":0.7972,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9952,"spearman":0.8042,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9969,"spearman":0.8322,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8652,"pearson":0.3897,"spearman":0.3174,"kendall_tau_b":0.2183},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8652,"pearson":0.519,"spearman":0.3537,"kendall_tau_b":0.2456},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8652,"pearson":0.511,"spearman":0.3453,"kendall_tau_b":0.2394},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8652,"pearson":0.5762,"spearman":0.4056,"kendall_tau_b":0.2837},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8652,"pearson":0.4198,"spearman":0.3292,"kendall_tau_b":0.2285},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8652,"pearson":0.5715,"spearman":0.3652,"kendall_tau_b":0.2551},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8652,"pearson":0.5635,"spearman":0.3574,"kendall_tau_b":0.2494},{"testset":"wmt20","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":8652,"pearson":0.6409,"spearman":0.41,"kendall_tau_b":0.2881},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8247,"spearman":0.6713,"kendall_tau_b":0.5152,"pairwise_accuracy":0.7576},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8256,"spearman":0.6434,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8327,"spearman":0.6434,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.978,"spearman":0.972,"kendall_tau_b":0.9091,"pairwise_accuracy":0.9545},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8385,"spearman":0.6713,"kendall_tau_b":0.5152,"pairwise_accuracy":0.7576},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.841,"spearman":0.6434,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8474,"spearman":0.6434,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9837,"spearman":0.972,"kendall_tau_b":0.9091,"pairwise_accuracy":0.9545},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14577,"pearson":0.3712,"spearman":0.349,"kendall_tau_b":0.2397},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14577,"pearson":0.4081,"spearman":0.376,"kendall_tau_b":0.2593},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14577,"pearson":0.4151,"spearman":0.3815,"kendall_tau_b":0.2636},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14577,"pearson":0.6993,"spearman":0.6256,"kendall_tau_b":0.4487},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14577,"pearson":0.3255,"spearman":0.3084,"kendall_tau_b":0.2107},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14577,"pearson":0.3598,"spearman":0.33,"kendall_tau_b":0.226},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14577,"pearson":0.3654,"spearman":0.3348,"kendall_tau_b":0.2296},{"testset":"wmt20","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14577,"pearson":0.6136,"spearman":0.5464,"kendall_tau_b":0.3842},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.8307,"spearman":0.8214,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.8529,"spearman":0.8929,"kendall_tau_b":0.8095,"pairwise_accuracy":0.9048},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.8419,"spearman":0.8929,"kendall_tau_b":0.8095,"pairwise_accuracy":0.9048},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.9655,"spearman":0.8571,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9279,"spearman":0.9253,"kendall_tau_b":0.8022,"pairwise_accuracy":0.9011},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9619,"spearman":0.956,"kendall_tau_b":0.8681,"pairwise_accuracy":0.9341},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9579,"spearman":0.956,"kendall_tau_b":0.8681,"pairwise_accuracy":0.9341},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9721,"spearman":0.9385,"kendall_tau_b":0.8462,"pairwise_accuracy":0.9231},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9295,"spearman":0.8813,"kendall_tau_b":0.7582,"pairwise_accuracy":0.8791},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.966,"spearman":0.9297,"kendall_tau_b":0.8242,"pairwise_accuracy":0.9121},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9607,"spearman":0.9297,"kendall_tau_b":0.8242,"pairwise_accuracy":0.9121},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9722,"spearman":0.8769,"kendall_tau_b":0.7582,"pairwise_accuracy":0.8791},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":9926,"pearson":0.1874,"spearman":0.1779,"kendall_tau_b":0.1224},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":9926,"pearson":0.2369,"spearman":0.2454,"kendall_tau_b":0.1695},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":9926,"pearson":0.2328,"spearman":0.2377,"kendall_tau_b":0.1643},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":9926,"pearson":0.4539,"spearman":0.5291,"kendall_tau_b":0.3764},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":9730,"pearson":0.2689,"spearman":0.2465,"kendall_tau_b":0.1672},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":9730,"pearson":0.3673,"spearman":0.298,"kendall_tau_b":0.2031},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":9730,"pearson":0.3606,"spearman":0.2922,"kendall_tau_b":0.1992},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":9730,"pearson":0.5794,"spearman":0.4926,"kendall_tau_b":0.3437},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":9730,"pearson":0.2306,"spearman":0.1967,"kendall_tau_b":0.1341},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":9730,"pearson":0.3105,"spearman":0.232,"kendall_tau_b":0.1586},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":9730,"pearson":0.3064,"spearman":0.2296,"kendall_tau_b":0.157},{"testset":"wmt20","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":9730,"pearson":0.4965,"spearman":0.383,"kendall_tau_b":0.2648},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.1629,"spearman":0.3636,"kendall_tau_b":0.2364,"pairwise_accuracy":0.6182},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.3499,"spearman":0.4182,"kendall_tau_b":0.3455,"pairwise_accuracy":0.6727},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.328,"spearman":0.4,"kendall_tau_b":0.3091,"pairwise_accuracy":0.6545},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8598,"spearman":0.4636,"kendall_tau_b":0.3818,"pairwise_accuracy":0.6909},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.2157,"spearman":0.0,"kendall_tau_b":-0.0909,"pairwise_accuracy":0.4545},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.3035,"spearman":0.0364,"kendall_tau_b":-0.0545,"pairwise_accuracy":0.4727},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.2917,"spearman":0.0091,"kendall_tau_b":-0.0909,"pairwise_accuracy":0.4545},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8063,"spearman":0.0455,"kendall_tau_b":-0.0182,"pairwise_accuracy":0.4909},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9981,"pearson":0.1833,"spearman":0.238,"kendall_tau_b":0.1615},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9981,"pearson":0.3631,"spearman":0.382,"kendall_tau_b":0.2651},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9981,"pearson":0.3383,"spearman":0.3747,"kendall_tau_b":0.2588},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9981,"pearson":0.3082,"spearman":0.3043,"kendall_tau_b":0.2111},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9981,"pearson":0.2007,"spearman":0.236,"kendall_tau_b":0.1612},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9981,"pearson":0.349,"spearman":0.3502,"kendall_tau_b":0.2441},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9981,"pearson":0.3294,"spearman":0.3439,"kendall_tau_b":0.2383},{"testset":"wmt20","pair":"en-iu","pair_verbatim":"en-iu","src":"en","tgt":"iu","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9981,"pearson":0.2812,"spearman":0.2981,"kendall_tau_b":0.2077},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9455,"spearman":0.9364,"kendall_tau_b":0.8182,"pairwise_accuracy":0.9091},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9513,"spearman":0.9364,"kendall_tau_b":0.8182,"pairwise_accuracy":0.9091},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9518,"spearman":0.9,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9738,"spearman":0.9091,"kendall_tau_b":0.7455,"pairwise_accuracy":0.8727},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9493,"spearman":0.9273,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9554,"spearman":0.9273,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9555,"spearman":0.8909,"kendall_tau_b":0.7455,"pairwise_accuracy":0.8727},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9714,"spearman":0.8909,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9578,"pearson":0.3779,"spearman":0.3803,"kendall_tau_b":0.2606},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9578,"pearson":0.3703,"spearman":0.3611,"kendall_tau_b":0.248},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9578,"pearson":0.3695,"spearman":0.3604,"kendall_tau_b":0.2475},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9578,"pearson":0.6396,"spearman":0.6177,"kendall_tau_b":0.4413},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9578,"pearson":0.3613,"spearman":0.3655,"kendall_tau_b":0.2506},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9578,"pearson":0.3547,"spearman":0.3477,"kendall_tau_b":0.2387},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9578,"pearson":0.3537,"spearman":0.3468,"kendall_tau_b":0.238},{"testset":"wmt20","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9578,"pearson":0.5945,"spearman":0.5768,"kendall_tau_b":0.4084},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9426,"spearman":0.7363,"kendall_tau_b":0.5824,"pairwise_accuracy":0.7912},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9571,"spearman":0.7319,"kendall_tau_b":0.6044,"pairwise_accuracy":0.8022},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.956,"spearman":0.7319,"kendall_tau_b":0.6044,"pairwise_accuracy":0.8022},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9812,"spearman":0.8681,"kendall_tau_b":0.7363,"pairwise_accuracy":0.8681},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.917,"spearman":0.7538,"kendall_tau_b":0.6044,"pairwise_accuracy":0.8022},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9314,"spearman":0.767,"kendall_tau_b":0.6264,"pairwise_accuracy":0.8132},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9302,"spearman":0.767,"kendall_tau_b":0.6264,"pairwise_accuracy":0.8132},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9757,"spearman":0.833,"kendall_tau_b":0.6703,"pairwise_accuracy":0.8352},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":10574,"pearson":0.2767,"spearman":0.2752,"kendall_tau_b":0.1867},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":10574,"pearson":0.3435,"spearman":0.3353,"kendall_tau_b":0.2296},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":10574,"pearson":0.3449,"spearman":0.334,"kendall_tau_b":0.2288},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":10574,"pearson":0.6095,"spearman":0.5877,"kendall_tau_b":0.4182},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":10574,"pearson":0.2443,"spearman":0.2447,"kendall_tau_b":0.1662},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":10574,"pearson":0.3059,"spearman":0.2973,"kendall_tau_b":0.2037},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":10574,"pearson":0.3063,"spearman":0.2954,"kendall_tau_b":0.2024},{"testset":"wmt20","pair":"en-pl","pair_verbatim":"en-pl","src":"en","tgt":"pl","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":10574,"pearson":0.5495,"spearman":0.5211,"kendall_tau_b":0.3654},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":9,"pearson":0.9801,"spearman":0.95,"kendall_tau_b":0.8889,"pairwise_accuracy":0.9444},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":9,"pearson":0.9816,"spearman":0.9667,"kendall_tau_b":0.8889,"pairwise_accuracy":0.9444},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":9,"pearson":0.9829,"spearman":0.9667,"kendall_tau_b":0.8889,"pairwise_accuracy":0.9444},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":9,"pearson":0.9251,"spearman":0.8833,"kendall_tau_b":0.7222,"pairwise_accuracy":0.8611},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":9,"pearson":0.9623,"spearman":0.9,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":9,"pearson":0.9527,"spearman":0.95,"kendall_tau_b":0.8889,"pairwise_accuracy":0.9444},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":9,"pearson":0.9608,"spearman":0.95,"kendall_tau_b":0.8889,"pairwise_accuracy":0.9444},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":9,"pearson":0.8378,"spearman":0.8167,"kendall_tau_b":0.7222,"pairwise_accuracy":0.8611},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":9,"n_points":11228,"pearson":0.2443,"spearman":0.2377,"kendall_tau_b":0.1598},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":9,"n_points":11228,"pearson":0.2734,"spearman":0.2677,"kendall_tau_b":0.1813},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":9,"n_points":11228,"pearson":0.2767,"spearman":0.2682,"kendall_tau_b":0.1817},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":9,"n_points":11228,"pearson":0.5372,"spearman":0.5281,"kendall_tau_b":0.3681},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":9,"n_points":11228,"pearson":0.1779,"spearman":0.1877,"kendall_tau_b":0.1264},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":9,"n_points":11228,"pearson":0.196,"spearman":0.2091,"kendall_tau_b":0.1413},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":9,"n_points":11228,"pearson":0.1993,"spearman":0.2106,"kendall_tau_b":0.1422},{"testset":"wmt20","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":9,"n_points":11228,"pearson":0.3913,"spearman":0.4136,"kendall_tau_b":0.2836},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.8801,"spearman":0.9286,"kendall_tau_b":0.8286,"pairwise_accuracy":0.9143},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.9367,"spearman":0.9357,"kendall_tau_b":0.8095,"pairwise_accuracy":0.9048},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.9288,"spearman":0.9393,"kendall_tau_b":0.8286,"pairwise_accuracy":0.9143},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.9443,"spearman":0.9036,"kendall_tau_b":0.7714,"pairwise_accuracy":0.8857},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.8617,"spearman":0.9321,"kendall_tau_b":0.8286,"pairwise_accuracy":0.9143},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.9312,"spearman":0.9393,"kendall_tau_b":0.8095,"pairwise_accuracy":0.9048},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.9234,"spearman":0.9429,"kendall_tau_b":0.8286,"pairwise_accuracy":0.9143},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":15,"pearson":0.9358,"spearman":0.8821,"kendall_tau_b":0.7714,"pairwise_accuracy":0.8857},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":7891,"pearson":0.2685,"spearman":0.3479,"kendall_tau_b":0.2361},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":7891,"pearson":0.4883,"spearman":0.458,"kendall_tau_b":0.3192},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":7891,"pearson":0.4791,"spearman":0.4539,"kendall_tau_b":0.316},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":7891,"pearson":0.6601,"spearman":0.6338,"kendall_tau_b":0.4531},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":7891,"pearson":0.2378,"spearman":0.3141,"kendall_tau_b":0.2134},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":7891,"pearson":0.4287,"spearman":0.4071,"kendall_tau_b":0.2822},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":7891,"pearson":0.4208,"spearman":0.4032,"kendall_tau_b":0.2792},{"testset":"wmt20","pair":"en-ta","pair_verbatim":"en-ta","src":"en","tgt":"ta","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":15,"n_points":7891,"pearson":0.5919,"spearman":0.5722,"kendall_tau_b":0.4048},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9283,"spearman":0.8741,"kendall_tau_b":0.7273,"pairwise_accuracy":0.8636},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9228,"spearman":0.8741,"kendall_tau_b":0.7273,"pairwise_accuracy":0.8636},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8778,"spearman":0.8741,"kendall_tau_b":0.7273,"pairwise_accuracy":0.8636},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.0066,"spearman":0.6923,"kendall_tau_b":0.6061,"pairwise_accuracy":0.803},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9337,"spearman":0.9441,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.9285,"spearman":0.9441,"kendall_tau_b":0.8485,"pairwise_accuracy":0.9242},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.8791,"spearman":0.9231,"kendall_tau_b":0.7879,"pairwise_accuracy":0.8939},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"pearson":0.0432,"spearman":0.7413,"kendall_tau_b":0.7273,"pairwise_accuracy":0.8636},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14985,"pearson":0.3404,"spearman":0.3325,"kendall_tau_b":0.2262},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14985,"pearson":0.3311,"spearman":0.3298,"kendall_tau_b":0.2241},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14985,"pearson":0.3235,"spearman":0.3235,"kendall_tau_b":0.2198},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14985,"pearson":0.3275,"spearman":0.4357,"kendall_tau_b":0.3045},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14985,"pearson":0.2878,"spearman":0.2769,"kendall_tau_b":0.1876},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14985,"pearson":0.2807,"spearman":0.2757,"kendall_tau_b":0.1866},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14985,"pearson":0.2735,"spearman":0.2698,"kendall_tau_b":0.1825},{"testset":"wmt20","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":12,"n_points":14985,"pearson":0.2747,"spearman":0.3599,"kendall_tau_b":0.2475},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.5688,"spearman":0.3909,"kendall_tau_b":0.3091,"pairwise_accuracy":0.6545},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.7292,"spearman":0.6091,"kendall_tau_b":0.4182,"pairwise_accuracy":0.7091},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.7263,"spearman":0.6091,"kendall_tau_b":0.4182,"pairwise_accuracy":0.7091},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.852,"spearman":0.8182,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.5247,"spearman":0.2455,"kendall_tau_b":0.2,"pairwise_accuracy":0.6},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.6738,"spearman":0.4455,"kendall_tau_b":0.3091,"pairwise_accuracy":0.6545},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.6637,"spearman":0.4455,"kendall_tau_b":0.3091,"pairwise_accuracy":0.6545},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8246,"spearman":0.7091,"kendall_tau_b":0.5273,"pairwise_accuracy":0.7636},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":12097,"pearson":0.4044,"spearman":0.421,"kendall_tau_b":0.2928},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":12097,"pearson":0.4465,"spearman":0.4497,"kendall_tau_b":0.3158},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":12097,"pearson":0.4437,"spearman":0.4488,"kendall_tau_b":0.3149},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":12097,"pearson":0.4767,"spearman":0.4962,"kendall_tau_b":0.3503},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":12097,"pearson":0.3681,"spearman":0.4031,"kendall_tau_b":0.2825},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":12097,"pearson":0.4128,"spearman":0.4341,"kendall_tau_b":0.3074},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":12097,"pearson":0.4099,"spearman":0.4323,"kendall_tau_b":0.3058},{"testset":"wmt20","pair":"iu-en","pair_verbatim":"iu-en","src":"iu","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":12097,"pearson":0.4474,"spearman":0.4773,"kendall_tau_b":0.3394},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.9685,"spearman":0.9273,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.9677,"spearman":0.9394,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.9736,"spearman":0.9273,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.9644,"spearman":0.9273,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.9725,"spearman":0.9273,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.9612,"spearman":0.8061,"kendall_tau_b":0.6889,"pairwise_accuracy":0.8444},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.9657,"spearman":0.8303,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":10,"pearson":0.9683,"spearman":0.8667,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":8940,"pearson":0.2276,"spearman":0.2445,"kendall_tau_b":0.1657},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":8940,"pearson":0.295,"spearman":0.2852,"kendall_tau_b":0.195},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":8940,"pearson":0.2954,"spearman":0.2868,"kendall_tau_b":0.1961},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":8940,"pearson":0.3505,"spearman":0.3381,"kendall_tau_b":0.2324},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":8940,"pearson":0.2086,"spearman":0.238,"kendall_tau_b":0.1619},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":8940,"pearson":0.2813,"spearman":0.2826,"kendall_tau_b":0.1935},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":8940,"pearson":0.2822,"spearman":0.2841,"kendall_tau_b":0.1946},{"testset":"wmt20","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":10,"n_points":8940,"pearson":0.3308,"spearman":0.3264,"kendall_tau_b":0.2246},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.969,"spearman":0.8571,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.9775,"spearman":0.8214,"kendall_tau_b":0.619,"pairwise_accuracy":0.8095},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.9827,"spearman":0.8214,"kendall_tau_b":0.619,"pairwise_accuracy":0.8095},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.9711,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.9801,"spearman":0.9286,"kendall_tau_b":0.8095,"pairwise_accuracy":0.9048},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.9821,"spearman":0.8571,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.9877,"spearman":0.8571,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":7,"pearson":0.9743,"spearman":0.9643,"kendall_tau_b":0.9048,"pairwise_accuracy":0.9524},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":6659,"pearson":0.3262,"spearman":0.3528,"kendall_tau_b":0.2405},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":6659,"pearson":0.3991,"spearman":0.4039,"kendall_tau_b":0.2783},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":6659,"pearson":0.396,"spearman":0.4012,"kendall_tau_b":0.2763},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":6659,"pearson":0.4697,"spearman":0.4742,"kendall_tau_b":0.3305},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":6659,"pearson":0.3077,"spearman":0.3307,"kendall_tau_b":0.2262},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":6659,"pearson":0.3832,"spearman":0.3776,"kendall_tau_b":0.2599},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":6659,"pearson":0.3803,"spearman":0.3754,"kendall_tau_b":0.2583},{"testset":"wmt20","pair":"km-en","pair_verbatim":"km-en","src":"km","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":7,"n_points":6659,"pearson":0.4561,"spearman":0.4491,"kendall_tau_b":0.3115},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.5491,"spearman":0.4637,"kendall_tau_b":0.4066,"pairwise_accuracy":0.7033},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.5281,"spearman":0.4066,"kendall_tau_b":0.3626,"pairwise_accuracy":0.6813},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.538,"spearman":0.4813,"kendall_tau_b":0.4066,"pairwise_accuracy":0.7033},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.5914,"spearman":0.5209,"kendall_tau_b":0.4066,"pairwise_accuracy":0.7033},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.2421,"spearman":0.2791,"kendall_tau_b":0.1429,"pairwise_accuracy":0.5714},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.2448,"spearman":0.2264,"kendall_tau_b":0.0989,"pairwise_accuracy":0.5495},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.252,"spearman":0.2703,"kendall_tau_b":0.1429,"pairwise_accuracy":0.5714},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.4013,"spearman":0.433,"kendall_tau_b":0.2747,"pairwise_accuracy":0.6374},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":11822,"pearson":0.1925,"spearman":0.1995,"kendall_tau_b":0.135},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":11822,"pearson":0.219,"spearman":0.2205,"kendall_tau_b":0.1497},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":11822,"pearson":0.2196,"spearman":0.2224,"kendall_tau_b":0.151},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":11822,"pearson":0.2723,"spearman":0.2852,"kendall_tau_b":0.195},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":11822,"pearson":0.1536,"spearman":0.1755,"kendall_tau_b":0.1193},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":11822,"pearson":0.1846,"spearman":0.201,"kendall_tau_b":0.1371},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":11822,"pearson":0.1843,"spearman":0.2011,"kendall_tau_b":0.1371},{"testset":"wmt20","pair":"pl-en","pair_verbatim":"pl-en","src":"pl","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":11822,"pearson":0.2484,"spearman":0.2702,"kendall_tau_b":0.1851},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":6,"pearson":0.8881,"spearman":0.7714,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":6,"pearson":0.8976,"spearman":0.8286,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":6,"pearson":0.8996,"spearman":0.8286,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":6,"pearson":0.9414,"spearman":0.9429,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":6,"pearson":0.8514,"spearman":0.7714,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":6,"pearson":0.8542,"spearman":0.8286,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":6,"pearson":0.8577,"spearman":0.8286,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":6,"pearson":0.9248,"spearman":0.9429,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":6,"n_points":7138,"pearson":0.2387,"spearman":0.2364,"kendall_tau_b":0.1598},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":6,"n_points":7138,"pearson":0.2829,"spearman":0.273,"kendall_tau_b":0.1857},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":6,"n_points":7138,"pearson":0.2813,"spearman":0.2723,"kendall_tau_b":0.1852},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":6,"n_points":7138,"pearson":0.3143,"spearman":0.3172,"kendall_tau_b":0.2175},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":6,"n_points":7138,"pearson":0.2075,"spearman":0.2044,"kendall_tau_b":0.1384},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":6,"n_points":7138,"pearson":0.2496,"spearman":0.2375,"kendall_tau_b":0.1618},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":6,"n_points":7138,"pearson":0.2485,"spearman":0.2378,"kendall_tau_b":0.1619},{"testset":"wmt20","pair":"ps-en","pair_verbatim":"ps-en","src":"ps","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":6,"n_points":7138,"pearson":0.2936,"spearman":0.2844,"kendall_tau_b":0.1947},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8843,"spearman":0.6,"kendall_tau_b":0.4545,"pairwise_accuracy":0.7273},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8899,"spearman":0.8091,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8943,"spearman":0.7727,"kendall_tau_b":0.6727,"pairwise_accuracy":0.8364},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.9227,"spearman":0.7182,"kendall_tau_b":0.5636,"pairwise_accuracy":0.7818},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8434,"spearman":0.5636,"kendall_tau_b":0.4182,"pairwise_accuracy":0.7091},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8589,"spearman":0.6636,"kendall_tau_b":0.5273,"pairwise_accuracy":0.7636},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8643,"spearman":0.6727,"kendall_tau_b":0.5636,"pairwise_accuracy":0.7818},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"pearson":0.8533,"spearman":0.6182,"kendall_tau_b":0.4545,"pairwise_accuracy":0.7273},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9863,"pearson":0.209,"spearman":0.2117,"kendall_tau_b":0.1431},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9863,"pearson":0.2347,"spearman":0.227,"kendall_tau_b":0.1538},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9863,"pearson":0.236,"spearman":0.229,"kendall_tau_b":0.1551},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9863,"pearson":0.3046,"spearman":0.2992,"kendall_tau_b":0.2049},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9863,"pearson":0.1731,"spearman":0.1863,"kendall_tau_b":0.1274},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9863,"pearson":0.1941,"spearman":0.1936,"kendall_tau_b":0.1329},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9863,"pearson":0.1954,"spearman":0.195,"kendall_tau_b":0.1338},{"testset":"wmt20","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":11,"n_points":9863,"pearson":0.2812,"spearman":0.2641,"kendall_tau_b":0.1823},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9158,"spearman":0.8505,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9512,"spearman":0.8637,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9531,"spearman":0.8637,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.8802,"spearman":0.8198,"kendall_tau_b":0.6264,"pairwise_accuracy":0.8132},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.8792,"spearman":0.8066,"kendall_tau_b":0.6044,"pairwise_accuracy":0.8022},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9085,"spearman":0.8637,"kendall_tau_b":0.6703,"pairwise_accuracy":0.8352},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.9105,"spearman":0.8637,"kendall_tau_b":0.6703,"pairwise_accuracy":0.8352},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"pearson":0.8614,"spearman":0.8813,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":7578,"pearson":0.2192,"spearman":0.2591,"kendall_tau_b":0.1755},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":7578,"pearson":0.2839,"spearman":0.2821,"kendall_tau_b":0.1929},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":7578,"pearson":0.2842,"spearman":0.2873,"kendall_tau_b":0.1966},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":7578,"pearson":0.3517,"spearman":0.3573,"kendall_tau_b":0.2452},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":7578,"pearson":0.2008,"spearman":0.2347,"kendall_tau_b":0.1597},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":7578,"pearson":0.2545,"spearman":0.2526,"kendall_tau_b":0.1725},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":7578,"pearson":0.2549,"spearman":0.2564,"kendall_tau_b":0.1751},{"testset":"wmt20","pair":"ta-en","pair_verbatim":"ta-en","src":"ta","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":14,"n_points":7578,"pearson":0.3323,"spearman":0.3366,"kendall_tau_b":0.2316},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":8,"pearson":0.36,"spearman":0.5714,"kendall_tau_b":0.3571,"pairwise_accuracy":0.6786},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":8,"pearson":0.5226,"spearman":0.3571,"kendall_tau_b":0.2143,"pairwise_accuracy":0.6071},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":8,"pearson":0.5642,"spearman":0.4286,"kendall_tau_b":0.2857,"pairwise_accuracy":0.6429},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":8,"pearson":0.8888,"spearman":0.8333,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.9559,"spearman":0.9324,"kendall_tau_b":0.8333,"pairwise_accuracy":0.9167},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.9761,"spearman":0.9324,"kendall_tau_b":0.8333,"pairwise_accuracy":0.9167},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.9746,"spearman":0.9471,"kendall_tau_b":0.85,"pairwise_accuracy":0.925},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.9519,"spearman":0.8676,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.8718,"spearman":0.6853,"kendall_tau_b":0.4833,"pairwise_accuracy":0.7417},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.8858,"spearman":0.7647,"kendall_tau_b":0.5833,"pairwise_accuracy":0.7917},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.8858,"spearman":0.7706,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":16,"pearson":0.8506,"spearman":0.6265,"kendall_tau_b":0.4833,"pairwise_accuracy":0.7417},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":8,"n_points":16000,"pearson":0.2409,"spearman":0.2561,"kendall_tau_b":0.1735},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":8,"n_points":16000,"pearson":0.2427,"spearman":0.2879,"kendall_tau_b":0.1952},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":8,"n_points":16000,"pearson":0.2464,"spearman":0.29,"kendall_tau_b":0.1968},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":8,"n_points":16000,"pearson":0.4991,"spearman":0.5959,"kendall_tau_b":0.421},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":27537,"pearson":0.2423,"spearman":0.2599,"kendall_tau_b":0.1768},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":27537,"pearson":0.2819,"spearman":0.2842,"kendall_tau_b":0.194},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":27537,"pearson":0.2808,"spearman":0.2844,"kendall_tau_b":0.1941},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":27537,"pearson":0.3144,"spearman":0.3238,"kendall_tau_b":0.2216},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":27537,"pearson":0.2073,"spearman":0.2275,"kendall_tau_b":0.1552},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":27537,"pearson":0.2434,"spearman":0.2492,"kendall_tau_b":0.1706},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":27537,"pearson":0.2424,"spearman":0.2496,"kendall_tau_b":0.1709},{"testset":"wmt20","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"ref","computed":false,"n_sys":16,"n_points":27537,"pearson":0.2827,"spearman":0.2883,"kendall_tau_b":0.1979},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9007,"spearman":0.7667,"kendall_tau_b":0.6111,"pairwise_accuracy":0.8056},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9413,"spearman":0.8908,"kendall_tau_b":0.8003,"pairwise_accuracy":0.8611},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.9355,"spearman":0.8667,"kendall_tau_b":0.7222,"pairwise_accuracy":0.8611},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9458,"spearman":0.9167,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.8684,"spearman":0.5667,"kendall_tau_b":0.4444,"pairwise_accuracy":0.7222},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9071,"spearman":0.7479,"kendall_tau_b":0.6288,"pairwise_accuracy":0.7778},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.8997,"spearman":0.7,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9162,"spearman":0.8833,"kendall_tau_b":0.7222,"pairwise_accuracy":0.8611},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4461,"pearson":0.0843,"spearman":0.0905,"kendall_tau_b":0.0621},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4461,"pearson":0.1089,"spearman":0.1022,"kendall_tau_b":0.0702},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":4461,"pearson":0.1072,"spearman":0.1005,"kendall_tau_b":0.0691},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4461,"pearson":0.189,"spearman":0.1916,"kendall_tau_b":0.1323},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4461,"pearson":0.0901,"spearman":0.082,"kendall_tau_b":0.0563},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4461,"pearson":0.1125,"spearman":0.0969,"kendall_tau_b":0.0669},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":4461,"pearson":0.1109,"spearman":0.0949,"kendall_tau_b":0.0655},{"testset":"wmt21.flores","pair":"bn-hi","pair_verbatim":"bn-hi","src":"bn","tgt":"hi","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4461,"pearson":0.1692,"spearman":0.1551,"kendall_tau_b":0.1075},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.784,"spearman":0.2833,"kendall_tau_b":0.1667,"pairwise_accuracy":0.5833},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.872,"spearman":0.2833,"kendall_tau_b":0.1667,"pairwise_accuracy":0.5833},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.876,"spearman":0.2833,"kendall_tau_b":0.1667,"pairwise_accuracy":0.5833},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9645,"spearman":0.5,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.7834,"spearman":0.2333,"kendall_tau_b":0.1111,"pairwise_accuracy":0.5556},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.871,"spearman":0.2333,"kendall_tau_b":0.1111,"pairwise_accuracy":0.5556},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.875,"spearman":0.2333,"kendall_tau_b":0.1111,"pairwise_accuracy":0.5556},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9638,"spearman":0.4333,"kendall_tau_b":0.2778,"pairwise_accuracy":0.6389},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4512,"pearson":0.1115,"spearman":0.1137,"kendall_tau_b":0.0829},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4512,"pearson":0.175,"spearman":0.1291,"kendall_tau_b":0.0946},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":4512,"pearson":0.1691,"spearman":0.1258,"kendall_tau_b":0.0921},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4512,"pearson":0.3397,"spearman":0.2523,"kendall_tau_b":0.1857},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4512,"pearson":0.1112,"spearman":0.1263,"kendall_tau_b":0.0952},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4512,"pearson":0.1758,"spearman":0.1508,"kendall_tau_b":0.1137},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":4512,"pearson":0.1699,"spearman":0.1466,"kendall_tau_b":0.1105},{"testset":"wmt21.flores","pair":"hi-bn","pair_verbatim":"hi-bn","src":"hi","tgt":"bn","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":4512,"pearson":0.3429,"spearman":0.2883,"kendall_tau_b":0.2182},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":6,"pearson":0.9794,"spearman":0.9429,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":6,"pearson":0.9976,"spearman":0.9429,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":6,"pearson":0.9957,"spearman":0.9429,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":6,"pearson":0.9988,"spearman":0.8857,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":6,"pearson":0.9737,"spearman":0.8857,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":6,"pearson":0.9959,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":6,"pearson":0.9934,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":6,"pearson":0.9995,"spearman":0.9429,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":6,"n_points":2968,"pearson":0.2202,"spearman":0.2483,"kendall_tau_b":0.1706},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":6,"n_points":2968,"pearson":0.3313,"spearman":0.3271,"kendall_tau_b":0.2271},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":6,"n_points":2968,"pearson":0.3226,"spearman":0.3236,"kendall_tau_b":0.2249},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":6,"n_points":2968,"pearson":0.3363,"spearman":0.3371,"kendall_tau_b":0.2358},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":6,"n_points":2968,"pearson":0.185,"spearman":0.2118,"kendall_tau_b":0.1463},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":6,"n_points":2968,"pearson":0.2749,"spearman":0.2763,"kendall_tau_b":0.1928},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":6,"n_points":2968,"pearson":0.2669,"spearman":0.273,"kendall_tau_b":0.1909},{"testset":"wmt21.flores","pair":"xh-zu","pair_verbatim":"xh-zu","src":"xh","tgt":"zu","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":6,"n_points":2968,"pearson":0.2803,"spearman":0.277,"kendall_tau_b":0.1928},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9027,"spearman":0.7,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9986,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"pearson":0.9959,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9958,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9027,"spearman":0.7,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9986,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"pearson":0.9959,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9958,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":2502,"pearson":0.2548,"spearman":0.306,"kendall_tau_b":0.2118},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":2502,"pearson":0.413,"spearman":0.3968,"kendall_tau_b":0.2772},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"n_points":2502,"pearson":0.3997,"spearman":0.3912,"kendall_tau_b":0.2735},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":2502,"pearson":0.5421,"spearman":0.5116,"kendall_tau_b":0.366},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":2502,"pearson":0.2548,"spearman":0.306,"kendall_tau_b":0.2118},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":2502,"pearson":0.413,"spearman":0.3968,"kendall_tau_b":0.2772},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"n_points":2502,"pearson":0.3997,"spearman":0.3912,"kendall_tau_b":0.2735},{"testset":"wmt21.flores","pair":"zu-xh","pair_verbatim":"zu-xh","src":"zu","tgt":"xh","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":2502,"pearson":0.5421,"spearman":0.5116,"kendall_tau_b":0.366},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.5503,"spearman":0.5476,"kendall_tau_b":0.4286,"pairwise_accuracy":0.7143},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.5618,"spearman":0.619,"kendall_tau_b":0.5,"pairwise_accuracy":0.75},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"pearson":0.5782,"spearman":0.619,"kendall_tau_b":0.5,"pairwise_accuracy":0.75},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.6264,"spearman":0.6905,"kendall_tau_b":0.5714,"pairwise_accuracy":0.7857},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.1994,"spearman":0.0952,"kendall_tau_b":0.0714,"pairwise_accuracy":0.5357},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.2021,"spearman":0.1667,"kendall_tau_b":0.1429,"pairwise_accuracy":0.5714},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"pearson":0.2004,"spearman":0.1667,"kendall_tau_b":0.1429,"pairwise_accuracy":0.5714},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.3406,"spearman":0.4524,"kendall_tau_b":0.3571,"pairwise_accuracy":0.6786},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":7242,"pearson":0.0091,"spearman":-0.0028,"kendall_tau_b":-0.0019},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":7242,"pearson":-0.0055,"spearman":-0.0087,"kendall_tau_b":-0.0057},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"n_points":7242,"pearson":-0.0032,"spearman":-0.0065,"kendall_tau_b":-0.0043},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":7242,"pearson":-0.0183,"spearman":-0.0108,"kendall_tau_b":-0.0072},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":7242,"pearson":-0.0092,"spearman":-0.0163,"kendall_tau_b":-0.011},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":7242,"pearson":-0.0246,"spearman":-0.0226,"kendall_tau_b":-0.0151},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"n_points":7242,"pearson":-0.0236,"spearman":-0.0227,"kendall_tau_b":-0.0152},{"testset":"wmt21.news","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":7242,"pearson":-0.0197,"spearman":-0.0137,"kendall_tau_b":-0.0092},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.2305,"spearman":0.2246,"kendall_tau_b":0.193,"pairwise_accuracy":0.5965},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.3568,"spearman":0.3224,"kendall_tau_b":0.2502,"pairwise_accuracy":0.5789},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":19,"pearson":0.3559,"spearman":0.2912,"kendall_tau_b":0.2281,"pairwise_accuracy":0.614},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.3541,"spearman":0.2456,"kendall_tau_b":0.1813,"pairwise_accuracy":0.5906},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":-0.1259,"spearman":-0.0579,"kendall_tau_b":-0.076,"pairwise_accuracy":0.462},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.0323,"spearman":0.0983,"kendall_tau_b":0.0305,"pairwise_accuracy":0.4737},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":19,"pearson":0.0323,"spearman":0.0404,"kendall_tau_b":0.0058,"pairwise_accuracy":0.5029},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.1373,"spearman":0.1667,"kendall_tau_b":0.0994,"pairwise_accuracy":0.5497},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":18409,"pearson":0.0114,"spearman":0.0176,"kendall_tau_b":0.0118},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":18409,"pearson":0.0141,"spearman":0.0222,"kendall_tau_b":0.0149},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":19,"n_points":18409,"pearson":0.0146,"spearman":0.0237,"kendall_tau_b":0.0159},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":18409,"pearson":-0.0147,"spearman":-0.0112,"kendall_tau_b":-0.0075},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":18409,"pearson":0.0025,"spearman":0.0056,"kendall_tau_b":0.0038},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":18409,"pearson":0.0081,"spearman":0.012,"kendall_tau_b":0.0081},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":19,"n_points":18409,"pearson":0.0082,"spearman":0.0129,"kendall_tau_b":0.0087},{"testset":"wmt21.news","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":18409,"pearson":-0.0148,"spearman":-0.0127,"kendall_tau_b":-0.0085},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.4373,"spearman":0.7,"kendall_tau_b":0.5,"pairwise_accuracy":0.75},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.4977,"spearman":0.6667,"kendall_tau_b":0.4444,"pairwise_accuracy":0.7222},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.4866,"spearman":0.65,"kendall_tau_b":0.3889,"pairwise_accuracy":0.6944},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.8338,"spearman":0.8333,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.2175,"spearman":0.6167,"kendall_tau_b":0.5,"pairwise_accuracy":0.75},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.3364,"spearman":0.5667,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.3134,"spearman":0.5833,"kendall_tau_b":0.3889,"pairwise_accuracy":0.6944},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.7462,"spearman":0.7667,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8258,"pearson":0.1572,"spearman":0.1579,"kendall_tau_b":0.1065},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8258,"pearson":0.2089,"spearman":0.19,"kendall_tau_b":0.1287},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":8258,"pearson":0.2091,"spearman":0.1916,"kendall_tau_b":0.1297},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8258,"pearson":0.3597,"spearman":0.3335,"kendall_tau_b":0.2286},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8258,"pearson":0.1146,"spearman":0.098,"kendall_tau_b":0.0661},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8258,"pearson":0.1463,"spearman":0.1172,"kendall_tau_b":0.0794},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":8258,"pearson":0.1464,"spearman":0.1186,"kendall_tau_b":0.0803},{"testset":"wmt21.news","pair":"de-fr","pair_verbatim":"de-fr","src":"de","tgt":"fr","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8258,"pearson":0.2516,"spearman":0.2108,"kendall_tau_b":0.1435},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9727,"spearman":0.9515,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9702,"spearman":0.997,"kendall_tau_b":0.9888,"pairwise_accuracy":0.9778},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.9691,"spearman":0.9879,"kendall_tau_b":0.9556,"pairwise_accuracy":0.9778},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9752,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.972,"spearman":0.9152,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.972,"spearman":0.9787,"kendall_tau_b":0.9439,"pairwise_accuracy":0.9556},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.971,"spearman":0.9636,"kendall_tau_b":0.9111,"pairwise_accuracy":0.9556},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9767,"spearman":0.9879,"kendall_tau_b":0.9556,"pairwise_accuracy":0.9778},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":10006,"pearson":0.2396,"spearman":0.2402,"kendall_tau_b":0.1628},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":10006,"pearson":0.2942,"spearman":0.2812,"kendall_tau_b":0.1916},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":10006,"pearson":0.2941,"spearman":0.2815,"kendall_tau_b":0.1921},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":10006,"pearson":0.6545,"spearman":0.5936,"kendall_tau_b":0.4217},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":10006,"pearson":0.2372,"spearman":0.2344,"kendall_tau_b":0.1588},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":10006,"pearson":0.2862,"spearman":0.2707,"kendall_tau_b":0.184},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":10006,"pearson":0.2872,"spearman":0.2719,"kendall_tau_b":0.1851},{"testset":"wmt21.news","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":10006,"pearson":0.6135,"spearman":0.5417,"kendall_tau_b":0.3798},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9146,"spearman":0.9231,"kendall_tau_b":0.7949,"pairwise_accuracy":0.8974},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8572,"spearman":0.8407,"kendall_tau_b":0.641,"pairwise_accuracy":0.8205},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":0.8342,"spearman":0.9011,"kendall_tau_b":0.7436,"pairwise_accuracy":0.8718},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8461,"spearman":0.8571,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.7542,"spearman":0.6812,"kendall_tau_b":0.5158,"pairwise_accuracy":0.7579},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.7687,"spearman":0.6526,"kendall_tau_b":0.4947,"pairwise_accuracy":0.7474},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":20,"pearson":0.8186,"spearman":0.7083,"kendall_tau_b":0.5368,"pairwise_accuracy":0.7684},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.8767,"spearman":0.7865,"kendall_tau_b":0.6316,"pairwise_accuracy":0.8158},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.731,"spearman":0.7173,"kendall_tau_b":0.5474,"pairwise_accuracy":0.7737},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.7527,"spearman":0.6992,"kendall_tau_b":0.5474,"pairwise_accuracy":0.7737},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":20,"pearson":0.7891,"spearman":0.7218,"kendall_tau_b":0.5684,"pairwise_accuracy":0.7842},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.869,"spearman":0.7504,"kendall_tau_b":0.5789,"pairwise_accuracy":0.7895},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6851,"pearson":0.0952,"spearman":0.11,"kendall_tau_b":0.0828},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6851,"pearson":0.0999,"spearman":0.1355,"kendall_tau_b":0.1017},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":6851,"pearson":0.0999,"spearman":0.1339,"kendall_tau_b":0.1005},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6851,"pearson":0.2958,"spearman":0.3234,"kendall_tau_b":0.2462},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":12988,"pearson":0.1194,"spearman":0.1168,"kendall_tau_b":0.0785},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":12988,"pearson":0.1483,"spearman":0.1387,"kendall_tau_b":0.0937},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":20,"n_points":12988,"pearson":0.1493,"spearman":0.1378,"kendall_tau_b":0.093},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-z","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":12988,"pearson":0.3226,"spearman":0.2834,"kendall_tau_b":0.1935},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":12988,"pearson":0.0891,"spearman":0.0759,"kendall_tau_b":0.0516},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":12988,"pearson":0.1177,"spearman":0.0976,"kendall_tau_b":0.0665},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":20,"n_points":12988,"pearson":0.1171,"spearman":0.096,"kendall_tau_b":0.0654},{"testset":"wmt21.news","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":12988,"pearson":0.2586,"spearman":0.1979,"kendall_tau_b":0.1351},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.7794,"spearman":0.7714,"kendall_tau_b":0.5604,"pairwise_accuracy":0.7802},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.7485,"spearman":0.7713,"kendall_tau_b":0.5732,"pairwise_accuracy":0.7582},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.7636,"spearman":0.789,"kendall_tau_b":0.6044,"pairwise_accuracy":0.8022},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.8641,"spearman":0.8945,"kendall_tau_b":0.7363,"pairwise_accuracy":0.8681},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.7553,"spearman":0.7319,"kendall_tau_b":0.5165,"pairwise_accuracy":0.7582},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.7274,"spearman":0.7138,"kendall_tau_b":0.5057,"pairwise_accuracy":0.7253},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.7299,"spearman":0.7231,"kendall_tau_b":0.5165,"pairwise_accuracy":0.7582},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.794,"spearman":0.7846,"kendall_tau_b":0.6044,"pairwise_accuracy":0.8022},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":10812,"pearson":0.1796,"spearman":0.1844,"kendall_tau_b":0.124},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":10812,"pearson":0.269,"spearman":0.2428,"kendall_tau_b":0.1645},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":10812,"pearson":0.2665,"spearman":0.2417,"kendall_tau_b":0.1638},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":10812,"pearson":0.3614,"spearman":0.333,"kendall_tau_b":0.2276},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":10812,"pearson":0.1575,"spearman":0.1494,"kendall_tau_b":0.1012},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":10812,"pearson":0.2377,"spearman":0.1956,"kendall_tau_b":0.1329},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":10812,"pearson":0.2352,"spearman":0.1952,"kendall_tau_b":0.1326},{"testset":"wmt21.news","pair":"en-ha","pair_verbatim":"en-ha","src":"en","tgt":"ha","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":10812,"pearson":0.314,"spearman":0.2623,"kendall_tau_b":0.1792},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9556,"spearman":0.8636,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9538,"spearman":0.8455,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"pearson":0.9533,"spearman":0.8455,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9493,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9521,"spearman":0.9727,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9489,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"pearson":0.9474,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9276,"spearman":0.8636,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":10838,"pearson":0.3133,"spearman":0.3099,"kendall_tau_b":0.2109},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":10838,"pearson":0.3885,"spearman":0.3813,"kendall_tau_b":0.2612},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"n_points":10838,"pearson":0.3837,"spearman":0.3754,"kendall_tau_b":0.257},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":10838,"pearson":0.6279,"spearman":0.6251,"kendall_tau_b":0.4428},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":10838,"pearson":0.2626,"spearman":0.2588,"kendall_tau_b":0.1758},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":10838,"pearson":0.3216,"spearman":0.3143,"kendall_tau_b":0.214},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"n_points":10838,"pearson":0.3181,"spearman":0.3103,"kendall_tau_b":0.2111},{"testset":"wmt21.news","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":10838,"pearson":0.527,"spearman":0.5168,"kendall_tau_b":0.3596},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":-0.6075,"spearman":-0.5929,"kendall_tau_b":-0.4286,"pairwise_accuracy":0.2857},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.9662,"spearman":0.9286,"kendall_tau_b":0.8286,"pairwise_accuracy":0.9143},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"pearson":0.9046,"spearman":0.925,"kendall_tau_b":0.8095,"pairwise_accuracy":0.9048},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.9808,"spearman":0.9393,"kendall_tau_b":0.8095,"pairwise_accuracy":0.9048},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":-0.5954,"spearman":-0.5,"kendall_tau_b":-0.3714,"pairwise_accuracy":0.3143},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.9564,"spearman":0.9536,"kendall_tau_b":0.8476,"pairwise_accuracy":0.9238},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"pearson":0.8846,"spearman":0.925,"kendall_tau_b":0.7905,"pairwise_accuracy":0.8952},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.9821,"spearman":0.975,"kendall_tau_b":0.9048,"pairwise_accuracy":0.9524},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":14856,"pearson":-0.054,"spearman":-0.0879,"kendall_tau_b":-0.0705},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":14856,"pearson":0.287,"spearman":0.2726,"kendall_tau_b":0.1848},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"n_points":14856,"pearson":0.288,"spearman":0.274,"kendall_tau_b":0.1858},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":14856,"pearson":0.5528,"spearman":0.522,"kendall_tau_b":0.3647},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":14856,"pearson":-0.0476,"spearman":-0.0685,"kendall_tau_b":-0.0551},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":14856,"pearson":0.2622,"spearman":0.2464,"kendall_tau_b":0.1671},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"n_points":14856,"pearson":0.2627,"spearman":0.2475,"kendall_tau_b":0.1678},{"testset":"wmt21.news","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":14856,"pearson":0.4956,"spearman":0.4474,"kendall_tau_b":0.3092},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.5071,"spearman":0.3451,"kendall_tau_b":0.2308,"pairwise_accuracy":0.6154},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.7829,"spearman":0.644,"kendall_tau_b":0.4945,"pairwise_accuracy":0.7473},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.6882,"spearman":0.6044,"kendall_tau_b":0.4066,"pairwise_accuracy":0.7033},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.6548,"spearman":0.5648,"kendall_tau_b":0.3626,"pairwise_accuracy":0.6813},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9577,"spearman":0.8667,"kendall_tau_b":0.7222,"pairwise_accuracy":0.8611},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9573,"spearman":0.9833,"kendall_tau_b":0.9444,"pairwise_accuracy":0.9722},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.9571,"spearman":0.9667,"kendall_tau_b":0.8889,"pairwise_accuracy":0.9444},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.982,"spearman":0.9833,"kendall_tau_b":0.9444,"pairwise_accuracy":0.9722},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.959,"spearman":0.8833,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9274,"spearman":0.9667,"kendall_tau_b":0.8889,"pairwise_accuracy":0.9444},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.9572,"spearman":0.9833,"kendall_tau_b":0.9444,"pairwise_accuracy":0.9722},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9637,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7378,"pearson":0.1669,"spearman":0.1575,"kendall_tau_b":0.1201},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7378,"pearson":0.2702,"spearman":0.2514,"kendall_tau_b":0.1932},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":7378,"pearson":0.2553,"spearman":0.2405,"kendall_tau_b":0.1846},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7378,"pearson":0.3695,"spearman":0.3992,"kendall_tau_b":0.3072},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8485,"pearson":0.2149,"spearman":0.2085,"kendall_tau_b":0.1402},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8485,"pearson":0.2759,"spearman":0.2742,"kendall_tau_b":0.1857},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":8485,"pearson":0.2725,"spearman":0.2697,"kendall_tau_b":0.1824},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-z","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8485,"pearson":0.4264,"spearman":0.4853,"kendall_tau_b":0.3375},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8485,"pearson":0.1901,"spearman":0.1799,"kendall_tau_b":0.1213},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8485,"pearson":0.2387,"spearman":0.2378,"kendall_tau_b":0.1612},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":8485,"pearson":0.2364,"spearman":0.2328,"kendall_tau_b":0.1576},{"testset":"wmt21.news","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":8485,"pearson":0.3785,"spearman":0.4235,"kendall_tau_b":0.2918},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":29,"pearson":-0.1237,"spearman":-0.0094,"kendall_tau_b":0.0246,"pairwise_accuracy":0.5123},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":29,"pearson":0.5492,"spearman":0.6737,"kendall_tau_b":0.5167,"pairwise_accuracy":0.7537},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":29,"pearson":0.3902,"spearman":0.4512,"kendall_tau_b":0.3695,"pairwise_accuracy":0.6847},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":29,"pearson":0.9343,"spearman":0.8754,"kendall_tau_b":0.734,"pairwise_accuracy":0.867},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":29,"pearson":-0.1191,"spearman":-0.0547,"kendall_tau_b":-0.0493,"pairwise_accuracy":0.4754},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":29,"pearson":0.3658,"spearman":0.4096,"kendall_tau_b":0.2843,"pairwise_accuracy":0.6379},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":29,"pearson":0.251,"spearman":0.2281,"kendall_tau_b":0.1675,"pairwise_accuracy":0.5837},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":29,"pearson":0.8008,"spearman":0.6739,"kendall_tau_b":0.5222,"pairwise_accuracy":0.7611},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":29,"n_points":27005,"pearson":-0.0217,"spearman":-0.0663,"kendall_tau_b":-0.0519},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":29,"n_points":27005,"pearson":0.1733,"spearman":0.1698,"kendall_tau_b":0.1147},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":29,"n_points":27005,"pearson":0.1757,"spearman":0.1739,"kendall_tau_b":0.1176},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":29,"n_points":27005,"pearson":0.4353,"spearman":0.3957,"kendall_tau_b":0.2725},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":29,"n_points":27005,"pearson":-0.0082,"spearman":-0.0454,"kendall_tau_b":-0.0357},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":29,"n_points":27005,"pearson":0.147,"spearman":0.1429,"kendall_tau_b":0.0963},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":29,"n_points":27005,"pearson":0.1496,"spearman":0.146,"kendall_tau_b":0.0984},{"testset":"wmt21.news","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":29,"n_points":27005,"pearson":0.3542,"spearman":0.3039,"kendall_tau_b":0.207},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.6532,"spearman":0.8333,"kendall_tau_b":0.7222,"pairwise_accuracy":0.8611},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.6461,"spearman":0.8833,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.6509,"spearman":0.8833,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.7877,"spearman":0.9,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.5613,"spearman":0.7667,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.5728,"spearman":0.7333,"kendall_tau_b":0.6111,"pairwise_accuracy":0.8056},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.5628,"spearman":0.7333,"kendall_tau_b":0.6111,"pairwise_accuracy":0.8056},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.4785,"spearman":0.2833,"kendall_tau_b":0.2778,"pairwise_accuracy":0.6389},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":7196,"pearson":0.136,"spearman":0.0969,"kendall_tau_b":0.0653},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":7196,"pearson":0.1803,"spearman":0.1262,"kendall_tau_b":0.0854},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":7196,"pearson":0.1778,"spearman":0.1257,"kendall_tau_b":0.085},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":7196,"pearson":0.314,"spearman":0.2514,"kendall_tau_b":0.1715},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":7196,"pearson":0.129,"spearman":0.1018,"kendall_tau_b":0.0699},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":7196,"pearson":0.1477,"spearman":0.1023,"kendall_tau_b":0.0703},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":7196,"pearson":0.1466,"spearman":0.1035,"kendall_tau_b":0.0711},{"testset":"wmt21.news","pair":"fr-de","pair_verbatim":"fr-de","src":"fr","tgt":"de","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":7196,"pearson":0.2287,"spearman":0.17,"kendall_tau_b":0.1172},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9356,"spearman":0.9385,"kendall_tau_b":0.8022,"pairwise_accuracy":0.9011},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9236,"spearman":0.9141,"kendall_tau_b":0.7667,"pairwise_accuracy":0.8681},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.9294,"spearman":0.9253,"kendall_tau_b":0.8022,"pairwise_accuracy":0.9011},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9385,"spearman":0.9516,"kendall_tau_b":0.8462,"pairwise_accuracy":0.9231},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.82,"spearman":0.8242,"kendall_tau_b":0.6264,"pairwise_accuracy":0.8132},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.8096,"spearman":0.8128,"kendall_tau_b":0.6556,"pairwise_accuracy":0.8132},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.8133,"spearman":0.8066,"kendall_tau_b":0.6264,"pairwise_accuracy":0.8132},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.8815,"spearman":0.8681,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":13171,"pearson":0.0446,"spearman":0.0453,"kendall_tau_b":0.0302},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":13171,"pearson":0.0422,"spearman":0.0483,"kendall_tau_b":0.0323},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":13171,"pearson":0.0443,"spearman":0.0496,"kendall_tau_b":0.0331},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":13171,"pearson":0.0425,"spearman":0.0466,"kendall_tau_b":0.0311},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":13171,"pearson":0.0385,"spearman":0.0375,"kendall_tau_b":0.0253},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":13171,"pearson":0.0376,"spearman":0.0402,"kendall_tau_b":0.027},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":13171,"pearson":0.0393,"spearman":0.0414,"kendall_tau_b":0.0278},{"testset":"wmt21.news","pair":"ha-en","pair_verbatim":"ha-en","src":"ha","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":13171,"pearson":0.0329,"spearman":0.0379,"kendall_tau_b":0.0254},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8585,"spearman":0.9394,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8539,"spearman":0.8997,"kendall_tau_b":0.809,"pairwise_accuracy":0.8889},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.8579,"spearman":0.9394,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8658,"spearman":0.9636,"kendall_tau_b":0.9111,"pairwise_accuracy":0.9556},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.7612,"spearman":0.4788,"kendall_tau_b":0.2889,"pairwise_accuracy":0.6444},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.7705,"spearman":0.3465,"kendall_tau_b":0.1798,"pairwise_accuracy":0.5778},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.7694,"spearman":0.4061,"kendall_tau_b":0.2444,"pairwise_accuracy":0.6222},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.7746,"spearman":0.3818,"kendall_tau_b":0.2,"pairwise_accuracy":0.6},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8903,"pearson":0.0486,"spearman":0.0473,"kendall_tau_b":0.0317},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8903,"pearson":0.0523,"spearman":0.0541,"kendall_tau_b":0.0362},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":8903,"pearson":0.0528,"spearman":0.0547,"kendall_tau_b":0.0366},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8903,"pearson":0.0576,"spearman":0.0591,"kendall_tau_b":0.0396},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8903,"pearson":0.0433,"spearman":0.0413,"kendall_tau_b":0.0278},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8903,"pearson":0.0371,"spearman":0.0384,"kendall_tau_b":0.0258},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":8903,"pearson":0.039,"spearman":0.0406,"kendall_tau_b":0.0272},{"testset":"wmt21.news","pair":"is-en","pair_verbatim":"is-en","src":"is","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8903,"pearson":0.0461,"spearman":0.0449,"kendall_tau_b":0.0302},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.7867,"spearman":0.6475,"kendall_tau_b":0.5105,"pairwise_accuracy":0.75},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.8309,"spearman":0.6819,"kendall_tau_b":0.5378,"pairwise_accuracy":0.7583},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":16,"pearson":0.8297,"spearman":0.677,"kendall_tau_b":0.5105,"pairwise_accuracy":0.75},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.8321,"spearman":0.7476,"kendall_tau_b":0.5607,"pairwise_accuracy":0.775},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.4347,"spearman":0.3444,"kendall_tau_b":0.2427,"pairwise_accuracy":0.6167},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.4814,"spearman":0.3918,"kendall_tau_b":0.2857,"pairwise_accuracy":0.6333},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":16,"pearson":0.4874,"spearman":0.3885,"kendall_tau_b":0.2762,"pairwise_accuracy":0.6333},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.5944,"spearman":0.5475,"kendall_tau_b":0.3933,"pairwise_accuracy":0.6917},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":14482,"pearson":0.0262,"spearman":0.0151,"kendall_tau_b":0.0101},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":14482,"pearson":0.0306,"spearman":0.0229,"kendall_tau_b":0.0153},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":16,"n_points":14482,"pearson":0.031,"spearman":0.0237,"kendall_tau_b":0.0158},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":14482,"pearson":0.0305,"spearman":0.0241,"kendall_tau_b":0.0161},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":14482,"pearson":0.0131,"spearman":0.0098,"kendall_tau_b":0.0066},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":14482,"pearson":0.0114,"spearman":0.0157,"kendall_tau_b":0.0106},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":16,"n_points":14482,"pearson":0.0124,"spearman":0.0167,"kendall_tau_b":0.0112},{"testset":"wmt21.news","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":14482,"pearson":0.0092,"spearman":0.0135,"kendall_tau_b":0.0091},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.5123,"spearman":0.1758,"kendall_tau_b":0.0222,"pairwise_accuracy":0.5111},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.5928,"spearman":0.3587,"kendall_tau_b":0.2697,"pairwise_accuracy":0.6222},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.5905,"spearman":0.3212,"kendall_tau_b":0.2,"pairwise_accuracy":0.6},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.7108,"spearman":0.4061,"kendall_tau_b":0.2444,"pairwise_accuracy":0.6222},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":-0.0242,"spearman":-0.1758,"kendall_tau_b":-0.2,"pairwise_accuracy":0.4},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.0672,"spearman":0.0547,"kendall_tau_b":0.0449,"pairwise_accuracy":0.5111},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.0606,"spearman":0.0061,"kendall_tau_b":-0.0222,"pairwise_accuracy":0.4889},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.177,"spearman":0.103,"kendall_tau_b":0.0222,"pairwise_accuracy":0.5111},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8926,"pearson":0.0055,"spearman":0.0107,"kendall_tau_b":0.0071},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8926,"pearson":0.0053,"spearman":0.0218,"kendall_tau_b":0.0147},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":8926,"pearson":0.0048,"spearman":0.0208,"kendall_tau_b":0.014},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8926,"pearson":-0.0026,"spearman":0.005,"kendall_tau_b":0.0035},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8926,"pearson":-0.0194,"spearman":-0.0049,"kendall_tau_b":-0.0032},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8926,"pearson":-0.0194,"spearman":0.0053,"kendall_tau_b":0.0037},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":8926,"pearson":-0.0201,"spearman":0.0047,"kendall_tau_b":0.0032},{"testset":"wmt21.news","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8926,"pearson":-0.0159,"spearman":-0.0045,"kendall_tau_b":-0.0029},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.6216,"spearman":0.4615,"kendall_tau_b":0.359,"pairwise_accuracy":0.6795},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.4953,"spearman":0.3297,"kendall_tau_b":0.2308,"pairwise_accuracy":0.6154},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":0.3129,"spearman":0.2637,"kendall_tau_b":0.2051,"pairwise_accuracy":0.6026},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.6175,"spearman":0.4286,"kendall_tau_b":0.3077,"pairwise_accuracy":0.6538},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":23,"pearson":0.6529,"spearman":0.4901,"kendall_tau_b":0.3597,"pairwise_accuracy":0.6798},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":23,"pearson":0.6972,"spearman":0.5514,"kendall_tau_b":0.4387,"pairwise_accuracy":0.7194},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":23,"pearson":0.7229,"spearman":0.5949,"kendall_tau_b":0.4625,"pairwise_accuracy":0.7312},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":23,"pearson":0.7313,"spearman":0.6334,"kendall_tau_b":0.4783,"pairwise_accuracy":0.7391},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":23,"pearson":0.2622,"spearman":0.0217,"kendall_tau_b":0.0198,"pairwise_accuracy":0.5099},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":23,"pearson":0.3808,"spearman":0.1907,"kendall_tau_b":0.1462,"pairwise_accuracy":0.5731},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":23,"pearson":0.4203,"spearman":0.2292,"kendall_tau_b":0.17,"pairwise_accuracy":0.585},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":23,"pearson":0.3547,"spearman":0.0642,"kendall_tau_b":0.0909,"pairwise_accuracy":0.5455},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8450,"pearson":0.2092,"spearman":0.2333,"kendall_tau_b":0.1648},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8450,"pearson":0.2087,"spearman":0.2413,"kendall_tau_b":0.1708},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":8450,"pearson":0.2136,"spearman":0.2472,"kendall_tau_b":0.1748},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8450,"pearson":0.421,"spearman":0.4809,"kendall_tau_b":0.3485},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":23,"n_points":36128,"pearson":0.0057,"spearman":0.0061,"kendall_tau_b":0.0041},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":23,"n_points":36128,"pearson":0.0081,"spearman":0.0116,"kendall_tau_b":0.0077},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":23,"n_points":36128,"pearson":0.0066,"spearman":0.0098,"kendall_tau_b":0.0065},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":23,"n_points":36128,"pearson":0.0004,"spearman":0.0022,"kendall_tau_b":0.0015},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":23,"n_points":36128,"pearson":0.0095,"spearman":0.0045,"kendall_tau_b":0.003},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":23,"n_points":36128,"pearson":0.0106,"spearman":0.01,"kendall_tau_b":0.0067},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":23,"n_points":36128,"pearson":0.0096,"spearman":0.008,"kendall_tau_b":0.0054},{"testset":"wmt21.news","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-raw","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":23,"n_points":36128,"pearson":0.0131,"spearman":0.0048,"kendall_tau_b":0.0033},{"testset":"wmt21.tedtalks","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.6201,"spearman":0.5275,"kendall_tau_b":0.3846,"pairwise_accuracy":0.6923},{"testset":"wmt21.tedtalks","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.4709,"spearman":0.4011,"kendall_tau_b":0.2821,"pairwise_accuracy":0.641},{"testset":"wmt21.tedtalks","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":0.4819,"spearman":0.5659,"kendall_tau_b":0.4359,"pairwise_accuracy":0.7179},{"testset":"wmt21.tedtalks","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.7801,"spearman":0.7857,"kendall_tau_b":0.6154,"pairwise_accuracy":0.8077},{"testset":"wmt21.tedtalks","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6877,"pearson":0.1515,"spearman":0.1479,"kendall_tau_b":0.1133},{"testset":"wmt21.tedtalks","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6877,"pearson":0.158,"spearman":0.192,"kendall_tau_b":0.1465},{"testset":"wmt21.tedtalks","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":6877,"pearson":0.165,"spearman":0.195,"kendall_tau_b":0.1489},{"testset":"wmt21.tedtalks","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6877,"pearson":0.3079,"spearman":0.347,"kendall_tau_b":0.2672},{"testset":"wmt21.tedtalks","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.8285,"spearman":0.8505,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt21.tedtalks","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.8248,"spearman":0.8505,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt21.tedtalks","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.8576,"spearman":0.8374,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt21.tedtalks","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.8715,"spearman":0.8198,"kendall_tau_b":0.6484,"pairwise_accuracy":0.8242},{"testset":"wmt21.tedtalks","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7168,"pearson":0.1539,"spearman":0.1505,"kendall_tau_b":0.1122},{"testset":"wmt21.tedtalks","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7168,"pearson":0.2679,"spearman":0.248,"kendall_tau_b":0.1888},{"testset":"wmt21.tedtalks","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":7168,"pearson":0.2649,"spearman":0.2483,"kendall_tau_b":0.1888},{"testset":"wmt21.tedtalks","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7168,"pearson":0.3161,"spearman":0.3647,"kendall_tau_b":0.2741},{"testset":"wmt21.tedtalks","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":-0.3694,"spearman":-0.3571,"kendall_tau_b":-0.359,"pairwise_accuracy":0.3205},{"testset":"wmt21.tedtalks","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":-0.3263,"spearman":-0.2033,"kendall_tau_b":-0.2051,"pairwise_accuracy":0.3974},{"testset":"wmt21.tedtalks","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":-0.3394,"spearman":-0.2582,"kendall_tau_b":-0.2564,"pairwise_accuracy":0.3718},{"testset":"wmt21.tedtalks","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":-0.2274,"spearman":-0.0879,"kendall_tau_b":-0.1026,"pairwise_accuracy":0.4487},{"testset":"wmt21.tedtalks","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"sentBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6877,"pearson":0.0971,"spearman":0.0809,"kendall_tau_b":0.0605},{"testset":"wmt21.tedtalks","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6877,"pearson":0.1107,"spearman":0.108,"kendall_tau_b":0.0814},{"testset":"wmt21.tedtalks","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":6877,"pearson":0.1111,"spearman":0.109,"kendall_tau_b":0.0821},{"testset":"wmt21.tedtalks","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-DA_2021","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6877,"pearson":0.1992,"spearman":0.2067,"kendall_tau_b":0.1561},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.9642,"spearman":0.8818,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.9619,"spearman":0.8818,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.9599,"spearman":0.8818,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refB","computed":true,"n_sys":11,"pearson":0.9608,"spearman":0.8818,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.9261,"spearman":0.9,"kendall_tau_b":0.7455,"pairwise_accuracy":0.8727},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.9299,"spearman":0.8636,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.8009,"spearman":0.4727,"kendall_tau_b":0.3091,"pairwise_accuracy":0.6545},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.8119,"spearman":0.4727,"kendall_tau_b":0.3091,"pairwise_accuracy":0.6545},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.8064,"spearman":0.4727,"kendall_tau_b":0.3091,"pairwise_accuracy":0.6545},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refB","computed":true,"n_sys":11,"pearson":0.8067,"spearman":0.4727,"kendall_tau_b":0.3091,"pairwise_accuracy":0.6545},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.8209,"spearman":0.5727,"kendall_tau_b":0.4182,"pairwise_accuracy":0.7091},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refB","computed":false,"n_sys":11,"pearson":0.8229,"spearman":0.5636,"kendall_tau_b":0.3818,"pairwise_accuracy":0.6909},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.0548,"spearman":0.0673,"kendall_tau_b":0.045},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.0536,"spearman":0.0654,"kendall_tau_b":0.0438},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.057,"spearman":0.0658,"kendall_tau_b":0.0441},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refB","computed":true,"n_sys":11,"n_points":14723,"pearson":0.0578,"spearman":0.0676,"kendall_tau_b":0.0453},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.0483,"spearman":0.054,"kendall_tau_b":0.0362},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.0499,"spearman":0.0449,"kendall_tau_b":0.03},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.0492,"spearman":0.0641,"kendall_tau_b":0.0431},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.0494,"spearman":0.0642,"kendall_tau_b":0.0431},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.0484,"spearman":0.0621,"kendall_tau_b":0.0417},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refB","computed":true,"n_sys":11,"n_points":14723,"pearson":0.0493,"spearman":0.0633,"kendall_tau_b":0.0425},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.0496,"spearman":0.0461,"kendall_tau_b":0.031},{"testset":"wmt22","pair":"cs-en","pair_verbatim":"cs-en","src":"cs","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refB","computed":false,"n_sys":11,"n_points":14723,"pearson":0.0458,"spearman":0.0382,"kendall_tau_b":0.0256},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9043,"spearman":0.7576,"kendall_tau_b":0.6444,"pairwise_accuracy":0.8222},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9084,"spearman":0.7697,"kendall_tau_b":0.6889,"pairwise_accuracy":0.8444},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9443,"spearman":0.7212,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.9417,"spearman":0.7333,"kendall_tau_b":0.6444,"pairwise_accuracy":0.8222},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9845,"spearman":0.9515,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9879,"spearman":0.9515,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.89,"spearman":0.7091,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8882,"spearman":0.7576,"kendall_tau_b":0.6444,"pairwise_accuracy":0.8222},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9269,"spearman":0.6848,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.9247,"spearman":0.7091,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9872,"spearman":0.9394,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9886,"spearman":0.9394,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11574,"pearson":0.2527,"spearman":0.2358,"kendall_tau_b":0.1593},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11574,"pearson":0.3007,"spearman":0.2757,"kendall_tau_b":0.1872},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11574,"pearson":0.3778,"spearman":0.3298,"kendall_tau_b":0.2249},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":11574,"pearson":0.3631,"spearman":0.3197,"kendall_tau_b":0.2178},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11574,"pearson":0.5799,"spearman":0.505,"kendall_tau_b":0.3523},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11574,"pearson":0.5472,"spearman":0.4795,"kendall_tau_b":0.332},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11594,"pearson":0.224,"spearman":0.1954,"kendall_tau_b":0.1332},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11594,"pearson":0.2698,"spearman":0.2341,"kendall_tau_b":0.1604},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11594,"pearson":0.3432,"spearman":0.284,"kendall_tau_b":0.1952},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":11594,"pearson":0.329,"spearman":0.2736,"kendall_tau_b":0.1879},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11594,"pearson":0.5405,"spearman":0.442,"kendall_tau_b":0.3092},{"testset":"wmt22","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":11594,"pearson":0.5149,"spearman":0.4245,"kendall_tau_b":0.2952},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.5326,"spearman":0.6333,"kendall_tau_b":0.5,"pairwise_accuracy":0.75},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.5585,"spearman":0.7167,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.5156,"spearman":0.5,"kendall_tau_b":0.3889,"pairwise_accuracy":0.6944},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.5223,"spearman":0.6167,"kendall_tau_b":0.4444,"pairwise_accuracy":0.7222},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.5733,"spearman":0.6,"kendall_tau_b":0.4444,"pairwise_accuracy":0.7222},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.5899,"spearman":0.6,"kendall_tau_b":0.4444,"pairwise_accuracy":0.7222},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.3524,"spearman":0.2167,"kendall_tau_b":0.2222,"pairwise_accuracy":0.6111},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.4045,"spearman":0.55,"kendall_tau_b":0.5,"pairwise_accuracy":0.75},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.3545,"spearman":0.0167,"kendall_tau_b":0.1111,"pairwise_accuracy":0.5556},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.3557,"spearman":0.1167,"kendall_tau_b":0.1667,"pairwise_accuracy":0.5833},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.4463,"spearman":0.3833,"kendall_tau_b":0.2778,"pairwise_accuracy":0.6389},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.4898,"spearman":0.5833,"kendall_tau_b":0.3889,"pairwise_accuracy":0.6944},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.0122,"spearman":0.0154,"kendall_tau_b":0.0104},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.0133,"spearman":0.0148,"kendall_tau_b":0.01},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.0148,"spearman":0.017,"kendall_tau_b":0.0114},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":15505,"pearson":0.0146,"spearman":0.0174,"kendall_tau_b":0.0117},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.0175,"spearman":0.0157,"kendall_tau_b":0.0105},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.0164,"spearman":0.0152,"kendall_tau_b":0.0101},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.0164,"spearman":0.0132,"kendall_tau_b":0.0089},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.0198,"spearman":0.0141,"kendall_tau_b":0.0095},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.028,"spearman":0.0253,"kendall_tau_b":0.017},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":15505,"pearson":0.0268,"spearman":0.0241,"kendall_tau_b":0.0163},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.0381,"spearman":0.0278,"kendall_tau_b":0.0187},{"testset":"wmt22","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":15505,"pearson":0.0349,"spearman":0.0212,"kendall_tau_b":0.0142},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.7219,"spearman":0.7939,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.6153,"spearman":0.697,"kendall_tau_b":0.5111,"pairwise_accuracy":0.7556},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.727,"spearman":0.7818,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refB","computed":true,"n_sys":10,"pearson":0.7364,"spearman":0.7455,"kendall_tau_b":0.5111,"pairwise_accuracy":0.7556},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.9461,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.9554,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.6664,"spearman":0.8182,"kendall_tau_b":0.6444,"pairwise_accuracy":0.8222},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.5492,"spearman":0.697,"kendall_tau_b":0.4667,"pairwise_accuracy":0.7333},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.6892,"spearman":0.7939,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refB","computed":true,"n_sys":10,"pearson":0.6922,"spearman":0.7697,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.8314,"spearman":0.9636,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refB","computed":false,"n_sys":10,"pearson":0.853,"spearman":0.9636,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.2296,"spearman":0.2238,"kendall_tau_b":0.1513},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.2325,"spearman":0.218,"kendall_tau_b":0.1484},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.2405,"spearman":0.234,"kendall_tau_b":0.1598},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refB","computed":true,"n_sys":10,"n_points":12050,"pearson":0.2402,"spearman":0.2314,"kendall_tau_b":0.1578},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.5499,"spearman":0.5066,"kendall_tau_b":0.3521},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.5177,"spearman":0.4932,"kendall_tau_b":0.3416},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.1906,"spearman":0.1973,"kendall_tau_b":0.1356},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.193,"spearman":0.1925,"kendall_tau_b":0.1332},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.202,"spearman":0.2119,"kendall_tau_b":0.1467},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refB","computed":true,"n_sys":10,"n_points":12050,"pearson":0.2006,"spearman":0.2083,"kendall_tau_b":0.144},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.4426,"spearman":0.4532,"kendall_tau_b":0.3173},{"testset":"wmt22","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refB","computed":false,"n_sys":10,"n_points":12050,"pearson":0.412,"spearman":0.4382,"kendall_tau_b":0.3062},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.5403,"spearman":0.6396,"kendall_tau_b":0.4945,"pairwise_accuracy":0.7473},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.6887,"spearman":0.6088,"kendall_tau_b":0.4505,"pairwise_accuracy":0.7253},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.7127,"spearman":0.6615,"kendall_tau_b":0.4945,"pairwise_accuracy":0.7473},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.6679,"spearman":0.6615,"kendall_tau_b":0.4945,"pairwise_accuracy":0.7473},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.916,"spearman":0.7495,"kendall_tau_b":0.6044,"pairwise_accuracy":0.8022},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9486,"spearman":0.833,"kendall_tau_b":0.6044,"pairwise_accuracy":0.8022},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.7293,"spearman":0.8,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9075,"spearman":0.9333,"kendall_tau_b":0.8333,"pairwise_accuracy":0.9167},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9457,"spearman":0.9833,"kendall_tau_b":0.9444,"pairwise_accuracy":0.9722},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.9635,"spearman":0.9833,"kendall_tau_b":0.9444,"pairwise_accuracy":0.9722},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.7933,"spearman":0.7667,"kendall_tau_b":0.5,"pairwise_accuracy":0.75},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.8622,"spearman":0.8833,"kendall_tau_b":0.7222,"pairwise_accuracy":0.8611},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.4933,"spearman":0.4667,"kendall_tau_b":0.3889,"pairwise_accuracy":0.6944},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.6562,"spearman":0.7833,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.8105,"spearman":0.7333,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.7789,"spearman":0.7333,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.593,"spearman":0.55,"kendall_tau_b":0.4444,"pairwise_accuracy":0.7222},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.7127,"spearman":0.75,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":18410,"pearson":0.2029,"spearman":0.2353,"kendall_tau_b":0.1775},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":18410,"pearson":0.2162,"spearman":0.2529,"kendall_tau_b":0.1906},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":18410,"pearson":0.2422,"spearman":0.2966,"kendall_tau_b":0.2237},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":18410,"pearson":0.2415,"spearman":0.2953,"kendall_tau_b":0.2227},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":18410,"pearson":0.5343,"spearman":0.4977,"kendall_tau_b":0.3834},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":18410,"pearson":0.5746,"spearman":0.4898,"kendall_tau_b":0.3779},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.1441,"spearman":0.1502,"kendall_tau_b":0.1006},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.1683,"spearman":0.1628,"kendall_tau_b":0.1091},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.2051,"spearman":0.2019,"kendall_tau_b":0.1359},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":10980,"pearson":0.2005,"spearman":0.1971,"kendall_tau_b":0.1325},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.3951,"spearman":0.3172,"kendall_tau_b":0.2146},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.407,"spearman":0.3349,"kendall_tau_b":0.2271},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.1236,"spearman":0.099,"kendall_tau_b":0.0694},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.1414,"spearman":0.0986,"kendall_tau_b":0.0693},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.1691,"spearman":0.1203,"kendall_tau_b":0.0847},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":10980,"pearson":0.1649,"spearman":0.1186,"kendall_tau_b":0.0834},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.3018,"spearman":0.1808,"kendall_tau_b":0.1273},{"testset":"wmt22","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":10980,"pearson":0.3106,"spearman":0.2025,"kendall_tau_b":0.143},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9577,"spearman":0.8571,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9461,"spearman":0.9286,"kendall_tau_b":0.7857,"pairwise_accuracy":0.8929},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9563,"spearman":0.9524,"kendall_tau_b":0.8571,"pairwise_accuracy":0.9286},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"pearson":0.9525,"spearman":0.9524,"kendall_tau_b":0.8571,"pairwise_accuracy":0.9286},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9102,"spearman":0.9048,"kendall_tau_b":0.7857,"pairwise_accuracy":0.8929},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9474,"spearman":0.9048,"kendall_tau_b":0.7857,"pairwise_accuracy":0.8929},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9194,"spearman":0.7857,"kendall_tau_b":0.6429,"pairwise_accuracy":0.8214},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9057,"spearman":0.8571,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9197,"spearman":0.9048,"kendall_tau_b":0.7857,"pairwise_accuracy":0.8929},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"pearson":0.9153,"spearman":0.9048,"kendall_tau_b":0.7857,"pairwise_accuracy":0.8929},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9395,"spearman":0.9524,"kendall_tau_b":0.8571,"pairwise_accuracy":0.9286},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9607,"spearman":0.9286,"kendall_tau_b":0.8571,"pairwise_accuracy":0.9286},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.2512,"spearman":0.2869,"kendall_tau_b":0.195},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.2704,"spearman":0.2879,"kendall_tau_b":0.1966},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.2969,"spearman":0.308,"kendall_tau_b":0.2116},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"n_points":9252,"pearson":0.2995,"spearman":0.3087,"kendall_tau_b":0.2118},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.5786,"spearman":0.5385,"kendall_tau_b":0.3772},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.5537,"spearman":0.5236,"kendall_tau_b":0.3663},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.2254,"spearman":0.2577,"kendall_tau_b":0.179},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.2423,"spearman":0.2523,"kendall_tau_b":0.1762},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.2619,"spearman":0.2634,"kendall_tau_b":0.185},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"n_points":9252,"pearson":0.264,"spearman":0.2657,"kendall_tau_b":0.1864},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.512,"spearman":0.4471,"kendall_tau_b":0.3156},{"testset":"wmt22","pair":"en-hr","pair_verbatim":"en-hr","src":"en","tgt":"hr","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9252,"pearson":0.4899,"spearman":0.439,"kendall_tau_b":0.3099},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.3363,"spearman":0.4176,"kendall_tau_b":0.2308,"pairwise_accuracy":0.6154},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8899,"spearman":0.6868,"kendall_tau_b":0.5385,"pairwise_accuracy":0.7692},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9547,"spearman":0.8956,"kendall_tau_b":0.7692,"pairwise_accuracy":0.8846},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":0.9231,"spearman":0.8297,"kendall_tau_b":0.641,"pairwise_accuracy":0.8205},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9227,"spearman":0.7747,"kendall_tau_b":0.641,"pairwise_accuracy":0.8205},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8959,"spearman":0.7143,"kendall_tau_b":0.5897,"pairwise_accuracy":0.7949},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.2817,"spearman":0.2527,"kendall_tau_b":0.2308,"pairwise_accuracy":0.6154},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8624,"spearman":0.6209,"kendall_tau_b":0.4872,"pairwise_accuracy":0.7436},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.931,"spearman":0.8407,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":0.9001,"spearman":0.8791,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9016,"spearman":0.8901,"kendall_tau_b":0.7436,"pairwise_accuracy":0.8718},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8853,"spearman":0.8846,"kendall_tau_b":0.7949,"pairwise_accuracy":0.8974},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.0498,"spearman":0.041,"kendall_tau_b":0.033},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.212,"spearman":0.2169,"kendall_tau_b":0.1457},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.2461,"spearman":0.2382,"kendall_tau_b":0.1617},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":15353,"pearson":0.2436,"spearman":0.2379,"kendall_tau_b":0.1615},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.4368,"spearman":0.4176,"kendall_tau_b":0.2859},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.4103,"spearman":0.435,"kendall_tau_b":0.2993},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.06,"spearman":0.0463,"kendall_tau_b":0.0376},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.2018,"spearman":0.1963,"kendall_tau_b":0.1332},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.2248,"spearman":0.2077,"kendall_tau_b":0.1423},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":15353,"pearson":0.2229,"spearman":0.2074,"kendall_tau_b":0.1421},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.362,"spearman":0.335,"kendall_tau_b":0.2295},{"testset":"wmt22","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":15353,"pearson":0.3446,"spearman":0.3513,"kendall_tau_b":0.2421},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.808,"spearman":0.7,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9878,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9692,"spearman":0.6,"kendall_tau_b":0.4,"pairwise_accuracy":0.7},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"pearson":0.9667,"spearman":0.6,"kendall_tau_b":0.4,"pairwise_accuracy":0.7},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9173,"spearman":0.6,"kendall_tau_b":0.4,"pairwise_accuracy":0.7},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9108,"spearman":0.6,"kendall_tau_b":0.4,"pairwise_accuracy":0.7},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.8041,"spearman":0.7,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9894,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9691,"spearman":0.6,"kendall_tau_b":0.4,"pairwise_accuracy":0.7},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"pearson":0.966,"spearman":0.6,"kendall_tau_b":0.4,"pairwise_accuracy":0.7},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9216,"spearman":0.6,"kendall_tau_b":0.4,"pairwise_accuracy":0.7},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9135,"spearman":0.6,"kendall_tau_b":0.4,"pairwise_accuracy":0.7},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1468,"pearson":0.0648,"spearman":0.0523,"kendall_tau_b":0.036},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1468,"pearson":0.1186,"spearman":0.1289,"kendall_tau_b":0.0886},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1468,"pearson":0.1388,"spearman":0.147,"kendall_tau_b":0.101},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"n_points":1468,"pearson":0.1311,"spearman":0.1402,"kendall_tau_b":0.0964},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1468,"pearson":0.0659,"spearman":0.1134,"kendall_tau_b":0.0782},{"testset":"wmt22","pair":"en-liv","pair_verbatim":"en-liv","src":"en","tgt":"liv","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1468,"pearson":0.0291,"spearman":0.0312,"kendall_tau_b":0.0213},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.7238,"spearman":0.7143,"kendall_tau_b":0.581,"pairwise_accuracy":0.7905},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8186,"spearman":0.8429,"kendall_tau_b":0.6952,"pairwise_accuracy":0.8476},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.815,"spearman":0.8143,"kendall_tau_b":0.6571,"pairwise_accuracy":0.8286},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"pearson":0.7944,"spearman":0.7929,"kendall_tau_b":0.6571,"pairwise_accuracy":0.8286},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8997,"spearman":0.8857,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.9488,"spearman":0.9214,"kendall_tau_b":0.8286,"pairwise_accuracy":0.9143},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.756,"spearman":0.6182,"kendall_tau_b":0.4545,"pairwise_accuracy":0.7273},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.7826,"spearman":0.7545,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.8688,"spearman":0.8636,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"pearson":0.8544,"spearman":0.8182,"kendall_tau_b":0.6727,"pairwise_accuracy":0.8364},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9608,"spearman":0.9636,"kendall_tau_b":0.8909,"pairwise_accuracy":0.9455},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9811,"spearman":0.9909,"kendall_tau_b":0.9636,"pairwise_accuracy":0.9818},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.6492,"spearman":0.5727,"kendall_tau_b":0.4182,"pairwise_accuracy":0.7091},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.6902,"spearman":0.7636,"kendall_tau_b":0.5636,"pairwise_accuracy":0.7818},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.8132,"spearman":0.8364,"kendall_tau_b":0.6727,"pairwise_accuracy":0.8364},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"pearson":0.7945,"spearman":0.7909,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9225,"spearman":0.9273,"kendall_tau_b":0.8545,"pairwise_accuracy":0.9273},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9627,"spearman":0.9818,"kendall_tau_b":0.9273,"pairwise_accuracy":0.9636},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":19725,"pearson":0.1601,"spearman":0.1927,"kendall_tau_b":0.1404},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":19725,"pearson":0.1715,"spearman":0.2092,"kendall_tau_b":0.1528},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":19725,"pearson":0.1682,"spearman":0.2296,"kendall_tau_b":0.1675},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"n_points":19725,"pearson":0.1648,"spearman":0.2247,"kendall_tau_b":0.1638},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":19725,"pearson":0.4685,"spearman":0.5338,"kendall_tau_b":0.3996},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":19725,"pearson":0.4955,"spearman":0.5601,"kendall_tau_b":0.4198},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.2127,"spearman":0.1904,"kendall_tau_b":0.1282},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.2374,"spearman":0.2061,"kendall_tau_b":0.1399},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.2606,"spearman":0.2347,"kendall_tau_b":0.1599},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"n_points":13156,"pearson":0.2565,"spearman":0.2278,"kendall_tau_b":0.1548},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.5303,"spearman":0.4773,"kendall_tau_b":0.3325},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.5139,"spearman":0.4895,"kendall_tau_b":0.3399},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.1993,"spearman":0.1813,"kendall_tau_b":0.1223},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.2197,"spearman":0.1954,"kendall_tau_b":0.1324},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.2418,"spearman":0.2246,"kendall_tau_b":0.1526},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"n_points":13156,"pearson":0.2368,"spearman":0.217,"kendall_tau_b":0.1472},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.4937,"spearman":0.4368,"kendall_tau_b":0.3031},{"testset":"wmt22","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":13156,"pearson":0.4849,"spearman":0.4518,"kendall_tau_b":0.3124},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.7689,"spearman":0.9048,"kendall_tau_b":0.7857,"pairwise_accuracy":0.8929},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.824,"spearman":0.9286,"kendall_tau_b":0.8571,"pairwise_accuracy":0.9286},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9132,"spearman":0.9048,"kendall_tau_b":0.7857,"pairwise_accuracy":0.8929},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"pearson":0.909,"spearman":0.9048,"kendall_tau_b":0.7857,"pairwise_accuracy":0.8929},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9418,"spearman":0.9524,"kendall_tau_b":0.8571,"pairwise_accuracy":0.9286},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9693,"spearman":0.9524,"kendall_tau_b":0.8571,"pairwise_accuracy":0.9286},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.7519,"spearman":0.8095,"kendall_tau_b":0.6429,"pairwise_accuracy":0.8214},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.8143,"spearman":0.8333,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.8948,"spearman":0.8095,"kendall_tau_b":0.6429,"pairwise_accuracy":0.8214},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"pearson":0.8916,"spearman":0.8095,"kendall_tau_b":0.6429,"pairwise_accuracy":0.8214},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9126,"spearman":0.881,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":8,"pearson":0.9389,"spearman":0.881,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.2455,"spearman":0.2363,"kendall_tau_b":0.1595},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.2635,"spearman":0.2437,"kendall_tau_b":0.1659},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.2976,"spearman":0.2841,"kendall_tau_b":0.1943},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"n_points":9672,"pearson":0.2909,"spearman":0.2743,"kendall_tau_b":0.1874},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.558,"spearman":0.543,"kendall_tau_b":0.3792},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.52,"spearman":0.5161,"kendall_tau_b":0.3591},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.2388,"spearman":0.2213,"kendall_tau_b":0.1497},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.2521,"spearman":0.2273,"kendall_tau_b":0.1546},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.2759,"spearman":0.2585,"kendall_tau_b":0.1769},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":8,"n_points":9672,"pearson":0.2708,"spearman":0.2511,"kendall_tau_b":0.1716},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.5104,"spearman":0.4735,"kendall_tau_b":0.328},{"testset":"wmt22","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":8,"n_points":9672,"pearson":0.4736,"spearman":0.4554,"kendall_tau_b":0.3152},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.1208,"spearman":0.0909,"kendall_tau_b":0.0909,"pairwise_accuracy":0.5455},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.3058,"spearman":0.1748,"kendall_tau_b":0.1515,"pairwise_accuracy":0.5758},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.3031,"spearman":0.1888,"kendall_tau_b":0.1212,"pairwise_accuracy":0.5606},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.2931,"spearman":0.3357,"kendall_tau_b":0.2121,"pairwise_accuracy":0.6061},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8461,"spearman":0.7622,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8751,"spearman":0.8252,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.0649,"spearman":0.1608,"kendall_tau_b":0.1515,"pairwise_accuracy":0.5758},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.2077,"spearman":0.2587,"kendall_tau_b":0.2121,"pairwise_accuracy":0.6061},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.2104,"spearman":0.2657,"kendall_tau_b":0.2424,"pairwise_accuracy":0.6212},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.203,"spearman":0.3287,"kendall_tau_b":0.2121,"pairwise_accuracy":0.6061},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.7655,"spearman":0.7483,"kendall_tau_b":0.5758,"pairwise_accuracy":0.7879},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8383,"spearman":0.7762,"kendall_tau_b":0.5758,"pairwise_accuracy":0.7879},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.0693,"spearman":0.0295,"kendall_tau_b":0.0233},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.1352,"spearman":0.0985,"kendall_tau_b":0.0672},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.1194,"spearman":0.0774,"kendall_tau_b":0.0529},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":14172,"pearson":0.1251,"spearman":0.0792,"kendall_tau_b":0.0542},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.3488,"spearman":0.3737,"kendall_tau_b":0.2557},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.3375,"spearman":0.402,"kendall_tau_b":0.2762},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.0558,"spearman":0.0401,"kendall_tau_b":0.0321},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.0945,"spearman":0.0739,"kendall_tau_b":0.0502},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.0991,"spearman":0.0747,"kendall_tau_b":0.0508},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":14172,"pearson":0.0994,"spearman":0.0746,"kendall_tau_b":0.0508},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.2365,"spearman":0.228,"kendall_tau_b":0.1555},{"testset":"wmt22","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":14172,"pearson":0.2237,"spearman":0.2281,"kendall_tau_b":0.156},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9517,"spearman":0.8022,"kendall_tau_b":0.6484,"pairwise_accuracy":0.8242},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9635,"spearman":0.7319,"kendall_tau_b":0.5385,"pairwise_accuracy":0.7692},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9907,"spearman":0.8418,"kendall_tau_b":0.6703,"pairwise_accuracy":0.8352},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.9897,"spearman":0.8505,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9873,"spearman":0.7187,"kendall_tau_b":0.5165,"pairwise_accuracy":0.7582},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.99,"spearman":0.7143,"kendall_tau_b":0.4945,"pairwise_accuracy":0.7473},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9342,"spearman":0.2527,"kendall_tau_b":0.1429,"pairwise_accuracy":0.5714},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9491,"spearman":0.2703,"kendall_tau_b":0.1648,"pairwise_accuracy":0.5824},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9828,"spearman":0.3231,"kendall_tau_b":0.2088,"pairwise_accuracy":0.6044},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.9807,"spearman":0.2967,"kendall_tau_b":0.1868,"pairwise_accuracy":0.5934},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9761,"spearman":0.1121,"kendall_tau_b":0.0549,"pairwise_accuracy":0.5275},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9782,"spearman":0.1209,"kendall_tau_b":0.0769,"pairwise_accuracy":0.5385},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.1008,"spearman":0.1368,"kendall_tau_b":0.0913},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.1112,"spearman":0.1435,"kendall_tau_b":0.0956},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.1714,"spearman":0.1417,"kendall_tau_b":0.0942},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":23326,"pearson":0.1723,"spearman":0.143,"kendall_tau_b":0.095},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.2305,"spearman":0.1367,"kendall_tau_b":0.0906},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.2471,"spearman":0.1365,"kendall_tau_b":0.0906},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.0944,"spearman":0.1207,"kendall_tau_b":0.081},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.1031,"spearman":0.1273,"kendall_tau_b":0.0852},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.1588,"spearman":0.1246,"kendall_tau_b":0.0832},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":23326,"pearson":0.1597,"spearman":0.1259,"kendall_tau_b":0.0841},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.2105,"spearman":0.118,"kendall_tau_b":0.0787},{"testset":"wmt22","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":23326,"pearson":0.2242,"spearman":0.1176,"kendall_tau_b":0.0786},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9785,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9792,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.989,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"pearson":0.9883,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9935,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9038,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9786,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9793,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9884,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"pearson":0.9877,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.9943,"spearman":1.0,"kendall_tau_b":1.0,"pairwise_accuracy":1.0},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":5,"pearson":0.907,"spearman":0.9,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1480,"pearson":0.1785,"spearman":0.2049,"kendall_tau_b":0.1386},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1480,"pearson":0.1954,"spearman":0.2182,"kendall_tau_b":0.1484},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1480,"pearson":0.2573,"spearman":0.271,"kendall_tau_b":0.184},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":5,"n_points":1480,"pearson":0.2571,"spearman":0.27,"kendall_tau_b":0.1837},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1480,"pearson":0.263,"spearman":0.2741,"kendall_tau_b":0.1864},{"testset":"wmt22","pair":"liv-en","pair_verbatim":"liv-en","src":"liv","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":5,"n_points":1480,"pearson":0.1667,"spearman":0.161,"kendall_tau_b":0.1083},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9703,"spearman":0.7939,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9697,"spearman":0.8182,"kendall_tau_b":0.6889,"pairwise_accuracy":0.8444},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9711,"spearman":0.8303,"kendall_tau_b":0.6444,"pairwise_accuracy":0.8222},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.9715,"spearman":0.8424,"kendall_tau_b":0.6889,"pairwise_accuracy":0.8444},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9487,"spearman":0.5636,"kendall_tau_b":0.4667,"pairwise_accuracy":0.7333},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9339,"spearman":0.3333,"kendall_tau_b":0.2889,"pairwise_accuracy":0.6444},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8425,"spearman":0.5152,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8314,"spearman":0.4424,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8273,"spearman":0.5394,"kendall_tau_b":0.3778,"pairwise_accuracy":0.6889},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.8288,"spearman":0.5152,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.857,"spearman":0.3455,"kendall_tau_b":0.2889,"pairwise_accuracy":0.6444},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8556,"spearman":0.3333,"kendall_tau_b":0.2889,"pairwise_accuracy":0.6444},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.0267,"spearman":0.0277,"kendall_tau_b":0.0185},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.0299,"spearman":0.0312,"kendall_tau_b":0.0209},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.029,"spearman":0.0283,"kendall_tau_b":0.0189},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":18776,"pearson":0.0294,"spearman":0.0288,"kendall_tau_b":0.0192},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.0304,"spearman":0.0187,"kendall_tau_b":0.0125},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.0364,"spearman":0.0177,"kendall_tau_b":0.0117},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.0198,"spearman":0.0207,"kendall_tau_b":0.0139},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.0238,"spearman":0.0275,"kendall_tau_b":0.0184},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.0191,"spearman":0.0224,"kendall_tau_b":0.0151},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":18776,"pearson":0.0196,"spearman":0.0225,"kendall_tau_b":0.0151},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.031,"spearman":0.0188,"kendall_tau_b":0.0126},{"testset":"wmt22","pair":"ru-en","pair_verbatim":"ru-en","src":"ru","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":18776,"pearson":0.0345,"spearman":0.0165,"kendall_tau_b":0.011},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.4705,"spearman":0.5155,"kendall_tau_b":0.352},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.5236,"spearman":0.5437,"kendall_tau_b":0.3729},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.5911,"spearman":0.5993,"kendall_tau_b":0.4186},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":2,"n_points":2146,"pearson":0.5893,"spearman":0.6013,"kendall_tau_b":0.4197},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.6463,"spearman":0.6528,"kendall_tau_b":0.4565},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.5412,"spearman":0.5468,"kendall_tau_b":0.3737},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.4761,"spearman":0.519,"kendall_tau_b":0.3605},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.5315,"spearman":0.5489,"kendall_tau_b":0.3828},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.6017,"spearman":0.6043,"kendall_tau_b":0.4295},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":2,"n_points":2146,"pearson":0.5984,"spearman":0.605,"kendall_tau_b":0.4295},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.6627,"spearman":0.6591,"kendall_tau_b":0.4703},{"testset":"wmt22","pair":"sah-ru","pair_verbatim":"sah-ru","src":"sah","tgt":"ru","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":2,"n_points":2146,"pearson":0.5588,"spearman":0.5511,"kendall_tau_b":0.3815},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9875,"spearman":0.8909,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9822,"spearman":0.8424,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9732,"spearman":0.8909,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.977,"spearman":0.8909,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9484,"spearman":0.8424,"kendall_tau_b":0.6889,"pairwise_accuracy":0.8444},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9463,"spearman":0.8545,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9822,"spearman":0.9515,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9743,"spearman":0.9152,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9789,"spearman":0.9515,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.9806,"spearman":0.9515,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9587,"spearman":0.9273,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.961,"spearman":0.9394,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8594,"pearson":0.2747,"spearman":0.2752,"kendall_tau_b":0.1862},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8594,"pearson":0.2915,"spearman":0.284,"kendall_tau_b":0.1929},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8594,"pearson":0.3106,"spearman":0.3019,"kendall_tau_b":0.2053},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":8594,"pearson":0.3105,"spearman":0.3017,"kendall_tau_b":0.2051},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8594,"pearson":0.5454,"spearman":0.5333,"kendall_tau_b":0.3721},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":8594,"pearson":0.5061,"spearman":0.4868,"kendall_tau_b":0.3376},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":3512,"pearson":0.233,"spearman":0.2167,"kendall_tau_b":0.1503},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":3512,"pearson":0.2574,"spearman":0.2312,"kendall_tau_b":0.1618},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":3512,"pearson":0.2822,"spearman":0.2441,"kendall_tau_b":0.1713},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":3512,"pearson":0.2799,"spearman":0.2424,"kendall_tau_b":0.17},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":3512,"pearson":0.5234,"spearman":0.4575,"kendall_tau_b":0.3264},{"testset":"wmt22","pair":"uk-cs","pair_verbatim":"uk-cs","src":"uk","tgt":"cs","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":3512,"pearson":0.4904,"spearman":0.4088,"kendall_tau_b":0.288},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.908,"spearman":0.6833,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.8895,"spearman":0.6167,"kendall_tau_b":0.5,"pairwise_accuracy":0.75},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.8977,"spearman":0.6833,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.8976,"spearman":0.6833,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9257,"spearman":0.85,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.9373,"spearman":0.8833,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.6484,"spearman":0.7,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.7139,"spearman":0.6,"kendall_tau_b":0.5,"pairwise_accuracy":0.75},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.6881,"spearman":0.7,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.6894,"spearman":0.7,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.7139,"spearman":0.6,"kendall_tau_b":0.4444,"pairwise_accuracy":0.7222},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"pearson":0.7155,"spearman":0.6333,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":0.0097,"spearman":0.0102,"kendall_tau_b":0.0068},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":0.0101,"spearman":0.0101,"kendall_tau_b":0.0067},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":0.0061,"spearman":0.0066,"kendall_tau_b":0.0044},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":16470,"pearson":0.0075,"spearman":0.0087,"kendall_tau_b":0.0058},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":0.0141,"spearman":0.0075,"kendall_tau_b":0.005},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt-z","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":0.0178,"spearman":0.0103,"kendall_tau_b":0.0068},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":0.0044,"spearman":0.0104,"kendall_tau_b":0.007},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":0.0035,"spearman":0.0094,"kendall_tau_b":0.0063},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":-0.0001,"spearman":0.004,"kendall_tau_b":0.0027},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":16470,"pearson":0.0016,"spearman":0.0056,"kendall_tau_b":0.0038},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":0.0048,"spearman":0.0025,"kendall_tau_b":0.0017},{"testset":"wmt22","pair":"uk-en","pair_verbatim":"uk-en","src":"uk","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":9,"n_points":16470,"pearson":0.0031,"spearman":0.0029,"kendall_tau_b":0.0019},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.5945,"spearman":0.3275,"kendall_tau_b":0.2308,"pairwise_accuracy":0.6154},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.7275,"spearman":0.4374,"kendall_tau_b":0.3407,"pairwise_accuracy":0.6703},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.6303,"spearman":0.3934,"kendall_tau_b":0.2967,"pairwise_accuracy":0.6484},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.6251,"spearman":0.3495,"kendall_tau_b":0.2747,"pairwise_accuracy":0.6374},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9418,"spearman":0.8593,"kendall_tau_b":0.7363,"pairwise_accuracy":0.8681},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.92,"spearman":0.8418,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.6399,"spearman":0.7343,"kendall_tau_b":0.5758,"pairwise_accuracy":0.7879},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.6028,"spearman":0.5944,"kendall_tau_b":0.4545,"pairwise_accuracy":0.7273},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.571,"spearman":0.6084,"kendall_tau_b":0.4545,"pairwise_accuracy":0.7273},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.582,"spearman":0.6084,"kendall_tau_b":0.4545,"pairwise_accuracy":0.7273},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.3817,"spearman":0.2098,"kendall_tau_b":0.1515,"pairwise_accuracy":0.5758},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.2658,"spearman":0.0979,"kendall_tau_b":0.0909,"pairwise_accuracy":0.5455},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.7957,"spearman":0.8112,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8657,"spearman":0.8531,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8063,"spearman":0.8182,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.8156,"spearman":0.8182,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8866,"spearman":0.8042,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8609,"spearman":0.7902,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8595,"spearman":0.8252,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8907,"spearman":0.8531,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.881,"spearman":0.8741,"kendall_tau_b":0.7576,"pairwise_accuracy":0.8788},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.8817,"spearman":0.8741,"kendall_tau_b":0.7576,"pairwise_accuracy":0.8788},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.893,"spearman":0.8252,"kendall_tau_b":0.5758,"pairwise_accuracy":0.7879},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8067,"spearman":0.7413,"kendall_tau_b":0.5152,"pairwise_accuracy":0.7576},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.5626,"spearman":0.6084,"kendall_tau_b":0.4242,"pairwise_accuracy":0.7121},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.5167,"spearman":0.6084,"kendall_tau_b":0.4242,"pairwise_accuracy":0.7121},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.568,"spearman":0.6294,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.5757,"spearman":0.6294,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.1349,"spearman":0.2657,"kendall_tau_b":0.2424,"pairwise_accuracy":0.6212},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":-0.0612,"spearman":0.0559,"kendall_tau_b":0.0606,"pairwise_accuracy":0.5303},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":26250,"pearson":0.1747,"spearman":0.1973,"kendall_tau_b":0.145},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":26250,"pearson":0.1719,"spearman":0.1907,"kendall_tau_b":0.14},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":26250,"pearson":0.1541,"spearman":0.2003,"kendall_tau_b":0.147},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":26250,"pearson":0.16,"spearman":0.2029,"kendall_tau_b":0.149},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":26250,"pearson":0.5846,"spearman":0.5635,"kendall_tau_b":0.4279},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":26250,"pearson":0.5812,"spearman":0.5581,"kendall_tau_b":0.427},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.0277,"spearman":0.0278,"kendall_tau_b":0.0186},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.0292,"spearman":0.0296,"kendall_tau_b":0.0197},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.0286,"spearman":0.0266,"kendall_tau_b":0.0177},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":19234,"pearson":0.0299,"spearman":0.0279,"kendall_tau_b":0.0186},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.007,"spearman":0.0093,"kendall_tau_b":0.0061},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-z","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.0072,"spearman":0.0085,"kendall_tau_b":0.0056},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.1785,"spearman":0.1437,"kendall_tau_b":0.0972},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.1727,"spearman":0.1286,"kendall_tau_b":0.0872},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.1954,"spearman":0.1344,"kendall_tau_b":0.0928},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":13416,"pearson":0.1977,"spearman":0.1387,"kendall_tau_b":0.0956},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.4187,"spearman":0.4642,"kendall_tau_b":0.3181},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise-z","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.3743,"spearman":0.4633,"kendall_tau_b":0.317},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.1413,"spearman":0.1133,"kendall_tau_b":0.0773},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.1364,"spearman":0.1008,"kendall_tau_b":0.0688},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.1524,"spearman":0.1031,"kendall_tau_b":0.0713},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":13416,"pearson":0.1545,"spearman":0.1061,"kendall_tau_b":0.0733},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.3636,"spearman":0.3931,"kendall_tau_b":0.2713},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt-appraise","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":13416,"pearson":0.3161,"spearman":0.3897,"kendall_tau_b":0.2676},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.0296,"spearman":0.036,"kendall_tau_b":0.0242},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.0336,"spearman":0.0392,"kendall_tau_b":0.0264},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.0354,"spearman":0.0375,"kendall_tau_b":0.0252},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":19234,"pearson":0.0362,"spearman":0.0386,"kendall_tau_b":0.0259},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.0012,"spearman":0.0032,"kendall_tau_b":0.0022},{"testset":"wmt22","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"wmt","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"metricx_xxl_MQM_2020","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":19234,"pearson":0.0,"spearman":-0.0041,"kendall_tau_b":-0.0027},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.5481,"spearman":0.511,"kendall_tau_b":0.3846,"pairwise_accuracy":0.6923},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.6765,"spearman":0.6044,"kendall_tau_b":0.4872,"pairwise_accuracy":0.7436},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.6369,"spearman":0.6044,"kendall_tau_b":0.4872,"pairwise_accuracy":0.7436},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":0.6222,"spearman":0.6099,"kendall_tau_b":0.5128,"pairwise_accuracy":0.7564},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8993,"spearman":0.8791,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8789,"spearman":0.8571,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":13117,"pearson":0.1836,"spearman":0.1754,"kendall_tau_b":0.119},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":13117,"pearson":0.2213,"spearman":0.199,"kendall_tau_b":0.1359},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":13117,"pearson":0.2727,"spearman":0.2452,"kendall_tau_b":0.1684},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":13117,"pearson":0.2629,"spearman":0.2369,"kendall_tau_b":0.1625},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":13117,"pearson":0.4542,"spearman":0.3985,"kendall_tau_b":0.277},{"testset":"wmt23","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"da-sqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":13117,"pearson":0.3951,"spearman":0.3927,"kendall_tau_b":0.2692},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8965,"spearman":0.9286,"kendall_tau_b":0.7949,"pairwise_accuracy":0.8974},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9038,"spearman":0.9066,"kendall_tau_b":0.7179,"pairwise_accuracy":0.859},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8601,"spearman":0.9066,"kendall_tau_b":0.7436,"pairwise_accuracy":0.8718},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":0.8685,"spearman":0.9066,"kendall_tau_b":0.7436,"pairwise_accuracy":0.8718},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9528,"spearman":0.9396,"kendall_tau_b":0.7949,"pairwise_accuracy":0.8974},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8311,"spearman":0.8297,"kendall_tau_b":0.641,"pairwise_accuracy":0.8205},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6617,"pearson":0.2697,"spearman":0.2348,"kendall_tau_b":0.1633},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6617,"pearson":0.2906,"spearman":0.2544,"kendall_tau_b":0.177},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6617,"pearson":0.3277,"spearman":0.2649,"kendall_tau_b":0.1847},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":6617,"pearson":0.3203,"spearman":0.2587,"kendall_tau_b":0.1805},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6617,"pearson":0.4796,"spearman":0.4365,"kendall_tau_b":0.3044},{"testset":"wmt23","pair":"de-en","pair_verbatim":"de-en","src":"de","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":6617,"pearson":0.3823,"spearman":0.4146,"kendall_tau_b":0.2846},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.5186,"spearman":0.3643,"kendall_tau_b":0.2381,"pairwise_accuracy":0.619},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.4962,"spearman":0.3714,"kendall_tau_b":0.2762,"pairwise_accuracy":0.6381},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.4495,"spearman":0.2571,"kendall_tau_b":0.1619,"pairwise_accuracy":0.581},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"pearson":0.4629,"spearman":0.2571,"kendall_tau_b":0.1619,"pairwise_accuracy":0.581},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8646,"spearman":0.8321,"kendall_tau_b":0.6762,"pairwise_accuracy":0.8381},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8914,"spearman":0.9107,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.1857,"spearman":0.1853,"kendall_tau_b":0.1265},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.1986,"spearman":0.1944,"kendall_tau_b":0.1338},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.2006,"spearman":0.2123,"kendall_tau_b":0.1463},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"n_points":16470,"pearson":0.2012,"spearman":0.209,"kendall_tau_b":0.144},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.3771,"spearman":0.4074,"kendall_tau_b":0.2828},{"testset":"wmt23","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"da-sqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.3606,"spearman":0.4213,"kendall_tau_b":0.2894},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9165,"spearman":0.9091,"kendall_tau_b":0.7879,"pairwise_accuracy":0.8939},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9195,"spearman":0.8951,"kendall_tau_b":0.7576,"pairwise_accuracy":0.8788},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8657,"spearman":0.7413,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.8717,"spearman":0.7762,"kendall_tau_b":0.6667,"pairwise_accuracy":0.8333},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9903,"spearman":0.986,"kendall_tau_b":0.9394,"pairwise_accuracy":0.9697},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9769,"spearman":0.9301,"kendall_tau_b":0.8182,"pairwise_accuracy":0.9091},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9582,"spearman":0.8671,"kendall_tau_b":0.7273,"pairwise_accuracy":0.8636},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9704,"spearman":0.8671,"kendall_tau_b":0.697,"pairwise_accuracy":0.8485},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9454,"spearman":0.7063,"kendall_tau_b":0.5758,"pairwise_accuracy":0.7879},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.9475,"spearman":0.7343,"kendall_tau_b":0.6061,"pairwise_accuracy":0.803},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9533,"spearman":0.965,"kendall_tau_b":0.8788,"pairwise_accuracy":0.9394},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8549,"spearman":0.9091,"kendall_tau_b":0.7576,"pairwise_accuracy":0.8788},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":5520,"pearson":0.1921,"spearman":0.1929,"kendall_tau_b":0.1366},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":5520,"pearson":0.237,"spearman":0.2624,"kendall_tau_b":0.1866},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":5520,"pearson":0.232,"spearman":0.2617,"kendall_tau_b":0.186},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":5520,"pearson":0.2253,"spearman":0.2503,"kendall_tau_b":0.1777},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":5520,"pearson":0.4324,"spearman":0.5784,"kendall_tau_b":0.4184},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":5520,"pearson":0.5855,"spearman":0.6819,"kendall_tau_b":0.5063},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":6588,"pearson":0.275,"spearman":0.2132,"kendall_tau_b":0.1468},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":6588,"pearson":0.3236,"spearman":0.2546,"kendall_tau_b":0.1754},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":6588,"pearson":0.3742,"spearman":0.2546,"kendall_tau_b":0.1757},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":6588,"pearson":0.3653,"spearman":0.2516,"kendall_tau_b":0.1735},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":6588,"pearson":0.4958,"spearman":0.4194,"kendall_tau_b":0.2897},{"testset":"wmt23","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"da-sqm","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":6588,"pearson":0.3886,"spearman":0.3851,"kendall_tau_b":0.2626},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.8333,"spearman":0.6353,"kendall_tau_b":0.4667,"pairwise_accuracy":0.7333},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.9608,"spearman":0.8706,"kendall_tau_b":0.7667,"pairwise_accuracy":0.8833},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.9663,"spearman":0.9088,"kendall_tau_b":0.8333,"pairwise_accuracy":0.9167},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":16,"pearson":0.9397,"spearman":0.8794,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.9694,"spearman":0.9588,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":0.9156,"spearman":0.9353,"kendall_tau_b":0.8,"pairwise_accuracy":0.9},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":17568,"pearson":0.0008,"spearman":-0.0361,"kendall_tau_b":-0.0287},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":17568,"pearson":0.1904,"spearman":0.2027,"kendall_tau_b":0.1375},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":17568,"pearson":0.2202,"spearman":0.1927,"kendall_tau_b":0.1324},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":16,"n_points":17568,"pearson":0.2185,"spearman":0.193,"kendall_tau_b":0.1326},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":17568,"pearson":0.4624,"spearman":0.4115,"kendall_tau_b":0.2847},{"testset":"wmt23","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"da-sqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":17568,"pearson":0.4013,"spearman":0.4113,"kendall_tau_b":0.2822},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8037,"spearman":0.7964,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.905,"spearman":0.5321,"kendall_tau_b":0.4286,"pairwise_accuracy":0.7143},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.9337,"spearman":0.525,"kendall_tau_b":0.3905,"pairwise_accuracy":0.6952},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"pearson":0.9163,"spearman":0.5857,"kendall_tau_b":0.4667,"pairwise_accuracy":0.7333},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.9947,"spearman":0.9536,"kendall_tau_b":0.8667,"pairwise_accuracy":0.9333},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8847,"spearman":0.9107,"kendall_tau_b":0.7714,"pairwise_accuracy":0.8857},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.0925,"spearman":0.0572,"kendall_tau_b":0.0454},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.3274,"spearman":0.2787,"kendall_tau_b":0.1923},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.3261,"spearman":0.2691,"kendall_tau_b":0.1863},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"n_points":16470,"pearson":0.321,"spearman":0.2739,"kendall_tau_b":0.1896},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.5747,"spearman":0.4296,"kendall_tau_b":0.3012},{"testset":"wmt23","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"da-sqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":16470,"pearson":0.4113,"spearman":0.402,"kendall_tau_b":0.2758},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.7147,"spearman":0.7343,"kendall_tau_b":0.5152,"pairwise_accuracy":0.7576},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.7283,"spearman":0.7063,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.7304,"spearman":0.7063,"kendall_tau_b":0.4848,"pairwise_accuracy":0.7424},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.7302,"spearman":0.7343,"kendall_tau_b":0.5152,"pairwise_accuracy":0.7576},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8812,"spearman":0.9021,"kendall_tau_b":0.7273,"pairwise_accuracy":0.8636},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9117,"spearman":0.9371,"kendall_tau_b":0.7879,"pairwise_accuracy":0.8939},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":9840,"pearson":0.2247,"spearman":0.2638,"kendall_tau_b":0.1969},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":9840,"pearson":0.2347,"spearman":0.2759,"kendall_tau_b":0.206},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":9840,"pearson":0.2441,"spearman":0.3002,"kendall_tau_b":0.2242},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":9840,"pearson":0.2412,"spearman":0.296,"kendall_tau_b":0.2211},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":9840,"pearson":0.4169,"spearman":0.4819,"kendall_tau_b":0.364},{"testset":"wmt23","pair":"he-en","pair_verbatim":"he-en","src":"he","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":9840,"pearson":0.5683,"spearman":0.5412,"kendall_tau_b":0.4137},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9389,"spearman":0.8873,"kendall_tau_b":0.75,"pairwise_accuracy":0.875},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9546,"spearman":0.9142,"kendall_tau_b":0.7794,"pairwise_accuracy":0.8897},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9656,"spearman":0.8922,"kendall_tau_b":0.7941,"pairwise_accuracy":0.8971},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":17,"pearson":0.9635,"spearman":0.8848,"kendall_tau_b":0.75,"pairwise_accuracy":0.875},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9677,"spearman":0.9534,"kendall_tau_b":0.8529,"pairwise_accuracy":0.9265},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9178,"spearman":0.9314,"kendall_tau_b":0.7794,"pairwise_accuracy":0.8897},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":19040,"pearson":0.2207,"spearman":0.2242,"kendall_tau_b":0.1525},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":19040,"pearson":0.226,"spearman":0.2232,"kendall_tau_b":0.1518},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":19040,"pearson":0.2921,"spearman":0.2577,"kendall_tau_b":0.1768},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":17,"n_points":19040,"pearson":0.2897,"spearman":0.2561,"kendall_tau_b":0.1756},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":19040,"pearson":0.4453,"spearman":0.3916,"kendall_tau_b":0.2719},{"testset":"wmt23","pair":"ja-en","pair_verbatim":"ja-en","src":"ja","tgt":"en","human":"da-sqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":19040,"pearson":0.3319,"spearman":0.3589,"kendall_tau_b":0.2456},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.7341,"spearman":0.7679,"kendall_tau_b":0.5619,"pairwise_accuracy":0.781},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.7721,"spearman":0.7607,"kendall_tau_b":0.581,"pairwise_accuracy":0.7905},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8091,"spearman":0.7179,"kendall_tau_b":0.5238,"pairwise_accuracy":0.7619},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"pearson":0.798,"spearman":0.7214,"kendall_tau_b":0.5429,"pairwise_accuracy":0.7714},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.898,"spearman":0.9071,"kendall_tau_b":0.7143,"pairwise_accuracy":0.8571},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8734,"spearman":0.9321,"kendall_tau_b":0.7905,"pairwise_accuracy":0.8952},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.6095,"spearman":0.5536,"kendall_tau_b":0.4095,"pairwise_accuracy":0.7048},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.6509,"spearman":0.55,"kendall_tau_b":0.3905,"pairwise_accuracy":0.6952},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.7013,"spearman":0.5607,"kendall_tau_b":0.3714,"pairwise_accuracy":0.6857},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"pearson":0.6883,"spearman":0.5643,"kendall_tau_b":0.3905,"pairwise_accuracy":0.6952},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8106,"spearman":0.7679,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.7348,"spearman":0.8036,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":17655,"pearson":0.1194,"spearman":0.1206,"kendall_tau_b":0.0847},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":17655,"pearson":0.1083,"spearman":0.0967,"kendall_tau_b":0.068},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":17655,"pearson":0.0635,"spearman":0.0648,"kendall_tau_b":0.0452},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"n_points":17655,"pearson":0.0645,"spearman":0.064,"kendall_tau_b":0.0447},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":17655,"pearson":0.3962,"spearman":0.4656,"kendall_tau_b":0.3328},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":17655,"pearson":0.6247,"spearman":0.6064,"kendall_tau_b":0.4486},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":13260,"pearson":0.1339,"spearman":0.0977,"kendall_tau_b":0.0663},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"f200spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":13260,"pearson":0.1387,"spearman":0.0916,"kendall_tau_b":0.0625},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":13260,"pearson":0.1683,"spearman":0.0931,"kendall_tau_b":0.0637},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"n_points":13260,"pearson":0.1632,"spearman":0.0901,"kendall_tau_b":0.0617},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":13260,"pearson":0.3778,"spearman":0.2658,"kendall_tau_b":0.183},{"testset":"wmt23","pair":"zh-en","pair_verbatim":"zh-en","src":"zh","tgt":"en","human":"da-sqm","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"MetricX-23","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":13260,"pearson":0.2644,"spearman":0.279,"kendall_tau_b":0.1897},{"testset":"wmt23.sent","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"pearson":0.6263,"spearman":0.6,"kendall_tau_b":0.4444,"pairwise_accuracy":0.7222},{"testset":"wmt23.sent","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":9,"n_points":14049,"pearson":0.1736,"spearman":0.2121,"kendall_tau_b":0.1582},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.5207,"spearman":0.5364,"kendall_tau_b":0.4182,"pairwise_accuracy":0.7091},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9151,"spearman":0.9,"kendall_tau_b":0.7818,"pairwise_accuracy":0.8909},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.7938,"spearman":0.8727,"kendall_tau_b":0.7455,"pairwise_accuracy":0.8727},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"pearson":0.7981,"spearman":0.8545,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.9204,"spearman":0.9364,"kendall_tau_b":0.8182,"pairwise_accuracy":0.9091},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":11,"pearson":0.8986,"spearman":0.8818,"kendall_tau_b":0.7091,"pairwise_accuracy":0.8545},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":21505,"pearson":0.2775,"spearman":0.266,"kendall_tau_b":0.1949},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":21505,"pearson":0.3281,"spearman":0.3002,"kendall_tau_b":0.2196},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":21505,"pearson":0.4241,"spearman":0.3189,"kendall_tau_b":0.2339},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":11,"n_points":21505,"pearson":0.4174,"spearman":0.3154,"kendall_tau_b":0.2313},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":21505,"pearson":0.5479,"spearman":0.388,"kendall_tau_b":0.2832},{"testset":"wmt24","pair":"cs-uk","pair_verbatim":"cs-uk","src":"cs","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":11,"n_points":21505,"pearson":0.5435,"spearman":0.3981,"kendall_tau_b":0.2897},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.5882,"spearman":0.5893,"kendall_tau_b":0.4667,"pairwise_accuracy":0.7333},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.6281,"spearman":0.6143,"kendall_tau_b":0.4476,"pairwise_accuracy":0.7238},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.6372,"spearman":0.5464,"kendall_tau_b":0.4476,"pairwise_accuracy":0.7238},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"pearson":0.6339,"spearman":0.5429,"kendall_tau_b":0.4476,"pairwise_accuracy":0.7238},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.8495,"spearman":0.8929,"kendall_tau_b":0.7524,"pairwise_accuracy":0.8762},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":15,"pearson":0.9122,"spearman":0.8821,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":8580,"pearson":0.2172,"spearman":0.2461,"kendall_tau_b":0.172},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":8580,"pearson":0.25,"spearman":0.2556,"kendall_tau_b":0.18},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":8580,"pearson":0.2808,"spearman":0.2697,"kendall_tau_b":0.1913},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":15,"n_points":8580,"pearson":0.2831,"spearman":0.269,"kendall_tau_b":0.1907},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":8580,"pearson":0.4803,"spearman":0.4421,"kendall_tau_b":0.317},{"testset":"wmt24","pair":"en-cs","pair_verbatim":"en-cs","src":"en","tgt":"cs","human":"esa","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":15,"n_points":8580,"pearson":0.5111,"spearman":0.4762,"kendall_tau_b":0.34},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9161,"spearman":0.5882,"kendall_tau_b":0.4559,"pairwise_accuracy":0.7279},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9461,"spearman":0.652,"kendall_tau_b":0.5147,"pairwise_accuracy":0.7574},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9256,"spearman":0.5833,"kendall_tau_b":0.4706,"pairwise_accuracy":0.7353},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":17,"pearson":0.9257,"spearman":0.5735,"kendall_tau_b":0.4559,"pairwise_accuracy":0.7279},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9922,"spearman":0.9534,"kendall_tau_b":0.8529,"pairwise_accuracy":0.9265},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":17,"pearson":0.9912,"spearman":0.9167,"kendall_tau_b":0.8088,"pairwise_accuracy":0.9044},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":8279,"pearson":0.1599,"spearman":0.1659,"kendall_tau_b":0.1218},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":8279,"pearson":0.1733,"spearman":0.1828,"kendall_tau_b":0.1344},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":8279,"pearson":0.1709,"spearman":0.1947,"kendall_tau_b":0.1428},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":17,"n_points":8279,"pearson":0.1671,"spearman":0.1844,"kendall_tau_b":0.1354},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":8279,"pearson":0.4058,"spearman":0.3997,"kendall_tau_b":0.298},{"testset":"wmt24","pair":"en-de","pair_verbatim":"en-de","src":"en","tgt":"de","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":17,"n_points":8279,"pearson":0.5563,"spearman":0.4869,"kendall_tau_b":0.3689},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.5097,"spearman":0.2,"kendall_tau_b":0.1429,"pairwise_accuracy":0.5714},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.6115,"spearman":0.2835,"kendall_tau_b":0.1868,"pairwise_accuracy":0.5934},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.6062,"spearman":0.3187,"kendall_tau_b":0.2747,"pairwise_accuracy":0.6374},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.5966,"spearman":0.2879,"kendall_tau_b":0.2308,"pairwise_accuracy":0.6154},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9583,"spearman":0.7143,"kendall_tau_b":0.5385,"pairwise_accuracy":0.7692},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9821,"spearman":0.8374,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"sys","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.6905,"spearman":0.5879,"kendall_tau_b":0.5128,"pairwise_accuracy":0.7564},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"sys","preferred":false,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.7866,"spearman":0.6813,"kendall_tau_b":0.5641,"pairwise_accuracy":0.7821},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"sys","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.7389,"spearman":0.5604,"kendall_tau_b":0.4359,"pairwise_accuracy":0.7179},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"sys","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":0.7371,"spearman":0.5989,"kendall_tau_b":0.4872,"pairwise_accuracy":0.7436},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"sys","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9767,"spearman":0.9121,"kendall_tau_b":0.7692,"pairwise_accuracy":0.8846},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"sys","preferred":false,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9757,"spearman":0.8956,"kendall_tau_b":0.7692,"pairwise_accuracy":0.8846},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":8722,"pearson":0.1457,"spearman":0.1802,"kendall_tau_b":0.1406},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":8722,"pearson":0.1549,"spearman":0.1952,"kendall_tau_b":0.1526},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":8722,"pearson":0.1479,"spearman":0.1894,"kendall_tau_b":0.148},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":8722,"pearson":0.1481,"spearman":0.1919,"kendall_tau_b":0.1499},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":8722,"pearson":0.3798,"spearman":0.3291,"kendall_tau_b":0.2596},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":8722,"pearson":0.5513,"spearman":0.3752,"kendall_tau_b":0.2974},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"seg","preferred":false,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.2548,"spearman":0.2296,"kendall_tau_b":0.166},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"seg","preferred":false,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.2754,"spearman":0.2423,"kendall_tau_b":0.1756},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"seg","preferred":false,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.3289,"spearman":0.2492,"kendall_tau_b":0.1809},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"seg","preferred":false,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":8255,"pearson":0.3247,"spearman":0.2517,"kendall_tau_b":0.1826},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"seg","preferred":false,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.4606,"spearman":0.3646,"kendall_tau_b":0.2664},{"testset":"wmt24","pair":"en-es","pair_verbatim":"en-es","src":"en","tgt":"es","human":"esa","level":"seg","preferred":false,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.4957,"spearman":0.3873,"kendall_tau_b":0.2811},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8988,"spearman":0.7212,"kendall_tau_b":0.6444,"pairwise_accuracy":0.8222},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9138,"spearman":0.7939,"kendall_tau_b":0.7333,"pairwise_accuracy":0.8667},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9705,"spearman":0.7818,"kendall_tau_b":0.6889,"pairwise_accuracy":0.8444},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.9665,"spearman":0.7818,"kendall_tau_b":0.6889,"pairwise_accuracy":0.8444},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9069,"spearman":0.9758,"kendall_tau_b":0.9111,"pairwise_accuracy":0.9556},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9927,"spearman":0.9758,"kendall_tau_b":0.9111,"pairwise_accuracy":0.9556},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.0913,"spearman":0.144,"kendall_tau_b":0.1007},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.1176,"spearman":0.1389,"kendall_tau_b":0.0975},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.1919,"spearman":0.1383,"kendall_tau_b":0.0978},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":6350,"pearson":0.1877,"spearman":0.1379,"kendall_tau_b":0.0974},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.3388,"spearman":0.3074,"kendall_tau_b":0.2178},{"testset":"wmt24","pair":"en-hi","pair_verbatim":"en-hi","src":"en","tgt":"hi","human":"esa","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.5334,"spearman":0.3876,"kendall_tau_b":0.2748},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9341,"spearman":0.9273,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.943,"spearman":0.9394,"kendall_tau_b":0.8222,"pairwise_accuracy":0.9111},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9308,"spearman":0.903,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.9307,"spearman":0.903,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.948,"spearman":0.903,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9644,"spearman":0.8303,"kendall_tau_b":0.6889,"pairwise_accuracy":0.8444},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.3347,"spearman":0.3386,"kendall_tau_b":0.2382},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.3604,"spearman":0.3483,"kendall_tau_b":0.2477},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.356,"spearman":0.3353,"kendall_tau_b":0.2414},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":6350,"pearson":0.3651,"spearman":0.343,"kendall_tau_b":0.2464},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.4923,"spearman":0.4746,"kendall_tau_b":0.3423},{"testset":"wmt24","pair":"en-is","pair_verbatim":"en-is","src":"en","tgt":"is","human":"esa","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.5728,"spearman":0.5599,"kendall_tau_b":0.4016},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8693,"spearman":0.7483,"kendall_tau_b":0.6061,"pairwise_accuracy":0.803},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8827,"spearman":0.8042,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8805,"spearman":0.7762,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.8098,"spearman":0.7902,"kendall_tau_b":0.6364,"pairwise_accuracy":0.8182},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.81,"spearman":0.4965,"kendall_tau_b":0.4242,"pairwise_accuracy":0.7121},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.9296,"spearman":0.7762,"kendall_tau_b":0.6061,"pairwise_accuracy":0.803},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.1761,"spearman":0.1868,"kendall_tau_b":0.1298},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.1675,"spearman":0.191,"kendall_tau_b":0.1326},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.1985,"spearman":0.1829,"kendall_tau_b":0.1277},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":7620,"pearson":0.187,"spearman":0.1798,"kendall_tau_b":0.1255},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.4184,"spearman":0.3104,"kendall_tau_b":0.2183},{"testset":"wmt24","pair":"en-ja","pair_verbatim":"en-ja","src":"en","tgt":"ja","human":"esa","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.4189,"spearman":0.3489,"kendall_tau_b":0.2436},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.7429,"spearman":0.6703,"kendall_tau_b":0.5897,"pairwise_accuracy":0.7949},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8146,"spearman":0.7692,"kendall_tau_b":0.641,"pairwise_accuracy":0.8205},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.8041,"spearman":0.7088,"kendall_tau_b":0.5128,"pairwise_accuracy":0.7564},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"pearson":0.7941,"spearman":0.6099,"kendall_tau_b":0.4872,"pairwise_accuracy":0.7436},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9467,"spearman":0.989,"kendall_tau_b":0.9487,"pairwise_accuracy":0.9744},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":13,"pearson":0.9679,"spearman":0.9396,"kendall_tau_b":0.8718,"pairwise_accuracy":0.9359},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.1336,"spearman":0.1152,"kendall_tau_b":0.0822},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.1783,"spearman":0.1386,"kendall_tau_b":0.0993},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.2452,"spearman":0.1682,"kendall_tau_b":0.121},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":13,"n_points":8255,"pearson":0.2338,"spearman":0.1607,"kendall_tau_b":0.1156},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.3952,"spearman":0.3352,"kendall_tau_b":0.2413},{"testset":"wmt24","pair":"en-ru","pair_verbatim":"en-ru","src":"en","tgt":"ru","human":"esa","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":13,"n_points":8255,"pearson":0.4473,"spearman":0.4004,"kendall_tau_b":0.2889},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.4967,"spearman":0.5152,"kendall_tau_b":0.3778,"pairwise_accuracy":0.6889},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.4455,"spearman":0.6242,"kendall_tau_b":0.4667,"pairwise_accuracy":0.7333},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.6348,"spearman":0.6727,"kendall_tau_b":0.5111,"pairwise_accuracy":0.7556},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"pearson":0.6177,"spearman":0.6727,"kendall_tau_b":0.5111,"pairwise_accuracy":0.7556},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.8531,"spearman":0.7455,"kendall_tau_b":0.6,"pairwise_accuracy":0.8},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":10,"pearson":0.9624,"spearman":0.8424,"kendall_tau_b":0.6889,"pairwise_accuracy":0.8444},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.1662,"spearman":0.1479,"kendall_tau_b":0.109},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.199,"spearman":0.1676,"kendall_tau_b":0.1239},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.2284,"spearman":0.1833,"kendall_tau_b":0.1364},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":10,"n_points":6350,"pearson":0.2259,"spearman":0.1792,"kendall_tau_b":0.1331},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.3701,"spearman":0.3001,"kendall_tau_b":0.2226},{"testset":"wmt24","pair":"en-uk","pair_verbatim":"en-uk","src":"en","tgt":"uk","human":"esa","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":10,"n_points":6350,"pearson":0.3642,"spearman":0.3108,"kendall_tau_b":0.23},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.6351,"spearman":0.4895,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.6356,"spearman":0.5455,"kendall_tau_b":0.3939,"pairwise_accuracy":0.697},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.6405,"spearman":0.4895,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"pearson":0.5061,"spearman":0.5804,"kendall_tau_b":0.4242,"pairwise_accuracy":0.7121},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8794,"spearman":0.8531,"kendall_tau_b":0.7273,"pairwise_accuracy":0.8636},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":12,"pearson":0.8908,"spearman":0.7552,"kendall_tau_b":0.6061,"pairwise_accuracy":0.803},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.1932,"spearman":0.1792,"kendall_tau_b":0.1251},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.1769,"spearman":0.1811,"kendall_tau_b":0.1261},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.178,"spearman":0.1783,"kendall_tau_b":0.1248},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":12,"n_points":7620,"pearson":0.162,"spearman":0.1754,"kendall_tau_b":0.1226},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.3988,"spearman":0.3521,"kendall_tau_b":0.2459},{"testset":"wmt24","pair":"en-zh","pair_verbatim":"en-zh","src":"en","tgt":"zh","human":"esa","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":12,"n_points":7620,"pearson":0.3485,"spearman":0.3738,"kendall_tau_b":0.2598},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.4282,"spearman":0.4242,"kendall_tau_b":0.3407,"pairwise_accuracy":0.6703},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.7028,"spearman":0.6132,"kendall_tau_b":0.5385,"pairwise_accuracy":0.7692},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.6286,"spearman":0.6484,"kendall_tau_b":0.5824,"pairwise_accuracy":0.7912},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"pearson":0.6316,"spearman":0.6484,"kendall_tau_b":0.5824,"pairwise_accuracy":0.7912},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9317,"spearman":0.8462,"kendall_tau_b":0.6923,"pairwise_accuracy":0.8462},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":14,"pearson":0.9755,"spearman":0.9341,"kendall_tau_b":0.8242,"pairwise_accuracy":0.9121},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7840,"pearson":0.0793,"spearman":0.0948,"kendall_tau_b":0.0845},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7840,"pearson":0.2304,"spearman":0.322,"kendall_tau_b":0.242},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7840,"pearson":0.2052,"spearman":0.2853,"kendall_tau_b":0.2144},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":14,"n_points":7840,"pearson":0.1974,"spearman":0.2898,"kendall_tau_b":0.2177},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET-22","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7840,"pearson":0.4099,"spearman":0.4601,"kendall_tau_b":0.3476},{"testset":"wmt24","pair":"ja-zh","pair_verbatim":"ja-zh","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-24-Hybrid","upstream_ref":"refA","computed":false,"n_sys":14,"n_points":7840,"pearson":0.4635,"spearman":0.5131,"kendall_tau_b":0.3895},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.8048,"spearman":0.8105,"kendall_tau_b":0.6316,"pairwise_accuracy":0.8158},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.8423,"spearman":0.8496,"kendall_tau_b":0.6947,"pairwise_accuracy":0.8474},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.8713,"spearman":0.8511,"kendall_tau_b":0.6737,"pairwise_accuracy":0.8368},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":20,"pearson":0.8534,"spearman":0.8511,"kendall_tau_b":0.6737,"pairwise_accuracy":0.8368},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.6158,"spearman":0.6707,"kendall_tau_b":0.5053,"pairwise_accuracy":0.7526},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":20,"pearson":0.8333,"spearman":0.8451,"kendall_tau_b":0.6526,"pairwise_accuracy":0.8263},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":4620,"pearson":0.2106,"spearman":0.2056,"kendall_tau_b":0.1399},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":4620,"pearson":0.2212,"spearman":0.2001,"kendall_tau_b":0.1369},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":4620,"pearson":0.2589,"spearman":0.2148,"kendall_tau_b":0.1472},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":20,"n_points":4620,"pearson":0.2438,"spearman":0.2089,"kendall_tau_b":0.1429},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":4620,"pearson":0.3029,"spearman":0.2428,"kendall_tau_b":0.1655},{"testset":"wmt25","pair":"cs-de","pair_verbatim":"cs-de_DE","src":"cs","tgt":"de","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":20,"n_points":4620,"pearson":0.3258,"spearman":0.3441,"kendall_tau_b":0.2363},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.3511,"spearman":0.4613,"kendall_tau_b":0.3203,"pairwise_accuracy":0.6601},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.4279,"spearman":0.5542,"kendall_tau_b":0.3987,"pairwise_accuracy":0.6993},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.4116,"spearman":0.5335,"kendall_tau_b":0.3595,"pairwise_accuracy":0.6797},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.3968,"spearman":0.4902,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.6054,"spearman":0.6367,"kendall_tau_b":0.5425,"pairwise_accuracy":0.7712},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8467,"spearman":0.8349,"kendall_tau_b":0.6471,"pairwise_accuracy":0.8235},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":4050,"pearson":0.0332,"spearman":0.0429,"kendall_tau_b":0.0292},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":4050,"pearson":0.111,"spearman":0.1233,"kendall_tau_b":0.0844},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":4050,"pearson":0.1566,"spearman":0.1584,"kendall_tau_b":0.1085},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":4050,"pearson":0.1214,"spearman":0.1337,"kendall_tau_b":0.0911},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":4050,"pearson":0.3093,"spearman":0.3111,"kendall_tau_b":0.2143},{"testset":"wmt25","pair":"cs-uk","pair_verbatim":"cs-uk_UA","src":"cs","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":4050,"pearson":0.4542,"spearman":0.4005,"kendall_tau_b":0.2801},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.961,"spearman":0.839,"kendall_tau_b":0.6732,"pairwise_accuracy":0.8366},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.9508,"spearman":0.8452,"kendall_tau_b":0.6863,"pairwise_accuracy":0.8431},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.9225,"spearman":0.8555,"kendall_tau_b":0.6601,"pairwise_accuracy":0.8301},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.9442,"spearman":0.8658,"kendall_tau_b":0.6993,"pairwise_accuracy":0.8497},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.5173,"spearman":0.5129,"kendall_tau_b":0.3464,"pairwise_accuracy":0.6732},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":-0.6242,"spearman":-0.612,"kendall_tau_b":-0.4248,"pairwise_accuracy":0.2876},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":0.4461,"spearman":0.4809,"kendall_tau_b":0.3565},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":0.524,"spearman":0.5318,"kendall_tau_b":0.3971},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":0.4186,"spearman":0.4275,"kendall_tau_b":0.3169},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3582,"pearson":0.4805,"spearman":0.487,"kendall_tau_b":0.3619},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":0.3287,"spearman":0.3369,"kendall_tau_b":0.2471},{"testset":"wmt25","pair":"en-ar","pair_verbatim":"en-ar_EG","src":"en","tgt":"ar","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":-0.318,"spearman":-0.3545,"kendall_tau_b":-0.2543},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.794,"spearman":0.8741,"kendall_tau_b":0.7778,"pairwise_accuracy":0.8889},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.6888,"spearman":0.839,"kendall_tau_b":0.6993,"pairwise_accuracy":0.8497},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.6876,"spearman":0.8101,"kendall_tau_b":0.6471,"pairwise_accuracy":0.8235},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.7414,"spearman":0.839,"kendall_tau_b":0.6863,"pairwise_accuracy":0.8431},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.5944,"spearman":0.6966,"kendall_tau_b":0.5033,"pairwise_accuracy":0.7516},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.049,"spearman":0.4035,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.291,"spearman":0.3546,"kendall_tau_b":0.2444},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.3631,"spearman":0.3993,"kendall_tau_b":0.2765},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.4176,"spearman":0.4077,"kendall_tau_b":0.2812},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3402,"pearson":0.4325,"spearman":0.4181,"kendall_tau_b":0.2889},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.3468,"spearman":0.2748,"kendall_tau_b":0.1854},{"testset":"wmt25","pair":"en-bho","pair_verbatim":"en-bho_IN","src":"en","tgt":"bho","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.2221,"spearman":0.2713,"kendall_tau_b":0.2099},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.6485,"spearman":0.6491,"kendall_tau_b":0.4503,"pairwise_accuracy":0.7251},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.6578,"spearman":0.707,"kendall_tau_b":0.4971,"pairwise_accuracy":0.7485},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.7032,"spearman":0.7825,"kendall_tau_b":0.5789,"pairwise_accuracy":0.7895},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":19,"pearson":0.7015,"spearman":0.7667,"kendall_tau_b":0.5556,"pairwise_accuracy":0.7778},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.479,"spearman":0.5404,"kendall_tau_b":0.3801,"pairwise_accuracy":0.6901},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.6003,"spearman":0.6632,"kendall_tau_b":0.5088,"pairwise_accuracy":0.7544},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":4028,"pearson":0.2358,"spearman":0.2308,"kendall_tau_b":0.1554},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":4028,"pearson":0.2534,"spearman":0.249,"kendall_tau_b":0.1681},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":4028,"pearson":0.2256,"spearman":0.2386,"kendall_tau_b":0.161},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":19,"n_points":4028,"pearson":0.2265,"spearman":0.2368,"kendall_tau_b":0.1596},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":4028,"pearson":0.2927,"spearman":0.3099,"kendall_tau_b":0.2101},{"testset":"wmt25","pair":"en-cs","pair_verbatim":"en-cs_CZ","src":"en","tgt":"cs","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":4028,"pearson":0.2492,"spearman":0.3699,"kendall_tau_b":0.2525},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.7836,"spearman":0.8019,"kendall_tau_b":0.6601,"pairwise_accuracy":0.8301},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8545,"spearman":0.8431,"kendall_tau_b":0.6993,"pairwise_accuracy":0.8497},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8345,"spearman":0.8865,"kendall_tau_b":0.7386,"pairwise_accuracy":0.8693},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.8433,"spearman":0.8927,"kendall_tau_b":0.7647,"pairwise_accuracy":0.8824},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.7055,"spearman":0.6264,"kendall_tau_b":0.451,"pairwise_accuracy":0.7255},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.7768,"spearman":0.7028,"kendall_tau_b":0.5294,"pairwise_accuracy":0.7647},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3420,"pearson":0.2669,"spearman":0.236,"kendall_tau_b":0.1602},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3420,"pearson":0.3255,"spearman":0.3021,"kendall_tau_b":0.206},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3420,"pearson":0.3294,"spearman":0.2985,"kendall_tau_b":0.2043},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3420,"pearson":0.3279,"spearman":0.2962,"kendall_tau_b":0.2025},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3420,"pearson":0.351,"spearman":0.3526,"kendall_tau_b":0.2402},{"testset":"wmt25","pair":"en-et","pair_verbatim":"en-et_EE","src":"en","tgt":"et","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3420,"pearson":0.4368,"spearman":0.4526,"kendall_tau_b":0.3124},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.9269,"spearman":0.9401,"kendall_tau_b":0.817,"pairwise_accuracy":0.9085},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.9441,"spearman":0.9567,"kendall_tau_b":0.8431,"pairwise_accuracy":0.9216},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.9231,"spearman":0.9236,"kendall_tau_b":0.7908,"pairwise_accuracy":0.8954},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.9315,"spearman":0.9484,"kendall_tau_b":0.817,"pairwise_accuracy":0.9085},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.753,"spearman":0.8906,"kendall_tau_b":0.7386,"pairwise_accuracy":0.8693},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8297,"spearman":0.87,"kendall_tau_b":0.7516,"pairwise_accuracy":0.8758},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3636,"pearson":0.4262,"spearman":0.4153,"kendall_tau_b":0.2871},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3636,"pearson":0.4902,"spearman":0.4862,"kendall_tau_b":0.3379},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3636,"pearson":0.4944,"spearman":0.5171,"kendall_tau_b":0.3619},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3636,"pearson":0.4832,"spearman":0.5084,"kendall_tau_b":0.3567},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3636,"pearson":0.5007,"spearman":0.4896,"kendall_tau_b":0.3399},{"testset":"wmt25","pair":"en-is","pair_verbatim":"en-is_IS","src":"en","tgt":"is","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3636,"pearson":0.5536,"spearman":0.5551,"kendall_tau_b":0.3906},{"testset":"wmt25","pair":"en-it","pair_verbatim":"en-it_IT","src":"en","tgt":"it","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":-0.2649,"spearman":-0.2405,"kendall_tau_b":-0.1765,"pairwise_accuracy":0.4118},{"testset":"wmt25","pair":"en-it","pair_verbatim":"en-it_IT","src":"en","tgt":"it","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.6583,"spearman":0.5913,"kendall_tau_b":0.4902,"pairwise_accuracy":0.7451},{"testset":"wmt25","pair":"en-it","pair_verbatim":"en-it_IT","src":"en","tgt":"it","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3870,"pearson":0.0636,"spearman":0.1483,"kendall_tau_b":0.1},{"testset":"wmt25","pair":"en-it","pair_verbatim":"en-it_IT","src":"en","tgt":"it","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3870,"pearson":0.4098,"spearman":0.3866,"kendall_tau_b":0.2656},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8371,"spearman":0.8019,"kendall_tau_b":0.6078,"pairwise_accuracy":0.8039},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8442,"spearman":0.8349,"kendall_tau_b":0.634,"pairwise_accuracy":0.817},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8645,"spearman":0.8535,"kendall_tau_b":0.6732,"pairwise_accuracy":0.8366},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.5927,"spearman":0.7936,"kendall_tau_b":0.5948,"pairwise_accuracy":0.7974},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":-0.0252,"spearman":0.0237,"kendall_tau_b":0.0065,"pairwise_accuracy":0.5033},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.3556,"spearman":0.3003,"kendall_tau_b":0.2026,"pairwise_accuracy":0.6013},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.1535,"spearman":0.1348,"kendall_tau_b":0.0915},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.1278,"spearman":0.1101,"kendall_tau_b":0.0746},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.1521,"spearman":0.1391,"kendall_tau_b":0.0942},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3402,"pearson":0.1319,"spearman":0.1052,"kendall_tau_b":0.0712},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.1458,"spearman":0.1316,"kendall_tau_b":0.0892},{"testset":"wmt25","pair":"en-ja","pair_verbatim":"en-ja_JP","src":"en","tgt":"ja","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3402,"pearson":0.2834,"spearman":0.2465,"kendall_tau_b":0.1681},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.8864,"spearman":0.8737,"kendall_tau_b":0.6725,"pairwise_accuracy":0.8363},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.8711,"spearman":0.8772,"kendall_tau_b":0.6959,"pairwise_accuracy":0.848},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.8745,"spearman":0.8877,"kendall_tau_b":0.7076,"pairwise_accuracy":0.8538},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":19,"pearson":0.8813,"spearman":0.8807,"kendall_tau_b":0.6959,"pairwise_accuracy":0.848},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.5511,"spearman":0.5632,"kendall_tau_b":0.462,"pairwise_accuracy":0.731},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":19,"pearson":0.7689,"spearman":0.8105,"kendall_tau_b":0.6374,"pairwise_accuracy":0.8187},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":2128,"pearson":0.0982,"spearman":0.0258,"kendall_tau_b":0.0196},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":2128,"pearson":0.0409,"spearman":-0.0945,"kendall_tau_b":-0.0673},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":2128,"pearson":0.0977,"spearman":0.0285,"kendall_tau_b":0.0223},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":19,"n_points":2128,"pearson":0.0957,"spearman":0.032,"kendall_tau_b":0.0248},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":2128,"pearson":0.2643,"spearman":0.3502,"kendall_tau_b":0.2539},{"testset":"wmt25","pair":"en-ko","pair_verbatim":"en-ko_KR","src":"en","tgt":"ko","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":19,"n_points":2128,"pearson":0.2311,"spearman":0.0331,"kendall_tau_b":0.0265},{"testset":"wmt25","pair":"en-mas","pair_verbatim":"en-mas_KE","src":"en","tgt":"mas","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.4991,"spearman":0.4737,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt25","pair":"en-mas","pair_verbatim":"en-mas_KE","src":"en","tgt":"mas","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25","upstream_ref":"refA","computed":false,"n_sys":16,"pearson":-0.0944,"spearman":-0.1647,"kendall_tau_b":-0.1167,"pairwise_accuracy":0.4417},{"testset":"wmt25","pair":"en-mas","pair_verbatim":"en-mas_KE","src":"en","tgt":"mas","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3330,"pearson":0.0561,"spearman":0.0878,"kendall_tau_b":0.0665},{"testset":"wmt25","pair":"en-mas","pair_verbatim":"en-mas_KE","src":"en","tgt":"mas","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25","upstream_ref":"refA","computed":false,"n_sys":16,"n_points":2960,"pearson":-0.0578,"spearman":-0.0852,"kendall_tau_b":-0.0653},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.26,"spearman":0.0361,"kendall_tau_b":0.0458,"pairwise_accuracy":0.5229},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.3868,"spearman":0.1723,"kendall_tau_b":0.1503,"pairwise_accuracy":0.5752},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.2171,"spearman":0.0175,"kendall_tau_b":0.0196,"pairwise_accuracy":0.5098},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.2296,"spearman":0.0217,"kendall_tau_b":0.0065,"pairwise_accuracy":0.5033},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.1853,"spearman":0.1641,"kendall_tau_b":0.1373,"pairwise_accuracy":0.5686},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.5054,"spearman":0.581,"kendall_tau_b":0.4771,"pairwise_accuracy":0.7386},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3600,"pearson":0.1092,"spearman":0.1139,"kendall_tau_b":0.0778},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3600,"pearson":0.16,"spearman":0.165,"kendall_tau_b":0.112},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3600,"pearson":0.0894,"spearman":0.1011,"kendall_tau_b":0.0681},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3600,"pearson":0.0926,"spearman":0.1122,"kendall_tau_b":0.0761},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3600,"pearson":0.2766,"spearman":0.2926,"kendall_tau_b":0.2002},{"testset":"wmt25","pair":"en-ru","pair_verbatim":"en-ru_RU","src":"en","tgt":"ru","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3600,"pearson":0.4078,"spearman":0.402,"kendall_tau_b":0.276},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8154,"spearman":0.8989,"kendall_tau_b":0.7386,"pairwise_accuracy":0.8693},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8127,"spearman":0.8865,"kendall_tau_b":0.7516,"pairwise_accuracy":0.8758},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.7297,"spearman":0.87,"kendall_tau_b":0.7255,"pairwise_accuracy":0.8627},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.7392,"spearman":0.8762,"kendall_tau_b":0.7386,"pairwise_accuracy":0.8693},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8746,"spearman":0.8287,"kendall_tau_b":0.6601,"pairwise_accuracy":0.8301},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.9151,"spearman":0.8349,"kendall_tau_b":0.6471,"pairwise_accuracy":0.8235},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3708,"pearson":0.5014,"spearman":0.5154,"kendall_tau_b":0.3583},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3708,"pearson":0.5275,"spearman":0.5389,"kendall_tau_b":0.3761},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3708,"pearson":0.4639,"spearman":0.518,"kendall_tau_b":0.356},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3708,"pearson":0.4763,"spearman":0.522,"kendall_tau_b":0.36},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3708,"pearson":0.6042,"spearman":0.5275,"kendall_tau_b":0.3719},{"testset":"wmt25","pair":"en-sr","pair_verbatim":"en-sr_Cyrl_RS","src":"en","tgt":"sr","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3708,"pearson":0.6292,"spearman":0.5442,"kendall_tau_b":0.3863},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.5461,"spearman":0.4964,"kendall_tau_b":0.3725,"pairwise_accuracy":0.6863},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.5819,"spearman":0.5397,"kendall_tau_b":0.3856,"pairwise_accuracy":0.6928},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.6767,"spearman":0.6182,"kendall_tau_b":0.4902,"pairwise_accuracy":0.7451},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.6637,"spearman":0.612,"kendall_tau_b":0.4771,"pairwise_accuracy":0.7386},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.4984,"spearman":0.5624,"kendall_tau_b":0.3987,"pairwise_accuracy":0.6993},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.6063,"spearman":0.7771,"kendall_tau_b":0.5686,"pairwise_accuracy":0.7843},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":0.1546,"spearman":0.1118,"kendall_tau_b":0.0765},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":0.1753,"spearman":0.13,"kendall_tau_b":0.0888},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":0.1644,"spearman":0.119,"kendall_tau_b":0.0821},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3582,"pearson":0.1586,"spearman":0.1177,"kendall_tau_b":0.0814},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":0.2899,"spearman":0.2628,"kendall_tau_b":0.1796},{"testset":"wmt25","pair":"en-uk","pair_verbatim":"en-uk_UA","src":"en","tgt":"uk","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3582,"pearson":0.4217,"spearman":0.4173,"kendall_tau_b":0.2901},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.4589,"spearman":0.4159,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.5404,"spearman":0.3725,"kendall_tau_b":0.2941,"pairwise_accuracy":0.6471},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.514,"spearman":0.4365,"kendall_tau_b":0.3333,"pairwise_accuracy":0.6667},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.5184,"spearman":0.3829,"kendall_tau_b":0.2941,"pairwise_accuracy":0.6471},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.2988,"spearman":0.3478,"kendall_tau_b":0.268,"pairwise_accuracy":0.634},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.6077,"spearman":0.6037,"kendall_tau_b":0.4379,"pairwise_accuracy":0.719},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3564,"pearson":0.214,"spearman":0.2104,"kendall_tau_b":0.1438},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3564,"pearson":0.2321,"spearman":0.2333,"kendall_tau_b":0.1597},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3564,"pearson":0.2024,"spearman":0.2105,"kendall_tau_b":0.1438},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":3564,"pearson":0.2422,"spearman":0.2555,"kendall_tau_b":0.1752},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3564,"pearson":0.2414,"spearman":0.2699,"kendall_tau_b":0.1836},{"testset":"wmt25","pair":"en-zh","pair_verbatim":"en-zh_CN","src":"en","tgt":"zh","human":"esa-merged","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":3564,"pearson":0.2824,"spearman":0.2825,"kendall_tau_b":0.1922},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8608,"spearman":0.8308,"kendall_tau_b":0.6209,"pairwise_accuracy":0.8105},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8788,"spearman":0.8431,"kendall_tau_b":0.6078,"pairwise_accuracy":0.8039},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.8767,"spearman":0.87,"kendall_tau_b":0.6863,"pairwise_accuracy":0.8431},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"pearson":0.8018,"spearman":0.8514,"kendall_tau_b":0.6732,"pairwise_accuracy":0.8366},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.7105,"spearman":0.6636,"kendall_tau_b":0.4771,"pairwise_accuracy":0.7386},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"sys","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"pearson":0.7559,"spearman":0.614,"kendall_tau_b":0.4641,"pairwise_accuracy":0.732},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"bleu","upstream_metric":"BLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":2844,"pearson":0.2207,"spearman":0.2371,"kendall_tau_b":0.168},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"spbleu","upstream_metric":"spBLEU","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":2844,"pearson":0.225,"spearman":0.2458,"kendall_tau_b":0.1741},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plain","upstream_metric":"chrF","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":2844,"pearson":0.2219,"spearman":0.2623,"kendall_tau_b":0.1854},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"chrf_plus_plus","upstream_metric":"chrF++ (computed, sacrebleu word_order=2)","upstream_ref":"refA","computed":true,"n_sys":18,"n_points":2844,"pearson":0.1916,"spearman":0.2376,"kendall_tau_b":0.1688},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"comet_score","upstream_metric":"COMET22","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":2844,"pearson":0.236,"spearman":0.2456,"kendall_tau_b":0.1741},{"testset":"wmt25","pair":"ja-zh","pair_verbatim":"ja-zh_CN","src":"ja","tgt":"zh","human":"mqm","level":"seg","preferred":true,"metric":"metricx_score","upstream_metric":"MetricX-25-Ref","upstream_ref":"refA","computed":false,"n_sys":18,"n_points":2844,"pearson":0.4524,"spearman":0.3692,"kendall_tau_b":0.2653}],"excluded":[{"testset":"wmt22","pair":"en-liv","human":"wmt-appraise-z","level":"seg","metric":"bleu","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"en-liv","human":"wmt-appraise-z","level":"seg","metric":"spbleu","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"en-liv","human":"wmt-appraise-z","level":"seg","metric":"chrf_plain","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"en-liv","human":"wmt-appraise-z","level":"seg","metric":"comet_score","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"en-liv","human":"wmt-appraise-z","level":"seg","metric":"metricx_score","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"liv-en","human":"wmt-appraise-z","level":"seg","metric":"bleu","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"liv-en","human":"wmt-appraise-z","level":"seg","metric":"spbleu","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"liv-en","human":"wmt-appraise-z","level":"seg","metric":"chrf_plain","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"liv-en","human":"wmt-appraise-z","level":"seg","metric":"comet_score","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"liv-en","human":"wmt-appraise-z","level":"seg","metric":"metricx_score","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise-z","level":"sys","metric":"bleu","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise-z","level":"sys","metric":"spbleu","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise-z","level":"sys","metric":"chrf_plain","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise-z","level":"sys","metric":"comet_score","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise-z","level":"sys","metric":"metricx_score","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise","level":"sys","metric":"bleu","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise","level":"sys","metric":"spbleu","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise","level":"sys","metric":"chrf_plain","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise","level":"sys","metric":"comet_score","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt22","pair":"sah-ru","human":"wmt-appraise","level":"sys","metric":"metricx_score","reason":"degenerate cell (fewer than 3 aligned systems / 10 aligned segments, zero variance, or ragged segment counts)"},{"testset":"wmt24pp","reason":"no allowlisted human judgments (e.g. wmt24pp ships references/system outputs but no human scores)"}],"counts":{"cells":1810,"preferred_cells":1052,"testsets":10,"pairs":57,"families":11,"excluded":21}}