pearmut 0.1.3__py3-none-any.whl → 0.2.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pearmut/app.py +20 -4
- pearmut/assignment.py +1 -5
- pearmut/cli.py +1 -1
- pearmut/static/assets/style.css +7 -0
- pearmut/static/dashboard.bundle.js +1 -1
- pearmut/static/dashboard.html +1 -1
- pearmut/static/listwise.bundle.js +1 -1
- pearmut/static/listwise.html +1 -1
- pearmut/static/pointwise.bundle.js +1 -1
- pearmut/static/pointwise.html +1 -1
- {pearmut-0.1.3.dist-info → pearmut-0.2.0.dist-info}/METADATA +30 -6
- pearmut-0.2.0.dist-info/RECORD +19 -0
- pearmut-0.1.3.dist-info/RECORD +0 -19
- {pearmut-0.1.3.dist-info → pearmut-0.2.0.dist-info}/WHEEL +0 -0
- {pearmut-0.1.3.dist-info → pearmut-0.2.0.dist-info}/entry_points.txt +0 -0
- {pearmut-0.1.3.dist-info → pearmut-0.2.0.dist-info}/licenses/LICENSE +0 -0
- {pearmut-0.1.3.dist-info → pearmut-0.2.0.dist-info}/top_level.txt +0 -0
pearmut/static/pointwise.html
CHANGED
|
@@ -66,4 +66,4 @@
|
|
|
66
66
|
direction: rtl;
|
|
67
67
|
width: 16px;
|
|
68
68
|
height: 200px;
|
|
69
|
-
}</style><script defer="defer" src="pointwise.bundle.js"></script></head><body><div style="max-width: 1600px; min-width: 900px; margin-left: auto; margin-right: auto; margin-top: 20px; padding-left: 10px;"><div class="white-box" style="margin-right: 30px; background-color: #e7e2cf; padding: 5px 15px 5px 5px;"><span id="instructions_global" style="display: inline-block; font-size: 11pt; width: calc(100% - 170px);"><ul id="instructions_spans"
|
|
69
|
+
}</style><script defer="defer" src="pointwise.bundle.js"></script></head><body><div style="max-width: 1600px; min-width: 900px; margin-left: auto; margin-right: auto; margin-top: 20px; padding-left: 10px;"><div class="white-box" style="margin-right: 30px; background-color: #e7e2cf; padding: 5px 15px 5px 5px;"><span id="instructions_global" style="display: inline-block; font-size: 11pt; width: calc(100% - 170px);"><ul><li id="instructions_spans">Error spans:<ul><li><strong>Select</strong> the part of translation where you have identified a <strong>translation error</strong> (drag or click start & end).</li><li><strong>Click</strong> on the highlight to change error severity (minor/major) or remove the highlight.</li></ul>Choose error severity:<ul><li><span class="instruction_sev" id="instruction_sev_minor">Minor errors:</span> Style, grammar, word choice could be better or more natural.</li><li><span class="instruction_sev" id="instruction_sev_major">Major errors:</span>: The meaning is changed significantly and/or the part is really hard to understand.</li></ul><strong>Tip</strong>: Highlight the word or general area of the error (it doesn't need to be exact). Use separate highlights for different errors.<br></li><li id="instructions_score">Score the translation: Please use the slider and set an overall score based on meaning preservation and general quality:</li><ul><li>0: <strong>No meaning preserved</strong>: most information is lost.</li><li>33%: <strong>Some meaning preserved</strong>: major gaps and narrative issues.</li><li>66%: <strong>Most meaning preserved</strong>: minor issues with grammar or consistency.</li><li>100%: <strong>Perfect</strong>: meaning and grammar align completely with the source.</li></ul><li id="instructions_categories">Error types: After highlighting an error fragment, you will be asked to select the specific error type (main category and subcategory). If you are unsure about which errors fall under which categories, please consult the <a href="https://themqm.org/the-mqm-typology/" style="font-weight: bold; text-decoration: none; color: black;">typology definitions</a>.</li></ul></span><div style="width: 170px; display: inline-block; vertical-align: top; text-align: right; padding-top: 5px;"><span id="time" style="width: 135px; text-align: left; display: inline-block; font-size: 11pt;" title="Approximation of total annotation time.">Time: 0m</span> <input type="button" value="⚙️" id="button_settings" style="height: 1.5em; width: 30px;"><br><br><div id="progress" style="text-align: center;"></div><br><br><input type="button" value="Next 🛠️" id="button_next" disabled="disabled" style="width: 170px; height: 2.5em;" title="Finish annotating all examples first."> <input type="button" value="skip tutorial" id="button_skip_tutorial" style="width: 170px; font-size: 11pt; height: 30px; margin-top: 10px; display: none;" title="Skip tutorial only if you completed it already."></div></div><div id="settings_div" class="white-box" style="margin-right: 20px; margin-top: 10px; display: none; background-color: #e7e2cf; font-size: 11pt;"><input type="checkbox" id="settings_approximate_alignment"> <label for="settings_approximate_alignment">Show approximate alignment</label></div><div id="output_div" style="margin-top: 100px;"></div><br><br><br></div></body></html>
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: pearmut
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.2.0
|
|
4
4
|
Summary: A tool for evaluation of model outputs, primarily MT.
|
|
5
5
|
Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
|
|
6
6
|
License: apache-2.0
|
|
@@ -42,11 +42,11 @@ You do not need to clone this repository. Simply install with pip and run locall
|
|
|
42
42
|
# install the package
|
|
43
43
|
pip install pearmut
|
|
44
44
|
# download two campaign definitions
|
|
45
|
-
wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/
|
|
46
|
-
wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/
|
|
45
|
+
wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/esa_encs.json
|
|
46
|
+
wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/da_enuk.json
|
|
47
47
|
# load them into pearmut
|
|
48
|
-
pearmut add
|
|
49
|
-
pearmut add
|
|
48
|
+
pearmut add esa_encs.json
|
|
49
|
+
pearmut add da_enuk.json
|
|
50
50
|
# start pearmut (will show management links)
|
|
51
51
|
pearmut run
|
|
52
52
|
```
|
|
@@ -145,6 +145,30 @@ For **listwise** template, `error_spans` is a 2D array where each inner array co
|
|
|
145
145
|
|
|
146
146
|
See [examples/esaai_prefilled.json](examples/esaai_prefilled.json) for a complete example.
|
|
147
147
|
|
|
148
|
+
## Tutorial and Attention Checks
|
|
149
|
+
|
|
150
|
+
You can add validation rules to items for tutorials or attention checks. Items with `validation` field will be checked before submission:
|
|
151
|
+
|
|
152
|
+
```python
|
|
153
|
+
{
|
|
154
|
+
"src": "The quick brown fox jumps.",
|
|
155
|
+
"tgt": "Rychlá hnědá liška skáče.",
|
|
156
|
+
"validation": {
|
|
157
|
+
"warning": "Please set score between 70-80.", # shown on failure (omit for silent logging)
|
|
158
|
+
"score": [70, 80], # required score range [min, max]
|
|
159
|
+
"error_spans": [{"start_i": [0, 2], "end_i": [4, 8], "severity": "minor"}], # expected spans
|
|
160
|
+
"allow_skip": true # show "skip tutorial" button
|
|
161
|
+
}
|
|
162
|
+
}
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
- Tutorial items: Include `allow_skip: true` and `warning` to let users skip after seeing the feedback
|
|
166
|
+
- Loud attention checks: Include `warning` without `allow_skip` to force users to retry
|
|
167
|
+
- Silent attention checks: Omit `warning` to silently log failures without user notification (useful for quality control with bad translations)
|
|
168
|
+
For listwise template, `validation` is an array where each element corresponds to a candidate.
|
|
169
|
+
The dashboard shows failed/total validation checks per user.
|
|
170
|
+
See [examples/tutorial_pointwise.json](examples/tutorial_pointwise.json) and [examples/tutorial_listwise.json](examples/tutorial_listwise.json) for complete examples.
|
|
171
|
+
|
|
148
172
|
## Single-stream Assignment
|
|
149
173
|
|
|
150
174
|
We also support a simple allocation where all annotators draw from the same pool (`single-stream`). Items are randomly assigned to annotators from the pool of unfinished items:
|
|
@@ -260,7 +284,7 @@ If you use this work in your paper, please cite as:
|
|
|
260
284
|
```bibtex
|
|
261
285
|
@misc{zouhar2025pearmut,
|
|
262
286
|
author={Vilém Zouhar},
|
|
263
|
-
title={Pearmut
|
|
287
|
+
title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
|
|
264
288
|
url={https://github.com/zouharvi/pearmut/},
|
|
265
289
|
year={2025},
|
|
266
290
|
}
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
pearmut/app.py,sha256=D4wk5HjEDFtkDakSWUBOb8sKRsbi_dBw3yyL1n6jhpQ,7957
|
|
2
|
+
pearmut/assignment.py,sha256=Sycq-_6BTjpm7KPSZ02zX9aTZxOr-zaxW5QbZpQlqV8,10415
|
|
3
|
+
pearmut/cli.py,sha256=xB05Fq8Ic1ucSxHWYBTtqWssFz0FwoLzHO7RFAG2vcc,7684
|
|
4
|
+
pearmut/utils.py,sha256=gk8b4biPc9TTvZiQMQ_8xh1_FsWuwrhtPzeK3NpzhZc,2902
|
|
5
|
+
pearmut/static/dashboard.bundle.js,sha256=NWGQfd0kXkSkpElCukPrMIPJROE8mMIkvhRwHHMzuAA,91528
|
|
6
|
+
pearmut/static/dashboard.html,sha256=lleOeCqjaCHM5ZG45Q5eM8vWxW65CTmJR3PEJbUKREE,1790
|
|
7
|
+
pearmut/static/index.html,sha256=ieCRLK83MVe-f-gtjYiOlvE-kKd8VnFF2xgyi6FoZpU,872
|
|
8
|
+
pearmut/static/listwise.bundle.js,sha256=UEb1smJt4kgeZU2FUqyc7jWGYQCzV8ri-1bZJXBxGHY,104819
|
|
9
|
+
pearmut/static/listwise.html,sha256=1z83PNGRR_4NEQ8kYxP19Aem_ew5CAKhKtcn2zxGL3M,5212
|
|
10
|
+
pearmut/static/pointwise.bundle.js,sha256=zd8U5tyYb3-IhF_07njSB9Nkab76ZYTj70Q1YPBlKkU,107171
|
|
11
|
+
pearmut/static/pointwise.html,sha256=lvplPE-9RxA-IFWkvzMEVGdroHN68qK9hvzMSuj-mmo,5009
|
|
12
|
+
pearmut/static/assets/favicon.svg,sha256=gVPxdBlyfyJVkiMfh8WLaiSyH4lpwmKZs8UiOeX8YW4,7347
|
|
13
|
+
pearmut/static/assets/style.css,sha256=SARZqqovP_2s9S5ENI7dxJ6Hacz-ztQ2zn2Hn7DwoJU,4089
|
|
14
|
+
pearmut-0.2.0.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
|
|
15
|
+
pearmut-0.2.0.dist-info/METADATA,sha256=fffWQPgx2ytlttm1vmij2gphQz9bDmnVhRakenrsyeM,12266
|
|
16
|
+
pearmut-0.2.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
|
|
17
|
+
pearmut-0.2.0.dist-info/entry_points.txt,sha256=eEA9LVWsS3neQbMvL_nMvEw8I0oFudw8nQa1iqxOiWM,45
|
|
18
|
+
pearmut-0.2.0.dist-info/top_level.txt,sha256=CdgtUM-SKQDt6o5g0QreO-_7XTBP9_wnHMS1P-Rl5Go,8
|
|
19
|
+
pearmut-0.2.0.dist-info/RECORD,,
|
pearmut-0.1.3.dist-info/RECORD
DELETED
|
@@ -1,19 +0,0 @@
|
|
|
1
|
-
pearmut/app.py,sha256=ymRlnpKrWSiwdc51Tw4PBDDFFOY1bmdeU-xJ2VlOl-Q,7393
|
|
2
|
-
pearmut/assignment.py,sha256=aOQNlGYzzPNgunAmIIwlcF4qY-l-w6Wmy7hGquArAsc,10623
|
|
3
|
-
pearmut/cli.py,sha256=mV76uw6BywckbU7QEKIKTboukcALEdZp7l-kskJnBVA,7683
|
|
4
|
-
pearmut/utils.py,sha256=gk8b4biPc9TTvZiQMQ_8xh1_FsWuwrhtPzeK3NpzhZc,2902
|
|
5
|
-
pearmut/static/dashboard.bundle.js,sha256=6389gsHLCFh6JqiKdU3ng-Lm6VICRvfJgCSYM61H75U,91257
|
|
6
|
-
pearmut/static/dashboard.html,sha256=tUP1yYvbKySRz0mxFtGq2Si4hTMhJkUCWeTpnq91Nf4,1789
|
|
7
|
-
pearmut/static/index.html,sha256=ieCRLK83MVe-f-gtjYiOlvE-kKd8VnFF2xgyi6FoZpU,872
|
|
8
|
-
pearmut/static/listwise.bundle.js,sha256=Qcz3TSA8C5QRFI-ui47y99WF87wf_4tMKHZ3TyfiYa8,103790
|
|
9
|
-
pearmut/static/listwise.html,sha256=MNS4gV1Fqx7JXZikLhrWgL0z1OPdqgumlOfTcmGnXEI,5212
|
|
10
|
-
pearmut/static/pointwise.bundle.js,sha256=doa3DC8n9L7IIV2ttWxV-TBKVMQHgjTQgSR3Pjozy3k,106133
|
|
11
|
-
pearmut/static/pointwise.html,sha256=dhmfgpWvCFB833Y4kj08_aBZyCN33SayYcS1ckL2-FU,5009
|
|
12
|
-
pearmut/static/assets/favicon.svg,sha256=gVPxdBlyfyJVkiMfh8WLaiSyH4lpwmKZs8UiOeX8YW4,7347
|
|
13
|
-
pearmut/static/assets/style.css,sha256=-B-RySjt8qccqkwvLT0PDy6IRoE1xytLLKAFtR_S-Tg,3967
|
|
14
|
-
pearmut-0.1.3.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
|
|
15
|
-
pearmut-0.1.3.dist-info/METADATA,sha256=XhlUE5eAzWzZ1MQX4RmPQuM5Kijk_LwYahgQvTbmmp4,10990
|
|
16
|
-
pearmut-0.1.3.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
|
|
17
|
-
pearmut-0.1.3.dist-info/entry_points.txt,sha256=eEA9LVWsS3neQbMvL_nMvEw8I0oFudw8nQa1iqxOiWM,45
|
|
18
|
-
pearmut-0.1.3.dist-info/top_level.txt,sha256=CdgtUM-SKQDt6o5g0QreO-_7XTBP9_wnHMS1P-Rl5Go,8
|
|
19
|
-
pearmut-0.1.3.dist-info/RECORD,,
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|