pearmut 0.2.8__py3-none-any.whl → 0.2.10__py3-none-any.whl

{pearmut-0.2.8.dist-info → pearmut-0.2.10.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.2.8
+Version: 0.2.10
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: MIT
@@ -47,9 +47,13 @@ Dynamic: license-file
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
 - [CLI Commands](#cli-commands)
+- [Terminology](#terminology)
 - [Development](#development)
 - [Citation](#citation)
 
+
+**Error Span** — A highlighted segment of text marked as containing an error, with optional severity (`minor`, `major`, `neutral`) and MQM category labels.
+
 ## Quick Start
 
 Install and run locally without cloning:
@@ -278,12 +282,54 @@ Management link (shown when adding campaigns or running server) provides:
 - Task progress reset (data preserved)
 - Download progress and annotations
 
-<img width="800" alt="Management dashboard" src="https://github.com/user-attachments/assets/800a1741-5f41-47ac-9d5d-5cbf6abfc0e6" />
+<img width="1000" alt="Management dashboard" src="https://github.com/user-attachments/assets/8953252c-d7b1-428c-a974-5bc7501457c7" />
 
 Completion tokens are shown at annotation end for verification (download correct tokens from dashboard). Incorrect tokens can be shown if quality control fails.
 
 <img width="500" alt="Token on completion" src="https://github.com/user-attachments/assets/40eb904c-f47a-4011-aa63-9a4f1c501549" />
 
+### Model Results Display
+
+Add `&results` to dashboard URL to show model rankings (requires valid token).
+Items need `model` field (pointwise) or `models` field (listwise) and the `protocol_score` needs to be enable such that the `score` can be used for the ranking:
+```python
+{"doc_id": "1", "model": "CommandA", "src": "...", "tgt": "..."}
+{"doc_id": "2", "models": ["CommandA", "Claude"], "src": "...", "tgt": ["...", "..."]}
+```
+See an example in [Campaign Management](#campaign-management)
+
+
+## Terminology
+
+- **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
+  - **Campaign File**: A JSON file that defines the campaign configuration, including the campaign ID, assignment type, protocol settings, and annotation data.
+  - **Campaign ID**: A unique identifier for a campaign (e.g., `"wmt25_#_en-cs_CZ"`). Used to reference and manage specific campaigns.
+- **Task**: A unit of work assigned to a user. In task-based assignment, each task consists of a predefined set of items for a specific user.
+- **Item** — A single annotation unit within a task. For translation evaluation, an item typically represents a document (source text and target translation). Items can contain text, images, audio, or video.
+- **Document** — A collection of one or more segments (sentence pairs or text units) that are evaluated together as a single item.
+- **User** / **Annotator**: A person who performs annotations in a campaign. Each user is identified by a unique user ID and accesses the campaign through a unique URL.
+- **Attention Check** — A validation item with known correct answers used to ensure annotator quality. Can be:
+  - **Loud**: Shows warning message and forces retry on failure
+  - **Silent**: Logs failures without notifying the user (for quality control analysis)
+  - **Token** — A completion code shown to users when they finish their annotations. Tokens verify the completion and whether the user passed quality control checks:
+    - **Pass Token** (`token_pass`): Shown when user meets validation thresholds
+    - **Fail Token** (`token_fail`): Shown when user fails to meet validation requirements
+- **Tutorial**: An instructional validation item that teaches users how to annotate. Includes `allow_skip: true` to let users skip if they have seen it before.
+- **Validation**: Quality control rules attached to items that check if annotations match expected criteria (score ranges, error span locations, etc.). Used for tutorials and attention checks.
+- **Model**: The system or model that generated the output being evaluated (e.g., `"GPT-4"`, `"Claude"`). Used for tracking and ranking model performance.
+- **Dashboard**: The management interface that shows campaign progress, annotator statistics, access links, and allows downloading annotations. Accessed via a special management URL with token authentication.
+- **Protocol**: The annotation scheme defining what data is collected:
+  - **Score**: Numeric quality rating (0-100)
+  - **Error Spans**: Text highlights marking errors
+  - **Error Categories**: MQM taxonomy labels for errors
+- **Template**: The annotation interface type:
+  - **Pointwise**: Evaluate one output at a time
+  - **Listwise**: Compare multiple outputs simultaneously
+- **Assignment**: The method for distributing items to users:
+  - **Task-based**: Each user has predefined items
+  - **Single-stream**: Users draw from a shared pool with random assignment
+  - **Dynamic**: Work in progress
+
 ## Development
 
 Server responds to data-only requests from frontend (no template coupling). Frontend served from pre-built `static/` on install.
@@ -295,7 +341,7 @@ cd pearmut
 npm install web/ --prefix web/
 npm run build --prefix web/
 # optionally keep running indefinitely to auto-rebuild
-npm watch build --prefix web/
+npm run watch --prefix web/
 
 # Install as editable
 pip3 install -e .
@@ -323,7 +369,7 @@ If you use this work in your paper, please cite as following.
     author={Vilém Zouhar},
     title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
-    year={2025},
+    year={2026},
 }
 ```
 

pearmut-0.2.10.dist-info/RECORD

@@ -0,0 +1,19 @@
+pearmut/app.py,sha256=fDec9vOX-ExgnlctoHlI19btxmPlOoSvFURiAHzGMmw,10437
+pearmut/assignment.py,sha256=GvulwsPEguA_rNZB58bDKYy1wVZX9j4vnmbrKH4m0Mo,10963
+pearmut/cli.py,sha256=oaViD9vlvFW3WIs2Ncm5DSUWRqObDZ1iTcAt1HAdaYg,17686
+pearmut/utils.py,sha256=TWcbdTehg4CNwCpc5FuEOszpQM464LY0IQHHE_Sq1Zg,5293
+pearmut/static/dashboard.bundle.js,sha256=IWoc8wETOX53ibPvcdEcnNUsJtugAJhn6NGYr9KB7OQ,100268
+pearmut/static/dashboard.html,sha256=w1xNgLakDMxzp9iDp18SOoKHO10kB7ldvzuuwsC0zxk,2694
+pearmut/static/index.html,sha256=SC5M-NSTnJh1UNHCC5VOP0TKkmhNn6MHlY6L4GDacpA,849
+pearmut/static/listwise.bundle.js,sha256=vX5lMUxxjll7wT1NsZjZEIiruKuSkrCY6Vt6YIKeJ58,109635
+pearmut/static/listwise.html,sha256=4A0a_GMVIjJmqT3lhJMT9huqvwgvrRfztt0KA0lJxKI,5308
+pearmut/static/pointwise.bundle.js,sha256=ylaXUnTp_v2h8LeQthqR22nlIoKEF-ryoyqQFRoC-nk,108974
+pearmut/static/pointwise.html,sha256=2NZYyjpznXP2b4GMeDcrjRYI5hZ45l7QgI-RQjkRUqs,5024
+pearmut/static/assets/favicon.svg,sha256=gVPxdBlyfyJVkiMfh8WLaiSyH4lpwmKZs8UiOeX8YW4,7347
+pearmut/static/assets/style.css,sha256=BrPnXTDr8hQ0M8T-EJlExddChzIFotlerBYMx2B8GDk,4136
+pearmut-0.2.10.dist-info/licenses/LICENSE,sha256=GtR6RcTdRn-P23h5pKFuWSLZrLPD0ytHAwSOBt7aLpI,1071
+pearmut-0.2.10.dist-info/METADATA,sha256=FLuvt03haoPPaDBBSzh9dKYFCBGVkP8JJ5udkOOc3yA,15676
+pearmut-0.2.10.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+pearmut-0.2.10.dist-info/entry_points.txt,sha256=eEA9LVWsS3neQbMvL_nMvEw8I0oFudw8nQa1iqxOiWM,45
+pearmut-0.2.10.dist-info/top_level.txt,sha256=CdgtUM-SKQDt6o5g0QreO-_7XTBP9_wnHMS1P-Rl5Go,8
+pearmut-0.2.10.dist-info/RECORD,,

pearmut-0.2.8.dist-info/RECORD

@@ -1,19 +0,0 @@
-pearmut/app.py,sha256=QrAAMQI8L922AD6biirItlwDB6gT90e1HTBH_txkPyM,8298
-pearmut/assignment.py,sha256=GvulwsPEguA_rNZB58bDKYy1wVZX9j4vnmbrKH4m0Mo,10963
-pearmut/cli.py,sha256=gZRmniwOHHtWS1rInUU1_zuuq3nybkW3dlMjJtyvlBM,17620
-pearmut/utils.py,sha256=TWcbdTehg4CNwCpc5FuEOszpQM464LY0IQHHE_Sq1Zg,5293
-pearmut/static/dashboard.bundle.js,sha256=9eHVyQd65pu1AJUyipKu8fWyOw5x1Jmf0KvbtAWuZm4,98636
-pearmut/static/dashboard.html,sha256=fN-B0jyeezMZP4qisGA7lmQem-FqvfDP1i5ziErQK2M,2120
-pearmut/static/index.html,sha256=SC5M-NSTnJh1UNHCC5VOP0TKkmhNn6MHlY6L4GDacpA,849
-pearmut/static/listwise.bundle.js,sha256=dR-CQ9r8OBhz71Usv9Lfg1RT9OMndiKZEvqPMBuDUho,109029
-pearmut/static/listwise.html,sha256=4A0a_GMVIjJmqT3lhJMT9huqvwgvrRfztt0KA0lJxKI,5308
-pearmut/static/pointwise.bundle.js,sha256=gX6bfzPqOK0xMTqcZtbrLM9TbICVbiicD5c4b2bw-AM,111274
-pearmut/static/pointwise.html,sha256=2NZYyjpznXP2b4GMeDcrjRYI5hZ45l7QgI-RQjkRUqs,5024
-pearmut/static/assets/favicon.svg,sha256=gVPxdBlyfyJVkiMfh8WLaiSyH4lpwmKZs8UiOeX8YW4,7347
-pearmut/static/assets/style.css,sha256=BrPnXTDr8hQ0M8T-EJlExddChzIFotlerBYMx2B8GDk,4136
-pearmut-0.2.8.dist-info/licenses/LICENSE,sha256=GtR6RcTdRn-P23h5pKFuWSLZrLPD0ytHAwSOBt7aLpI,1071
-pearmut-0.2.8.dist-info/METADATA,sha256=qrMERkMaAMsYmGjJm0Dny2tG6AWw3m3YVg5b7aGRVJ4,11934
-pearmut-0.2.8.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-pearmut-0.2.8.dist-info/entry_points.txt,sha256=eEA9LVWsS3neQbMvL_nMvEw8I0oFudw8nQa1iqxOiWM,45
-pearmut-0.2.8.dist-info/top_level.txt,sha256=CdgtUM-SKQDt6o5g0QreO-_7XTBP9_wnHMS1P-Rl5Go,8
-pearmut-0.2.8.dist-info/RECORD,,