pearmut 0.2.9__py3-none-any.whl → 0.2.10__py3-none-any.whl

{pearmut-0.2.9.dist-info → pearmut-0.2.10.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.2.9
+Version: 0.2.10
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: MIT
@@ -47,9 +47,13 @@ Dynamic: license-file
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
 - [CLI Commands](#cli-commands)
+- [Terminology](#terminology)
 - [Development](#development)
 - [Citation](#citation)
 
+
+**Error Span** — A highlighted segment of text marked as containing an error, with optional severity (`minor`, `major`, `neutral`) and MQM category labels.
+
 ## Quick Start
 
 Install and run locally without cloning:
@@ -294,6 +298,38 @@ Items need `model` field (pointwise) or `models` field (listwise) and the `proto
 ```
 See an example in [Campaign Management](#campaign-management)
 
+
+## Terminology
+
+- **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
+  - **Campaign File**: A JSON file that defines the campaign configuration, including the campaign ID, assignment type, protocol settings, and annotation data.
+  - **Campaign ID**: A unique identifier for a campaign (e.g., `"wmt25_#_en-cs_CZ"`). Used to reference and manage specific campaigns.
+- **Task**: A unit of work assigned to a user. In task-based assignment, each task consists of a predefined set of items for a specific user.
+- **Item** — A single annotation unit within a task. For translation evaluation, an item typically represents a document (source text and target translation). Items can contain text, images, audio, or video.
+- **Document** — A collection of one or more segments (sentence pairs or text units) that are evaluated together as a single item.
+- **User** / **Annotator**: A person who performs annotations in a campaign. Each user is identified by a unique user ID and accesses the campaign through a unique URL.
+- **Attention Check** — A validation item with known correct answers used to ensure annotator quality. Can be:
+  - **Loud**: Shows warning message and forces retry on failure
+  - **Silent**: Logs failures without notifying the user (for quality control analysis)
+  - **Token** — A completion code shown to users when they finish their annotations. Tokens verify the completion and whether the user passed quality control checks:
+    - **Pass Token** (`token_pass`): Shown when user meets validation thresholds
+    - **Fail Token** (`token_fail`): Shown when user fails to meet validation requirements
+- **Tutorial**: An instructional validation item that teaches users how to annotate. Includes `allow_skip: true` to let users skip if they have seen it before.
+- **Validation**: Quality control rules attached to items that check if annotations match expected criteria (score ranges, error span locations, etc.). Used for tutorials and attention checks.
+- **Model**: The system or model that generated the output being evaluated (e.g., `"GPT-4"`, `"Claude"`). Used for tracking and ranking model performance.
+- **Dashboard**: The management interface that shows campaign progress, annotator statistics, access links, and allows downloading annotations. Accessed via a special management URL with token authentication.
+- **Protocol**: The annotation scheme defining what data is collected:
+  - **Score**: Numeric quality rating (0-100)
+  - **Error Spans**: Text highlights marking errors
+  - **Error Categories**: MQM taxonomy labels for errors
+- **Template**: The annotation interface type:
+  - **Pointwise**: Evaluate one output at a time
+  - **Listwise**: Compare multiple outputs simultaneously
+- **Assignment**: The method for distributing items to users:
+  - **Task-based**: Each user has predefined items
+  - **Single-stream**: Users draw from a shared pool with random assignment
+  - **Dynamic**: Work in progress
+
 ## Development
 
 Server responds to data-only requests from frontend (no template coupling). Frontend served from pre-built `static/` on install.
@@ -333,7 +369,7 @@ If you use this work in your paper, please cite as following.
     author={Vilém Zouhar},
     title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
-    year={2025},
+    year={2026},
 }
 ```
 

pearmut-0.2.10.dist-info/RECORD

@@ -0,0 +1,19 @@
+pearmut/app.py,sha256=fDec9vOX-ExgnlctoHlI19btxmPlOoSvFURiAHzGMmw,10437
+pearmut/assignment.py,sha256=GvulwsPEguA_rNZB58bDKYy1wVZX9j4vnmbrKH4m0Mo,10963
+pearmut/cli.py,sha256=oaViD9vlvFW3WIs2Ncm5DSUWRqObDZ1iTcAt1HAdaYg,17686
+pearmut/utils.py,sha256=TWcbdTehg4CNwCpc5FuEOszpQM464LY0IQHHE_Sq1Zg,5293
+pearmut/static/dashboard.bundle.js,sha256=IWoc8wETOX53ibPvcdEcnNUsJtugAJhn6NGYr9KB7OQ,100268
+pearmut/static/dashboard.html,sha256=w1xNgLakDMxzp9iDp18SOoKHO10kB7ldvzuuwsC0zxk,2694
+pearmut/static/index.html,sha256=SC5M-NSTnJh1UNHCC5VOP0TKkmhNn6MHlY6L4GDacpA,849
+pearmut/static/listwise.bundle.js,sha256=vX5lMUxxjll7wT1NsZjZEIiruKuSkrCY6Vt6YIKeJ58,109635
+pearmut/static/listwise.html,sha256=4A0a_GMVIjJmqT3lhJMT9huqvwgvrRfztt0KA0lJxKI,5308
+pearmut/static/pointwise.bundle.js,sha256=ylaXUnTp_v2h8LeQthqR22nlIoKEF-ryoyqQFRoC-nk,108974
+pearmut/static/pointwise.html,sha256=2NZYyjpznXP2b4GMeDcrjRYI5hZ45l7QgI-RQjkRUqs,5024
+pearmut/static/assets/favicon.svg,sha256=gVPxdBlyfyJVkiMfh8WLaiSyH4lpwmKZs8UiOeX8YW4,7347
+pearmut/static/assets/style.css,sha256=BrPnXTDr8hQ0M8T-EJlExddChzIFotlerBYMx2B8GDk,4136
+pearmut-0.2.10.dist-info/licenses/LICENSE,sha256=GtR6RcTdRn-P23h5pKFuWSLZrLPD0ytHAwSOBt7aLpI,1071
+pearmut-0.2.10.dist-info/METADATA,sha256=FLuvt03haoPPaDBBSzh9dKYFCBGVkP8JJ5udkOOc3yA,15676
+pearmut-0.2.10.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+pearmut-0.2.10.dist-info/entry_points.txt,sha256=eEA9LVWsS3neQbMvL_nMvEw8I0oFudw8nQa1iqxOiWM,45
+pearmut-0.2.10.dist-info/top_level.txt,sha256=CdgtUM-SKQDt6o5g0QreO-_7XTBP9_wnHMS1P-Rl5Go,8
+pearmut-0.2.10.dist-info/RECORD,,

pearmut-0.2.9.dist-info/RECORD

@@ -1,19 +0,0 @@
-pearmut/app.py,sha256=FV7pEf-TzMmATLLpPMIjWOwznxaND8IYX_jO_s0h9oQ,10236
-pearmut/assignment.py,sha256=GvulwsPEguA_rNZB58bDKYy1wVZX9j4vnmbrKH4m0Mo,10963
-pearmut/cli.py,sha256=oaViD9vlvFW3WIs2Ncm5DSUWRqObDZ1iTcAt1HAdaYg,17686
-pearmut/utils.py,sha256=TWcbdTehg4CNwCpc5FuEOszpQM464LY0IQHHE_Sq1Zg,5293
-pearmut/static/dashboard.bundle.js,sha256=9cjhcY57uEOaJNcrOdn4lpMpQIfsRtPlr-rcQBQ92K0,100268
-pearmut/static/dashboard.html,sha256=w1xNgLakDMxzp9iDp18SOoKHO10kB7ldvzuuwsC0zxk,2694
-pearmut/static/index.html,sha256=SC5M-NSTnJh1UNHCC5VOP0TKkmhNn6MHlY6L4GDacpA,849
-pearmut/static/listwise.bundle.js,sha256=BqHFM-o5Eg0FvCaa2-BoLtWffiH1F0yJV76wXRlGOEM,109635
-pearmut/static/listwise.html,sha256=4A0a_GMVIjJmqT3lhJMT9huqvwgvrRfztt0KA0lJxKI,5308
-pearmut/static/pointwise.bundle.js,sha256=cuCT-quJzsRzorS8ojaQ0uXbDix-niS1txKOhq8QirE,108974
-pearmut/static/pointwise.html,sha256=2NZYyjpznXP2b4GMeDcrjRYI5hZ45l7QgI-RQjkRUqs,5024
-pearmut/static/assets/favicon.svg,sha256=gVPxdBlyfyJVkiMfh8WLaiSyH4lpwmKZs8UiOeX8YW4,7347
-pearmut/static/assets/style.css,sha256=BrPnXTDr8hQ0M8T-EJlExddChzIFotlerBYMx2B8GDk,4136
-pearmut-0.2.9.dist-info/licenses/LICENSE,sha256=GtR6RcTdRn-P23h5pKFuWSLZrLPD0ytHAwSOBt7aLpI,1071
-pearmut-0.2.9.dist-info/METADATA,sha256=_qNEVQtaI1wAaECmY8PTTvmHQzs5j6ks67A5VaZWZy8,12429
-pearmut-0.2.9.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-pearmut-0.2.9.dist-info/entry_points.txt,sha256=eEA9LVWsS3neQbMvL_nMvEw8I0oFudw8nQa1iqxOiWM,45
-pearmut-0.2.9.dist-info/top_level.txt,sha256=CdgtUM-SKQDt6o5g0QreO-_7XTBP9_wnHMS1P-Rl5Go,8
-pearmut-0.2.9.dist-info/RECORD,,