pearmut 0.2.9__tar.gz → 0.2.10__tar.gz

{pearmut-0.2.9 → pearmut-0.2.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.2.9
+Version: 0.2.10
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: MIT
@@ -47,9 +47,13 @@ Dynamic: license-file
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
 - [CLI Commands](#cli-commands)
+- [Terminology](#terminology)
 - [Development](#development)
 - [Citation](#citation)
 
+
+**Error Span** — A highlighted segment of text marked as containing an error, with optional severity (`minor`, `major`, `neutral`) and MQM category labels.
+
 ## Quick Start
 
 Install and run locally without cloning:
@@ -294,6 +298,38 @@ Items need `model` field (pointwise) or `models` field (listwise) and the `proto
 ```
 See an example in [Campaign Management](#campaign-management)
 
+
+## Terminology
+
+- **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
+  - **Campaign File**: A JSON file that defines the campaign configuration, including the campaign ID, assignment type, protocol settings, and annotation data.
+  - **Campaign ID**: A unique identifier for a campaign (e.g., `"wmt25_#_en-cs_CZ"`). Used to reference and manage specific campaigns.
+- **Task**: A unit of work assigned to a user. In task-based assignment, each task consists of a predefined set of items for a specific user.
+- **Item** — A single annotation unit within a task. For translation evaluation, an item typically represents a document (source text and target translation). Items can contain text, images, audio, or video.
+- **Document** — A collection of one or more segments (sentence pairs or text units) that are evaluated together as a single item.
+- **User** / **Annotator**: A person who performs annotations in a campaign. Each user is identified by a unique user ID and accesses the campaign through a unique URL.
+- **Attention Check** — A validation item with known correct answers used to ensure annotator quality. Can be:
+  - **Loud**: Shows warning message and forces retry on failure
+  - **Silent**: Logs failures without notifying the user (for quality control analysis)
+  - **Token** — A completion code shown to users when they finish their annotations. Tokens verify the completion and whether the user passed quality control checks:
+    - **Pass Token** (`token_pass`): Shown when user meets validation thresholds
+    - **Fail Token** (`token_fail`): Shown when user fails to meet validation requirements
+- **Tutorial**: An instructional validation item that teaches users how to annotate. Includes `allow_skip: true` to let users skip if they have seen it before.
+- **Validation**: Quality control rules attached to items that check if annotations match expected criteria (score ranges, error span locations, etc.). Used for tutorials and attention checks.
+- **Model**: The system or model that generated the output being evaluated (e.g., `"GPT-4"`, `"Claude"`). Used for tracking and ranking model performance.
+- **Dashboard**: The management interface that shows campaign progress, annotator statistics, access links, and allows downloading annotations. Accessed via a special management URL with token authentication.
+- **Protocol**: The annotation scheme defining what data is collected:
+  - **Score**: Numeric quality rating (0-100)
+  - **Error Spans**: Text highlights marking errors
+  - **Error Categories**: MQM taxonomy labels for errors
+- **Template**: The annotation interface type:
+  - **Pointwise**: Evaluate one output at a time
+  - **Listwise**: Compare multiple outputs simultaneously
+- **Assignment**: The method for distributing items to users:
+  - **Task-based**: Each user has predefined items
+  - **Single-stream**: Users draw from a shared pool with random assignment
+  - **Dynamic**: Work in progress
+
 ## Development
 
 Server responds to data-only requests from frontend (no template coupling). Frontend served from pre-built `static/` on install.
@@ -333,7 +369,7 @@ If you use this work in your paper, please cite as following.
     author={Vilém Zouhar},
     title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
-    year={2025},
+    year={2026},
 }
 ```
 

{pearmut-0.2.9 → pearmut-0.2.10}/README.md

@@ -27,9 +27,13 @@
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
 - [CLI Commands](#cli-commands)
+- [Terminology](#terminology)
 - [Development](#development)
 - [Citation](#citation)
 
+
+**Error Span** — A highlighted segment of text marked as containing an error, with optional severity (`minor`, `major`, `neutral`) and MQM category labels.
+
 ## Quick Start
 
 Install and run locally without cloning:
@@ -274,6 +278,38 @@ Items need `model` field (pointwise) or `models` field (listwise) and the `proto
 ```
 See an example in [Campaign Management](#campaign-management)
 
+
+## Terminology
+
+- **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
+  - **Campaign File**: A JSON file that defines the campaign configuration, including the campaign ID, assignment type, protocol settings, and annotation data.
+  - **Campaign ID**: A unique identifier for a campaign (e.g., `"wmt25_#_en-cs_CZ"`). Used to reference and manage specific campaigns.
+- **Task**: A unit of work assigned to a user. In task-based assignment, each task consists of a predefined set of items for a specific user.
+- **Item** — A single annotation unit within a task. For translation evaluation, an item typically represents a document (source text and target translation). Items can contain text, images, audio, or video.
+- **Document** — A collection of one or more segments (sentence pairs or text units) that are evaluated together as a single item.
+- **User** / **Annotator**: A person who performs annotations in a campaign. Each user is identified by a unique user ID and accesses the campaign through a unique URL.
+- **Attention Check** — A validation item with known correct answers used to ensure annotator quality. Can be:
+  - **Loud**: Shows warning message and forces retry on failure
+  - **Silent**: Logs failures without notifying the user (for quality control analysis)
+  - **Token** — A completion code shown to users when they finish their annotations. Tokens verify the completion and whether the user passed quality control checks:
+    - **Pass Token** (`token_pass`): Shown when user meets validation thresholds
+    - **Fail Token** (`token_fail`): Shown when user fails to meet validation requirements
+- **Tutorial**: An instructional validation item that teaches users how to annotate. Includes `allow_skip: true` to let users skip if they have seen it before.
+- **Validation**: Quality control rules attached to items that check if annotations match expected criteria (score ranges, error span locations, etc.). Used for tutorials and attention checks.
+- **Model**: The system or model that generated the output being evaluated (e.g., `"GPT-4"`, `"Claude"`). Used for tracking and ranking model performance.
+- **Dashboard**: The management interface that shows campaign progress, annotator statistics, access links, and allows downloading annotations. Accessed via a special management URL with token authentication.
+- **Protocol**: The annotation scheme defining what data is collected:
+  - **Score**: Numeric quality rating (0-100)
+  - **Error Spans**: Text highlights marking errors
+  - **Error Categories**: MQM taxonomy labels for errors
+- **Template**: The annotation interface type:
+  - **Pointwise**: Evaluate one output at a time
+  - **Listwise**: Compare multiple outputs simultaneously
+- **Assignment**: The method for distributing items to users:
+  - **Task-based**: Each user has predefined items
+  - **Single-stream**: Users draw from a shared pool with random assignment
+  - **Dynamic**: Work in progress
+
 ## Development
 
 Server responds to data-only requests from frontend (no template coupling). Frontend served from pre-built `static/` on install.
@@ -313,7 +349,7 @@ If you use this work in your paper, please cite as following.
     author={Vilém Zouhar},
     title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
-    year={2025},
+    year={2026},
 }
 ```
 

{pearmut-0.2.9 → pearmut-0.2.10}/pearmut.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.2.9
+Version: 0.2.10
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: MIT
@@ -47,9 +47,13 @@ Dynamic: license-file
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
 - [CLI Commands](#cli-commands)
+- [Terminology](#terminology)
 - [Development](#development)
 - [Citation](#citation)
 
+
+**Error Span** — A highlighted segment of text marked as containing an error, with optional severity (`minor`, `major`, `neutral`) and MQM category labels.
+
 ## Quick Start
 
 Install and run locally without cloning:
@@ -294,6 +298,38 @@ Items need `model` field (pointwise) or `models` field (listwise) and the `proto
 ```
 See an example in [Campaign Management](#campaign-management)
 
+
+## Terminology
+
+- **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
+  - **Campaign File**: A JSON file that defines the campaign configuration, including the campaign ID, assignment type, protocol settings, and annotation data.
+  - **Campaign ID**: A unique identifier for a campaign (e.g., `"wmt25_#_en-cs_CZ"`). Used to reference and manage specific campaigns.
+- **Task**: A unit of work assigned to a user. In task-based assignment, each task consists of a predefined set of items for a specific user.
+- **Item** — A single annotation unit within a task. For translation evaluation, an item typically represents a document (source text and target translation). Items can contain text, images, audio, or video.
+- **Document** — A collection of one or more segments (sentence pairs or text units) that are evaluated together as a single item.
+- **User** / **Annotator**: A person who performs annotations in a campaign. Each user is identified by a unique user ID and accesses the campaign through a unique URL.
+- **Attention Check** — A validation item with known correct answers used to ensure annotator quality. Can be:
+  - **Loud**: Shows warning message and forces retry on failure
+  - **Silent**: Logs failures without notifying the user (for quality control analysis)
+  - **Token** — A completion code shown to users when they finish their annotations. Tokens verify the completion and whether the user passed quality control checks:
+    - **Pass Token** (`token_pass`): Shown when user meets validation thresholds
+    - **Fail Token** (`token_fail`): Shown when user fails to meet validation requirements
+- **Tutorial**: An instructional validation item that teaches users how to annotate. Includes `allow_skip: true` to let users skip if they have seen it before.
+- **Validation**: Quality control rules attached to items that check if annotations match expected criteria (score ranges, error span locations, etc.). Used for tutorials and attention checks.
+- **Model**: The system or model that generated the output being evaluated (e.g., `"GPT-4"`, `"Claude"`). Used for tracking and ranking model performance.
+- **Dashboard**: The management interface that shows campaign progress, annotator statistics, access links, and allows downloading annotations. Accessed via a special management URL with token authentication.
+- **Protocol**: The annotation scheme defining what data is collected:
+  - **Score**: Numeric quality rating (0-100)
+  - **Error Spans**: Text highlights marking errors
+  - **Error Categories**: MQM taxonomy labels for errors
+- **Template**: The annotation interface type:
+  - **Pointwise**: Evaluate one output at a time
+  - **Listwise**: Compare multiple outputs simultaneously
+- **Assignment**: The method for distributing items to users:
+  - **Task-based**: Each user has predefined items
+  - **Single-stream**: Users draw from a shared pool with random assignment
+  - **Dynamic**: Work in progress
+
 ## Development
 
 Server responds to data-only requests from frontend (no template coupling). Frontend served from pre-built `static/` on install.
@@ -333,7 +369,7 @@ If you use this work in your paper, please cite as following.
     author={Vilém Zouhar},
     title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
-    year={2025},
+    year={2026},
 }
 ```
 

{pearmut-0.2.9 → pearmut-0.2.10}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pearmut"
-version = "0.2.9"
+version = "0.2.10"
 description = "A tool for evaluation of model outputs, primarily MT."
 readme = "README.md"
 license = { text = "MIT" }

{pearmut-0.2.9 → pearmut-0.2.10}/server/app.py

@@ -291,7 +291,11 @@ async def _download_annotations(
             with open(output_path, "r") as f:
                 output[campaign_id] = [json.loads(x) for x in f.readlines()]
 
-    return JSONResponse(content=output, status_code=200)
+    return JSONResponse(
+        content=output,
+        status_code=200,
+        headers={"Content-Disposition": 'inline; filename="annotations.json"'}
+    )
 
 
 @app.get("/download-progress")
@@ -315,7 +319,11 @@ async def _download_progress(
 
         output[cid] = progress_data[cid]
 
-    return JSONResponse(content=output, status_code=200)
+    return JSONResponse(
+        content=output,
+        status_code=200,
+        headers={"Content-Disposition": 'inline; filename="progress.json"'}
+    )
 
 
 static_dir = f"{os.path.dirname(os.path.abspath(__file__))}/static/"