pearmut 0.2.8__tar.gz → 0.2.10__tar.gz

{pearmut-0.2.8 → pearmut-0.2.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.2.8
+Version: 0.2.10
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: MIT
@@ -47,9 +47,13 @@ Dynamic: license-file
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
 - [CLI Commands](#cli-commands)
+- [Terminology](#terminology)
 - [Development](#development)
 - [Citation](#citation)
 
+
+**Error Span** — A highlighted segment of text marked as containing an error, with optional severity (`minor`, `major`, `neutral`) and MQM category labels.
+
 ## Quick Start
 
 Install and run locally without cloning:
@@ -278,12 +282,54 @@ Management link (shown when adding campaigns or running server) provides:
 - Task progress reset (data preserved)
 - Download progress and annotations
 
-<img width="800" alt="Management dashboard" src="https://github.com/user-attachments/assets/800a1741-5f41-47ac-9d5d-5cbf6abfc0e6" />
+<img width="1000" alt="Management dashboard" src="https://github.com/user-attachments/assets/8953252c-d7b1-428c-a974-5bc7501457c7" />
 
 Completion tokens are shown at annotation end for verification (download correct tokens from dashboard). Incorrect tokens can be shown if quality control fails.
 
 <img width="500" alt="Token on completion" src="https://github.com/user-attachments/assets/40eb904c-f47a-4011-aa63-9a4f1c501549" />
 
+### Model Results Display
+
+Add `&results` to dashboard URL to show model rankings (requires valid token).
+Items need `model` field (pointwise) or `models` field (listwise) and the `protocol_score` needs to be enable such that the `score` can be used for the ranking:
+```python
+{"doc_id": "1", "model": "CommandA", "src": "...", "tgt": "..."}
+{"doc_id": "2", "models": ["CommandA", "Claude"], "src": "...", "tgt": ["...", "..."]}
+```
+See an example in [Campaign Management](#campaign-management)
+
+
+## Terminology
+
+- **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
+  - **Campaign File**: A JSON file that defines the campaign configuration, including the campaign ID, assignment type, protocol settings, and annotation data.
+  - **Campaign ID**: A unique identifier for a campaign (e.g., `"wmt25_#_en-cs_CZ"`). Used to reference and manage specific campaigns.
+- **Task**: A unit of work assigned to a user. In task-based assignment, each task consists of a predefined set of items for a specific user.
+- **Item** — A single annotation unit within a task. For translation evaluation, an item typically represents a document (source text and target translation). Items can contain text, images, audio, or video.
+- **Document** — A collection of one or more segments (sentence pairs or text units) that are evaluated together as a single item.
+- **User** / **Annotator**: A person who performs annotations in a campaign. Each user is identified by a unique user ID and accesses the campaign through a unique URL.
+- **Attention Check** — A validation item with known correct answers used to ensure annotator quality. Can be:
+  - **Loud**: Shows warning message and forces retry on failure
+  - **Silent**: Logs failures without notifying the user (for quality control analysis)
+  - **Token** — A completion code shown to users when they finish their annotations. Tokens verify the completion and whether the user passed quality control checks:
+    - **Pass Token** (`token_pass`): Shown when user meets validation thresholds
+    - **Fail Token** (`token_fail`): Shown when user fails to meet validation requirements
+- **Tutorial**: An instructional validation item that teaches users how to annotate. Includes `allow_skip: true` to let users skip if they have seen it before.
+- **Validation**: Quality control rules attached to items that check if annotations match expected criteria (score ranges, error span locations, etc.). Used for tutorials and attention checks.
+- **Model**: The system or model that generated the output being evaluated (e.g., `"GPT-4"`, `"Claude"`). Used for tracking and ranking model performance.
+- **Dashboard**: The management interface that shows campaign progress, annotator statistics, access links, and allows downloading annotations. Accessed via a special management URL with token authentication.
+- **Protocol**: The annotation scheme defining what data is collected:
+  - **Score**: Numeric quality rating (0-100)
+  - **Error Spans**: Text highlights marking errors
+  - **Error Categories**: MQM taxonomy labels for errors
+- **Template**: The annotation interface type:
+  - **Pointwise**: Evaluate one output at a time
+  - **Listwise**: Compare multiple outputs simultaneously
+- **Assignment**: The method for distributing items to users:
+  - **Task-based**: Each user has predefined items
+  - **Single-stream**: Users draw from a shared pool with random assignment
+  - **Dynamic**: Work in progress
+
 ## Development
 
 Server responds to data-only requests from frontend (no template coupling). Frontend served from pre-built `static/` on install.
@@ -295,7 +341,7 @@ cd pearmut
 npm install web/ --prefix web/
 npm run build --prefix web/
 # optionally keep running indefinitely to auto-rebuild
-npm watch build --prefix web/
+npm run watch --prefix web/
 
 # Install as editable
 pip3 install -e .
@@ -323,7 +369,7 @@ If you use this work in your paper, please cite as following.
     author={Vilém Zouhar},
     title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
-    year={2025},
+    year={2026},
 }
 ```
 

{pearmut-0.2.8 → pearmut-0.2.10}/README.md

@@ -27,9 +27,13 @@
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
 - [CLI Commands](#cli-commands)
+- [Terminology](#terminology)
 - [Development](#development)
 - [Citation](#citation)
 
+
+**Error Span** — A highlighted segment of text marked as containing an error, with optional severity (`minor`, `major`, `neutral`) and MQM category labels.
+
 ## Quick Start
 
 Install and run locally without cloning:
@@ -258,12 +262,54 @@ Management link (shown when adding campaigns or running server) provides:
 - Task progress reset (data preserved)
 - Download progress and annotations
 
-<img width="800" alt="Management dashboard" src="https://github.com/user-attachments/assets/800a1741-5f41-47ac-9d5d-5cbf6abfc0e6" />
+<img width="1000" alt="Management dashboard" src="https://github.com/user-attachments/assets/8953252c-d7b1-428c-a974-5bc7501457c7" />
 
 Completion tokens are shown at annotation end for verification (download correct tokens from dashboard). Incorrect tokens can be shown if quality control fails.
 
 <img width="500" alt="Token on completion" src="https://github.com/user-attachments/assets/40eb904c-f47a-4011-aa63-9a4f1c501549" />
 
+### Model Results Display
+
+Add `&results` to dashboard URL to show model rankings (requires valid token).
+Items need `model` field (pointwise) or `models` field (listwise) and the `protocol_score` needs to be enable such that the `score` can be used for the ranking:
+```python
+{"doc_id": "1", "model": "CommandA", "src": "...", "tgt": "..."}
+{"doc_id": "2", "models": ["CommandA", "Claude"], "src": "...", "tgt": ["...", "..."]}
+```
+See an example in [Campaign Management](#campaign-management)
+
+
+## Terminology
+
+- **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
+  - **Campaign File**: A JSON file that defines the campaign configuration, including the campaign ID, assignment type, protocol settings, and annotation data.
+  - **Campaign ID**: A unique identifier for a campaign (e.g., `"wmt25_#_en-cs_CZ"`). Used to reference and manage specific campaigns.
+- **Task**: A unit of work assigned to a user. In task-based assignment, each task consists of a predefined set of items for a specific user.
+- **Item** — A single annotation unit within a task. For translation evaluation, an item typically represents a document (source text and target translation). Items can contain text, images, audio, or video.
+- **Document** — A collection of one or more segments (sentence pairs or text units) that are evaluated together as a single item.
+- **User** / **Annotator**: A person who performs annotations in a campaign. Each user is identified by a unique user ID and accesses the campaign through a unique URL.
+- **Attention Check** — A validation item with known correct answers used to ensure annotator quality. Can be:
+  - **Loud**: Shows warning message and forces retry on failure
+  - **Silent**: Logs failures without notifying the user (for quality control analysis)
+  - **Token** — A completion code shown to users when they finish their annotations. Tokens verify the completion and whether the user passed quality control checks:
+    - **Pass Token** (`token_pass`): Shown when user meets validation thresholds
+    - **Fail Token** (`token_fail`): Shown when user fails to meet validation requirements
+- **Tutorial**: An instructional validation item that teaches users how to annotate. Includes `allow_skip: true` to let users skip if they have seen it before.
+- **Validation**: Quality control rules attached to items that check if annotations match expected criteria (score ranges, error span locations, etc.). Used for tutorials and attention checks.
+- **Model**: The system or model that generated the output being evaluated (e.g., `"GPT-4"`, `"Claude"`). Used for tracking and ranking model performance.
+- **Dashboard**: The management interface that shows campaign progress, annotator statistics, access links, and allows downloading annotations. Accessed via a special management URL with token authentication.
+- **Protocol**: The annotation scheme defining what data is collected:
+  - **Score**: Numeric quality rating (0-100)
+  - **Error Spans**: Text highlights marking errors
+  - **Error Categories**: MQM taxonomy labels for errors
+- **Template**: The annotation interface type:
+  - **Pointwise**: Evaluate one output at a time
+  - **Listwise**: Compare multiple outputs simultaneously
+- **Assignment**: The method for distributing items to users:
+  - **Task-based**: Each user has predefined items
+  - **Single-stream**: Users draw from a shared pool with random assignment
+  - **Dynamic**: Work in progress
+
 ## Development
 
 Server responds to data-only requests from frontend (no template coupling). Frontend served from pre-built `static/` on install.
@@ -275,7 +321,7 @@ cd pearmut
 npm install web/ --prefix web/
 npm run build --prefix web/
 # optionally keep running indefinitely to auto-rebuild
-npm watch build --prefix web/
+npm run watch --prefix web/
 
 # Install as editable
 pip3 install -e .
@@ -303,8 +349,8 @@ If you use this work in your paper, please cite as following.
     author={Vilém Zouhar},
     title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
-    year={2025},
+    year={2026},
 }
 ```
 
-Contributions are welcome! Please reach out to [Vilém Zouhar](mailto:vilem.zouhar@gmail.com).
+Contributions are welcome! Please reach out to [Vilém Zouhar](mailto:vilem.zouhar@gmail.com).

{pearmut-0.2.8 → pearmut-0.2.10}/pearmut.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.2.8
+Version: 0.2.10
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: MIT
@@ -47,9 +47,13 @@ Dynamic: license-file
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
 - [CLI Commands](#cli-commands)
+- [Terminology](#terminology)
 - [Development](#development)
 - [Citation](#citation)
 
+
+**Error Span** — A highlighted segment of text marked as containing an error, with optional severity (`minor`, `major`, `neutral`) and MQM category labels.
+
 ## Quick Start
 
 Install and run locally without cloning:
@@ -278,12 +282,54 @@ Management link (shown when adding campaigns or running server) provides:
 - Task progress reset (data preserved)
 - Download progress and annotations
 
-<img width="800" alt="Management dashboard" src="https://github.com/user-attachments/assets/800a1741-5f41-47ac-9d5d-5cbf6abfc0e6" />
+<img width="1000" alt="Management dashboard" src="https://github.com/user-attachments/assets/8953252c-d7b1-428c-a974-5bc7501457c7" />
 
 Completion tokens are shown at annotation end for verification (download correct tokens from dashboard). Incorrect tokens can be shown if quality control fails.
 
 <img width="500" alt="Token on completion" src="https://github.com/user-attachments/assets/40eb904c-f47a-4011-aa63-9a4f1c501549" />
 
+### Model Results Display
+
+Add `&results` to dashboard URL to show model rankings (requires valid token).
+Items need `model` field (pointwise) or `models` field (listwise) and the `protocol_score` needs to be enable such that the `score` can be used for the ranking:
+```python
+{"doc_id": "1", "model": "CommandA", "src": "...", "tgt": "..."}
+{"doc_id": "2", "models": ["CommandA", "Claude"], "src": "...", "tgt": ["...", "..."]}
+```
+See an example in [Campaign Management](#campaign-management)
+
+
+## Terminology
+
+- **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
+  - **Campaign File**: A JSON file that defines the campaign configuration, including the campaign ID, assignment type, protocol settings, and annotation data.
+  - **Campaign ID**: A unique identifier for a campaign (e.g., `"wmt25_#_en-cs_CZ"`). Used to reference and manage specific campaigns.
+- **Task**: A unit of work assigned to a user. In task-based assignment, each task consists of a predefined set of items for a specific user.
+- **Item** — A single annotation unit within a task. For translation evaluation, an item typically represents a document (source text and target translation). Items can contain text, images, audio, or video.
+- **Document** — A collection of one or more segments (sentence pairs or text units) that are evaluated together as a single item.
+- **User** / **Annotator**: A person who performs annotations in a campaign. Each user is identified by a unique user ID and accesses the campaign through a unique URL.
+- **Attention Check** — A validation item with known correct answers used to ensure annotator quality. Can be:
+  - **Loud**: Shows warning message and forces retry on failure
+  - **Silent**: Logs failures without notifying the user (for quality control analysis)
+  - **Token** — A completion code shown to users when they finish their annotations. Tokens verify the completion and whether the user passed quality control checks:
+    - **Pass Token** (`token_pass`): Shown when user meets validation thresholds
+    - **Fail Token** (`token_fail`): Shown when user fails to meet validation requirements
+- **Tutorial**: An instructional validation item that teaches users how to annotate. Includes `allow_skip: true` to let users skip if they have seen it before.
+- **Validation**: Quality control rules attached to items that check if annotations match expected criteria (score ranges, error span locations, etc.). Used for tutorials and attention checks.
+- **Model**: The system or model that generated the output being evaluated (e.g., `"GPT-4"`, `"Claude"`). Used for tracking and ranking model performance.
+- **Dashboard**: The management interface that shows campaign progress, annotator statistics, access links, and allows downloading annotations. Accessed via a special management URL with token authentication.
+- **Protocol**: The annotation scheme defining what data is collected:
+  - **Score**: Numeric quality rating (0-100)
+  - **Error Spans**: Text highlights marking errors
+  - **Error Categories**: MQM taxonomy labels for errors
+- **Template**: The annotation interface type:
+  - **Pointwise**: Evaluate one output at a time
+  - **Listwise**: Compare multiple outputs simultaneously
+- **Assignment**: The method for distributing items to users:
+  - **Task-based**: Each user has predefined items
+  - **Single-stream**: Users draw from a shared pool with random assignment
+  - **Dynamic**: Work in progress
+
 ## Development
 
 Server responds to data-only requests from frontend (no template coupling). Frontend served from pre-built `static/` on install.
@@ -295,7 +341,7 @@ cd pearmut
 npm install web/ --prefix web/
 npm run build --prefix web/
 # optionally keep running indefinitely to auto-rebuild
-npm watch build --prefix web/
+npm run watch --prefix web/
 
 # Install as editable
 pip3 install -e .
@@ -323,7 +369,7 @@ If you use this work in your paper, please cite as following.
     author={Vilém Zouhar},
     title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
-    year={2025},
+    year={2026},
 }
 ```
 

{pearmut-0.2.8 → pearmut-0.2.10}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pearmut"
-version = "0.2.8"
+version = "0.2.10"
 description = "A tool for evaluation of model outputs, primarily MT."
 readme = "README.md"
 license = { text = "MIT" }

{pearmut-0.2.8 → pearmut-0.2.10}/server/app.py

@@ -1,5 +1,7 @@
+import collections
 import json
 import os
+import statistics
 from typing import Any
 
 from fastapi import FastAPI, Query
@@ -12,6 +14,7 @@ from .assignment import get_i_item, get_next_item, reset_task, update_progress
 from .utils import (
     ROOT,
     check_validation_threshold,
+    get_db_log,
     load_progress_data,
     save_db_payload,
     save_progress_data,
@@ -191,6 +194,58 @@ async def _dashboard_data(request: DashboardDataRequest):
     )
 
 
+class DashboardResultsRequest(BaseModel):
+    campaign_id: str
+    token: str
+
+
+@app.post("/dashboard-results")
+async def _dashboard_results(request: DashboardResultsRequest):
+    campaign_id = request.campaign_id
+    token = request.token
+
+    if campaign_id not in progress_data:
+        return JSONResponse(content="Unknown campaign ID", status_code=400)
+    
+    # Check if token is valid
+    if token != tasks_data[campaign_id]["token"]:
+        return JSONResponse(content="Invalid token", status_code=400)
+
+    # Compute model scores from annotations
+    model_scores = collections.defaultdict(dict)
+    
+    # Iterate through all tasks to find items with 'model' field
+    log = get_db_log(campaign_id)
+    for entry in log:
+        if "item" not in entry or "annotations" not in entry:
+            continue
+        for item, annotation in zip(entry["item"], entry["annotations"]):
+            if "model" in item:
+                # pointwise
+                if "score" in annotation:
+                    # make sure to only keep the latest score for each item
+                    # json.dumps(item) creates a unique item key
+                    model_scores[item["model"]][json.dumps(item)] = annotation["score"]
+            elif "models" in item:
+                # listwise
+                for model, annotation_cand in zip(item["models"], annotation):
+                    if "score" in annotation_cand:
+                        model_scores[model][json.dumps(item)] = (
+                            annotation_cand["score"]
+                        )
+
+    results = [
+        {
+            "model": model,
+            "score": statistics.mean(scores.values()),
+            "count": len(scores),
+        }
+        for model, scores in model_scores.items()
+    ]
+    results.sort(key=lambda x: x["score"], reverse=True)
+    return JSONResponse(content=results, status_code=200)
+
+
 class ResetTaskRequest(BaseModel):
     campaign_id: str
     user_id: str
@@ -236,7 +291,11 @@ async def _download_annotations(
             with open(output_path, "r") as f:
                 output[campaign_id] = [json.loads(x) for x in f.readlines()]
 
-    return JSONResponse(content=output, status_code=200)
+    return JSONResponse(
+        content=output,
+        status_code=200,
+        headers={"Content-Disposition": 'inline; filename="annotations.json"'}
+    )
 
 
 @app.get("/download-progress")
@@ -260,7 +319,11 @@ async def _download_progress(
 
         output[cid] = progress_data[cid]
 
-    return JSONResponse(content=output, status_code=200)
+    return JSONResponse(
+        content=output,
+        status_code=200,
+        headers={"Content-Disposition": 'inline; filename="progress.json"'}
+    )
 
 
 static_dir = f"{os.path.dirname(os.path.abspath(__file__))}/static/"

{pearmut-0.2.8 → pearmut-0.2.10}/server/cli.py

@@ -41,7 +41,9 @@ def _run(args_unknown):
             args.server + "/dashboard.html?" + "&".join([
                 f"campaign_id={urllib.parse.quote_plus(campaign_id)}&token={campaign_data["token"]}"
                 for campaign_id, campaign_data in tasks_data.items()
-            ])
+            ]),
+            # this is important to flush
+            flush=True,
         )
 
     uvicorn.run(