pearmut 1.0.1__tar.gz → 1.0.3__tar.gz

{pearmut-1.0.1 → pearmut-1.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 1.0.1
+Version: 1.0.3
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: MIT
@@ -19,7 +19,7 @@ Provides-Extra: dev
 Requires-Dist: pytest; extra == "dev"
 Dynamic: license-file
 
-# 🍐Pearmut &nbsp; &nbsp; [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut) [![PyPI download/month](https://img.shields.io/pypi/dm/pearmut.svg)](https://pypi.python.org/pypi/pearmut/) [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/) [![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml)
+# 🍐Pearmut <br> [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut) [![PyPI download/month](https://img.shields.io/pypi/dm/pearmut.svg)](https://pypi.python.org/pypi/pearmut/) [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/) [![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml) [![arXiv](https://img.shields.io/badge/arXiv-2601.02933-b31b1b.svg?style=flat)](https://arxiv.org/abs/2601.02933)
 
 **Platform for Evaluation and Reviewing of Multilingual Tasks**: Evaluate model outputs for translation and NLP tasks with support for multimodal data (text, video, audio, images) and multiple annotation protocols ([DA](https://aclanthology.org/N15-1124/), [ESA](https://aclanthology.org/2024.wmt-1.131/), [ESA<sup>AI</sup>](https://aclanthology.org/2025.naacl-long.255/), [MQM](https://doi.org/10.1162/tacl_a_00437), and more!).
 
@@ -35,12 +35,15 @@ Dynamic: license-file
   - [Assignment Types](#assignment-types)
 - [Advanced Features](#advanced-features)
   - [Pre-filled Error Spans (ESA<sup>AI</sup>)](#pre-filled-error-spans-esaai)
+  - [Custom MQM Taxonomy](#custom-mqm-taxonomy)
   - [Tutorial and Attention Checks](#tutorial-and-attention-checks)
+  - [Form Items for User Metadata](#form-items-for-user-metadata)
   - [Pre-defined User IDs and Tokens](#pre-defined-user-ids-and-tokens)
   - [Multimodal Annotations](#multimodal-annotations)
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
   - [Custom Completion Messages](#custom-completion-messages)
+  - [Prolific Integration](#prolific-integration)
 - [CLI Commands](#cli-commands)
 - [Terminology](#terminology)
 - [Development](#development)
@@ -141,6 +144,22 @@ The `shuffle` parameter in campaign `info` controls this behavior:
   "data": [...]
 }
 ```
+Documents in `data_welcome` are not shuffled and so don't require to have the same models in all documents.
+
+### Showing Model Names
+
+By default, model names are hidden to avoid biasing annotators. To display model names on top of each output block, set `show_model_names` to `true`:
+```python
+{
+  "info": {
+    "assignment": "task-based",
+    "protocol": "ESA",
+    "show_model_names": true  # Default: false.
+  },
+  "campaign_id": "my_campaign",
+  "data": [...]
+}
+```
 
 ### Custom Score Sliders
 
@@ -163,6 +182,52 @@ For multi-dimensional evaluation tasks (e.g., assessing fluency on a Likert scal
 
 When `sliders` is specified, only the custom sliders are shown. Each slider must have `name`, `min`, `max`, and `step` properties. All sliders must be answered before proceeding.
 
+### Textfield for Post-editing/Translation
+
+Enable a textfield for post-editing or translation tasks using the `textfield` parameter in `info`. The textfield content is stored in annotations alongside scores and error spans.
+
+```python
+{
+  "info": {
+    "protocol": "DA",
+    "textfield": "prefilled"  # Options: null, "hidden", "visible", "prefilled"
+  }
+}
+```
+
+**Textfield modes:**
+- `null` or omitted: No textfield (default)
+- `"hidden"`: Textfield hidden by default, shown by clicking a button
+- `"visible"`: Textfield always visible
+- `"prefilled"`: Textfield visible and pre-filled with model output for post-editing
+
+### Custom MQM Taxonomy
+
+For MQM protocol campaigns, you can define a custom error taxonomy instead of using the default MQM categories. Specify `mqm_categories` in the campaign `info` section as a dictionary mapping main categories to lists of subcategories:
+
+
+```python
+{
+  "info": {
+    "assignment": "task-based",
+    "protocol": "MQM",
+    "mqm_categories": {
+      "": [],                          # Empty selection option
+      "General": ["", "Accuracy", "Fluency"],
+      "Audio-specific": ["", "Inaudible", "Background noise", "Speaker overlap", "Misinterpretation"],
+      "Style": ["", "Awkward", "Embarassing"],
+      "Unknown": []                    # Category with no subcategories
+    }
+  },
+  "campaign_id": "custom_mqm_example",
+  "data": [...]
+}
+```
+
+If `mqm_categories` is not provided, the default MQM taxonomy will be used. The empty string key `""` provides an unselected state in the dropdown. Categories with empty subcategory lists (e.g., `"Style": []`) do not require a subcategory selection.
+
+See [examples/custom_mqm.json](examples/custom_mqm.json) for a complete example.
+
 ### Custom Instructions
 
 Set campaign-level instructions using the `instructions` field in `info` (supports HTML).
@@ -252,6 +317,34 @@ The `score_greaterthan` field specifies the index of the candidate that must hav
 See [examples/tutorial/esa_deen.json](examples/tutorial/esa_deen.json) for a mock campaign with a fully prepared ESA tutorial.
 To use it, simply extract the `data` attribute and prefix it to each task in your campaign.
 
+#### Universal Tutorial Items with `data_welcome`
+
+Use `data_welcome` to add tutorial items that users must complete before starting regular tasks. The structure is a list of documents (same as `data`). Welcome items have IDs `welcome_0`, `welcome_1`, etc. and are tracked separately via `progress_welcome`.
+
+### Form Items for User Metadata
+
+Collect user information (demographics, expertise) before annotation tasks using form items in `data_welcome`.
+Form items have `text` (label/question) and `form` (field type: `null`, `"string"`, `"number"`, `"choices"`, and `"script"`).
+Documents must be homogeneous: all form items or all evaluation items.
+
+```python
+{
+  "data_welcome": [
+    [
+      {"text": "What is your native language?", "form": "string"},
+      {"text": "Rate your expertise (1-10)", "form": "number"}
+    ]
+  ]
+}
+```
+
+<img width="400" alt="Screenshot of a user form" src="https://github.com/user-attachments/assets/2310e8dc-98e9-4abf-8a27-6781b0094efe" />
+
+
+It is possible to automatically collect additional information from the host system using `"script"` field type.
+Typically such a form document (or their sequence) would be stored in `"data_welcome"` such that it is both mandatory and show to all users.
+See [examples/user_info_form.json](examples/user_info_form.json).
+
 ### Single-stream Assignment
 
 All annotators draw from a shared pool with random assignment:
@@ -265,11 +358,14 @@ All annotators draw from a shared pool with random assignment:
         # ESA: error spans and scores
         "protocol": "ESA",
         "users": 50,                           # number of annotators (can also be a list, see below)
+        "docs_per_user": 10,                   # optional: show goodbye after N documents per user
     },
     "data": [...], # list of all items (shared among all annotators)
 }
 ```
 
+Set `docs_per_user` to limit how many documents each user annotates before seeing the goodbye message (for single-stream, this is the number of documents).
+
 ### Dynamic Assignment
 
 The `dynamic` assignment type intelligently selects items based on current model performance to focus annotation effort on top-performing models using contrastive comparisons.
@@ -286,11 +382,14 @@ All items must contain outputs from all models for this assignment type to work
         "dynamic_contrastive_models": 2,       # how many models to compare per item (optional, default: 1)
         "dynamic_first": 5,                    # annotations per model before dynamic kicks in (optional, default: 5)
         "dynamic_backoff": 0.1,                # probability of uniform sampling (optional, default: 0)
+        "docs_per_user": 20,                   # optional: show goodbye after N documents per user
     },
     "data": [...], # list of all items (shared among all annotators)
 }
 ```
 
+Set `docs_per_user` to limit how many documents each user annotates before seeing the goodbye message (for dynamic, this is roughly the number of documents × models).
+
 **How it works:**
 1. Initial phase: Each model gets `dynamic_first` annotations with fully random contrastive evaluation
 2. Dynamic phase: After the initial phase, top `dynamic_top` models (by average score) are identified
@@ -378,6 +477,14 @@ When tokens are supplied, the dashboard will try to show model rankings based on
 
 Customize the goodbye message shown to users when they complete all annotations using the `instructions_goodbye` field in campaign info. Supports arbitrary HTML for styling and formatting with variable replacement: `${TOKEN}` (completion token) and `${USER_ID}` (user ID). Default: `"If someone asks you for a token of completion, show them: ${TOKEN}"`.
 
+### Prolific Integration
+
+Use task-based assignment with Prolific. For each task, Pearmut generates a unique URL which can be uploaded to Prolific's interface. Add redirect (on completion) to `instructions_goodbye`:
+```json
+"instructions_goodbye": "<a href='https://app.prolific.com/submissions/complete?cc=${TOKEN}'>Click here to return to Prolific</a>"
+```
+The `${TOKEN}` is automatically replaced based on passing attention checks (see [Attention checks](#tutorial-and-attention-checks) and [Pre-defined tokens](#pre-defined-user-ids-and-tokens)).
+
 ## Terminology
 
 - **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
@@ -401,7 +508,7 @@ Customize the goodbye message shown to users when they complete all annotations
   - **Score**: Numeric quality rating (0-100)
   - **Error Spans**: Text highlights marking errors with severity (`minor`, `major`)
   - **Error Categories**: MQM taxonomy labels for errors
-- **Template**: The annotation interface type. The `basic` template supports comparing multiple outputs simultaneously.
+- **Template**: The annotation interface type. The `annotate` template supports comparing multiple outputs simultaneously.
 - **Assignment**: The method for distributing items to users:
   - **Task-based**: Each user has predefined items
   - **Single-stream**: Users draw from a shared pool with random assignment
@@ -432,7 +539,7 @@ pearmut run
 2. Add build rule to `webpack.config.js`
 3. Reference as `info->template` in campaign JSON
 
-See [web/src/basic.ts](web/src/basic.ts) for example.
+See [web/src/annotate.ts](web/src/annotate.ts) for example.
 
 ### Deployment
 
@@ -443,68 +550,15 @@ Run on public server or tunnel local port to public IP/domain and run locally.
 If you use this work in your paper, please cite as following.
 ```bibtex
 @misc{zouhar2026pearmut,
-  author = {Zouhar, Vilém},
-  title = {Pearmut: Human Evaluation of Translation Made Trivial},
-  year = {2026}
+      title={Pearmut: Human Evaluation of Translation Made Trivial}, 
+      author={Vilém Zouhar and Tom Kocmi},
+      year={2026},
+      eprint={2601.02933},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2601.02933}, 
 }
 ```
 
 Contributions are welcome! Please reach out to [Vilém Zouhar](mailto:vilem.zouhar@gmail.com).
-
-# Changelog
-
-- v1.0.1
-  - Support RTL languages
-  - Add boxes for references
-  - Add custom score sliders for multi-dimensional evaluation
-  - Make instructions customizable and protocol-dependent
-  - Support custom sliders
-  - Purge/reset whole tasks from dashboard
-  - Fix resetting individual users in single-stream/dynamic
-  - Fix notification stacking
-  - Add campaigns from dashboard
-- v0.3.3
-  - Rename `doc_id` to `item_id`
-  - Add Typst, LaTeX, and PDF export for model ranking tables. Hide them by default.
-  - Add dynamic assignment type with contrastive model comparison
-  - Add `instructions_goodbye` field with variable substitution
-  - Add visual anchors at 33% and 66% on sliders
-  - Add German→English ESA tutorial with attention checks
-  - Validate document model consistency before shuffle
-  - Fix UI block on any interaction
-- v0.3.2
-  - Revert seeding of user IDs
-  - Set ESA (Error Span Annotation) as default
-  - Update server IP address configuration
-  - Show approximate alignment by default
-  - Unify pointwise and listwise interfaces into `basic`
-  - Refactor protocol configuration (breaking change)
-- v0.2.11
-  - Add comment field in settings panel
-  - Add `score_gt` validation for listwise comparisons
-  - Add Content-Disposition headers for proper download filenames
-  - Add model results display to dashboard with rankings
-  - Add campaign file structure validation
-  - Purge command now unlinks assets
-- v0.2.6
-  - Add frozen annotation links feature for view-only mode
-  - Add word-level annotation mode toggle for error spans
-  - Add `[missing]` token support
-  - Improve frontend speed and cleanup toolboxes on item load
-  - Host assets via symlinks
-  - Add validation threshold for success/fail tokens
-  - Implement reset masking for annotations
-  - Allow pre-defined user IDs and tokens in campaign data
-- v0.1.1
-  - Set server defaults and add VM launch scripts
-  - Add warning dialog when navigating away with unsaved work
-  - Add tutorial validation support for pointwise and listwise
-  - Add ability to preview existing annotations via progress bar
-  - Add support for ESA<sup>AI</sup> pre-filled error_spans
-  - Rename pairwise to listwise and update layout
-  - Implement single-stream assignment type
-- v0.0.3
-  - Support multimodal inputs and outputs
-  - Add dashboard
-  - Implement ESA (Error Span Annotation) and MQM support
-
+See changes in [CHANGELOG.md](CHANGELOG.md).

{pearmut-1.0.1 → pearmut-1.0.3}/README.md

@@ -1,4 +1,4 @@
-# 🍐Pearmut     [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut) [![PyPI download/month](https://img.shields.io/pypi/dm/pearmut.svg)](https://pypi.python.org/pypi/pearmut/) [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/) [![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml)
+# 🍐Pearmut <br> [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut) [![PyPI download/month](https://img.shields.io/pypi/dm/pearmut.svg)](https://pypi.python.org/pypi/pearmut/) [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/) [![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml) [![arXiv](https://img.shields.io/badge/arXiv-2601.02933-b31b1b.svg?style=flat)](https://arxiv.org/abs/2601.02933)
 
 **Platform for Evaluation and Reviewing of Multilingual Tasks**: Evaluate model outputs for translation and NLP tasks with support for multimodal data (text, video, audio, images) and multiple annotation protocols ([DA](https://aclanthology.org/N15-1124/), [ESA](https://aclanthology.org/2024.wmt-1.131/), [ESA<sup>AI</sup>](https://aclanthology.org/2025.naacl-long.255/), [MQM](https://doi.org/10.1162/tacl_a_00437), and more!).
 
@@ -14,12 +14,15 @@
   - [Assignment Types](#assignment-types)
 - [Advanced Features](#advanced-features)
   - [Pre-filled Error Spans (ESA<sup>AI</sup>)](#pre-filled-error-spans-esaai)
+  - [Custom MQM Taxonomy](#custom-mqm-taxonomy)
   - [Tutorial and Attention Checks](#tutorial-and-attention-checks)
+  - [Form Items for User Metadata](#form-items-for-user-metadata)
   - [Pre-defined User IDs and Tokens](#pre-defined-user-ids-and-tokens)
   - [Multimodal Annotations](#multimodal-annotations)
   - [Hosting Assets](#hosting-assets)
 - [Campaign Management](#campaign-management)
   - [Custom Completion Messages](#custom-completion-messages)
+  - [Prolific Integration](#prolific-integration)
 - [CLI Commands](#cli-commands)
 - [Terminology](#terminology)
 - [Development](#development)
@@ -120,6 +123,22 @@ The `shuffle` parameter in campaign `info` controls this behavior:
   "data": [...]
 }
 ```
+Documents in `data_welcome` are not shuffled and so don't require to have the same models in all documents.
+
+### Showing Model Names
+
+By default, model names are hidden to avoid biasing annotators. To display model names on top of each output block, set `show_model_names` to `true`:
+```python
+{
+  "info": {
+    "assignment": "task-based",
+    "protocol": "ESA",
+    "show_model_names": true  # Default: false.
+  },
+  "campaign_id": "my_campaign",
+  "data": [...]
+}
+```
 
 ### Custom Score Sliders
 
@@ -142,6 +161,52 @@ For multi-dimensional evaluation tasks (e.g., assessing fluency on a Likert scal
 
 When `sliders` is specified, only the custom sliders are shown. Each slider must have `name`, `min`, `max`, and `step` properties. All sliders must be answered before proceeding.
 
+### Textfield for Post-editing/Translation
+
+Enable a textfield for post-editing or translation tasks using the `textfield` parameter in `info`. The textfield content is stored in annotations alongside scores and error spans.
+
+```python
+{
+  "info": {
+    "protocol": "DA",
+    "textfield": "prefilled"  # Options: null, "hidden", "visible", "prefilled"
+  }
+}
+```
+
+**Textfield modes:**
+- `null` or omitted: No textfield (default)
+- `"hidden"`: Textfield hidden by default, shown by clicking a button
+- `"visible"`: Textfield always visible
+- `"prefilled"`: Textfield visible and pre-filled with model output for post-editing
+
+### Custom MQM Taxonomy
+
+For MQM protocol campaigns, you can define a custom error taxonomy instead of using the default MQM categories. Specify `mqm_categories` in the campaign `info` section as a dictionary mapping main categories to lists of subcategories:
+
+
+```python
+{
+  "info": {
+    "assignment": "task-based",
+    "protocol": "MQM",
+    "mqm_categories": {
+      "": [],                          # Empty selection option
+      "General": ["", "Accuracy", "Fluency"],
+      "Audio-specific": ["", "Inaudible", "Background noise", "Speaker overlap", "Misinterpretation"],
+      "Style": ["", "Awkward", "Embarassing"],
+      "Unknown": []                    # Category with no subcategories
+    }
+  },
+  "campaign_id": "custom_mqm_example",
+  "data": [...]
+}
+```
+
+If `mqm_categories` is not provided, the default MQM taxonomy will be used. The empty string key `""` provides an unselected state in the dropdown. Categories with empty subcategory lists (e.g., `"Style": []`) do not require a subcategory selection.
+
+See [examples/custom_mqm.json](examples/custom_mqm.json) for a complete example.
+
 ### Custom Instructions
 
 Set campaign-level instructions using the `instructions` field in `info` (supports HTML).
@@ -231,6 +296,34 @@ The `score_greaterthan` field specifies the index of the candidate that must hav
 See [examples/tutorial/esa_deen.json](examples/tutorial/esa_deen.json) for a mock campaign with a fully prepared ESA tutorial.
 To use it, simply extract the `data` attribute and prefix it to each task in your campaign.
 
+#### Universal Tutorial Items with `data_welcome`
+
+Use `data_welcome` to add tutorial items that users must complete before starting regular tasks. The structure is a list of documents (same as `data`). Welcome items have IDs `welcome_0`, `welcome_1`, etc. and are tracked separately via `progress_welcome`.
+
+### Form Items for User Metadata
+
+Collect user information (demographics, expertise) before annotation tasks using form items in `data_welcome`.
+Form items have `text` (label/question) and `form` (field type: `null`, `"string"`, `"number"`, `"choices"`, and `"script"`).
+Documents must be homogeneous: all form items or all evaluation items.
+
+```python
+{
+  "data_welcome": [
+    [
+      {"text": "What is your native language?", "form": "string"},
+      {"text": "Rate your expertise (1-10)", "form": "number"}
+    ]
+  ]
+}
+```
+
+<img width="400" alt="Screenshot of a user form" src="https://github.com/user-attachments/assets/2310e8dc-98e9-4abf-8a27-6781b0094efe" />
+
+
+It is possible to automatically collect additional information from the host system using `"script"` field type.
+Typically such a form document (or their sequence) would be stored in `"data_welcome"` such that it is both mandatory and show to all users.
+See [examples/user_info_form.json](examples/user_info_form.json).
+
 ### Single-stream Assignment
 
 All annotators draw from a shared pool with random assignment:
@@ -244,11 +337,14 @@ All annotators draw from a shared pool with random assignment:
         # ESA: error spans and scores
         "protocol": "ESA",
         "users": 50,                           # number of annotators (can also be a list, see below)
+        "docs_per_user": 10,                   # optional: show goodbye after N documents per user
     },
     "data": [...], # list of all items (shared among all annotators)
 }
 ```
 
+Set `docs_per_user` to limit how many documents each user annotates before seeing the goodbye message (for single-stream, this is the number of documents).
+
 ### Dynamic Assignment
 
 The `dynamic` assignment type intelligently selects items based on current model performance to focus annotation effort on top-performing models using contrastive comparisons.
@@ -265,11 +361,14 @@ All items must contain outputs from all models for this assignment type to work
         "dynamic_contrastive_models": 2,       # how many models to compare per item (optional, default: 1)
         "dynamic_first": 5,                    # annotations per model before dynamic kicks in (optional, default: 5)
         "dynamic_backoff": 0.1,                # probability of uniform sampling (optional, default: 0)
+        "docs_per_user": 20,                   # optional: show goodbye after N documents per user
     },
     "data": [...], # list of all items (shared among all annotators)
 }
 ```
 
+Set `docs_per_user` to limit how many documents each user annotates before seeing the goodbye message (for dynamic, this is roughly the number of documents × models).
+
 **How it works:**
 1. Initial phase: Each model gets `dynamic_first` annotations with fully random contrastive evaluation
 2. Dynamic phase: After the initial phase, top `dynamic_top` models (by average score) are identified
@@ -357,6 +456,14 @@ When tokens are supplied, the dashboard will try to show model rankings based on
 
 Customize the goodbye message shown to users when they complete all annotations using the `instructions_goodbye` field in campaign info. Supports arbitrary HTML for styling and formatting with variable replacement: `${TOKEN}` (completion token) and `${USER_ID}` (user ID). Default: `"If someone asks you for a token of completion, show them: ${TOKEN}"`.
 
+### Prolific Integration
+
+Use task-based assignment with Prolific. For each task, Pearmut generates a unique URL which can be uploaded to Prolific's interface. Add redirect (on completion) to `instructions_goodbye`:
+```json
+"instructions_goodbye": "<a href='https://app.prolific.com/submissions/complete?cc=${TOKEN}'>Click here to return to Prolific</a>"
+```
+The `${TOKEN}` is automatically replaced based on passing attention checks (see [Attention checks](#tutorial-and-attention-checks) and [Pre-defined tokens](#pre-defined-user-ids-and-tokens)).
+
 ## Terminology
 
 - **Campaign**: An annotation project that contains configuration, data, and user assignments. Each campaign has a unique identifier and is defined in a JSON file.
@@ -380,7 +487,7 @@ Customize the goodbye message shown to users when they complete all annotations
   - **Score**: Numeric quality rating (0-100)
   - **Error Spans**: Text highlights marking errors with severity (`minor`, `major`)
   - **Error Categories**: MQM taxonomy labels for errors
-- **Template**: The annotation interface type. The `basic` template supports comparing multiple outputs simultaneously.
+- **Template**: The annotation interface type. The `annotate` template supports comparing multiple outputs simultaneously.
 - **Assignment**: The method for distributing items to users:
   - **Task-based**: Each user has predefined items
   - **Single-stream**: Users draw from a shared pool with random assignment
@@ -411,7 +518,7 @@ pearmut run
 2. Add build rule to `webpack.config.js`
 3. Reference as `info->template` in campaign JSON
 
-See [web/src/basic.ts](web/src/basic.ts) for example.
+See [web/src/annotate.ts](web/src/annotate.ts) for example.
 
 ### Deployment
 
@@ -422,68 +529,15 @@ Run on public server or tunnel local port to public IP/domain and run locally.
 If you use this work in your paper, please cite as following.
 ```bibtex
 @misc{zouhar2026pearmut,
-  author = {Zouhar, Vilém},
-  title = {Pearmut: Human Evaluation of Translation Made Trivial},
-  year = {2026}
+      title={Pearmut: Human Evaluation of Translation Made Trivial}, 
+      author={Vilém Zouhar and Tom Kocmi},
+      year={2026},
+      eprint={2601.02933},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2601.02933}, 
 }
 ```
 
 Contributions are welcome! Please reach out to [Vilém Zouhar](mailto:vilem.zouhar@gmail.com).
-
-# Changelog
-
-- v1.0.1
-  - Support RTL languages
-  - Add boxes for references
-  - Add custom score sliders for multi-dimensional evaluation
-  - Make instructions customizable and protocol-dependent
-  - Support custom sliders
-  - Purge/reset whole tasks from dashboard
-  - Fix resetting individual users in single-stream/dynamic
-  - Fix notification stacking
-  - Add campaigns from dashboard
-- v0.3.3
-  - Rename `doc_id` to `item_id`
-  - Add Typst, LaTeX, and PDF export for model ranking tables. Hide them by default.
-  - Add dynamic assignment type with contrastive model comparison
-  - Add `instructions_goodbye` field with variable substitution
-  - Add visual anchors at 33% and 66% on sliders
-  - Add German→English ESA tutorial with attention checks
-  - Validate document model consistency before shuffle
-  - Fix UI block on any interaction
-- v0.3.2
-  - Revert seeding of user IDs
-  - Set ESA (Error Span Annotation) as default
-  - Update server IP address configuration
-  - Show approximate alignment by default
-  - Unify pointwise and listwise interfaces into `basic`
-  - Refactor protocol configuration (breaking change)
-- v0.2.11
-  - Add comment field in settings panel
-  - Add `score_gt` validation for listwise comparisons
-  - Add Content-Disposition headers for proper download filenames
-  - Add model results display to dashboard with rankings
-  - Add campaign file structure validation
-  - Purge command now unlinks assets
-- v0.2.6
-  - Add frozen annotation links feature for view-only mode
-  - Add word-level annotation mode toggle for error spans
-  - Add `[missing]` token support
-  - Improve frontend speed and cleanup toolboxes on item load
-  - Host assets via symlinks
-  - Add validation threshold for success/fail tokens
-  - Implement reset masking for annotations
-  - Allow pre-defined user IDs and tokens in campaign data
-- v0.1.1
-  - Set server defaults and add VM launch scripts
-  - Add warning dialog when navigating away with unsaved work
-  - Add tutorial validation support for pointwise and listwise
-  - Add ability to preview existing annotations via progress bar
-  - Add support for ESA<sup>AI</sup> pre-filled error_spans
-  - Rename pairwise to listwise and update layout
-  - Implement single-stream assignment type
-- v0.0.3
-  - Support multimodal inputs and outputs
-  - Add dashboard
-  - Implement ESA (Error Span Annotation) and MQM support
-
+See changes in [CHANGELOG.md](CHANGELOG.md).