pearmut 0.1.2__tar.gz → 0.2.0__tar.gz

{pearmut-0.1.2 → pearmut-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.1.2
+Version: 0.2.0
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: apache-2.0
@@ -23,7 +23,7 @@ Dynamic: license-file
 
 Pearmut is a **Platform for Evaluation and Reviewing of Multilingual Tasks**.
 It evaluates model outputs, primarily translation but also various other NLP tasks.
-Supports multimodality (text, video, audio, images) and a variety of annotation protocols (DA, ESA, MQM, paired ESA, etc).
+Supports multimodality (text, video, audio, images) and a variety of annotation protocols ([DA](https://aclanthology.org/N15-1124/), [ESA](https://aclanthology.org/2024.wmt-1.131/), [ESA<sup>AI</sup>](https://aclanthology.org/2025.naacl-long.255/), [MQM](https://doi.org/10.1162/tacl_a_00437), paired ESA, etc).
 
 [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut)
 &nbsp;
@@ -31,7 +31,7 @@ Supports multimodality (text, video, audio, images) and a variety of annotation
 &nbsp;
 [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/)
 &nbsp;
-[![build status](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml)
+[![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml)
 
 <img width="1000" alt="Screenshot of ESA/MQM interface" src="https://github.com/user-attachments/assets/f14c91a5-44d7-4248-ada9-387e95ca59d0" />
 
@@ -42,11 +42,11 @@ You do not need to clone this repository. Simply install with pip and run locall
 # install the package
 pip install pearmut
 # download two campaign definitions
-wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/wmt25_%23_en-cs_CZ.json
-wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/wmt25_%23_cs-de_DE.json
+wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/esa_encs.json
+wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/da_enuk.json
 # load them into pearmut
-pearmut add wmt25_#_en-cs_CZ.json
-pearmut add wmt25_#_cs-de_DE.json
+pearmut add esa_encs.json
+pearmut add da_enuk.json
 # start pearmut (will show management links)
 pearmut run
 ```
@@ -115,6 +115,62 @@ For the standard ones (ESA, DA, MQM), we expect each item to be a dictionary (co
 ... # definition of another item (document)
 ```
 
+## Pre-filled Error Spans (ESA<sup>AI</sup> Support)
+
+For workflows where you want to provide pre-filled error annotations (e.g., ESA<sup>AI</sup>), you can include an `error_spans` key in each item.
+These spans will be loaded into the interface as existing annotations that users can review, modify, or delete.
+
+```python
+{
+  "src": "The quick brown fox jumps over the lazy dog.",
+  "tgt": "Rychlá hnědá liška skáče přes líného psa.",
+  "error_spans": [
+    {
+      "start_i": 0,         # character index start (inclusive)
+      "end_i": 5,           # character index end (inclusive)
+      "severity": "minor",  # "minor", "major", "neutral", or null
+      "category": null      # MQM category string or null
+    },
+    {
+      "start_i": 27,
+      "end_i": 32,
+      "severity": "major",
+      "category": null
+    }
+  ]
+}
+```
+
+For **listwise** template, `error_spans` is a 2D array where each inner array corresponds to error spans for that candidate.
+
+See [examples/esaai_prefilled.json](examples/esaai_prefilled.json) for a complete example.
+
+## Tutorial and Attention Checks
+
+You can add validation rules to items for tutorials or attention checks. Items with `validation` field will be checked before submission:
+
+```python
+{
+  "src": "The quick brown fox jumps.",
+  "tgt": "Rychlá hnědá liška skáče.",
+  "validation": {
+    "warning": "Please set score between 70-80.",  # shown on failure (omit for silent logging)
+    "score": [70, 80],                             # required score range [min, max]
+    "error_spans": [{"start_i": [0, 2], "end_i": [4, 8], "severity": "minor"}],  # expected spans
+    "allow_skip": true                             # show "skip tutorial" button
+  }
+}
+```
+
+- Tutorial items: Include `allow_skip: true` and `warning` to let users skip after seeing the feedback
+- Loud attention checks: Include `warning` without `allow_skip` to force users to retry
+- Silent attention checks: Omit `warning` to silently log failures without user notification (useful for quality control with bad translations)
+For listwise template, `validation` is an array where each element corresponds to a candidate. 
+The dashboard shows failed/total validation checks per user.
+See [examples/tutorial_pointwise.json](examples/tutorial_pointwise.json) and [examples/tutorial_listwise.json](examples/tutorial_listwise.json) for complete examples.
+
+## Single-stream Assignment
+
 We also support a simple allocation where all annotators draw from the same pool (`single-stream`). Items are randomly assigned to annotators from the pool of unfinished items:
 ```python
 {
@@ -138,7 +194,7 @@ We also support dynamic allocation of annotations (`dynamic`, not yet ⚠️), w
     "campaign_id": "my campaign 6",
     "info": {
         "assignment": "dynamic",
-        "template": "kway",
+        "template": "listwise",
         "protocol_k": 5,
         "num_users": 50,
     },
@@ -154,6 +210,25 @@ pearmut add my_campaign_4.json
 pearmut run
 ```
 
+## Campaign options
+
+In summary, you can select from the assignment types
+
+- `task-based`: each user has a predefined set of items
+- `single-stream`: all users are annotating together the same set of items
+- `dynamic`: WIP ⚠️
+
+and independently of that select your protocol template:
+
+- `pointwise`: evaluate a single output given a single output
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+- `listwise`: evaluate multiple outputs at the same time given a single output ⚠️
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+
 ## Campaign management
 
 When adding new campaigns or launching pearmut, a management link is shown that gives an overview of annotator progress but also an easy access to the annotation links or resetting the task progress (no data will be lost).
@@ -170,7 +245,7 @@ An intentionally incorrect token can be shown if the annotations don't pass qual
 
 We also support anything HTML-compatible both on the input and on the output.
 This includes embedded YouTube videos, or even simple `<video ` tags that point to some resource somewhere.
-For an example, try [examples/mock_multimodal.json](examples/mock_multimodal.json).
+For an example, try [examples/multimodal.json](examples/multimodal.json).
 Tip: make sure the elements are already appropriately styled.
 
 <img width="800" alt="Preview of multimodal elements in Pearmut" src="https://github.com/user-attachments/assets/f34a1a3e-ad95-4114-95ee-8a49e8003faf" />
@@ -209,7 +284,7 @@ If you use this work in your paper, please cite as:
 ```bibtex
 @misc{zouhar2025pearmut,
     author={Vilém Zouhar},
-    title={Pearmut🍐 Platform for Evaluation and Reviewing of Multilingual Tasks},
+    title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
     year={2025},
 }

{pearmut-0.1.2 → pearmut-0.2.0}/README.md

@@ -2,7 +2,7 @@
 
 Pearmut is a **Platform for Evaluation and Reviewing of Multilingual Tasks**.
 It evaluates model outputs, primarily translation but also various other NLP tasks.
-Supports multimodality (text, video, audio, images) and a variety of annotation protocols (DA, ESA, MQM, paired ESA, etc).
+Supports multimodality (text, video, audio, images) and a variety of annotation protocols ([DA](https://aclanthology.org/N15-1124/), [ESA](https://aclanthology.org/2024.wmt-1.131/), [ESA<sup>AI</sup>](https://aclanthology.org/2025.naacl-long.255/), [MQM](https://doi.org/10.1162/tacl_a_00437), paired ESA, etc).
 
 [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut)
 &nbsp;
@@ -10,7 +10,7 @@ Supports multimodality (text, video, audio, images) and a variety of annotation
 &nbsp;
 [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/)
 &nbsp;
-[![build status](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml)
+[![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml)
 
 <img width="1000" alt="Screenshot of ESA/MQM interface" src="https://github.com/user-attachments/assets/f14c91a5-44d7-4248-ada9-387e95ca59d0" />
 
@@ -21,11 +21,11 @@ You do not need to clone this repository. Simply install with pip and run locall
 # install the package
 pip install pearmut
 # download two campaign definitions
-wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/wmt25_%23_en-cs_CZ.json
-wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/wmt25_%23_cs-de_DE.json
+wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/esa_encs.json
+wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/da_enuk.json
 # load them into pearmut
-pearmut add wmt25_#_en-cs_CZ.json
-pearmut add wmt25_#_cs-de_DE.json
+pearmut add esa_encs.json
+pearmut add da_enuk.json
 # start pearmut (will show management links)
 pearmut run
 ```
@@ -94,6 +94,62 @@ For the standard ones (ESA, DA, MQM), we expect each item to be a dictionary (co
 ... # definition of another item (document)
 ```
 
+## Pre-filled Error Spans (ESA<sup>AI</sup> Support)
+
+For workflows where you want to provide pre-filled error annotations (e.g., ESA<sup>AI</sup>), you can include an `error_spans` key in each item.
+These spans will be loaded into the interface as existing annotations that users can review, modify, or delete.
+
+```python
+{
+  "src": "The quick brown fox jumps over the lazy dog.",
+  "tgt": "Rychlá hnědá liška skáče přes líného psa.",
+  "error_spans": [
+    {
+      "start_i": 0,         # character index start (inclusive)
+      "end_i": 5,           # character index end (inclusive)
+      "severity": "minor",  # "minor", "major", "neutral", or null
+      "category": null      # MQM category string or null
+    },
+    {
+      "start_i": 27,
+      "end_i": 32,
+      "severity": "major",
+      "category": null
+    }
+  ]
+}
+```
+
+For **listwise** template, `error_spans` is a 2D array where each inner array corresponds to error spans for that candidate.
+
+See [examples/esaai_prefilled.json](examples/esaai_prefilled.json) for a complete example.
+
+## Tutorial and Attention Checks
+
+You can add validation rules to items for tutorials or attention checks. Items with `validation` field will be checked before submission:
+
+```python
+{
+  "src": "The quick brown fox jumps.",
+  "tgt": "Rychlá hnědá liška skáče.",
+  "validation": {
+    "warning": "Please set score between 70-80.",  # shown on failure (omit for silent logging)
+    "score": [70, 80],                             # required score range [min, max]
+    "error_spans": [{"start_i": [0, 2], "end_i": [4, 8], "severity": "minor"}],  # expected spans
+    "allow_skip": true                             # show "skip tutorial" button
+  }
+}
+```
+
+- Tutorial items: Include `allow_skip: true` and `warning` to let users skip after seeing the feedback
+- Loud attention checks: Include `warning` without `allow_skip` to force users to retry
+- Silent attention checks: Omit `warning` to silently log failures without user notification (useful for quality control with bad translations)
+For listwise template, `validation` is an array where each element corresponds to a candidate. 
+The dashboard shows failed/total validation checks per user.
+See [examples/tutorial_pointwise.json](examples/tutorial_pointwise.json) and [examples/tutorial_listwise.json](examples/tutorial_listwise.json) for complete examples.
+
+## Single-stream Assignment
+
 We also support a simple allocation where all annotators draw from the same pool (`single-stream`). Items are randomly assigned to annotators from the pool of unfinished items:
 ```python
 {
@@ -117,7 +173,7 @@ We also support dynamic allocation of annotations (`dynamic`, not yet ⚠️), w
     "campaign_id": "my campaign 6",
     "info": {
         "assignment": "dynamic",
-        "template": "kway",
+        "template": "listwise",
         "protocol_k": 5,
         "num_users": 50,
     },
@@ -133,6 +189,25 @@ pearmut add my_campaign_4.json
 pearmut run
 ```
 
+## Campaign options
+
+In summary, you can select from the assignment types
+
+- `task-based`: each user has a predefined set of items
+- `single-stream`: all users are annotating together the same set of items
+- `dynamic`: WIP ⚠️
+
+and independently of that select your protocol template:
+
+- `pointwise`: evaluate a single output given a single output
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+- `listwise`: evaluate multiple outputs at the same time given a single output ⚠️
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+
 ## Campaign management
 
 When adding new campaigns or launching pearmut, a management link is shown that gives an overview of annotator progress but also an easy access to the annotation links or resetting the task progress (no data will be lost).
@@ -149,7 +224,7 @@ An intentionally incorrect token can be shown if the annotations don't pass qual
 
 We also support anything HTML-compatible both on the input and on the output.
 This includes embedded YouTube videos, or even simple `<video ` tags that point to some resource somewhere.
-For an example, try [examples/mock_multimodal.json](examples/mock_multimodal.json).
+For an example, try [examples/multimodal.json](examples/multimodal.json).
 Tip: make sure the elements are already appropriately styled.
 
 <img width="800" alt="Preview of multimodal elements in Pearmut" src="https://github.com/user-attachments/assets/f34a1a3e-ad95-4114-95ee-8a49e8003faf" />
@@ -188,7 +263,7 @@ If you use this work in your paper, please cite as:
 ```bibtex
 @misc{zouhar2025pearmut,
     author={Vilém Zouhar},
-    title={Pearmut🍐 Platform for Evaluation and Reviewing of Multilingual Tasks},
+    title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
     year={2025},
 }

{pearmut-0.1.2 → pearmut-0.2.0}/pearmut.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.1.2
+Version: 0.2.0
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: apache-2.0
@@ -23,7 +23,7 @@ Dynamic: license-file
 
 Pearmut is a **Platform for Evaluation and Reviewing of Multilingual Tasks**.
 It evaluates model outputs, primarily translation but also various other NLP tasks.
-Supports multimodality (text, video, audio, images) and a variety of annotation protocols (DA, ESA, MQM, paired ESA, etc).
+Supports multimodality (text, video, audio, images) and a variety of annotation protocols ([DA](https://aclanthology.org/N15-1124/), [ESA](https://aclanthology.org/2024.wmt-1.131/), [ESA<sup>AI</sup>](https://aclanthology.org/2025.naacl-long.255/), [MQM](https://doi.org/10.1162/tacl_a_00437), paired ESA, etc).
 
 [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut)
 &nbsp;
@@ -31,7 +31,7 @@ Supports multimodality (text, video, audio, images) and a variety of annotation
 &nbsp;
 [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/)
 &nbsp;
-[![build status](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml)
+[![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml)
 
 <img width="1000" alt="Screenshot of ESA/MQM interface" src="https://github.com/user-attachments/assets/f14c91a5-44d7-4248-ada9-387e95ca59d0" />
 
@@ -42,11 +42,11 @@ You do not need to clone this repository. Simply install with pip and run locall
 # install the package
 pip install pearmut
 # download two campaign definitions
-wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/wmt25_%23_en-cs_CZ.json
-wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/wmt25_%23_cs-de_DE.json
+wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/esa_encs.json
+wget https://raw.githubusercontent.com/zouharvi/pearmut/refs/heads/main/examples/da_enuk.json
 # load them into pearmut
-pearmut add wmt25_#_en-cs_CZ.json
-pearmut add wmt25_#_cs-de_DE.json
+pearmut add esa_encs.json
+pearmut add da_enuk.json
 # start pearmut (will show management links)
 pearmut run
 ```
@@ -115,6 +115,62 @@ For the standard ones (ESA, DA, MQM), we expect each item to be a dictionary (co
 ... # definition of another item (document)
 ```
 
+## Pre-filled Error Spans (ESA<sup>AI</sup> Support)
+
+For workflows where you want to provide pre-filled error annotations (e.g., ESA<sup>AI</sup>), you can include an `error_spans` key in each item.
+These spans will be loaded into the interface as existing annotations that users can review, modify, or delete.
+
+```python
+{
+  "src": "The quick brown fox jumps over the lazy dog.",
+  "tgt": "Rychlá hnědá liška skáče přes líného psa.",
+  "error_spans": [
+    {
+      "start_i": 0,         # character index start (inclusive)
+      "end_i": 5,           # character index end (inclusive)
+      "severity": "minor",  # "minor", "major", "neutral", or null
+      "category": null      # MQM category string or null
+    },
+    {
+      "start_i": 27,
+      "end_i": 32,
+      "severity": "major",
+      "category": null
+    }
+  ]
+}
+```
+
+For **listwise** template, `error_spans` is a 2D array where each inner array corresponds to error spans for that candidate.
+
+See [examples/esaai_prefilled.json](examples/esaai_prefilled.json) for a complete example.
+
+## Tutorial and Attention Checks
+
+You can add validation rules to items for tutorials or attention checks. Items with `validation` field will be checked before submission:
+
+```python
+{
+  "src": "The quick brown fox jumps.",
+  "tgt": "Rychlá hnědá liška skáče.",
+  "validation": {
+    "warning": "Please set score between 70-80.",  # shown on failure (omit for silent logging)
+    "score": [70, 80],                             # required score range [min, max]
+    "error_spans": [{"start_i": [0, 2], "end_i": [4, 8], "severity": "minor"}],  # expected spans
+    "allow_skip": true                             # show "skip tutorial" button
+  }
+}
+```
+
+- Tutorial items: Include `allow_skip: true` and `warning` to let users skip after seeing the feedback
+- Loud attention checks: Include `warning` without `allow_skip` to force users to retry
+- Silent attention checks: Omit `warning` to silently log failures without user notification (useful for quality control with bad translations)
+For listwise template, `validation` is an array where each element corresponds to a candidate. 
+The dashboard shows failed/total validation checks per user.
+See [examples/tutorial_pointwise.json](examples/tutorial_pointwise.json) and [examples/tutorial_listwise.json](examples/tutorial_listwise.json) for complete examples.
+
+## Single-stream Assignment
+
 We also support a simple allocation where all annotators draw from the same pool (`single-stream`). Items are randomly assigned to annotators from the pool of unfinished items:
 ```python
 {
@@ -138,7 +194,7 @@ We also support dynamic allocation of annotations (`dynamic`, not yet ⚠️), w
     "campaign_id": "my campaign 6",
     "info": {
         "assignment": "dynamic",
-        "template": "kway",
+        "template": "listwise",
         "protocol_k": 5,
         "num_users": 50,
     },
@@ -154,6 +210,25 @@ pearmut add my_campaign_4.json
 pearmut run
 ```
 
+## Campaign options
+
+In summary, you can select from the assignment types
+
+- `task-based`: each user has a predefined set of items
+- `single-stream`: all users are annotating together the same set of items
+- `dynamic`: WIP ⚠️
+
+and independently of that select your protocol template:
+
+- `pointwise`: evaluate a single output given a single output
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+- `listwise`: evaluate multiple outputs at the same time given a single output ⚠️
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+
 ## Campaign management
 
 When adding new campaigns or launching pearmut, a management link is shown that gives an overview of annotator progress but also an easy access to the annotation links or resetting the task progress (no data will be lost).
@@ -170,7 +245,7 @@ An intentionally incorrect token can be shown if the annotations don't pass qual
 
 We also support anything HTML-compatible both on the input and on the output.
 This includes embedded YouTube videos, or even simple `<video ` tags that point to some resource somewhere.
-For an example, try [examples/mock_multimodal.json](examples/mock_multimodal.json).
+For an example, try [examples/multimodal.json](examples/multimodal.json).
 Tip: make sure the elements are already appropriately styled.
 
 <img width="800" alt="Preview of multimodal elements in Pearmut" src="https://github.com/user-attachments/assets/f34a1a3e-ad95-4114-95ee-8a49e8003faf" />
@@ -209,7 +284,7 @@ If you use this work in your paper, please cite as:
 ```bibtex
 @misc{zouhar2025pearmut,
     author={Vilém Zouhar},
-    title={Pearmut🍐 Platform for Evaluation and Reviewing of Multilingual Tasks},
+    title={Pearmut: Platform for Evaluating and Reviewing of Multilingual Tasks},
     url={https://github.com/zouharvi/pearmut/},
     year={2025},
 }

{pearmut-0.1.2 → pearmut-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pearmut"
-version = "0.1.2"
+version = "0.2.0"
 description = "A tool for evaluation of model outputs, primarily MT."
 readme = "README.md"
 license = { text = "apache-2.0" }
@@ -40,6 +40,7 @@ build-backend = "setuptools.build_meta"
 [tool.setuptools.package-data]
 "pearmut" = ["static/**"]
 
+# managed by .github/workflows/publish.yml now but still can be built and pushed lcoally
 # rm -rf server/static/; npm install web/ --prefix web/; npm run build --prefix web/
 # rm -rf {build,dist,pearmut.egg-info}/*; python3 -m build 
 # python3 -m twine upload dist/* -u __token__

{pearmut-0.1.2 → pearmut-0.2.0}/server/app.py

@@ -8,8 +8,8 @@ from fastapi.responses import JSONResponse
 from fastapi.staticfiles import StaticFiles
 from pydantic import BaseModel
 
-from .assignment import get_next_item, reset_task, update_progress
-from .utils import ROOT, load_progress_data, save_progress_data
+from .assignment import get_i_item, get_next_item, reset_task, update_progress
+from .utils import ROOT, load_progress_data, save_db_payload, save_progress_data
 
 os.makedirs(f"{ROOT}/data/outputs", exist_ok=True)
 
@@ -36,7 +36,7 @@ class LogResponseRequest(BaseModel):
     campaign_id: str
     user_id: str
     item_i: int
-    payload: Any
+    payload: dict[str, Any]
 
 
 @app.post("/log-response")
@@ -45,6 +45,7 @@ async def _log_response(request: LogResponseRequest):
 
     campaign_id = request.campaign_id
     user_id = request.user_id
+    item_i = request.item_i
 
     if campaign_id not in progress_data:
         return JSONResponse(content={"error": "Unknown campaign ID"}, status_code=400)
@@ -52,8 +53,8 @@ async def _log_response(request: LogResponseRequest):
         return JSONResponse(content={"error": "Unknown user ID"}, status_code=400)
 
     # append response to the output log
-    with open(f"{ROOT}/data/outputs/{campaign_id}.jsonl", "a") as log_file:
-        log_file.write(json.dumps(request.payload, ensure_ascii=False) + "\n")
+    save_db_payload(campaign_id, request.payload | {
+                    "user_id": user_id, "item_i": item_i})
 
     # if actions were submitted, we can log time data
     if "actions" in request.payload:
@@ -68,7 +69,16 @@ async def _log_response(request: LogResponseRequest):
             for a, b in zip(times, times[1:])
         ])
 
-    update_progress(campaign_id, user_id, tasks_data, progress_data, request.item_i, request.payload)
+    # Initialize validation_checks if it doesn't exist
+    print(request.payload.keys())
+    if "validations" in request.payload:
+        if "validations" not in progress_data[campaign_id][user_id]:
+            progress_data[campaign_id][user_id]["validations"] = {}
+
+        progress_data[campaign_id][user_id]["validations"][request.item_i] = request.payload["validations"]
+
+    update_progress(campaign_id, user_id, tasks_data,
+                    progress_data, request.item_i, request.payload)
     save_progress_data(progress_data)
 
     return JSONResponse(content={"status": "ok"}, status_code=200)
@@ -97,6 +107,32 @@ async def _get_next_item(request: NextItemRequest):
     )
 
 
+class GetItemRequest(BaseModel):
+    campaign_id: str
+    user_id: str
+    item_i: int
+
+
+@app.post("/get-i-item")
+async def _get_i_item(request: GetItemRequest):
+    campaign_id = request.campaign_id
+    user_id = request.user_id
+    item_i = request.item_i
+
+    if campaign_id not in progress_data:
+        return JSONResponse(content={"error": "Unknown campaign ID"}, status_code=400)
+    if user_id not in progress_data[campaign_id]:
+        return JSONResponse(content={"error": "Unknown user ID"}, status_code=400)
+
+    return get_i_item(
+        campaign_id,
+        user_id,
+        tasks_data,
+        progress_data,
+        item_i,
+    )
+
+
 class DashboardDataRequest(BaseModel):
     campaign_id: str
     token: str | None = None
@@ -119,6 +155,11 @@ async def _dashboard_data(request: DashboardDataRequest):
     for user_id, user_val in progress_data[campaign_id].items():
         # shallow copy
         entry = dict(user_val)
+        entry["validations"] = [
+            all(v)
+            for v in list(entry.get("validations", {}).values())
+        ]
+        
 
         if not is_privileged:
             entry["token_correct"] = None
@@ -203,10 +244,11 @@ async def _download_progress(
 
 static_dir = f"{os.path.dirname(os.path.abspath(__file__))}/static/"
 if not os.path.exists(static_dir + "index.html"):
-    raise FileNotFoundError("Static directory not found. Please build the frontend first.")
+    raise FileNotFoundError(
+        "Static directory not found. Please build the frontend first.")
 
 app.mount(
     "/",
     StaticFiles(directory=static_dir, html=True, follow_symlink=True),
     name="static",
-)
+)