pearmut 0.1.2__tar.gz → 0.1.3__tar.gz

{pearmut-0.1.2 → pearmut-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.1.2
+Version: 0.1.3
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: apache-2.0
@@ -23,7 +23,7 @@ Dynamic: license-file
 
 Pearmut is a **Platform for Evaluation and Reviewing of Multilingual Tasks**.
 It evaluates model outputs, primarily translation but also various other NLP tasks.
-Supports multimodality (text, video, audio, images) and a variety of annotation protocols (DA, ESA, MQM, paired ESA, etc).
+Supports multimodality (text, video, audio, images) and a variety of annotation protocols ([DA](https://aclanthology.org/N15-1124/), [ESA](https://aclanthology.org/2024.wmt-1.131/), [ESA<sup>AI</sup>](https://aclanthology.org/2025.naacl-long.255/), [MQM](https://doi.org/10.1162/tacl_a_00437), paired ESA, etc).
 
 [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut)
 &nbsp;
@@ -31,7 +31,7 @@ Supports multimodality (text, video, audio, images) and a variety of annotation
 &nbsp;
 [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/)
 &nbsp;
-[![build status](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml)
+[![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml)
 
 <img width="1000" alt="Screenshot of ESA/MQM interface" src="https://github.com/user-attachments/assets/f14c91a5-44d7-4248-ada9-387e95ca59d0" />
 
@@ -115,6 +115,38 @@ For the standard ones (ESA, DA, MQM), we expect each item to be a dictionary (co
 ... # definition of another item (document)
 ```
 
+## Pre-filled Error Spans (ESA<sup>AI</sup> Support)
+
+For workflows where you want to provide pre-filled error annotations (e.g., ESA<sup>AI</sup>), you can include an `error_spans` key in each item.
+These spans will be loaded into the interface as existing annotations that users can review, modify, or delete.
+
+```python
+{
+  "src": "The quick brown fox jumps over the lazy dog.",
+  "tgt": "Rychlá hnědá liška skáče přes líného psa.",
+  "error_spans": [
+    {
+      "start_i": 0,         # character index start (inclusive)
+      "end_i": 5,           # character index end (inclusive)
+      "severity": "minor",  # "minor", "major", "neutral", or null
+      "category": null      # MQM category string or null
+    },
+    {
+      "start_i": 27,
+      "end_i": 32,
+      "severity": "major",
+      "category": null
+    }
+  ]
+}
+```
+
+For **listwise** template, `error_spans` is a 2D array where each inner array corresponds to error spans for that candidate.
+
+See [examples/esaai_prefilled.json](examples/esaai_prefilled.json) for a complete example.
+
+## Single-stream Assignment
+
 We also support a simple allocation where all annotators draw from the same pool (`single-stream`). Items are randomly assigned to annotators from the pool of unfinished items:
 ```python
 {
@@ -138,7 +170,7 @@ We also support dynamic allocation of annotations (`dynamic`, not yet ⚠️), w
     "campaign_id": "my campaign 6",
     "info": {
         "assignment": "dynamic",
-        "template": "kway",
+        "template": "listwise",
         "protocol_k": 5,
         "num_users": 50,
     },
@@ -154,6 +186,25 @@ pearmut add my_campaign_4.json
 pearmut run
 ```
 
+## Campaign options
+
+In summary, you can select from the assignment types
+
+- `task-based`: each user has a predefined set of items
+- `single-stream`: all users are annotating together the same set of items
+- `dynamic`: WIP ⚠️
+
+and independently of that select your protocol template:
+
+- `pointwise`: evaluate a single output given a single output
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+- `listwise`: evaluate multiple outputs at the same time given a single output ⚠️
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+
 ## Campaign management
 
 When adding new campaigns or launching pearmut, a management link is shown that gives an overview of annotator progress but also an easy access to the annotation links or resetting the task progress (no data will be lost).
@@ -170,7 +221,7 @@ An intentionally incorrect token can be shown if the annotations don't pass qual
 
 We also support anything HTML-compatible both on the input and on the output.
 This includes embedded YouTube videos, or even simple `<video ` tags that point to some resource somewhere.
-For an example, try [examples/mock_multimodal.json](examples/mock_multimodal.json).
+For an example, try [examples/multimodal.json](examples/multimodal.json).
 Tip: make sure the elements are already appropriately styled.
 
 <img width="800" alt="Preview of multimodal elements in Pearmut" src="https://github.com/user-attachments/assets/f34a1a3e-ad95-4114-95ee-8a49e8003faf" />

{pearmut-0.1.2 → pearmut-0.1.3}/README.md

@@ -2,7 +2,7 @@
 
 Pearmut is a **Platform for Evaluation and Reviewing of Multilingual Tasks**.
 It evaluates model outputs, primarily translation but also various other NLP tasks.
-Supports multimodality (text, video, audio, images) and a variety of annotation protocols (DA, ESA, MQM, paired ESA, etc).
+Supports multimodality (text, video, audio, images) and a variety of annotation protocols ([DA](https://aclanthology.org/N15-1124/), [ESA](https://aclanthology.org/2024.wmt-1.131/), [ESA<sup>AI</sup>](https://aclanthology.org/2025.naacl-long.255/), [MQM](https://doi.org/10.1162/tacl_a_00437), paired ESA, etc).
 
 [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut)
 &nbsp;
@@ -10,7 +10,7 @@ Supports multimodality (text, video, audio, images) and a variety of annotation
 &nbsp;
 [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/)
 &nbsp;
-[![build status](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml)
+[![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml)
 
 <img width="1000" alt="Screenshot of ESA/MQM interface" src="https://github.com/user-attachments/assets/f14c91a5-44d7-4248-ada9-387e95ca59d0" />
 
@@ -94,6 +94,38 @@ For the standard ones (ESA, DA, MQM), we expect each item to be a dictionary (co
 ... # definition of another item (document)
 ```
 
+## Pre-filled Error Spans (ESA<sup>AI</sup> Support)
+
+For workflows where you want to provide pre-filled error annotations (e.g., ESA<sup>AI</sup>), you can include an `error_spans` key in each item.
+These spans will be loaded into the interface as existing annotations that users can review, modify, or delete.
+
+```python
+{
+  "src": "The quick brown fox jumps over the lazy dog.",
+  "tgt": "Rychlá hnědá liška skáče přes líného psa.",
+  "error_spans": [
+    {
+      "start_i": 0,         # character index start (inclusive)
+      "end_i": 5,           # character index end (inclusive)
+      "severity": "minor",  # "minor", "major", "neutral", or null
+      "category": null      # MQM category string or null
+    },
+    {
+      "start_i": 27,
+      "end_i": 32,
+      "severity": "major",
+      "category": null
+    }
+  ]
+}
+```
+
+For **listwise** template, `error_spans` is a 2D array where each inner array corresponds to error spans for that candidate.
+
+See [examples/esaai_prefilled.json](examples/esaai_prefilled.json) for a complete example.
+
+## Single-stream Assignment
+
 We also support a simple allocation where all annotators draw from the same pool (`single-stream`). Items are randomly assigned to annotators from the pool of unfinished items:
 ```python
 {
@@ -117,7 +149,7 @@ We also support dynamic allocation of annotations (`dynamic`, not yet ⚠️), w
     "campaign_id": "my campaign 6",
     "info": {
         "assignment": "dynamic",
-        "template": "kway",
+        "template": "listwise",
         "protocol_k": 5,
         "num_users": 50,
     },
@@ -133,6 +165,25 @@ pearmut add my_campaign_4.json
 pearmut run
 ```
 
+## Campaign options
+
+In summary, you can select from the assignment types
+
+- `task-based`: each user has a predefined set of items
+- `single-stream`: all users are annotating together the same set of items
+- `dynamic`: WIP ⚠️
+
+and independently of that select your protocol template:
+
+- `pointwise`: evaluate a single output given a single output
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+- `listwise`: evaluate multiple outputs at the same time given a single output ⚠️
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+
 ## Campaign management
 
 When adding new campaigns or launching pearmut, a management link is shown that gives an overview of annotator progress but also an easy access to the annotation links or resetting the task progress (no data will be lost).
@@ -149,7 +200,7 @@ An intentionally incorrect token can be shown if the annotations don't pass qual
 
 We also support anything HTML-compatible both on the input and on the output.
 This includes embedded YouTube videos, or even simple `<video ` tags that point to some resource somewhere.
-For an example, try [examples/mock_multimodal.json](examples/mock_multimodal.json).
+For an example, try [examples/multimodal.json](examples/multimodal.json).
 Tip: make sure the elements are already appropriately styled.
 
 <img width="800" alt="Preview of multimodal elements in Pearmut" src="https://github.com/user-attachments/assets/f34a1a3e-ad95-4114-95ee-8a49e8003faf" />

{pearmut-0.1.2 → pearmut-0.1.3}/pearmut.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.1.2
+Version: 0.1.3
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: apache-2.0
@@ -23,7 +23,7 @@ Dynamic: license-file
 
 Pearmut is a **Platform for Evaluation and Reviewing of Multilingual Tasks**.
 It evaluates model outputs, primarily translation but also various other NLP tasks.
-Supports multimodality (text, video, audio, images) and a variety of annotation protocols (DA, ESA, MQM, paired ESA, etc).
+Supports multimodality (text, video, audio, images) and a variety of annotation protocols ([DA](https://aclanthology.org/N15-1124/), [ESA](https://aclanthology.org/2024.wmt-1.131/), [ESA<sup>AI</sup>](https://aclanthology.org/2025.naacl-long.255/), [MQM](https://doi.org/10.1162/tacl_a_00437), paired ESA, etc).
 
 [![PyPi version](https://badgen.net/pypi/v/pearmut/)](https://pypi.org/project/pearmut)
 &nbsp;
@@ -31,7 +31,7 @@ Supports multimodality (text, video, audio, images) and a variety of annotation
 &nbsp;
 [![PyPi license](https://badgen.net/pypi/license/pearmut/)](https://pypi.org/project/pearmut/)
 &nbsp;
-[![build status](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/ci.yml)
+[![build status](https://github.com/zouharvi/pearmut/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/pearmut/actions/workflows/test.yml)
 
 <img width="1000" alt="Screenshot of ESA/MQM interface" src="https://github.com/user-attachments/assets/f14c91a5-44d7-4248-ada9-387e95ca59d0" />
 
@@ -115,6 +115,38 @@ For the standard ones (ESA, DA, MQM), we expect each item to be a dictionary (co
 ... # definition of another item (document)
 ```
 
+## Pre-filled Error Spans (ESA<sup>AI</sup> Support)
+
+For workflows where you want to provide pre-filled error annotations (e.g., ESA<sup>AI</sup>), you can include an `error_spans` key in each item.
+These spans will be loaded into the interface as existing annotations that users can review, modify, or delete.
+
+```python
+{
+  "src": "The quick brown fox jumps over the lazy dog.",
+  "tgt": "Rychlá hnědá liška skáče přes líného psa.",
+  "error_spans": [
+    {
+      "start_i": 0,         # character index start (inclusive)
+      "end_i": 5,           # character index end (inclusive)
+      "severity": "minor",  # "minor", "major", "neutral", or null
+      "category": null      # MQM category string or null
+    },
+    {
+      "start_i": 27,
+      "end_i": 32,
+      "severity": "major",
+      "category": null
+    }
+  ]
+}
+```
+
+For **listwise** template, `error_spans` is a 2D array where each inner array corresponds to error spans for that candidate.
+
+See [examples/esaai_prefilled.json](examples/esaai_prefilled.json) for a complete example.
+
+## Single-stream Assignment
+
 We also support a simple allocation where all annotators draw from the same pool (`single-stream`). Items are randomly assigned to annotators from the pool of unfinished items:
 ```python
 {
@@ -138,7 +170,7 @@ We also support dynamic allocation of annotations (`dynamic`, not yet ⚠️), w
     "campaign_id": "my campaign 6",
     "info": {
         "assignment": "dynamic",
-        "template": "kway",
+        "template": "listwise",
         "protocol_k": 5,
         "num_users": 50,
     },
@@ -154,6 +186,25 @@ pearmut add my_campaign_4.json
 pearmut run
 ```
 
+## Campaign options
+
+In summary, you can select from the assignment types
+
+- `task-based`: each user has a predefined set of items
+- `single-stream`: all users are annotating together the same set of items
+- `dynamic`: WIP ⚠️
+
+and independently of that select your protocol template:
+
+- `pointwise`: evaluate a single output given a single output
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+- `listwise`: evaluate multiple outputs at the same time given a single output ⚠️
+  - `protocol_score`: ask for score 0 to 100
+  - `protocol_error_spans`: ask for highlighting error spans
+  - `protocol_error_categories`: ask for highlighting error categories
+
 ## Campaign management
 
 When adding new campaigns or launching pearmut, a management link is shown that gives an overview of annotator progress but also an easy access to the annotation links or resetting the task progress (no data will be lost).
@@ -170,7 +221,7 @@ An intentionally incorrect token can be shown if the annotations don't pass qual
 
 We also support anything HTML-compatible both on the input and on the output.
 This includes embedded YouTube videos, or even simple `<video ` tags that point to some resource somewhere.
-For an example, try [examples/mock_multimodal.json](examples/mock_multimodal.json).
+For an example, try [examples/multimodal.json](examples/multimodal.json).
 Tip: make sure the elements are already appropriately styled.
 
 <img width="800" alt="Preview of multimodal elements in Pearmut" src="https://github.com/user-attachments/assets/f34a1a3e-ad95-4114-95ee-8a49e8003faf" />

{pearmut-0.1.2 → pearmut-0.1.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pearmut"
-version = "0.1.2"
+version = "0.1.3"
 description = "A tool for evaluation of model outputs, primarily MT."
 readme = "README.md"
 license = { text = "apache-2.0" }

{pearmut-0.1.2 → pearmut-0.1.3}/server/app.py

@@ -8,8 +8,8 @@ from fastapi.responses import JSONResponse
 from fastapi.staticfiles import StaticFiles
 from pydantic import BaseModel
 
-from .assignment import get_next_item, reset_task, update_progress
-from .utils import ROOT, load_progress_data, save_progress_data
+from .assignment import get_i_item, get_next_item, reset_task, update_progress
+from .utils import ROOT, load_progress_data, save_db_payload, save_progress_data
 
 os.makedirs(f"{ROOT}/data/outputs", exist_ok=True)
 
@@ -36,7 +36,7 @@ class LogResponseRequest(BaseModel):
     campaign_id: str
     user_id: str
     item_i: int
-    payload: Any
+    payload: dict[str, Any]
 
 
 @app.post("/log-response")
@@ -45,6 +45,7 @@ async def _log_response(request: LogResponseRequest):
 
     campaign_id = request.campaign_id
     user_id = request.user_id
+    item_i = request.item_i
 
     if campaign_id not in progress_data:
         return JSONResponse(content={"error": "Unknown campaign ID"}, status_code=400)
@@ -52,8 +53,7 @@ async def _log_response(request: LogResponseRequest):
         return JSONResponse(content={"error": "Unknown user ID"}, status_code=400)
 
     # append response to the output log
-    with open(f"{ROOT}/data/outputs/{campaign_id}.jsonl", "a") as log_file:
-        log_file.write(json.dumps(request.payload, ensure_ascii=False) + "\n")
+    save_db_payload(campaign_id, request.payload | {"user_id": user_id, "item_i": item_i})
 
     # if actions were submitted, we can log time data
     if "actions" in request.payload:
@@ -97,6 +97,32 @@ async def _get_next_item(request: NextItemRequest):
     )
 
 
+class GetItemRequest(BaseModel):
+    campaign_id: str
+    user_id: str
+    item_i: int
+
+
+@app.post("/get-i-item")
+async def _get_i_item(request: GetItemRequest):
+    campaign_id = request.campaign_id
+    user_id = request.user_id
+    item_i = request.item_i
+
+    if campaign_id not in progress_data:
+        return JSONResponse(content={"error": "Unknown campaign ID"}, status_code=400)
+    if user_id not in progress_data[campaign_id]:
+        return JSONResponse(content={"error": "Unknown user ID"}, status_code=400)
+
+    return get_i_item(
+        campaign_id,
+        user_id,
+        tasks_data,
+        progress_data,
+        item_i,
+    )
+
+
 class DashboardDataRequest(BaseModel):
     campaign_id: str
     token: str | None = None

{pearmut-0.1.2 → pearmut-0.1.3}/server/assignment.py

@@ -3,6 +3,8 @@ from typing import Any
 
 from fastapi.responses import JSONResponse
 
+from .utils import get_db_log_item
+
 
 def _completed_response(
     progress_data: dict,
@@ -37,13 +39,121 @@ def get_next_item(
     if assignment == "task-based":
         return get_next_item_taskbased(campaign_id, user_id, tasks_data, progress_data)
     elif assignment == "single-stream":
-        return get_next_item_single_stream(campaign_id, user_id, tasks_data, progress_data)
+        return get_next_item_singlestream(campaign_id, user_id, tasks_data, progress_data)
     elif assignment == "dynamic":
         return get_next_item_dynamic(campaign_id, user_id, tasks_data, progress_data)
     else:
         return JSONResponse(content={"error": "Unknown campaign assignment type"}, status_code=400)
 
 
+def get_i_item(
+    campaign_id: str,
+    user_id: str,
+    tasks_data: dict,
+    progress_data: dict,
+    item_i: int,
+) -> JSONResponse:
+    """
+    Get a specific item by index for the user in the specified campaign.
+    """
+    assignment = tasks_data[campaign_id]["info"]["assignment"]
+    if assignment == "task-based":
+        return get_i_item_taskbased(campaign_id, user_id, tasks_data, progress_data, item_i)
+    elif assignment == "single-stream":
+        return get_i_item_singlestream(campaign_id, user_id, tasks_data, progress_data, item_i)
+    else:
+        return JSONResponse(content={"error": "Get item not supported for this assignment type"}, status_code=400)
+
+
+def get_i_item_taskbased(
+    campaign_id: str,
+    user_id: str,
+    data_all: dict,
+    progress_data: dict,
+    item_i: int,
+) -> JSONResponse:
+    """
+    Get specific item for task-based protocol.
+    """
+    user_progress = progress_data[campaign_id][user_id]
+    if all(user_progress["progress"]):
+        return _completed_response(progress_data, campaign_id, user_id)
+
+    # try to get existing annotations if any
+    items_existing = get_db_log_item(campaign_id, user_id, item_i)
+    if items_existing:
+        # get the latest ones
+        payload_existing = items_existing[-1]["annotations"]
+
+    if item_i < 0 or item_i >= len(data_all[campaign_id]["data"][user_id]):
+        return JSONResponse(
+            content={"status": "error", "message": "Item index out of range"},
+            status_code=400
+        )
+
+    return JSONResponse(
+        content={
+            "status": "ok",
+            "progress": user_progress["progress"],
+            "time": user_progress["time"],
+            "info": {
+                "item_i": item_i,
+            } | {
+                k: v
+                for k, v in data_all[campaign_id]["info"].items()
+                if k.startswith("protocol")
+            },
+            "payload": data_all[campaign_id]["data"][user_id][item_i]
+        } | ({"payload_existing": payload_existing} if items_existing else {}),
+        status_code=200
+    )
+
+
+def get_i_item_singlestream(
+    campaign_id: str,
+    user_id: str,
+    data_all: dict,
+    progress_data: dict,
+    item_i: int,
+) -> JSONResponse:
+    """
+    Get specific item for single-stream assignment.
+    """
+    user_progress = progress_data[campaign_id][user_id]
+    if all(user_progress["progress"]):
+        return _completed_response(progress_data, campaign_id, user_id)
+
+    # try to get existing annotations if any
+    # note the None user_id since it is shared
+    items_existing = get_db_log_item(campaign_id, None, item_i)
+    if items_existing:
+        # get the latest ones
+        payload_existing = items_existing[-1]["annotations"]
+
+    if item_i < 0 or item_i >= len(data_all[campaign_id]["data"]):
+        return JSONResponse(
+            content={"status": "error", "message": "Item index out of range"},
+            status_code=400
+        )
+
+    return JSONResponse(
+        content={
+            "status": "ok",
+            "progress": user_progress["progress"],
+            "time": user_progress["time"],
+            "info": {
+                "item_i": item_i,
+            } | {
+                k: v
+                for k, v in data_all[campaign_id]["info"].items()
+                if k.startswith("protocol")
+            },
+            "payload": data_all[campaign_id]["data"][item_i]
+        } | ({"payload_existing": payload_existing} if items_existing else {}),
+        status_code=200
+    )
+
+
 def get_next_item_taskbased(
     campaign_id: str,
     user_id: str,
@@ -51,7 +161,7 @@ def get_next_item_taskbased(
     progress_data: dict,
 ) -> JSONResponse:
     """
-    Get the next item for task-based protocol.
+    Get the next item for task-based assignment.
     """
     user_progress = progress_data[campaign_id][user_id]
     if all(user_progress["progress"]):
@@ -59,6 +169,13 @@ def get_next_item_taskbased(
 
     # find first incomplete item
     item_i = min([i for i, v in enumerate(user_progress["progress"]) if not v])
+
+    # try to get existing annotations if any
+    items_existing = get_db_log_item(campaign_id, user_id, item_i)
+    if items_existing:
+        # get the latest ones
+        payload_existing = items_existing[-1]["annotations"]
+
     return JSONResponse(
         content={
             "status": "ok",
@@ -71,23 +188,20 @@ def get_next_item_taskbased(
                 for k, v in data_all[campaign_id]["info"].items()
                 if k.startswith("protocol")
             },
-            "payload": data_all[campaign_id]["data"][user_id][item_i]},
+            "payload": data_all[campaign_id]["data"][user_id][item_i]
+        } | ({"payload_existing": payload_existing} if items_existing else {}),
         status_code=200
     )
 
 
-def get_next_item_dynamic(campaign_data: dict, user_id: str, progress_data: dict, data_all: dict):
-    raise NotImplementedError("Dynamic protocol is not implemented yet.")
-
-
-def get_next_item_single_stream(
+def get_next_item_singlestream(
     campaign_id: str,
     user_id: str,
     data_all: dict,
     progress_data: dict,
 ) -> JSONResponse:
     """
-    Get the next item for single-stream protocol.
+    Get the next item for single-stream assignment.
     In this mode, all users share the same pool of items.
     Items are randomly selected from unfinished items.
 
@@ -104,6 +218,13 @@ def get_next_item_single_stream(
     incomplete_indices = [i for i, v in enumerate(progress) if not v]
     item_i = random.choice(incomplete_indices)
 
+    # try to get existing annotations if any
+    # note the None user_id since it is shared
+    items_existing = get_db_log_item(campaign_id, None, item_i)
+    if items_existing:
+        # get the latest ones
+        payload_existing = items_existing[-1]["annotations"]
+
     return JSONResponse(
         content={
             "status": "ok",
@@ -116,11 +237,18 @@ def get_next_item_single_stream(
                 for k, v in data_all[campaign_id]["info"].items()
                 if k.startswith("protocol")
             },
-            "payload": data_all[campaign_id]["data"][item_i]},
+            "payload": data_all[campaign_id]["data"][item_i]
+        } | ({"payload_existing": payload_existing} if items_existing else {}),
         status_code=200
     )
 
 
+
+def get_next_item_dynamic(campaign_data: dict, user_id: str, progress_data: dict, data_all: dict):
+    raise NotImplementedError("Dynamic protocol is not implemented yet.")
+
+
+
 def _reset_user_time(progress_data: dict, campaign_id: str, user_id: str) -> None:
     """Reset time tracking fields for a user."""
     progress_data[campaign_id][user_id]["time"] = 0.0