pearmut 0.2.5__tar.gz → 0.2.6__tar.gz

{pearmut-0.2.5 → pearmut-0.2.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.2.5
+Version: 0.2.6
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: apache-2.0
@@ -67,7 +67,6 @@ One of the simplest ones, where each user has a pre-defined list of tasks (`task
     "protocol_score": true,                 # we want scores [0...100] for each segment
     "protocol_error_spans": true,           # we want error spans
     "protocol_error_categories": false,     # we do not want error span categories
-    "instructions": "Evaluate translation from en to cs_CZ",  # message to show to users
   },
   "campaign_id": "wmt25_#_en-cs_CZ",
   "data": [
@@ -76,6 +75,7 @@ One of the simplest ones, where each user has a pre-defined list of tasks (`task
       [
         # each evaluation item is a document
         {
+          "instructions": "Evaluate translation from en to cs_CZ",  # message to show to users above the first item
           "src": "This will be the year that Guinness loses its cool. Cheers to that!",
           "tgt": "Nevím přesně, kdy jsem to poprvé zaznamenal. Možná to bylo ve chvíli, ..."
         },
@@ -288,22 +288,25 @@ Pearmut provides the following commands:
 
 ## Hosting Assets
 
-If you need to host local assets (e.g., audio files, images, videos) via Pearmut, you can use the `assets` key in your campaign file.
-When present, this directory is symlinked to the `static/` directory so its contents become accessible from the server.
+To host local assets (e.g., audio files, images, videos), use the `assets` key in your campaign file:
 
 ```python
 {
     "campaign_id": "my_campaign",
     "info": { 
-      "assets": "videos",  # path to directory containing assets
+      "assets": {
+        "source": "videos",              # path to directory containing assets
+        "destination": "assets/my_videos"  # where to mount (must start with "assets/")
+      },
       ...
     },
     "data": [ ... ]
 }
 ```
 
-For example, if `videos` contains `audio.mp3`, it will be accessible at `localhost:8001/assets/videos/audio.mp3`.
-The path can be absolute or relative to your current working directory.
+Files in `videos/` become accessible at `localhost:8001/assets/my_videos/`.
+This creates a symlink, so the source directory must exist throughout the annotation period.
+If another campaign already uses the destination path, the add command will fail.
 
 ## Development
 
@@ -333,6 +336,8 @@ The `pearmut run` also accepts `--port` (default 8001).
 If you wish to create a new protocol (referenceable from `info->template`), simply create a new HTML and TS file in `web/src` and add a rule to `webpack.config.js` so that your template gets built.
 A template can call the server for data etc (see [web/src/pointwise.ts](web/src/pointwise.ts) as an exmple).
 
+To run the platform, you need to run this on some publicly-facing server, or run it locally and tunnel your port to a public IP/domain.
+
 ## Citation
 
 If you use this work in your paper, please cite as:

{pearmut-0.2.5 → pearmut-0.2.6}/README.md

@@ -47,7 +47,6 @@ One of the simplest ones, where each user has a pre-defined list of tasks (`task
     "protocol_score": true,                 # we want scores [0...100] for each segment
     "protocol_error_spans": true,           # we want error spans
     "protocol_error_categories": false,     # we do not want error span categories
-    "instructions": "Evaluate translation from en to cs_CZ",  # message to show to users
   },
   "campaign_id": "wmt25_#_en-cs_CZ",
   "data": [
@@ -56,6 +55,7 @@ One of the simplest ones, where each user has a pre-defined list of tasks (`task
       [
         # each evaluation item is a document
         {
+          "instructions": "Evaluate translation from en to cs_CZ",  # message to show to users above the first item
           "src": "This will be the year that Guinness loses its cool. Cheers to that!",
           "tgt": "Nevím přesně, kdy jsem to poprvé zaznamenal. Možná to bylo ve chvíli, ..."
         },
@@ -268,22 +268,25 @@ Pearmut provides the following commands:
 
 ## Hosting Assets
 
-If you need to host local assets (e.g., audio files, images, videos) via Pearmut, you can use the `assets` key in your campaign file.
-When present, this directory is symlinked to the `static/` directory so its contents become accessible from the server.
+To host local assets (e.g., audio files, images, videos), use the `assets` key in your campaign file:
 
 ```python
 {
     "campaign_id": "my_campaign",
     "info": { 
-      "assets": "videos",  # path to directory containing assets
+      "assets": {
+        "source": "videos",              # path to directory containing assets
+        "destination": "assets/my_videos"  # where to mount (must start with "assets/")
+      },
       ...
     },
     "data": [ ... ]
 }
 ```
 
-For example, if `videos` contains `audio.mp3`, it will be accessible at `localhost:8001/assets/videos/audio.mp3`.
-The path can be absolute or relative to your current working directory.
+Files in `videos/` become accessible at `localhost:8001/assets/my_videos/`.
+This creates a symlink, so the source directory must exist throughout the annotation period.
+If another campaign already uses the destination path, the add command will fail.
 
 ## Development
 
@@ -313,6 +316,8 @@ The `pearmut run` also accepts `--port` (default 8001).
 If you wish to create a new protocol (referenceable from `info->template`), simply create a new HTML and TS file in `web/src` and add a rule to `webpack.config.js` so that your template gets built.
 A template can call the server for data etc (see [web/src/pointwise.ts](web/src/pointwise.ts) as an exmple).
 
+To run the platform, you need to run this on some publicly-facing server, or run it locally and tunnel your port to a public IP/domain.
+
 ## Citation
 
 If you use this work in your paper, please cite as:

{pearmut-0.2.5 → pearmut-0.2.6}/pearmut.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pearmut
-Version: 0.2.5
+Version: 0.2.6
 Summary: A tool for evaluation of model outputs, primarily MT.
 Author-email: Vilém Zouhar <vilem.zouhar@gmail.com>
 License: apache-2.0
@@ -67,7 +67,6 @@ One of the simplest ones, where each user has a pre-defined list of tasks (`task
     "protocol_score": true,                 # we want scores [0...100] for each segment
     "protocol_error_spans": true,           # we want error spans
     "protocol_error_categories": false,     # we do not want error span categories
-    "instructions": "Evaluate translation from en to cs_CZ",  # message to show to users
   },
   "campaign_id": "wmt25_#_en-cs_CZ",
   "data": [
@@ -76,6 +75,7 @@ One of the simplest ones, where each user has a pre-defined list of tasks (`task
       [
         # each evaluation item is a document
         {
+          "instructions": "Evaluate translation from en to cs_CZ",  # message to show to users above the first item
           "src": "This will be the year that Guinness loses its cool. Cheers to that!",
           "tgt": "Nevím přesně, kdy jsem to poprvé zaznamenal. Možná to bylo ve chvíli, ..."
         },
@@ -288,22 +288,25 @@ Pearmut provides the following commands:
 
 ## Hosting Assets
 
-If you need to host local assets (e.g., audio files, images, videos) via Pearmut, you can use the `assets` key in your campaign file.
-When present, this directory is symlinked to the `static/` directory so its contents become accessible from the server.
+To host local assets (e.g., audio files, images, videos), use the `assets` key in your campaign file:
 
 ```python
 {
     "campaign_id": "my_campaign",
     "info": { 
-      "assets": "videos",  # path to directory containing assets
+      "assets": {
+        "source": "videos",              # path to directory containing assets
+        "destination": "assets/my_videos"  # where to mount (must start with "assets/")
+      },
       ...
     },
     "data": [ ... ]
 }
 ```
 
-For example, if `videos` contains `audio.mp3`, it will be accessible at `localhost:8001/assets/videos/audio.mp3`.
-The path can be absolute or relative to your current working directory.
+Files in `videos/` become accessible at `localhost:8001/assets/my_videos/`.
+This creates a symlink, so the source directory must exist throughout the annotation period.
+If another campaign already uses the destination path, the add command will fail.
 
 ## Development
 
@@ -333,6 +336,8 @@ The `pearmut run` also accepts `--port` (default 8001).
 If you wish to create a new protocol (referenceable from `info->template`), simply create a new HTML and TS file in `web/src` and add a rule to `webpack.config.js` so that your template gets built.
 A template can call the server for data etc (see [web/src/pointwise.ts](web/src/pointwise.ts) as an exmple).
 
+To run the platform, you need to run this on some publicly-facing server, or run it locally and tunnel your port to a public IP/domain.
+
 ## Citation
 
 If you use this work in your paper, please cite as:

{pearmut-0.2.5 → pearmut-0.2.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pearmut"
-version = "0.2.5"
+version = "0.2.6"
 description = "A tool for evaluation of model outputs, primarily MT."
 readme = "README.md"
 license = { text = "apache-2.0" }

{pearmut-0.2.5 → pearmut-0.2.6}/server/cli.py

@@ -12,6 +12,9 @@ import psutil
 
 from .utils import ROOT, load_progress_data, save_progress_data
 
+# Static directory path (constant for consistency)
+STATIC_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "static")
+
 os.makedirs(f"{ROOT}/data/tasks", exist_ok=True)
 load_progress_data(warn=None)
 
@@ -50,6 +53,40 @@ def _run(args_unknown):
     )
 
 
+def _validate_item_structure(items, template):
+    """
+    Validate that items have the correct structure.
+    Items should be lists of dictionaries with 'src' and 'tgt' keys.
+    
+    Args:
+        items: List of item dictionaries to validate
+        template: Template type ('pointwise' or 'listwise') for type validation
+    """
+    if not isinstance(items, list):
+        raise ValueError("Items must be a list")
+
+    for item in items:
+        if not isinstance(item, dict):
+            raise ValueError("Each item must be a dictionary with 'src' and 'tgt' keys")
+        if 'src' not in item or 'tgt' not in item:
+            raise ValueError("Each item must contain 'src' and 'tgt' keys")
+        
+        # Validate src is always a string
+        if not isinstance(item['src'], str):
+            raise ValueError("Item 'src' must be a string")
+        
+        # Validate tgt type based on template
+        if template == 'listwise':
+            if not isinstance(item['tgt'], list):
+                raise ValueError("Item 'tgt' must be a list for listwise template")
+            # Check that all elements in tgt list are strings
+            if not all(isinstance(t, str) for t in item['tgt']):
+                raise ValueError("All elements in 'tgt' list must be strings for listwise template")
+        elif template == 'pointwise':
+            if not isinstance(item['tgt'], str):
+                raise ValueError("Item 'tgt' must be a string for pointwise template")
+
+
 def _add_single_campaign(data_file, overwrite, server):
     """
     Add a single campaign from a JSON data file.
@@ -80,6 +117,7 @@ def _add_single_campaign(data_file, overwrite, server):
         raise ValueError("Campaign 'info' must contain 'template' field.")
 
     assignment = campaign_data["info"]["assignment"]
+    template = campaign_data["info"]["template"]
     # use random words for identifying users
     rng = random.Random(campaign_data["campaign_id"])
     rword = wonderwords.RandomWord(rng=rng)
@@ -96,6 +134,13 @@ def _add_single_campaign(data_file, overwrite, server):
         if not all(isinstance(task, list) for task in tasks):
             raise ValueError(
                 "Each task in task-based campaign 'data' must be a list of items.")
+        # Validate item structure for each task
+        for task_i, task in enumerate(tasks):
+            for doc_i, doc in enumerate(task):
+                try:
+                    _validate_item_structure(doc, template)
+                except ValueError as e:
+                    raise ValueError(f"Task {task_i}, document {doc_i}: {e}")
         num_users = len(tasks)
     elif assignment == "single-stream":
         tasks = campaign_data["data"]
@@ -105,6 +150,12 @@ def _add_single_campaign(data_file, overwrite, server):
         if not isinstance(campaign_data["data"], list):
             raise ValueError(
                 "Single-stream campaign 'data' must be a list of items.")
+        # Validate item structure for single-stream
+        for doc_i, doc in enumerate(tasks):
+            try:
+                _validate_item_structure(doc, template)
+            except ValueError as e:
+                raise ValueError(f"Document {doc_i}: {e}")
         if isinstance(users_spec, int):
             num_users = users_spec
         elif isinstance(users_spec, list):
@@ -199,30 +250,62 @@ def _add_single_campaign(data_file, overwrite, server):
 
     # Handle assets symlink if specified
     if "assets" in campaign_data["info"]:
-        assets_real_path = campaign_data["info"]["assets"]
+        assets_config = campaign_data["info"]["assets"]
+        
+        # assets must be a dictionary with source and destination keys
+        if not isinstance(assets_config, dict):
+            raise ValueError("Assets must be a dictionary with 'source' and 'destination' keys.")
+        if "source" not in assets_config or "destination" not in assets_config:
+            raise ValueError("Assets config must contain 'source' and 'destination' keys.")
+        
+        assets_source = assets_config["source"]
+        assets_destination = assets_config["destination"]
+        
+        # Validate destination starts with 'assets/'
+        if not assets_destination.startswith("assets/"):
+            raise ValueError(f"Assets destination '{assets_destination}' must start with 'assets/'.")
         
         # Resolve relative paths from the caller's current working directory
-        assets_real_path = os.path.abspath(assets_real_path)
+        assets_real_path = os.path.abspath(assets_source)
 
         if not os.path.isdir(assets_real_path):
-            raise ValueError(f"Assets path '{assets_real_path}' must be an existing directory.")
+            raise ValueError(f"Assets source path '{assets_real_path}' must be an existing directory.")
 
-        static_dir = f"{os.path.dirname(os.path.abspath(__file__))}/static"
-        dir_name = assets_real_path.split(os.sep)[-1]
-
-        if not os.path.isdir(static_dir):
+        if not os.path.isdir(STATIC_DIR):
             raise ValueError(
-                f"Static directory '{static_dir}' does not exist. "
+                f"Static directory '{STATIC_DIR}' does not exist. "
                 "Please build the frontend first."
             )
-        symlink_path = f"{static_dir}/assets/{dir_name}"
-
-        # Remove existing symlink if present and we are overriding
-        if os.path.exists(symlink_path):
+        
+        # Symlink path is based on the destination, stripping the 'assets/' prefix
+        symlink_path = f"{STATIC_DIR}/{assets_destination}"
+
+        # Remove existing symlink if present and we are overriding the same campaign
+        if os.path.lexists(symlink_path):
+            # Check if any other campaign is using this destination
+            current_campaign_id = campaign_data['campaign_id']
+            tasks_dir = f"{ROOT}/data/tasks"
+            if os.path.exists(tasks_dir):
+                for task_file in os.listdir(tasks_dir):
+                    if task_file.endswith('.json'):
+                        other_campaign_id = task_file[:-5]
+                        if other_campaign_id != current_campaign_id:
+                            with open(f"{tasks_dir}/{task_file}", "r") as f:
+                                other_campaign = json.load(f)
+                            other_assets = other_campaign.get("info", {}).get("assets")
+                            if other_assets and isinstance(other_assets, dict):
+                                if other_assets.get("destination") == assets_destination:
+                                    raise ValueError(
+                                        f"Assets destination '{assets_destination}' is already used by campaign '{other_campaign_id}'."
+                                    )
+            # Only allow overwrite if it's the same campaign
             if overwrite:
                 os.remove(symlink_path)
             else:
-                raise ValueError(f"Assets symlink '{symlink_path}' already exists.")
+                raise ValueError(f"Assets destination '{assets_destination}' is already taken.")
+        
+        # Ensure the assets directory exists
+        os.makedirs(f"{STATIC_DIR}/assets", exist_ok=True)
 
         os.symlink(assets_real_path, symlink_path, target_is_directory=True)
         print(f"Assets symlinked: {symlink_path} -> {assets_real_path}")
@@ -299,6 +382,20 @@ def main():
     elif args.command == 'purge':
         import shutil
 
+        def _unlink_assets(campaign_id):
+            """Unlink assets symlink for a campaign if it exists."""
+            task_file = f"{ROOT}/data/tasks/{campaign_id}.json"
+            if not os.path.exists(task_file):
+                return
+            with open(task_file, "r") as f:
+                campaign_data = json.load(f)
+            destination = campaign_data.get("info", {}).get("assets", {}).get("destination")
+            if destination:
+                symlink_path = f"{STATIC_DIR}/{destination}"
+                if os.path.islink(symlink_path):
+                    os.remove(symlink_path)
+                    print(f"Assets symlink removed: {symlink_path}")
+
         # Parse optional campaign name
         purge_args = argparse.ArgumentParser()
         purge_args.add_argument(
@@ -314,6 +411,8 @@ def main():
                 f"Are you sure you want to purge campaign '{campaign_id}'? This action cannot be undone. [y/n] "
             )
             if confirm.lower() == 'y':
+                # Unlink assets before removing task file
+                _unlink_assets(campaign_id)
                 # Remove task file
                 task_file = f"{ROOT}/data/tasks/{campaign_id}.json"
                 if os.path.exists(task_file):
@@ -336,6 +435,13 @@ def main():
                 "Are you sure you want to purge all campaign data? This action cannot be undone. [y/n] "
             )
             if confirm.lower() == 'y':
+                # Unlink all assets first
+                tasks_dir = f"{ROOT}/data/tasks"
+                if os.path.exists(tasks_dir):
+                    for task_file in os.listdir(tasks_dir):
+                        if task_file.endswith('.json'):
+                            campaign_id = task_file[:-5]
+                            _unlink_assets(campaign_id)
                 shutil.rmtree(f"{ROOT}/data/tasks", ignore_errors=True)
                 shutil.rmtree(f"{ROOT}/data/outputs", ignore_errors=True)
                 if os.path.exists(f"{ROOT}/data/progress.json"):