evalmc 0.0.1__tar.gz → 0.0.3__tar.gz

evalmc-0.0.3/PKG-INFO

@@ -0,0 +1,95 @@
+Metadata-Version: 2.4
+Name: evalmc
+Version: 0.0.3
+Summary: Submit ONNX policies and Docker agents to MyoChallenge
+Author-email: MyoChallenge Team <caggiano@gmail.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/MyoHub/mcEvalProd-CLI
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.0.0
+Requires-Dist: docker>=6.0.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: boto3>=1.26.0
+Dynamic: license-file
+
+# evalmc
+
+MyoChallenge submission CLI — upload ONNX policies or Docker agent images to
+[mc26.myosuite.org](https://mc26.myosuite.org).
+
+[![PyPI](https://img.shields.io/pypi/v/evalmc.svg)](https://pypi.org/project/evalmc/)
+[![Python](https://img.shields.io/pypi/pyversions/evalmc.svg)](https://pypi.org/project/evalmc/)
+[![License](https://img.shields.io/pypi/l/evalmc.svg)](https://pypi.org/project/evalmc/)
+
+```bash
+pip install -U evalmc
+```
+
+Default API: `https://api.mc.myosuite.org`
+
+---
+
+## Quick start
+
+```bash
+evalmc host --set-host https://api.mc.myosuite.org
+evalmc login
+evalmc status
+evalmc submit-onnx model.onnx -t chasetag -p 1
+```
+
+`evalmc login` stores the email and auth token from the dashboard.
+Credentials live in `~/.myochallenge/`.
+
+---
+
+## Commands
+
+| Command | Purpose |
+|---|---|
+| `evalmc host` | Show or set the API host |
+| `evalmc login` | Save email + auth token |
+| `evalmc status` | Confirm host and team |
+| `evalmc submit-onnx` | Upload an ONNX policy |
+| `evalmc nominate` | Pick a scored artifact for title-cycle finals |
+| `evalmc push` | Submit a Docker / ECR agent image |
+
+### ONNX
+
+Recommended for ChaseTag. The graph is `obs → action`; no `policy.py` required.
+
+```bash
+evalmc submit-onnx model.onnx -t chasetag -p 1
+```
+
+### Title-cycle nomination
+
+Before freeze, choose which scored in-window submission plays in finals.
+If you skip this, the platform uses your best score.
+
+```bash
+evalmc nominate -t chasetag -p 1 --submission-id 123
+```
+
+### Docker
+
+For custom Python around the policy. The image must ship `policy.py` and/or
+`model.onnx` under `/policy` (or `/app`).
+
+```bash
+evalmc push my-agent:tag -t chasetag -p 1
+```
+
+---
+
+Some tracks are invitation-only. Registration and submission are rejected until
+the team is on the whitelist. Leaderboards stay public.

evalmc-0.0.3/README.md

@@ -0,0 +1,72 @@
+# evalmc
+
+MyoChallenge submission CLI — upload ONNX policies or Docker agent images to
+[mc26.myosuite.org](https://mc26.myosuite.org).
+
+[![PyPI](https://img.shields.io/pypi/v/evalmc.svg)](https://pypi.org/project/evalmc/)
+[![Python](https://img.shields.io/pypi/pyversions/evalmc.svg)](https://pypi.org/project/evalmc/)
+[![License](https://img.shields.io/pypi/l/evalmc.svg)](https://pypi.org/project/evalmc/)
+
+```bash
+pip install -U evalmc
+```
+
+Default API: `https://api.mc.myosuite.org`
+
+---
+
+## Quick start
+
+```bash
+evalmc host --set-host https://api.mc.myosuite.org
+evalmc login
+evalmc status
+evalmc submit-onnx model.onnx -t chasetag -p 1
+```
+
+`evalmc login` stores the email and auth token from the dashboard.
+Credentials live in `~/.myochallenge/`.
+
+---
+
+## Commands
+
+| Command | Purpose |
+|---|---|
+| `evalmc host` | Show or set the API host |
+| `evalmc login` | Save email + auth token |
+| `evalmc status` | Confirm host and team |
+| `evalmc submit-onnx` | Upload an ONNX policy |
+| `evalmc nominate` | Pick a scored artifact for title-cycle finals |
+| `evalmc push` | Submit a Docker / ECR agent image |
+
+### ONNX
+
+Recommended for ChaseTag. The graph is `obs → action`; no `policy.py` required.
+
+```bash
+evalmc submit-onnx model.onnx -t chasetag -p 1
+```
+
+### Title-cycle nomination
+
+Before freeze, choose which scored in-window submission plays in finals.
+If you skip this, the platform uses your best score.
+
+```bash
+evalmc nominate -t chasetag -p 1 --submission-id 123
+```
+
+### Docker
+
+For custom Python around the policy. The image must ship `policy.py` and/or
+`model.onnx` under `/policy` (or `/app`).
+
+```bash
+evalmc push my-agent:tag -t chasetag -p 1
+```
+
+---
+
+Some tracks are invitation-only. Registration and submission are rejected until
+the team is on the whitelist. Leaderboards stay public.

{evalmc-0.0.1 → evalmc-0.0.3}/evalmc/check_challenge.py

@@ -18,6 +18,8 @@ def check_task_phase(api_url, task, phase):
                 "score_names": data.get("score_names"),
                 "display_name": data.get("display_name"),
                 "visible": data.get("visible"),
+                "evaluation_mode": data.get("evaluation_mode", "absolute"),
+                "access_mode": data.get("access_mode", "open"),
                 "created_at": data.get("created_at"),
             }
         else:
@@ -112,6 +114,7 @@ def main():
         total_score_metrics += tp.get("num_scores", 0)
         label = tp.get("display_name") or f"{tp['task']} phase {tp['phase']}"
         print(f"✅ {label}: {tp['num_scores']} score metrics")
+        print(f"    Format: {tp.get('evaluation_mode', 'absolute')} | Access: {tp.get('access_mode', 'open')}")
         print(f"    Score Names: {', '.join(tp.get('score_names', []))}")
         print()
 

evalmc-0.0.3/evalmc/config_paths.py

@@ -0,0 +1,9 @@
+"""Shared local config file locations for the evalmc CLI."""
+
+from pathlib import Path
+
+CONFIG_DIR = Path.home() / ".myochallenge"
+HOST_URL_FILE = CONFIG_DIR / "host_url"
+AUTH_TOKEN_FILE = CONFIG_DIR / "auth_token.json"
+
+CONFIG_DIR.mkdir(exist_ok=True)

{evalmc-0.0.1 → evalmc-0.0.3}/evalmc/host.py

@@ -4,16 +4,10 @@ MyoChallenge Host - Configure API host URL
 """
 
 import sys
-from pathlib import Path
 
 import click
 
-# Configuration file paths
-CONFIG_DIR = Path.home() / ".myochallenge"
-HOST_URL_FILE = CONFIG_DIR / "host_url"
-
-# Ensure config directory exists
-CONFIG_DIR.mkdir(exist_ok=True)
+from .config_paths import HOST_URL_FILE
 
 
 @click.command()

{evalmc-0.0.1 → evalmc-0.0.3}/evalmc/login.py

@@ -5,18 +5,11 @@ MyoChallenge Login - Interactive authentication and credential storage
 
 import json
 import sys
-from pathlib import Path
 
 import click
 import requests
 
-# Configuration file paths
-CONFIG_DIR = Path.home() / ".myochallenge"
-HOST_URL_FILE = CONFIG_DIR / "host_url"
-AUTH_TOKEN_FILE = CONFIG_DIR / "auth_token.json"
-
-# Ensure config directory exists
-CONFIG_DIR.mkdir(exist_ok=True)
+from .config_paths import AUTH_TOKEN_FILE, HOST_URL_FILE
 
 
 def get_default_host():

{evalmc-0.0.1 → evalmc-0.0.3}/evalmc/main.py

@@ -10,8 +10,8 @@ from .login import login
 from .set_token import set_token
 from .status import status
 from .submission import push
-from .submit import submit
 from .submit_onnx import submit_onnx
+from .nominate import nominate
 
 
 @click.group()
@@ -23,8 +23,7 @@ def main():
     Two submission paths are available:
 
     \b
-    evalmc submit policy.zip -p 1 -t tabletennis   # zip upload (recommended)
-    evalmc submit-onnx model.onnx -p 1 -t elbowflexion  # ONNX upload
+    evalmc submit-onnx model.onnx -p 1 -t elbowflexion  # ONNX (elbow / ChaseTag)
     evalmc push   my-image:tag -p 1 -t tabletennis  # Docker/ECR image push
     """
     pass
@@ -34,6 +33,6 @@ main.add_command(host)
 main.add_command(login)
 main.add_command(set_token)
 main.add_command(push)
-main.add_command(submit)
 main.add_command(submit_onnx)
+main.add_command(nominate)
 main.add_command(status)

evalmc-0.0.3/evalmc/nominate.py

@@ -0,0 +1,33 @@
+"""Select a scored title-cycle submission for the upcoming finals."""
+
+import click
+import requests
+
+from .login import get_default_host
+from .push_helpers import http_error_detail, resolve_credentials
+
+
+@click.command()
+@click.option("-t", "--task", required=True)
+@click.option("-p", "--phase", required=True, type=int)
+@click.option("--submission-id", required=True, type=int)
+@click.option("-e", "--email", default=None)
+@click.option("-a", "--auth-token", default=None)
+def nominate(task, phase, submission_id, email, auth_token):
+    """Use SUBMISSION_ID for this title cycle's finals."""
+    email, auth_token = resolve_credentials(email, auth_token)
+    try:
+        response = requests.put(
+            f"{get_default_host()}/title-cycle/{task}/{phase}/nominate",
+            json={"submission_id": submission_id},
+            headers={"X-User-Email": email, "X-Auth-Token": auth_token},
+            timeout=20,
+        )
+        response.raise_for_status()
+    except requests.exceptions.HTTPError as exc:
+        raise click.ClickException(
+            f"Could not nominate submission: {http_error_detail(exc.response)}"
+        )
+    except requests.exceptions.RequestException as exc:
+        raise click.ClickException(f"Could not reach the API: {exc}")
+    click.echo(f"Submission {submission_id} will be used for finals.")

{evalmc-0.0.1 → evalmc-0.0.3}/evalmc/push_helpers.py

@@ -4,6 +4,9 @@ from __future__ import annotations
 
 from urllib.parse import urlparse
 
+import click
+import requests
+
 POLICY_PATHS = (
     "/policy/policy.py",
     "/policy/model.onnx",
@@ -51,7 +54,7 @@ def image_not_found_help(image: str, build_hint: str | None = None) -> str:
     lines.extend(
         [
             "",
-            "For elbow flexion, zip/ONNX submission is usually simpler:",
+            "For elbow flexion, ONNX submission is usually simpler:",
             "  evalmc submit-onnx model.onnx -t elbowflexion -p 1",
         ]
     )
@@ -95,9 +98,8 @@ def local_push_remote_warning(api_base: str, local_tag: str) -> str:
         f"\n"
         f"Expected outcome: submission status 'submitted' → 'failed' (no leaderboard entry).\n"
         f"\n"
-        f"Recommended alternatives for elbow flexion:\n"
+        f"Recommended alternative for elbow flexion:\n"
         f"  evalmc submit-onnx model.onnx -t elbowflexion -p 1\n"
-        f"  evalmc submit policy.zip -t elbowflexion -p 1\n"
         f"\n"
         f"For Docker on production, the server needs USE_LOCAL_STORAGE=false and ECR push.\n"
         f"Contact the organizers if ECR upload is required."
@@ -121,6 +123,63 @@ def policy_missing_help(image: str) -> str:
     )
 
 
+def resolve_credentials(email: str | None, auth_token: str | None) -> tuple[str, str]:
+    """Fill in email/auth_token from stored credentials when omitted.
+
+    Raises click.ClickException if neither the args nor stored credentials
+    provide both values.
+    """
+    from .login import get_stored_auth
+
+    if not email or not auth_token:
+        stored = get_stored_auth()
+        if not stored:
+            raise click.ClickException(
+                "No credentials found. Run 'evalmc login' or pass --email and --auth-token."
+            )
+        email = email or stored["email"]
+        auth_token = auth_token or stored["auth_token"]
+    return email, auth_token
+
+
+def post_submission_file(
+    api_base: str,
+    endpoint: str,
+    task: str,
+    phase: int,
+    email: str,
+    auth_token: str,
+    file_field: str,
+    file_name: str,
+    file_obj,
+    content_type: str,
+) -> dict:
+    """POST a submission file to the API; raise ClickException on failure."""
+    try:
+        resp = requests.post(
+            f"{api_base}{endpoint}",
+            data={"task": task, "phase": phase, "email": email, "auth_token": auth_token},
+            files={file_field: (file_name, file_obj, content_type)},
+            timeout=120,
+        )
+        resp.raise_for_status()
+    except requests.exceptions.HTTPError as e:
+        detail = http_error_detail(e.response)
+        if e.response.status_code == 403 and "invitation-only" in detail.lower():
+            detail = f"{detail} Ask an organizer to approve your team for this track."
+        raise click.ClickException(f"Server error {e.response.status_code}: {detail}")
+    except requests.exceptions.RequestException as e:
+        raise click.ClickException(f"Request failed: {e}")
+    return resp.json()
+
+
+def echo_submission_result(data: dict) -> None:
+    click.echo("Submitted successfully!")
+    click.echo(f"  Submission ID : {data['submission_id']}")
+    click.echo(f"  Type          : {data['submission_type']}")
+    click.echo(f"  Status        : {data['status']}")
+
+
 def post_submit_local_remote_note(submission_id, local_tag: str, api_base: str) -> str:
     return (
         f"\n⚠️  Submission {submission_id} recorded, but evaluation may fail on the server.\n"
@@ -132,5 +191,5 @@ def post_submit_local_remote_note(submission_id, local_tag: str, api_base: str)
         f"  curl -s {api_base}/submissions/team/YOUR_TEAM | python3 -m json.tool\n"
         f"\n"
         f"If status becomes 'failed', the server could not run your local Docker tag.\n"
-        f"Use evalmc submit-onnx or evalmc submit (zip) instead."
+        f"Use evalmc submit-onnx instead."
     )

{evalmc-0.0.1 → evalmc-0.0.3}/evalmc/set_token.py

@@ -5,16 +5,10 @@ MyoChallenge Set Token - Directly configure authentication token
 
 import json
 import sys
-from pathlib import Path
 
 import click
 
-# Configuration file paths
-CONFIG_DIR = Path.home() / ".myochallenge"
-AUTH_TOKEN_FILE = CONFIG_DIR / "auth_token.json"
-
-# Ensure config directory exists
-CONFIG_DIR.mkdir(exist_ok=True)
+from .config_paths import AUTH_TOKEN_FILE
 
 
 @click.command()

{evalmc-0.0.1 → evalmc-0.0.3}/evalmc/submission.py

@@ -3,10 +3,13 @@
 MyoChallenge Submission Script - ECR-based Docker image submission
 """
 
+import json
+import shutil
 import subprocess
 import sys
+import tempfile
+import time
 import uuid
-from pathlib import Path
 
 import click
 import docker
@@ -25,14 +28,6 @@ from .push_helpers import (
     upload_mode_summary,
 )
 
-# Configuration file paths
-CONFIG_DIR = Path.home() / ".myochallenge"
-HOST_URL_FILE = CONFIG_DIR / "host_url"
-AUTH_TOKEN_FILE = CONFIG_DIR / "auth_token.json"
-
-# Ensure config directory exists
-CONFIG_DIR.mkdir(exist_ok=True)
-
 
 # Configuration
 LOCAL_DOCKER_REGISTRY_URI = "localhost:5000"
@@ -134,8 +129,8 @@ def _run_subprocess(cmd: list[str], step: str) -> None:
 @click.option(
     "--api-base",
     help="API base URL",
-    default="https://api.mc.myosuite.org",
-    show_default=True,
+    default=None,
+    show_default="stored host, or https://api.mc.myosuite.org",
 )
 @click.option(
     "--skip-policy-check",
@@ -177,7 +172,7 @@ def push(image, phase, task, email, auth_token, url, public, private, api_base,
     if not public and not private:
         private = True  # Default to private
 
-    api_base = get_default_host()
+    api_base = api_base or get_default_host()
 
     click.echo(f"🚀 Starting submission for {image} to {task} phase {phase}")
     click.echo(f"🌐 API host: {api_base}")
@@ -250,15 +245,6 @@ def push(image, phase, task, email, auth_token, url, public, private, api_base,
     except requests.exceptions.RequestException as e:
         _fail(f"Authentication request failed: {e}")
 
-    # Collect submission metadata (not used in ECR flow)
-    # submission_metadata = {
-    #     "is_public": json.dumps(public),
-    #     "method_name": "",
-    #     "method_description": "",
-    #     "project_url": "",
-    #     "publication_url": "",
-    # }
-
     # Generate unique tag for submission
     submission_tag = str(uuid.uuid4())
     click.echo(f"🏷️  Generated submission tag: {submission_tag}")
@@ -290,7 +276,7 @@ def push(image, phase, task, email, auth_token, url, public, private, api_base,
                 if not click.confirm("Continue with local Docker tag anyway?", default=False):
                     _fail(
                         "Submission cancelled",
-                        "Use evalmc submit-onnx or evalmc submit (zip), or re-run with --yes.",
+                        "Use evalmc submit-onnx, or re-run with --yes.",
                     )
             elif not is_local_api(api_base):
                 click.echo(local_push_remote_warning(api_base, local_tag))
@@ -327,6 +313,8 @@ def push(image, phase, task, email, auth_token, url, public, private, api_base,
                     "task": task,
                     "phase": phase,
                     "image_uri": image_uri,
+                    "email": email,
+                    "auth_token": auth_token,
                 },
                 timeout=10,
             )
@@ -376,6 +364,8 @@ def push(image, phase, task, email, auth_token, url, public, private, api_base,
                     "task": task,
                     "phase": phase,
                     "image_uri": image_uri,
+                    "email": email,
+                    "auth_token": auth_token,
                 },
                 timeout=10,
             )
@@ -412,8 +402,6 @@ def push(image, phase, task, email, auth_token, url, public, private, api_base,
                     f"Server returned: {ecr_data}",
                 )
 
-            import time
-
             submission_id = str(uuid.uuid4())[
                 :8
             ]  # Use first 8 chars of UUID for uniqueness
@@ -425,10 +413,6 @@ def push(image, phase, task, email, auth_token, url, public, private, api_base,
             click.echo(f"⏰ Timestamp: {timestamp}")
 
             try:
-                import json
-                import shutil
-                import tempfile
-
                 temp_dir = tempfile.mkdtemp()
                 dockerfile_path = f"{temp_dir}/Dockerfile"
 
@@ -485,6 +469,8 @@ def push(image, phase, task, email, auth_token, url, public, private, api_base,
                         "task": task,
                         "phase": phase,
                         "image_uri": image_uri,
+                        "email": email,
+                        "auth_token": auth_token,
                     },
                     timeout=10,
                 )

evalmc-0.0.3/evalmc/submit_onnx.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+"""evalmc submit-onnx — upload a standalone ONNX model to MyoChallenge."""
+
+from pathlib import Path
+
+import click
+
+from .login import get_default_host
+from .push_helpers import echo_submission_result, post_submission_file, resolve_credentials
+
+
+@click.command("submit-onnx")
+@click.argument("onnx_path", type=click.Path(exists=True, path_type=Path))
+@click.option("-p", "--phase", required=True, type=int, help="Challenge phase number")
+@click.option("-t", "--task", required=True, help="Challenge task name (e.g. elbowflexion)")
+@click.option("-e", "--email", default=None, help="Team email")
+@click.option("-a", "--auth-token", default=None, help="Auth token")
+def submit_onnx(onnx_path, phase, task, email, auth_token):
+    """Submit a standalone ONNX model (no policy.py required).
+
+    Works for elbow flexion and ChaseTag (obs→action graphs).
+
+    Example:
+      evalmc submit-onnx my_model.onnx -t elbowflexion -p 1
+      evalmc submit-onnx chase.onnx -t chasetag -p 1
+    """
+    if onnx_path.suffix.lower() != ".onnx":
+        raise click.ClickException("File must have .onnx extension")
+
+    api_base = get_default_host()
+    email, auth_token = resolve_credentials(email, auth_token)
+
+    click.echo(f"Submitting {onnx_path.name} → task={task} phase={phase}")
+
+    with open(onnx_path, "rb") as f:
+        data = post_submission_file(
+            api_base,
+            "/submit/onnx",
+            task,
+            phase,
+            email,
+            auth_token,
+            file_field="onnx_model",
+            file_name=onnx_path.name,
+            file_obj=f,
+            content_type="application/octet-stream",
+        )
+
+    echo_submission_result(data)

evalmc-0.0.3/evalmc.egg-info/PKG-INFO

@@ -0,0 +1,95 @@
+Metadata-Version: 2.4
+Name: evalmc
+Version: 0.0.3
+Summary: Submit ONNX policies and Docker agents to MyoChallenge
+Author-email: MyoChallenge Team <caggiano@gmail.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/MyoHub/mcEvalProd-CLI
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.0.0
+Requires-Dist: docker>=6.0.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: boto3>=1.26.0
+Dynamic: license-file
+
+# evalmc
+
+MyoChallenge submission CLI — upload ONNX policies or Docker agent images to
+[mc26.myosuite.org](https://mc26.myosuite.org).
+
+[![PyPI](https://img.shields.io/pypi/v/evalmc.svg)](https://pypi.org/project/evalmc/)
+[![Python](https://img.shields.io/pypi/pyversions/evalmc.svg)](https://pypi.org/project/evalmc/)
+[![License](https://img.shields.io/pypi/l/evalmc.svg)](https://pypi.org/project/evalmc/)
+
+```bash
+pip install -U evalmc
+```
+
+Default API: `https://api.mc.myosuite.org`
+
+---
+
+## Quick start
+
+```bash
+evalmc host --set-host https://api.mc.myosuite.org
+evalmc login
+evalmc status
+evalmc submit-onnx model.onnx -t chasetag -p 1
+```
+
+`evalmc login` stores the email and auth token from the dashboard.
+Credentials live in `~/.myochallenge/`.
+
+---
+
+## Commands
+
+| Command | Purpose |
+|---|---|
+| `evalmc host` | Show or set the API host |
+| `evalmc login` | Save email + auth token |
+| `evalmc status` | Confirm host and team |
+| `evalmc submit-onnx` | Upload an ONNX policy |
+| `evalmc nominate` | Pick a scored artifact for title-cycle finals |
+| `evalmc push` | Submit a Docker / ECR agent image |
+
+### ONNX
+
+Recommended for ChaseTag. The graph is `obs → action`; no `policy.py` required.
+
+```bash
+evalmc submit-onnx model.onnx -t chasetag -p 1
+```
+
+### Title-cycle nomination
+
+Before freeze, choose which scored in-window submission plays in finals.
+If you skip this, the platform uses your best score.
+
+```bash
+evalmc nominate -t chasetag -p 1 --submission-id 123
+```
+
+### Docker
+
+For custom Python around the policy. The image must ship `policy.py` and/or
+`model.onnx` under `/policy` (or `/app`).
+
+```bash
+evalmc push my-agent:tag -t chasetag -p 1
+```
+
+---
+
+Some tracks are invitation-only. Registration and submission are rejected until
+the team is on the whitelist. Leaderboards stay public.