aimx 0.3.0__tar.gz → 0.3.2__tar.gz

aimx-0.3.2/.agents/skills/aimx/SKILL.md

@@ -0,0 +1,123 @@
+---
+name: aimx
+description: Use when autoresearch, log_experiment, experiment analysis, or automatic iteration workflows need to inspect local Aim repositories with aimx, collect run params, metric summaries, traces, image evidence, compare training runs, or summarize model results without mutating Aim data.
+---
+
+# Aimx
+
+## Overview
+
+Use `aimx` as a read-only evidence collector for `autoresearch` `log_experiment`
+steps. Prefer JSON output so downstream agents can compare runs, explain model
+effects, and propose the next experiment from concrete Aim data.
+
+## Requirements
+
+- Require `aimx` in the Python environment that runs `log_experiment`.
+  Verify with `aimx --help` or `python -m aimx --help` before collecting data.
+- When `aimx` is missing, first identify and follow the user's current
+  dependency-management workflow from project files and instructions
+  (`pyproject.toml`, lockfiles, README, AGENTS/CLAUDE/GEMINI notes, CI config).
+  Use that manager to add or install `aimx`. In this repository, prefer
+  `uv sync`, `uv run ...`, and `uv add aimx` because repo policy requires `uv`.
+- If there is no project environment to modify and the user only needs a
+  standalone CLI, install `aimx` as a tool using the user's preferred tool
+  manager. Use `uv tool install aimx` only when `uv` is already the chosen or
+  accepted tool workflow.
+- Require read access to a local Aim repository path. Pass `--repo` explicitly
+  and keep collection commands read-only.
+- If native Aim availability matters for passthrough checks, run `aimx doctor`;
+  do not initialize, repair, migrate, or rewrite Aim repositories from this
+  skill.
+
+## Workflow
+
+1. Locate the Aim repository. Pass `--repo <repo-root-or-.aim>` explicitly; in
+   this repository, use `--repo data` or `--repo data/.aim` for local checks.
+2. Define the run scope as an AimQL expression. Start broad with
+   `run.hash != ''`, then narrow by `run.hash`, `run.experiment`, or `run.name`.
+3. Collect run parameters:
+
+   ```bash
+   aimx query params "<run-scope>" --repo <repo> --json
+   aimx query params "<run-scope>" --repo <repo> --json --param hparam.lr --param model
+   ```
+
+4. Collect metric summaries for candidate objective metrics:
+
+   ```bash
+   aimx query metrics "(<run-scope>) and metric.name == 'loss'" --repo <repo> --json
+   aimx query metrics "(<run-scope>) and metric.name != ''" --repo <repo> --json
+   ```
+
+5. Collect traces only for decisive metrics or suspected anomalies:
+
+   ```bash
+   aimx trace "(<run-scope>) and metric.name == 'loss'" --repo <repo> --json --tail 50
+   ```
+
+6. Collect image metadata when qualitative outputs matter:
+
+   ```bash
+   aimx query images "images" --repo <repo> --json --head 20
+   ```
+
+7. Emit a compact `log_experiment` record containing:
+
+   ```json
+   {
+     "repo": "<repo>",
+     "run_scope": "<AimQL>",
+     "params": {},
+     "metric_summary": {},
+     "trace_evidence": {},
+     "image_evidence": {},
+     "interpretation": {
+       "best_runs": [],
+       "regressions": [],
+       "confidence": "low|medium|high",
+       "next_experiments": []
+     }
+   }
+   ```
+
+## Interpretation Rules
+
+- Prefer validation, test, or held-out contexts over training contexts when
+  ranking runs.
+- Treat `aimx query metrics` as summary data: `last`, `min`, `max`, and step
+  counts. Use `aimx trace --json` when shape, stability, divergence, or late
+  improvement matters.
+- For minimization metrics such as loss or error, compare `min.value` and the
+  corresponding step. For maximization metrics such as accuracy, F1, AUC, or
+  IoU, compare `max.value`.
+- Separate incomplete or failed runs from strong results before drawing
+  conclusions. Very low step counts, missing params, or missing validation
+  metrics should reduce confidence.
+- Preserve read-only behavior. Do not run commands that initialize, repair,
+  migrate, delete, or rewrite Aim repositories during `log_experiment`.
+
+## Helper Script
+
+Use `scripts/collect_experiment_snapshot.py` when an agent needs one structured
+JSON bundle instead of several manual commands.
+
+```bash
+uv run python skills/aimx/scripts/collect_experiment_snapshot.py \
+  --repo data \
+  --base-expr "run.experiment == 'cloud-segmentation'" \
+  --metric loss \
+  --trace-metric loss \
+  --param hparam.lr \
+  --param model \
+  --pretty
+```
+
+The script uses the current Python interpreter as `python -m aimx` by default.
+Pass `--aimx "aimx"` or `--aimx "uv run aimx"` when a different launcher is
+needed. It writes only to stdout.
+
+## Reference
+
+Read `references/aimx-cli.md` for command details, JSON envelope shapes, and
+suggested `log_experiment` evidence fields.

aimx-0.3.2/.agents/skills/aimx/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Aimx"
+  short_description: "Collect Aim experiment evidence"
+  default_prompt: "Use $aimx to collect params, metrics, traces, and image evidence from an Aim repo before proposing the next experiment."

aimx-0.3.2/.agents/skills/aimx/references/aimx-cli.md

@@ -0,0 +1,233 @@
+# aimx CLI Reference for log_experiment
+
+Use this reference when building an `autoresearch` `log_experiment` step from a
+local Aim repository. All commands below are read-only inspection commands.
+
+## Installation Requirement
+
+`aimx` must be installed in the environment that runs the collection commands.
+Verify one of these before collecting:
+
+```bash
+aimx --help
+python -m aimx --help
+```
+
+If it is missing, inspect the current project first and use the dependency
+manager it already uses. Check project instructions, `pyproject.toml`, lockfiles,
+README setup docs, and CI config before choosing a command.
+
+Examples:
+
+```bash
+# uv-managed project
+uv add aimx
+uv sync
+
+# Poetry-managed project
+poetry add aimx
+
+# Pipenv-managed project
+pipenv install aimx
+
+# requirements.txt project
+python -m pip install aimx
+python -m pip freeze > requirements.txt
+```
+
+Use the project's normal lockfile/update flow. For this `aimx` repository,
+prefer `uv sync`, `uv run ...`, and `uv add aimx` because repo policy requires
+`uv`. For a standalone CLI with no project dependency to modify, use the user's
+preferred tool manager; use `uv tool install aimx` only when `uv` is already the
+chosen or accepted workflow.
+
+## Repository Paths
+
+`--repo` accepts either the Aim repository root or the `.aim` metadata
+directory. Prefer explicit repo paths so automation does not accidentally read a
+user-level or unrelated Aim repository.
+
+Examples:
+
+```bash
+aimx query params "run.hash != ''" --repo data --json
+aimx query params "run.hash != ''" --repo data/.aim --json
+```
+
+## AimQL Scoping
+
+Use AimQL in the expression argument. Common run scopes:
+
+```text
+run.hash != ''
+run.hash == 'eca37394'
+run.experiment == 'cloud-segmentation'
+run.name == 'resnet-ft-0420'
+```
+
+Metric expressions combine run fields with metric fields:
+
+```text
+(run.experiment == 'cloud-segmentation') and metric.name == 'loss'
+(run.hash == 'eca37394') and metric.name != ''
+```
+
+Short run hashes are expanded by `aimx` where supported.
+
+## Params
+
+Use params first to understand the design of each run.
+
+```bash
+aimx query params "<run-scope>" --repo <repo> --json
+aimx query params "<run-scope>" --repo <repo> --json --param hparam.lr --param model
+```
+
+JSON shape:
+
+```json
+{
+  "target": "params",
+  "repo": "data",
+  "expression": "run.hash != ''",
+  "runs_count": 2,
+  "param_keys": ["hparam.lr", "model"],
+  "runs": [
+    {
+      "hash": "full-run-hash",
+      "experiment": "experiment-name",
+      "name": "run-name",
+      "params": {"hparam.lr": 0.0001, "model": "ResNet"},
+      "missing_params": []
+    }
+  ]
+}
+```
+
+Use `missing_params` as a confidence signal when comparing runs.
+
+## Metric Summaries
+
+Use metric summaries to rank runs cheaply.
+
+```bash
+aimx query metrics "<metric-expr>" --repo <repo> --json
+aimx query metrics "<metric-expr>" --repo <repo> --json --steps 100:500
+aimx query metrics "<metric-expr>" --repo <repo> --json --epochs 1:10
+```
+
+JSON shape:
+
+```json
+{
+  "target": "metrics",
+  "repo": "data",
+  "expression": "metric.name == 'loss'",
+  "runs_count": 1,
+  "metrics_count": 2,
+  "runs": [
+    {
+      "hash": "full-run-hash",
+      "experiment": "experiment-name",
+      "name": "run-name",
+      "metrics": [
+        {
+          "name": "loss",
+          "context": {"subset": "val"},
+          "steps": 110,
+          "last": {"value": 0.43, "step": 110},
+          "min": {"value": 0.32, "step": 60},
+          "max": {"value": 0.48, "step": 107}
+        }
+      ]
+    }
+  ]
+}
+```
+
+Use `context` to distinguish train, val, test, dataset split, seed, or other
+metric dimensions.
+
+## Traces
+
+Use traces when a summary hides important behavior such as late overfitting,
+instability, divergence, or plateauing.
+
+```bash
+aimx trace "<metric-expr>" --repo <repo> --json
+aimx trace "<metric-expr>" --repo <repo> --json --tail 50
+aimx trace "<metric-expr>" --repo <repo> --json --steps 100:500 --every 5
+```
+
+JSON shape:
+
+```json
+[
+  {
+    "run": {
+      "hash": "full-run-hash",
+      "experiment": "experiment-name",
+      "name": "run-name"
+    },
+    "metric": "loss",
+    "context": {"subset": "val"},
+    "count": 50,
+    "steps": [1, 2, 3],
+    "epochs": [1.0, 2.0, 3.0],
+    "values": [0.9, 0.7, 0.5]
+  }
+]
+```
+
+If no metrics match, current `aimx trace --json` may print a text message
+instead of JSON. Treat that as no trace evidence rather than a parsing failure.
+
+## Images
+
+Use images for qualitative checks such as sample predictions, masks, generated
+outputs, confusion examples, or visual regressions.
+
+```bash
+aimx query images "images" --repo <repo> --json
+aimx query images "images" --repo <repo> --json --head 20
+aimx query images "images" --repo <repo> --json --epochs 10:50
+```
+
+JSON shape:
+
+```json
+{
+  "target": "images",
+  "repo": "data",
+  "expression": "images",
+  "count": 1,
+  "rows": [
+    {
+      "run_hash": "full-run-hash",
+      "experiment": "experiment-name",
+      "name": "example",
+      "context": {"epoch": 10, "subset": "val"}
+    }
+  ]
+}
+```
+
+Use image metadata in automated logs; render images manually only when the user
+asks for visual inspection.
+
+## log_experiment Evidence Fields
+
+Recommended fields for autoresearch output:
+
+- `run_scope`: AimQL expression and repo path used for evidence.
+- `params`: selected hyperparameters and model identifiers per run.
+- `metric_summary`: objective metric summaries per run and context.
+- `trace_evidence`: sampled value arrays for decisive metrics.
+- `image_evidence`: image row counts and representative contexts.
+- `ranking`: best run per objective, objective direction, and tie-breakers.
+- `regressions`: runs worse than baseline, incomplete runs, missing metrics, or
+  suspicious curves.
+- `next_experiments`: concrete parameter changes grounded in the evidence.
+
+Keep conclusions tied to the collected data. When metric direction is unknown,
+state the assumption before ranking.

aimx-0.3.2/.agents/skills/aimx/scripts/collect_experiment_snapshot.py

@@ -0,0 +1,217 @@
+#!/usr/bin/env python3
+"""Collect a read-only aimx experiment snapshot for autoresearch logs."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import shlex
+import subprocess
+import sys
+from dataclasses import dataclass
+from typing import Any
+
+
+@dataclass
+class CommandPayload:
+    argv: list[str]
+    ok: bool
+    data: Any | None = None
+    message: str | None = None
+    error: str | None = None
+
+    def as_dict(self) -> dict[str, Any]:
+        payload: dict[str, Any] = {"argv": self.argv, "ok": self.ok}
+        if self.data is not None:
+            payload["data"] = self.data
+        if self.message:
+            payload["message"] = self.message
+        if self.error:
+            payload["error"] = self.error
+        return payload
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Collect params, metric summaries, optional traces, and image metadata with aimx."
+    )
+    parser.add_argument("--repo", default=".", help="Aim repo root or .aim metadata path.")
+    parser.add_argument(
+        "--base-expr",
+        default="run.hash != ''",
+        help="AimQL run scope used for params and generated metric expressions.",
+    )
+    parser.add_argument(
+        "--metric",
+        action="append",
+        default=[],
+        help="Metric name to summarize; repeatable. Defaults to all metrics when no metric expression is supplied.",
+    )
+    parser.add_argument(
+        "--metric-expr",
+        action="append",
+        default=[],
+        help="Full AimQL metric expression to summarize; repeatable.",
+    )
+    parser.add_argument(
+        "--trace-metric",
+        action="append",
+        default=[],
+        help="Metric name to collect trace samples for; repeatable.",
+    )
+    parser.add_argument(
+        "--trace-expr",
+        action="append",
+        default=[],
+        help="Full AimQL trace expression; repeatable.",
+    )
+    parser.add_argument(
+        "--trace-tail",
+        type=int,
+        default=50,
+        help="Tail sample count for trace commands. Use 0 to disable tail sampling.",
+    )
+    parser.add_argument(
+        "--param",
+        action="append",
+        default=[],
+        help="Parameter key to select; repeatable. Omit to let aimx discover params.",
+    )
+    parser.add_argument(
+        "--include-images",
+        action="store_true",
+        help="Also collect image metadata with aimx query images.",
+    )
+    parser.add_argument(
+        "--image-expr",
+        default="images",
+        help="AimQL image expression used when --include-images is set.",
+    )
+    parser.add_argument(
+        "--image-head",
+        type=int,
+        default=20,
+        help="Limit image metadata rows when --include-images is set.",
+    )
+    parser.add_argument(
+        "--aimx",
+        default=f"{sys.executable} -m aimx",
+        help='Launcher for aimx, for example "aimx", "uv run aimx", or "python -m aimx".',
+    )
+    parser.add_argument("--timeout", type=int, default=60, help="Per-command timeout in seconds.")
+    parser.add_argument("--pretty", action="store_true", help="Pretty-print JSON output.")
+    return parser.parse_args()
+
+
+def aimql_string(value: str) -> str:
+    return "'" + value.replace("\\", "\\\\").replace("'", "\\'") + "'"
+
+
+def metric_expression(base_expr: str, metric_name: str) -> str:
+    return f"({base_expr}) and metric.name == {aimql_string(metric_name)}"
+
+
+def run_json(base_cmd: list[str], args: list[str], timeout: int) -> CommandPayload:
+    argv = [*base_cmd, *args]
+    try:
+        completed = subprocess.run(
+            argv,
+            check=False,
+            capture_output=True,
+            text=True,
+            timeout=timeout,
+        )
+    except FileNotFoundError as exc:
+        return CommandPayload(argv=argv, ok=False, error=str(exc))
+    except subprocess.TimeoutExpired as exc:
+        return CommandPayload(argv=argv, ok=False, error=f"Timed out after {timeout}s: {exc}")
+
+    stdout = completed.stdout.strip()
+    stderr = completed.stderr.strip()
+    if completed.returncode != 0:
+        return CommandPayload(
+            argv=argv,
+            ok=False,
+            message=stdout or None,
+            error=stderr or f"Command exited with status {completed.returncode}",
+        )
+
+    if not stdout:
+        return CommandPayload(argv=argv, ok=True, data=None)
+
+    try:
+        return CommandPayload(argv=argv, ok=True, data=json.loads(stdout))
+    except json.JSONDecodeError:
+        return CommandPayload(argv=argv, ok=True, message=stdout)
+
+
+def collect_snapshot(args: argparse.Namespace) -> tuple[dict[str, Any], bool]:
+    base_cmd = shlex.split(args.aimx)
+    metric_exprs = list(args.metric_expr)
+    metric_exprs.extend(metric_expression(args.base_expr, name) for name in args.metric)
+    if not metric_exprs:
+        metric_exprs.append(f"({args.base_expr}) and metric.name != ''")
+
+    trace_exprs = list(args.trace_expr)
+    trace_exprs.extend(metric_expression(args.base_expr, name) for name in args.trace_metric)
+
+    params_args = ["query", "params", args.base_expr, "--repo", args.repo, "--json"]
+    for key in args.param:
+        params_args.extend(["--param", key])
+
+    params = run_json(base_cmd, params_args, args.timeout)
+
+    metrics = [
+        run_json(base_cmd, ["query", "metrics", expr, "--repo", args.repo, "--json"], args.timeout)
+        for expr in metric_exprs
+    ]
+
+    trace_common = ["--repo", args.repo, "--json"]
+    if args.trace_tail > 0:
+        trace_common.extend(["--tail", str(args.trace_tail)])
+    traces = [
+        run_json(base_cmd, ["trace", expr, *trace_common], args.timeout)
+        for expr in trace_exprs
+    ]
+
+    images = None
+    if args.include_images:
+        image_args = [
+            "query",
+            "images",
+            args.image_expr,
+            "--repo",
+            args.repo,
+            "--json",
+            "--head",
+            str(args.image_head),
+        ]
+        images = run_json(base_cmd, image_args, args.timeout)
+
+    snapshot: dict[str, Any] = {
+        "repo": args.repo,
+        "base_expr": args.base_expr,
+        "read_only": True,
+        "params": params.as_dict(),
+        "metrics": [item.as_dict() for item in metrics],
+        "traces": [item.as_dict() for item in traces],
+        "images": images.as_dict() if images else None,
+    }
+
+    failures = [params, *metrics, *traces]
+    if images:
+        failures.append(images)
+    ok = all(item.ok for item in failures)
+    return snapshot, ok
+
+
+def main() -> int:
+    args = parse_args()
+    snapshot, ok = collect_snapshot(args)
+    indent = 2 if args.pretty else None
+    print(json.dumps(snapshot, indent=indent, sort_keys=args.pretty))
+    return 0 if ok else 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

{aimx-0.3.0 → aimx-0.3.2}/.gitignore

@@ -137,6 +137,8 @@ celerybeat.pid
 # Environments
 .env
 .envrc
+.env*
+.venv/
 .venv
 env/
 venv/
@@ -188,6 +190,12 @@ cython_debug/
 #  you could uncomment the following to ignore the entire vscode folder
 # .vscode/
 
+# OS / editor
+.DS_Store
+Thumbs.db
+*.tmp
+*.swp
+
 # Ruff stuff:
 .ruff_cache/
 
@@ -210,4 +218,4 @@ __marimo__/
 .specify/
 
 # aim
-data/
+data/

{aimx-0.3.0 → aimx-0.3.2}/AGENTS.md

@@ -77,6 +77,10 @@ Why `3.12`:
 - Python 3.12 for development, runtime support `>=3.10,<3.13` + Python standard library, native Aim CLI (external runtime prerequisite for delegated commands), pytest for test automation (001-aim-command-passthrough)
 - Python 3.12 for development, runtime support `>=3.10,<3.13` + Python standard library, Aim SDK from the dev dependency group for local development and tests (001-aim-command-passthrough)
 - Existing local Aim repositories on disk, including repo roots that contain a `.aim` metadata directory (001-aim-command-passthrough)
+- Python 3.12 for development, runtime support `>=3.10,<3.13` + `rich>=13.7`, `textual-image>=0.12.0` (already declared in (003-query-images-terminal-render)
+- Existing local Aim repositories (read-only). Image bytes are read (003-query-images-terminal-render)
+- Python 3.12 for development, runtime support `>=3.10,<3.13` + Python standard library, `numpy>=1.24`, `rich>=13.7`, `textual-image>=0.12.0`, existing Aim SDK usage for owned query commands; no new dependency planned (004-run-params-query)
+- Existing local Aim repositories on disk (read-only); run params are read from Aim run metadata attributes under `.aim` (004-run-params-query)
 
 ## Recent Changes
 - 001-aim-command-passthrough: Added Python 3.12 for development, runtime support `>=3.10,<3.13` + Python standard library, native Aim CLI (external runtime prerequisite for delegated commands), pytest for test automation