@inneranimalmedia/agentsam-sdk 1.7.0 → 1.8.0

package/python/docs/tooling.md

@@ -0,0 +1,67 @@
+# Host tooling for agentsam-sdk
+
+Python package is **stdlib-only** (`pip install -e .` adds no third-party deps).
+These **host** tools are expected on the operator/CI machine:
+
+| Tool | Required for | Install (macOS) | Check |
+|------|--------------|-----------------|-------|
+| **Python ≥ 3.10** | all modules + unittest | Homebrew / pyenv | `python3 --version` |
+| **jq** | filtering inventory / audit JSON on the CLI | `brew install jq` | `jq --version` |
+| **wrangler** (Cloudflare CLI) | `data.d1_bloat`, `data.agentsam_walk` via `D1Adapter` | `npm i -g wrangler` (or repo local) | `wrangler --version` |
+| **CLOUDFLARE_API_TOKEN** (env) | remote D1 execute | dashboard API token | `test -n "$CLOUDFLARE_API_TOKEN"` |
+| **AGENTSAM_D1_DB_NAME** (env) | D1 tools when `--db` omitted | set in shell / `.env.cloudflare` loader | — |
+
+Optional:
+
+| Tool | Why |
+|------|-----|
+| `git` | repo-root detection in some callers |
+| `node` / `npm` | IAM `npm run inventory:repo-size*` shims, wrangler install |
+
+## Quick host check
+
+From `agentsam-sdk/`:
+
+```bash
+./scripts/check-host-tooling.sh
+# or:
+./scripts/check-host-tooling.sh --require-d1   # also needs wrangler + token + db name
+```
+
+## jq recipes (inventory)
+
+```bash
+# category row for docs
+agentsam repository inventory --repo-root .. --format json \
+  | jq '.data.categories[] | select(.id=="docs")'
+
+# top 10 largest files
+agentsam repository inventory --repo-root .. --format json \
+  | jq '.data.largest_files[:10]'
+
+# totals only
+jq '.totals' /tmp/inv/repository-inventory.json
+
+# extensions over 100 files
+jq '.by_extension | to_entries | map(select(.value > 100))' \
+  /tmp/inv/repository-inventory.json
+```
+
+## jq recipes (per-file source bloat)
+
+```bash
+agentsam repository scan-bloat --root src --min-kb 10 --top 50 --json-envelope \
+  | jq -r '.files[] | "\(.size_kb)KB\t\(.path)"'
+```
+
+## jq recipes (D1 audits)
+
+```bash
+agentsam data d1-bloat --db "$AGENTSAM_D1_DB_NAME" --quick --format json \
+  --output-dir /tmp/d1-bloat
+jq '.data.tables[:10] | .[] | {name, rows}' /tmp/d1-bloat/*.json 2>/dev/null \
+  || jq '.' /tmp/d1-bloat/receipt-*.json | head
+```
+
+Exact JSON shapes vary by tool — prefer reading the written `*.json` under
+`--output-dir` over scraping stdout when chaining in CI.

package/python/protocol/README.md

@@ -0,0 +1,51 @@
+# agentsam-sdk protocol (LOCKED)
+
+Every Agent Sam **tool / feature SDK surface** is dual-homed. There is no
+“Python audits only in IAM” vs “JS CLI only on npm” split for productized
+tooling.
+
+| Home | Path | Role |
+|------|------|------|
+| **Main platform repo** | `inneranimalmedia/agentsam-sdk/` | Source of truth while building; ships with platform deploy |
+| **Published SDK repo** | `github.com/SamPrimeaux/agentsam-sdk` → npm `@inneranimalmedia/agentsam-sdk` | Same contract + modules for external / CLI consumers |
+
+## Rules
+
+1. **Author once, land twice.** New `agentsam_sdk.*` tools (data, repository,
+   readiness, history, code-intel, …) land in the monorepo package **and** are
+   mirrored into the published `agentsam-sdk` repo in the same change set / PR
+   pair. No silent one-sided land.
+2. **Contract is shared.** `runtime/contract` (`ToolInput` / `ToolResult` /
+   receipts, unixepoch, no hardcoded identity) is the protocol. Language may
+   differ (Python stdlib vs JS), but CLI verbs and JSON shapes must stay
+   aligned — see `protocol/dual-repo-sync.md`.
+3. **No drift.** Publishing npm without updating the monorepo copy (or the
+   reverse) is a protocol violation. Gate: version bump + changelog note that
+   lists mirrored paths.
+4. **Ship is two-sided.**
+   - Platform: Mac `npm run deploy:full` / `deploy:fast` (or GCP `ship:remote`)
+     when the monorepo copy or Worker wiring changed.
+   - SDK: **manual** npm publish / version bump on `agentsam-sdk` after the
+     mirror lands (operator-owned — do not assume CI auto-publishes).
+5. **Inspiration ≠ copy-paste secrets.** In-app surfaces (e.g. `src/core/code-indexer.js`,
+   AST-RAG Phase 1/2) are the reference implementations to port into portable
+   SDK modules — strip platform-only bindings, keep adapters for D1 / git /
+   Hyperdrive.
+
+## Layout (both homes)
+
+```
+agentsam-sdk/
+  protocol/           ← this law
+  python/agentsam_sdk ← Python portable tools (stdlib-first audits / inventory)
+  src/                ← JS CLI / scaffold (published npm entry)
+  packages/           ← optional workspace packages (not always in npm tarball)
+    agentsam-shell-kit/  ← @inneranimalmedia/agentsam-shell-kit (private until ready)
+  docs/gaps.md        ← port status vs IAM scripts + in-app indexers
+```
+
+**npm identity (LOCKED):** root publishable package is always
+`@inneranimalmedia/agentsam-sdk`. Do not overwrite root `package.json` with a
+workspace kit name. Fold UI kits under `packages/*`.
+
+Exact folder names may evolve; the dual-home + root-identity rules do not.

package/python/protocol/dual-repo-sync.md

@@ -0,0 +1,35 @@
+# Dual-repo sync checklist
+
+Use this every time you add or change an `agentsam_sdk` tool/feature.
+
+## Before coding
+
+- [ ] Name the module (`agentsam_sdk.<domain>.<tool>`) and CLI verb
+- [ ] Confirm it belongs in SDK (portable) vs platform-only Worker hot path
+- [ ] Note in-app inspiration path (if any), e.g. `src/core/code-indexer.js`
+
+## Land
+
+- [ ] Implement + tests in **`inneranimalmedia/agentsam-sdk/`**
+- [ ] Mirror the same module/CLI/docs into **`agentsam-sdk`** (npm repo)
+- [ ] Update `docs/gaps.md` (or equivalent) in **both** trees
+- [ ] Update `protocol/` only in monorepo if law changed; copy README blurb to npm
+
+## Publish / deploy (operator)
+
+- [ ] Platform: commit/push IAM → deploy by host (`deploy:full` / `ship:remote`)
+- [ ] SDK: bump `package.json` version in npm repo → `npm publish` (manual)
+- [ ] Record versions next to each other (PR description or receipt):  
+      `iam@<sha>` ↔ `@inneranimalmedia/agentsam-sdk@<semver>`
+
+## Drift signals (fail the PR)
+
+- Module exists only in one repo
+- CLI flag / JSON field renamed on one side only
+- README claims “this is not the npm package” / “audits don’t belong here”
+- npm publish without a same-day IAM mirror commit (or documented lag ticket)
+
+## jq / host tools
+
+Host tooling (`jq`, `wrangler`, Python ≥3.10) is documented in
+`docs/tooling.md`. Keep that file mirrored when recipes change.

package/python/pyproject.toml

@@ -0,0 +1,16 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "agentsam-sdk"
+version = "0.1.0"
+description = "Stdlib-only D1/repository audit toolkit for inneranimalmedia (Agent Sam)."
+requires-python = ">=3.10"
+dependencies = []
+
+[project.scripts]
+agentsam = "agentsam_sdk.cli:main"
+
+[tool.setuptools.packages.find]
+include = ["agentsam_sdk*"]

package/python/scripts/check-host-tooling.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# Verify host tools for agentsam-sdk (stdlib Python + jq + optional wrangler/D1).
+set -euo pipefail
+
+REQUIRE_D1=0
+for arg in "$@"; do
+  case "$arg" in
+    --require-d1) REQUIRE_D1=1 ;;
+    -h|--help)
+      echo "Usage: $0 [--require-d1]"
+      exit 0
+      ;;
+  esac
+done
+
+fail=0
+ok() { printf '  OK  %s\n' "$1"; }
+bad() { printf '  MISSING  %s\n' "$1"; fail=1; }
+
+echo "agentsam-sdk host tooling"
+
+if command -v python3 >/dev/null 2>&1; then
+  pyv=$(python3 -c 'import sys; print("%d.%d"%sys.version_info[:2])')
+  # shellcheck disable=SC2072
+  if python3 -c 'import sys; raise SystemExit(0 if sys.version_info >= (3,10) else 1)'; then
+    ok "python3 ($pyv)"
+  else
+    bad "python3 >= 3.10 (found $pyv)"
+  fi
+else
+  bad "python3"
+fi
+
+if command -v jq >/dev/null 2>&1; then
+  ok "jq ($(jq --version 2>&1))"
+else
+  bad "jq (brew install jq) — needed for JSON filters in docs/tooling.md"
+fi
+
+if [[ "$REQUIRE_D1" -eq 1 ]]; then
+  if command -v wrangler >/dev/null 2>&1 || command -v npx >/dev/null 2>&1; then
+    ok "wrangler/npx available"
+  else
+    bad "wrangler or npx"
+  fi
+  if [[ -n "${CLOUDFLARE_API_TOKEN:-}" ]]; then
+    ok "CLOUDFLARE_API_TOKEN set"
+  else
+    bad "CLOUDFLARE_API_TOKEN"
+  fi
+  if [[ -n "${AGENTSAM_D1_DB_NAME:-}" ]]; then
+    ok "AGENTSAM_D1_DB_NAME=${AGENTSAM_D1_DB_NAME}"
+  else
+    bad "AGENTSAM_D1_DB_NAME (or pass --db on each command)"
+  fi
+else
+  echo "  (skip D1 env — pass --require-d1 to enforce)"
+fi
+
+if [[ "$fail" -ne 0 ]]; then
+  echo "FAIL: install missing tools (see docs/tooling.md)"
+  exit 1
+fi
+echo "PASS"
+exit 0

package/python/tests/fixtures/sample_tables.json

@@ -0,0 +1,17 @@
+{
+  "agentsam_tool_call_log": {
+    "columns": [["id","INTEGER"],["input_json","TEXT"],["output_json","TEXT"],["created_at","TEXT"]],
+    "row_count": 250000,
+    "text_lengths": {"input_json": 6291456, "output_json": 12582912}
+  },
+  "agentsam_memory": {
+    "columns": [["id","INTEGER"],["key","TEXT"],["value","TEXT"],["updated_at","TEXT"]],
+    "row_count": 40,
+    "text_lengths": {"value": 12000}
+  },
+  "cms_pages": {
+    "columns": [["id","INTEGER"],["slug","TEXT"],["title","TEXT"]],
+    "row_count": 0,
+    "text_lengths": {}
+  }
+}

package/python/tests/fixtures.py

@@ -0,0 +1,95 @@
+"""Stub D1 shape shared by data/* tests -- no live D1 required.
+
+Mirrors a small slice of the real schema (a couple agentsam_* tables plus
+one bloated cms_ table) so d1_bloat / agentsam_walk logic can be exercised
+without wrangler or a network call.
+"""
+from __future__ import annotations
+
+FAKE_TABLES = [
+    "agentsam_tool_call_log",
+    "agentsam_workflow_runs",
+    "cms_pages",
+]
+
+FAKE_COLUMNS: dict[str, list[tuple[str, str]]] = {
+    "agentsam_tool_call_log": [
+        ("id", "TEXT"),
+        ("tool_name", "TEXT"),
+        ("input_json", "TEXT"),
+        ("output_json", "TEXT"),
+        ("created_at", "INTEGER"),
+    ],
+    "agentsam_workflow_runs": [
+        ("id", "TEXT"),
+        ("step_results_json", "TEXT"),
+        ("status", "TEXT"),
+        ("updated_at", "INTEGER"),
+    ],
+    "cms_pages": [
+        ("id", "TEXT"),
+        ("body", "TEXT"),
+        ("slug", "TEXT"),
+    ],
+}
+
+FAKE_ROW_COUNTS = {
+    "agentsam_tool_call_log": 150_000,
+    "agentsam_workflow_runs": 40,
+    "cms_pages": 12,
+}
+
+
+class FakeD1Adapter:
+    """Drop-in stand-in for agentsam_sdk.data.d1_adapter.D1Adapter.
+
+    Constructed the same way tests need it (via a classmethod matching
+    from_env's call signature) and implements only the methods d1_bloat.py /
+    agentsam_walk.py actually call.
+    """
+
+    def __init__(self, db_name: str = "fake-db"):
+        self.db_name = db_name
+
+    @classmethod
+    def from_env(cls, db_name=None, wrangler_config=None, repo_root=None):
+        return cls(db_name=db_name or "fake-db")
+
+    def list_tables(self, like=None):
+        if not like:
+            return list(FAKE_TABLES)
+        prefix = like.rstrip("%")
+        return [t for t in FAKE_TABLES if t.startswith(prefix)]
+
+    def database_size(self):
+        return "12.3 MB"
+
+    def table_columns(self, table):
+        return list(FAKE_COLUMNS.get(table, []))
+
+    def table_indexes(self, table):
+        return []
+
+    def foreign_keys(self, table):
+        return []
+
+    def row_count(self, table):
+        return FAKE_ROW_COUNTS.get(table, 0)
+
+    def query(self, sql):
+        # Only d1_bloat's SUM(LENGTH(...)) aggregate query shape is needed.
+        rc = FAKE_ROW_COUNTS.get(self._table_from_sql(sql), 0)
+        row = {"rc": rc}
+        for col in FAKE_COLUMNS.get(self._table_from_sql(sql), []):
+            name = col[0]
+            if name in sql:
+                row[name] = 200 * rc
+                row[f"m_{name}"] = 200
+        return [row]
+
+    @staticmethod
+    def _table_from_sql(sql: str) -> str:
+        for t in FAKE_TABLES:
+            if f'"{t}"' in sql:
+                return t
+        return ""

package/python/tests/test_agentsam_walk.py

@@ -0,0 +1,31 @@
+"""Unit tests for data.agentsam_walk -- pure-function logic, no live D1."""
+import unittest
+
+from agentsam_sdk.data.agentsam_walk import _capability_for, TableWalk, _render_markdown
+
+
+class TestCapabilityHeuristic(unittest.TestCase):
+    def test_tool_chain_maps_to_tools_commands_mcp(self):
+        self.assertEqual(_capability_for("agentsam_tool_chain"), "tools_commands_mcp")
+
+    def test_workflow_runs_maps_to_workflow_dag(self):
+        self.assertEqual(_capability_for("agentsam_workflow_runs"), "workflow_dag")
+
+    def test_unknown_table_is_uncategorized(self):
+        self.assertEqual(_capability_for("zzz_totally_unmatched_xyz"), "uncategorized")
+
+
+class TestRenderMarkdown(unittest.TestCase):
+    def test_groups_by_capability(self):
+        walks = [
+            TableWalk(name="agentsam_tool_chain", capability="tools_commands_mcp", row_count=10),
+            TableWalk(name="agentsam_memory", capability="memory_rag", row_count=5),
+        ]
+        md = _render_markdown(walks, "agentsam_")
+        self.assertIn("tools_commands_mcp", md)
+        self.assertIn("memory_rag", md)
+        self.assertIn("agentsam_tool_chain", md)
+
+
+if __name__ == "__main__":
+    unittest.main()

package/python/tests/test_contract.py

@@ -0,0 +1,32 @@
+"""Unit tests for runtime.contract -- receipts, no I/O beyond a tmp dir."""
+import tempfile
+import unittest
+from pathlib import Path
+
+from agentsam_sdk.runtime.contract import ToolInput, ToolResult, write_receipt, start_timer
+
+
+class TestReceipts(unittest.TestCase):
+    def test_write_receipt_creates_file(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            ti = ToolInput(output_dir=tmp)
+            started = start_timer()
+            result = ToolResult(
+                ok=True, tool="data.d1_bloat", mode="quick", request_id=ti.request_id,
+                started_at=started, finished_at=start_timer(), summary="test run",
+            )
+            path = write_receipt(result, ti.output_path())
+            self.assertIsNotNone(path)
+            self.assertTrue(Path(path).exists())
+
+    def test_write_receipt_noop_without_output_dir(self):
+        ti = ToolInput()
+        result = ToolResult(
+            ok=True, tool="data.d1_bloat", mode="quick", request_id=ti.request_id,
+            started_at=0.0, finished_at=1.0, summary="test",
+        )
+        self.assertIsNone(write_receipt(result, ti.output_path()))
+
+
+if __name__ == "__main__":
+    unittest.main()

package/python/tests/test_d1_bloat.py

@@ -0,0 +1,57 @@
+"""Unit tests for data.d1_bloat -- pure-function parsing, no live D1/network."""
+import unittest
+
+from agentsam_sdk.data.d1_bloat import (
+    ColStat, TableStat, _pick_bloat_columns, _flag_suspicious, _render_markdown, _fmt_bytes,
+)
+
+
+class TestBloatColumnPicking(unittest.TestCase):
+    def test_picks_json_and_body_columns(self):
+        cols = [("id", "INTEGER"), ("input_json", "TEXT"), ("output_json", "TEXT"), ("created_at", "TEXT")]
+        picked = _pick_bloat_columns(cols)
+        self.assertIn("input_json", picked)
+        self.assertIn("output_json", picked)
+        self.assertNotIn("id", picked)
+        self.assertNotIn("created_at", picked)
+
+    def test_skips_id_and_fk_like_columns(self):
+        cols = [("tenant_id", "TEXT"), ("workspace_id", "TEXT"), ("metadata", "TEXT")]
+        picked = _pick_bloat_columns(cols)
+        self.assertNotIn("tenant_id", picked)
+        self.assertNotIn("workspace_id", picked)
+        self.assertIn("metadata", picked)
+
+
+class TestFlagging(unittest.TestCase):
+    def test_flags_large_table_high_severity(self):
+        big = TableStat(name="agentsam_tool_call_log", row_count=250_000, text_bytes=18_874_368,
+                         est_bytes=18_874_368, columns=[ColStat(name="output_json", bytes=12_582_912)])
+        small = TableStat(name="cms_pages", row_count=0, text_bytes=0, est_bytes=0)
+        flags = _flag_suspicious([big, small])
+        names = {f["table"] for f in flags}
+        self.assertIn("agentsam_tool_call_log", names)
+        big_flag = next(f for f in flags if f["table"] == "agentsam_tool_call_log")
+        self.assertEqual(big_flag["severity"], "high")
+
+    def test_empty_table_not_flagged(self):
+        small = TableStat(name="cms_pages", row_count=0, text_bytes=0, est_bytes=0)
+        flags = _flag_suspicious([small])
+        self.assertEqual(flags, [])
+
+
+class TestFormatting(unittest.TestCase):
+    def test_fmt_bytes_scales(self):
+        self.assertEqual(_fmt_bytes(500), "500 B")
+        self.assertEqual(_fmt_bytes(2048), "2.0 KB")
+        self.assertEqual(_fmt_bytes(5 * 1024 * 1024), "5.00 MB")
+
+    def test_render_markdown_includes_table_names(self):
+        stats = [TableStat(name="agentsam_memory", row_count=40, text_bytes=12000, est_bytes=12000)]
+        md = _render_markdown(stats, "1.2 MB", "quick", 10, 1)
+        self.assertIn("agentsam_memory", md)
+        self.assertIn("D1 bloat audit", md)
+
+
+if __name__ == "__main__":
+    unittest.main()

package/python/tests/test_repository_inventory.py

@@ -0,0 +1,53 @@
+import unittest
+from tempfile import TemporaryDirectory
+from pathlib import Path
+
+from agentsam_sdk.repository import inventory
+from agentsam_sdk.runtime.contract import ToolInput
+
+
+class TestRepositoryInventory(unittest.TestCase):
+    def test_counts_files_by_category_size_and_extension(self):
+        with TemporaryDirectory() as repo, TemporaryDirectory() as out:
+            root = Path(repo)
+            (root / "src").mkdir()
+            (root / "src" / "a.py").write_text("x" * 100)
+            (root / "src" / "b.py").write_text("y" * 50)
+            (root / "docs").mkdir()
+            (root / "docs" / "c.md").write_text("z" * 20)
+            (root / "node_modules").mkdir()
+            (root / "node_modules" / "skip.js").write_text("skip me" * 1000)
+
+            ti = ToolInput(
+                params={"repo_root": str(root), "top": 5, "by_ext": True},
+                output_dir=out,
+            )
+            result = inventory.run(ti)
+
+            self.assertTrue(result.ok)
+            self.assertEqual(result.data["file_total"], 3)  # node_modules excluded
+            self.assertEqual(result.data["by_extension"][".py"], 2)
+            self.assertEqual(result.data["by_top_level_dir"]["src"], 2)
+            ids = {c["id"] for c in result.data["categories"]}
+            self.assertIn("worker_src", ids)
+            self.assertIn("docs", ids)
+            self.assertGreater(result.data["totals"]["bytes"], 0)
+            self.assertTrue((Path(out) / "repository-inventory.json").exists())
+            self.assertTrue((Path(out) / "repository-inventory.md").exists())
+            # largest list prefers bigger src file
+            self.assertEqual(result.data["largest_files"][0]["path"], "src/a.py")
+
+    def test_missing_repo_root_is_reported_not_raised(self):
+        ti = ToolInput(params={"repo_root": "/definitely/does/not/exist/xyz"})
+        result = inventory.run(ti)
+        self.assertFalse(result.ok)
+        self.assertEqual(result.error, "repo_root_not_found")
+
+    def test_categorize_helpers(self):
+        self.assertEqual(inventory.categorize(Path("dashboard/App.tsx")), "dashboard")
+        self.assertEqual(inventory.categorize(Path("README.md")), "root_misc")
+        self.assertIn("KiB", inventory.human_bytes(2048))
+
+
+if __name__ == "__main__":
+    unittest.main()

package/python/tests/test_scan_bloat.py

@@ -0,0 +1,31 @@
+"""Unit tests for repository.scan_bloat — no network."""
+from __future__ import annotations
+
+from pathlib import Path
+
+from agentsam_sdk.repository import scan_bloat
+from agentsam_sdk.runtime.contract import ToolInput
+
+
+def test_scan_ranks_by_size(tmp_path: Path) -> None:
+    (tmp_path / "small.js").write_text("x=1\n", encoding="utf-8")
+    (tmp_path / "big.js").write_text("x=1\n" * 500, encoding="utf-8")
+    (tmp_path / "skip.py").write_text("print(1)\n" * 200, encoding="utf-8")
+    rows = scan_bloat.scan(tmp_path)
+    assert len(rows) == 2
+    assert rows[0]["path"] == "big.js"
+    assert rows[0]["size_bytes"] > rows[1]["size_bytes"]
+
+
+def test_run_json_envelope(tmp_path: Path) -> None:
+    (tmp_path / "a.ts").write_text("// " + ("n" * 2000) + "\n", encoding="utf-8")
+    result = scan_bloat.run(
+        ToolInput(
+            mode="read-only",
+            params={"root": str(tmp_path), "top": 5, "min_kb": 0},
+        )
+    )
+    assert result.ok
+    assert result.tool == "repository.scan_bloat"
+    assert result.data["file_count"] == 1
+    assert result.data["files"][0]["path"] == "a.ts"