route66 0.1.1__tar.gz

route66-0.1.1/.gitignore

@@ -0,0 +1,58 @@
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+*.pyc
+*.pdb
+
+# Virtual environment (uv)
+.venv/
+venv/
+env/
+
+# Distribution / packaging
+dist/
+build/
+*.egg-info/
+*.egg
+MANIFEST
+
+# Testing
+.pytest_cache/
+.coverage
+coverage.xml
+htmlcov/
+.tox/
+
+# Linting (ruff)
+.ruff_cache/
+
+# Type checking
+.mypy_cache/
+.pytype/
+
+# Jupyter
+.ipynb_checkpoints/
+*.ipynb
+
+# Environment variables
+.env
+.env.*
+!.env.example
+
+# OS
+.DS_Store
+Thumbs.db
+desktop.ini
+
+# Editors
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# Logs
+*.log
+logs/

route66-0.1.1/PKG-INFO

@@ -0,0 +1,104 @@
+Metadata-Version: 2.4
+Name: route66
+Version: 0.1.1
+Summary: route66 Python SDK
+Author: route66 maintainers
+License: Proprietary
+Keywords: route66,sdk
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: twine>=5.1; extra == 'dev'
+Provides-Extra: semantic
+Requires-Dist: sentence-transformers>=3.0; extra == 'semantic'
+Description-Content-Type: text/markdown
+
+# route66
+
+**Intelligent Tool Router for LLM Agents** — a catalog-in library that,
+given a query and a large tool catalog, returns only the relevant tools,
+deduplicated, in an ordered execution plan, with policy-aware defaults.
+
+Built against the **`router-testkit`** hackathon grading contract
+(single-tool routing, ordered multi-tool plans, near-duplicate
+resolution, capability-based fallback widening, deprecation policy,
+out-of-scope refusal, and clarify-on-vague-destructive).
+
+## Layout
+
+```
+route66/
+├── tool_router/          # The POC library (facade + retrievers + dedup)
+├── testkit_adapter.py    # Bridges tool_router to the harness contract
+├── demo.py               # Toy 60-tool catalog demo of the POC
+├── metrics.py            # Recall/precision/token-savings eval on the toy set
+├── router-testkit/       # Hackathon input kit — DO NOT MODIFY
+│   ├── catalog/          #   64 mock tools · 13 clusters · dup/version metadata
+│   ├── test_cases/       #   29 scored cases across 11 categories
+│   ├── registry_guardrail/#  Intake validator fixtures
+│   └── harness/          #   run_benchmark.py (scorer) + baseline_router.py
+├── bench/                # Additional benchmarks (baseline, mutations, guardrail)
+├── specs/                # Spec-driven-development plan (SPEC-001..011)
+├── docs/                 # PRDs and the hackathon problem statement
+└── tests/                # Unit tests (populated as specs are executed)
+```
+
+## Baseline
+
+| Router                                | Cases passed | Token savings |
+|---------------------------------------|--------------|----------------|
+| Testkit reference `baseline_router`   | 7 / 29 (24%) | 94% |
+| `tool_router` (this repo, current)    | 5 / 29 (17%) | 90% |
+
+The gap is deliberate and mapped in `specs/`. See
+[`specs/README.md`](./specs/README.md) for the priority-ordered plan
+(P0 → P1 → P2) with acceptance criteria pinned to specific `TC-*`
+cases. Target after P0: ≥ 20/29. After P1: ≥ 24/29.
+
+## Quick start
+
+```bash
+# Install (uv-managed)
+uv sync
+
+# Score the reference baseline
+cd router-testkit/harness
+python run_benchmark.py --verbose
+
+# Score this project's router
+python run_benchmark.py --router testkit_adapter:MyRouter --verbose
+```
+
+The harness has no third-party dependencies; standard-library only.
+The router itself can run on `HashingEmbedder` (zero deps) or opt-in
+`SentenceTransformerEmbedder` via `uv sync --extra semantic`.
+
+## Where to read next
+
+1. **`specs/README.md`** — the executable plan. Every spec is pickable
+   independently and has acceptance criteria tied to testkit case IDs.
+2. **`router-testkit/README.md`** — the grading contract (authoritative).
+3. **`tool_router/README.md`** (below `tool_router/`) — the current POC
+   library docs. Will be reframed in SPEC-011.
+4. **`docs/hackathon-definition.md`** — the problem statement.
+5. **`docs/PRD.md`** — the initial PRD.
+
+## Toolchain
+
+- Python ≥ 3.14 (pinned in `.python-version`, managed by uv)
+- Ruff for lint + format
+- Pytest for unit tests (see `pyproject.toml`'s `[dev]` extra)
+
+## Dev
+
+```bash
+uv run ruff check .
+uv run ruff format .
+uv run pytest
+```

route66-0.1.1/README.md

@@ -0,0 +1,83 @@
+# route66
+
+**Intelligent Tool Router for LLM Agents** — a catalog-in library that,
+given a query and a large tool catalog, returns only the relevant tools,
+deduplicated, in an ordered execution plan, with policy-aware defaults.
+
+Built against the **`router-testkit`** hackathon grading contract
+(single-tool routing, ordered multi-tool plans, near-duplicate
+resolution, capability-based fallback widening, deprecation policy,
+out-of-scope refusal, and clarify-on-vague-destructive).
+
+## Layout
+
+```
+route66/
+├── tool_router/          # The POC library (facade + retrievers + dedup)
+├── testkit_adapter.py    # Bridges tool_router to the harness contract
+├── demo.py               # Toy 60-tool catalog demo of the POC
+├── metrics.py            # Recall/precision/token-savings eval on the toy set
+├── router-testkit/       # Hackathon input kit — DO NOT MODIFY
+│   ├── catalog/          #   64 mock tools · 13 clusters · dup/version metadata
+│   ├── test_cases/       #   29 scored cases across 11 categories
+│   ├── registry_guardrail/#  Intake validator fixtures
+│   └── harness/          #   run_benchmark.py (scorer) + baseline_router.py
+├── bench/                # Additional benchmarks (baseline, mutations, guardrail)
+├── specs/                # Spec-driven-development plan (SPEC-001..011)
+├── docs/                 # PRDs and the hackathon problem statement
+└── tests/                # Unit tests (populated as specs are executed)
+```
+
+## Baseline
+
+| Router                                | Cases passed | Token savings |
+|---------------------------------------|--------------|----------------|
+| Testkit reference `baseline_router`   | 7 / 29 (24%) | 94% |
+| `tool_router` (this repo, current)    | 5 / 29 (17%) | 90% |
+
+The gap is deliberate and mapped in `specs/`. See
+[`specs/README.md`](./specs/README.md) for the priority-ordered plan
+(P0 → P1 → P2) with acceptance criteria pinned to specific `TC-*`
+cases. Target after P0: ≥ 20/29. After P1: ≥ 24/29.
+
+## Quick start
+
+```bash
+# Install (uv-managed)
+uv sync
+
+# Score the reference baseline
+cd router-testkit/harness
+python run_benchmark.py --verbose
+
+# Score this project's router
+python run_benchmark.py --router testkit_adapter:MyRouter --verbose
+```
+
+The harness has no third-party dependencies; standard-library only.
+The router itself can run on `HashingEmbedder` (zero deps) or opt-in
+`SentenceTransformerEmbedder` via `uv sync --extra semantic`.
+
+## Where to read next
+
+1. **`specs/README.md`** — the executable plan. Every spec is pickable
+   independently and has acceptance criteria tied to testkit case IDs.
+2. **`router-testkit/README.md`** — the grading contract (authoritative).
+3. **`tool_router/README.md`** (below `tool_router/`) — the current POC
+   library docs. Will be reframed in SPEC-011.
+4. **`docs/hackathon-definition.md`** — the problem statement.
+5. **`docs/PRD.md`** — the initial PRD.
+
+## Toolchain
+
+- Python ≥ 3.14 (pinned in `.python-version`, managed by uv)
+- Ruff for lint + format
+- Pytest for unit tests (see `pyproject.toml`'s `[dev]` extra)
+
+## Dev
+
+```bash
+uv run ruff check .
+uv run ruff format .
+uv run pytest
+```

route66-0.1.1/bench/README.md

@@ -0,0 +1,115 @@
+# bench/ — Conduit Evaluation Suite
+
+This directory holds the benchmark that makes Conduit's claims falsifiable. Nothing in the PRD is defensible without the numbers this suite produces.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `tools.jsonl` | The 500-tool catalog. One tool per line. Ground truth for what the router picks *from*. |
+| `queries.jsonl` | The 200-query labeled eval set. Each query has a `gold_tools` list — the ground-truth tool(s) that should be selected. |
+| `queries_ood.jsonl` | 20-query out-of-distribution slice — sampled/authored from real oncall vernacular, held out from all tuning. This is the honest test. |
+| `run_baseline.py` | Long-context baseline runner (Sonnet 4.5 + prompt caching + all 500 tools in context). Produces `baseline_longcontext.md`. See §2.5 of PRD. |
+| `run_conduit.py` | Conduit routed pipeline runner. Produces `report.md`. |
+| `metrics.py` | Shared metric computations (precision@k, recall@k, token counting, cost). |
+| `baseline_longcontext.md` | Auto-generated report from `run_baseline.py`. **This file drives the §2.5 gate decision.** |
+| `report.md` | Auto-generated final report comparing all baselines. |
+
+## Schemas
+
+### `tools.jsonl` — one tool per line
+
+```json
+{
+  "id": "datadog.query_service_map",
+  "server": "datadog",
+  "name": "query_service_map",
+  "description": "Query the Datadog service map to identify upstream/downstream dependencies of a service. Useful for diagnosing latency propagation and identifying the source of a cascading failure.",
+  "params_summary": "service_name (str), time_window_minutes (int, default 15), env (str, optional)",
+  "returns_summary": "list of connected services with edge metadata (latency_p95, error_rate)",
+  "tags": ["datadog", "observability", "dependency-graph", "read"],
+  "policy_tags": ["read-only"],
+  "signature": "query_service_map(service_name: str, time_window_minutes: int = 15, env: str | None = None) -> ServiceMap"
+}
+```
+
+- `id` — globally-unique, dotted namespace `{server}.{name}`.
+- `signature` — the compressed form injected into prompts (see PRD §3.6).
+- `policy_tags` — used by §3.7 policy filter (P1).
+- `tags` — semantic tags; used for BM25 keyword match.
+
+### `queries.jsonl` — one labeled query per line
+
+```json
+{
+  "id": "q_001",
+  "archetype": "bad_deploy",
+  "difficulty": "easy",
+  "query": "Our checkout service p99 latency spiked after the 2:14pm deploy — can you find which commit and roll it back?",
+  "gold_tools": ["datadog.query_latency", "github.get_recent_deploys", "argocd.rollback"],
+  "gold_k": 3,
+  "notes": "Multi-step SRE flow. Router should return the diagnostic tool plus the deploy-history tool plus the mitigation tool."
+}
+```
+
+- `archetype` — one of 10 SRE incident archetypes (see below).
+- `difficulty` — `easy | medium | hard`. Hard queries use colloquial language, abbreviations, or ambiguous phrasing.
+- `gold_tools` — ordered list of tools that constitute the correct chain. Precision@k evaluates whether these appear in the top-k selection.
+- `gold_k` — expected number of tools for this query. Used to evaluate the dynamic-K gate (§3.5).
+
+### `queries_ood.jsonl` — same schema as `queries.jsonl`
+
+Sampled from real oncall vernacular. Held out from all tuning. If Conduit hits precision@3 ≥ 80% here, the "94/8" claim is real. If it doesn't, we don't ship the number.
+
+## Archetypes (10 SRE incident categories)
+
+| Archetype | Description |
+|---|---|
+| `bad_deploy` | Regression introduced by a recent deploy |
+| `memory_leak` | Slow-growing memory that eventually OOMs |
+| `dns_failure` | DNS resolution issues (misconfig, expired records, propagation) |
+| `cert_expiry` | TLS certificate expired or about to expire |
+| `db_failover` | Database primary crashed or is being failed over |
+| `cache_stampede` | Thundering-herd behavior after cache invalidation |
+| `quota_exhaustion` | API rate limit or cloud quota exceeded |
+| `secret_rotation` | Credentials rotated, breaking downstream services |
+| `capacity_spike` | Sudden traffic surge exceeding capacity |
+| `dependency_outage` | A third-party or upstream dependency is down |
+
+Each archetype targets 20 queries in `queries.jsonl` — 6 easy, 8 medium, 6 hard.
+
+## Metrics
+
+- **precision@k** — fraction of queries where any gold tool appears in the top-k selection.
+- **recall@k** — fraction of gold tools present in the top-k selection, averaged over queries.
+- **exact_match@k** — fraction of queries where *all* gold tools appear in the top-k.
+- **prompt_tokens** — median + p95 tokens sent to the LLM per query (before prompt-cache credit).
+- **effective_tokens** — median + p95 tokens billed after prompt caching.
+- **cost_per_query** — USD, using published Sonnet 4.5 pricing.
+- **latency_p50 / latency_p95** — end-to-end latency including LLM response.
+- **hallucination_rate** — fraction of tool calls where the LLM invented a tool name not in the catalog.
+- **end_to_end_success** — human-graded: did the response answer the user's question correctly?
+
+## Running
+
+```bash
+# 1. Long-context baseline (the §2.5 gate)
+python bench/run_baseline.py --tools bench/tools.jsonl --queries bench/queries.jsonl --out bench/baseline_longcontext.md
+
+# 2. Conduit routed pipeline (once §2.5 gate says "build")
+python bench/run_conduit.py --tools bench/tools.jsonl --queries bench/queries.jsonl --out bench/report.md
+
+# 3. Out-of-distribution slice (both pipelines)
+python bench/run_baseline.py --queries bench/queries_ood.jsonl --out bench/baseline_ood.md
+python bench/run_conduit.py  --queries bench/queries_ood.jsonl --out bench/report_ood.md
+```
+
+## The §2.5 gate
+
+After `run_baseline.py` completes, read `baseline_longcontext.md`. Follow the decision table in PRD §2.5:
+
+- Baseline precision@3 ≥ 88% → **strip retrieval, ship the long-context path.**
+- Baseline precision@3 80–88% → build retrieval, demote §3.7/§3.8 to P1.
+- Baseline precision@3 < 80% → build retrieval, mandatory validation gates.
+
+Do this by **Day 2 EOD**. Everything downstream depends on it.

route66-0.1.1/pyproject.toml

@@ -0,0 +1,66 @@
+[build-system]
+requires = ["hatchling>=1.25"]
+build-backend = "hatchling.build"
+
+[project]
+name = "route66"
+dynamic = ["version"]
+description = "route66 Python SDK"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "Proprietary" }
+authors = [{ name = "route66 maintainers" }]
+keywords = ["route66", "sdk"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Operating System :: OS Independent",
+    "Typing :: Typed",
+]
+dependencies = []
+
+[project.optional-dependencies]
+semantic = ["sentence-transformers>=3.0"]
+dev = [
+    "ruff>=0.5",
+    "pytest>=8.0",
+    "build>=1.2",
+    "twine>=5.1",
+    "pytest>=8.0",
+]
+
+[project.scripts]
+route66 = "route66.__main__:main"
+
+# ---------------------------------------------------------------------------
+# Build backend (hatchling) — src layout
+# ---------------------------------------------------------------------------
+[tool.hatch.version]
+# The version is stamped into src/route66/_version.py by the release pipeline
+# just before build. Keeping it in a dedicated file (instead of pyproject.toml)
+# lets each provider release its own version without touching this file.
+source = "regex"
+path = "src/route66/_version.py"
+pattern = "^__version__ = \"(?P<version>[^\"]+)\""
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/route66"]
+
+[tool.hatch.build.targets.sdist]
+include = [
+    "src/route66",
+    "README.md",
+    "pyproject.toml",
+    "release/",
+]
+
+# ---------------------------------------------------------------------------
+# Tooling
+# ---------------------------------------------------------------------------
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-ra"

route66-0.1.1/release/CHANGELOG.md

@@ -0,0 +1,23 @@
+# route66 — release changelog
+
+Each provider tracks its own version line. Entries are appended by the release
+pipeline (`ci/scripts/record_release.py`) so the file below is the source of
+truth for what shipped where.
+
+## generic
+
+<!-- generic:begin -->
+- _no releases yet_
+<!-- generic:end -->
+
+## gitlab (GitLab Package Registry)
+
+<!-- gitlab:begin -->
+- _no releases yet_
+<!-- gitlab:end -->
+
+## pypi (public PyPI)
+
+<!-- pypi:begin -->
+- _no releases yet_
+<!-- pypi:end -->

route66-0.1.1/release/README.md

@@ -0,0 +1,152 @@
+# route66 — release infrastructure
+
+route66 ships as a Python package to two provider registries, each with its
+own independently-managed version line, plus a `generic` line that fans out to
+both.
+
+## Providers
+
+| Provider  | Registry                                     | Version file                              | Auth in CI                  |
+| --------- | -------------------------------------------- | ----------------------------------------- | --------------------------- |
+| `gitlab`  | Private GitLab project Package Registry      | `release/versions/gitlab.json`            | `CI_JOB_TOKEN` (built-in)   |
+| `pypi`    | Public PyPI (`pypi.org`)                     | `release/versions/pypi.json`              | `PYPI_API_TOKEN` (variable) |
+| `generic` | Fans out to **both** providers with a shared | `release/versions/generic.json` (+ writes | uses provider tokens above  |
+|           | version number                               | into each provider file as well)          |                             |
+
+Each provider's file keeps a `current` version and an append-only `history`
+list. The release pipeline never rewrites a past entry — it only updates the
+one for the version that was just published (idempotent re-runs) or prepends a
+new entry.
+
+## Layout
+
+```
+route66/
+├── src/route66/            # SDK source (src-layout)
+│   ├── __init__.py
+│   ├── __main__.py
+│   └── _version.py         # stamped by ci/scripts/bump_version.py
+├── tests/
+├── ci/
+│   ├── release.gitlab-ci.yml   # included by .gitlab-ci.yml
+│   └── scripts/
+│       ├── _common.py
+│       ├── bump_version.py
+│       ├── publish.py
+│       └── record_release.py
+├── release/
+│   ├── CHANGELOG.md         # per-provider sections, auto-updated
+│   └── versions/
+│       ├── generic.json
+│       ├── gitlab.json
+│       └── pypi.json
+├── .gitlab-ci.yml
+└── pyproject.toml
+```
+
+## Continuous pipeline
+
+Runs on every push / MR:
+
+1. **lint** — `ruff check`.
+2. **test** — `pytest`.
+3. **build** — `python -m build` → uploads `dist/` as a job artifact.
+
+## On-demand release pipelines
+
+All three release jobs are `when: manual` and only appear on the default
+(protected) branch. Trigger from **CI/CD → Pipelines → Run pipeline**, choose
+the job, and (optionally) set variables:
+
+| Variable        | Default | Meaning                                              |
+| --------------- | ------- | ---------------------------------------------------- |
+| `BUMP`          | `patch` | One of `patch` / `minor` / `major`.                  |
+| `VERSION`       | *empty* | Explicit `x.y.z` — overrides `BUMP` when set.        |
+| `RELEASE_NOTES` | *empty* | Free-form text appended to the changelog entry.      |
+
+### `release:gitlab`
+
+Publishes to the private GitLab Package Registry using `CI_JOB_TOKEN`. No
+extra project variable required. Endpoint is derived from `CI_API_V4_URL` and
+`CI_PROJECT_ID` at runtime, so the same job works on any GitLab (SaaS or
+self-hosted).
+
+Install a released version:
+
+```bash
+pip install route66 \
+  --index-url https://__token__:<deploy-token>@<gitlab-host>/api/v4/projects/<project-id>/packages/pypi/simple
+```
+
+### `release:pypi`
+
+Publishes to public PyPI. Requires a **project-scoped** API token added as a
+masked + protected CI variable named `PYPI_API_TOKEN`.
+
+Install:
+
+```bash
+pip install route66
+```
+
+### `release:generic`
+
+Bumps the shared version in `release/versions/generic.json`, then fans out and
+publishes to `gitlab` **and** `pypi` with the same version number. Each
+provider's own history file gets an entry for that version too, so any
+provider can be inspected independently.
+
+## Version history model
+
+Every publish produces exactly one entry in the affected provider file(s):
+
+```jsonc
+{
+  "version": "1.4.2",
+  "released_at": "2026-07-14T12:34:56+00:00",
+  "commit": "<full sha>",
+  "pipeline": "<pipeline url>",
+  "artifacts": ["route66-1.4.2-py3-none-any.whl", "route66-1.4.2.tar.gz"],
+  "notes": "..."
+}
+```
+
+Because provider version files are independent, the `pypi` line can lag or
+lead the `gitlab` line freely (e.g. hotfix a private release without touching
+public PyPI). The `generic` line acts as a "canonical" release marker that
+guarantees both providers agree.
+
+## Required CI/CD variables
+
+Set these under **Settings → CI/CD → Variables** on the route66 project:
+
+| Name                | Scope                | Notes                                             |
+| ------------------- | -------------------- | ------------------------------------------------- |
+| `GITLAB_PUSH_TOKEN` | Protected + Masked   | Project access token with `write_repository`. Used to push the version-bump commit + tag back to the default branch. |
+| `PYPI_API_TOKEN`    | Protected + Masked   | Only needed for `release:pypi` and `release:generic`. |
+
+`CI_JOB_TOKEN` is provided automatically by GitLab and is already scoped to
+the project's Package Registry.
+
+## Tagging convention
+
+Each release creates an annotated tag of the form `{provider}/v{X.Y.Z}`, e.g.
+`gitlab/v1.4.2`, `pypi/v1.4.2`, `generic/v2.0.0`. That way `git tag --list`
+tells you every artifact that ever shipped for a given provider, and provider
+lines never collide.
+
+## Local dry-run
+
+To verify a bump without publishing:
+
+```bash
+python ci/scripts/bump_version.py --provider gitlab --kind patch
+cat release/versions/gitlab.json
+python -m build
+```
+
+Then discard the local changes:
+
+```bash
+git checkout -- src/route66/_version.py release/versions/gitlab.json
+```

route66-0.1.1/release/versions/generic.json

@@ -0,0 +1,15 @@
+{
+  "provider": "generic",
+  "description": "Shared/coordinated release version. Bumped by the generic release pipeline and published to every provider.",
+  "current": "0.1.0",
+  "history": [
+    {
+      "version": "0.1.0",
+      "released_at": null,
+      "commit": null,
+      "pipeline": null,
+      "artifacts": [],
+      "notes": "Initial scaffold. No published artifact yet."
+    }
+  ]
+}

route66-0.1.1/release/versions/gitlab.json

@@ -0,0 +1,16 @@
+{
+  "provider": "gitlab",
+  "description": "GitLab Package Registry (private, project-scoped PyPI index).",
+  "index_url_template": "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi",
+  "current": "0.1.0",
+  "history": [
+    {
+      "version": "0.1.0",
+      "released_at": null,
+      "commit": null,
+      "pipeline": null,
+      "artifacts": [],
+      "notes": "Initial scaffold. No published artifact yet."
+    }
+  ]
+}

route66-0.1.1/release/versions/pypi.json

@@ -0,0 +1,16 @@
+{
+  "provider": "pypi",
+  "description": "Public PyPI (pypi.org).",
+  "index_url": "https://upload.pypi.org/legacy/",
+  "current": "0.1.0",
+  "history": [
+    {
+      "version": "0.1.0",
+      "released_at": null,
+      "commit": null,
+      "pipeline": null,
+      "artifacts": [],
+      "notes": "Initial scaffold. No published artifact yet."
+    }
+  ]
+}

route66-0.1.1/router-testkit/README.md

@@ -0,0 +1,137 @@
+# AI Tool-Routing Hackathon — Input Test Kit
+
+A ready-to-run input set for building and grading an **intelligent tool-routing
+layer** that filters a large enterprise tool catalog down to a small,
+high-precision subset per query.
+
+Everything here is *input* for participants: a mock catalog, a scored test
+suite, registry-guardrail fixtures, catalog-churn scenarios, and a benchmark
+harness with a reference baseline router to beat.
+
+```
+router-testkit/
+├── catalog/
+│   └── tools_catalog.json          64 mock tools · 13 clusters · dup/version metadata
+├── registry_guardrail/
+│   └── submissions.json            valid + malformed intake fixtures (accept/reject)
+├── test_cases/
+│   ├── 01_single_tool.json         ── 29 scored query packets across 11 categories
+│   ├── 02_multi_tool_ordered.json
+│   ├── 03_deduplication.json
+│   ├── 04_repeat_count.json
+│   ├── 05_clarification.json
+│   ├── 06_fallback_widening.json
+│   ├── 07_backward_compat.json
+│   ├── 08_negative_out_of_scope.json
+│   ├── 09_cross_cluster.json
+│   ├── 10_positional_bias.json
+│   ├── 11_scope_overlap.json
+│   ├── catalog_mutations.json      add / remove / replace / upgrade resilience
+│   └── _index.json
+└── harness/
+    ├── run_benchmark.py            scorer (prints metrics; no dependencies)
+    └── baseline_router.py          naive reference router (REPLACE THIS)
+```
+
+## Quick start
+
+Standard-library Python only — nothing to install.
+
+```bash
+cd harness
+python run_benchmark.py                          # runs the baseline (~24% accuracy)
+python run_benchmark.py --router mymod:MyRouter  # run your own
+python run_benchmark.py --verbose                # also print why each case failed
+```
+
+Your router is any class with:
+
+```python
+class MyRouter:
+    def __init__(self, catalog): ...
+    def route(self, query, context=None) -> dict:
+        return {
+          "selected_tools": ["fin.get_revenue_report", ...],  # the K schemas you'd inject
+          "plan":           ["fin.get_revenue_report", ...],  # ordered call plan (ids may repeat)
+          "clarify":        False,                             # True => you asked instead of acting
+          "clarify_question": None,
+        }
+```
+
+## The catalog (`catalog/tools_catalog.json`)
+
+64 tools over 13 clusters (finance, communication, analytics, hr, identity,
+it_devops, crm, calendar, documents, cloud_storage, marketing, legal,
+data_export). It carries the metadata your router is expected to exploit:
+
+- `near_duplicate_groups` — deliberately overlapping tools. E.g. the classic
+  `get_user` / `fetch_profile` / `lookup_member` / `get_user_by_email` quartet,
+  two revenue fetchers, two chart makers, `create_ticket` vs `open_incident`.
+- `version_families` — `create_invoice` (deprecated v1) coexists with
+  `create_invoice_v2` (current). Each tool has `deprecated`, `replaces`,
+  `replaced_by`, `sunset_date`.
+- `cross_cluster_traps` — tools whose **name** points at the wrong cluster:
+  `email_report` lives in data_export (not communication); `send_campaign` is
+  a marketing blast, not a 1:1 email.
+- Per-tool `side_effects` (`read` / `write` / `destructive`) so you can be
+  conservative about routing destructive actions on vague queries.
+
+## The scored suite (`test_cases/*.json`)
+
+Each case has a machine-checkable `expected` block:
+
+| field | meaning |
+|---|---|
+| `tools_required` | `{tool_id, min_calls, max_calls}` — must be selected and called within the count band |
+| `order` | ordered list that must appear as a **subsequence** of your `plan` (or `null`) |
+| `forbidden` | tool ids that must NOT be selected. `"*"` means *no tool at all* |
+| `should_clarify` | router should ask, not act |
+| `allow_any_of` | dedup groups: pick **exactly one** id per group |
+
+Categories map 1:1 to the problem's pillars:
+
+- **single_tool / cross_cluster** — basic and multi-cluster routing accuracy.
+- **multi_tool_ordered** — the canonical *"pull revenue → make chart → email to
+  finance"* chain, with ordering enforced.
+- **deduplication** — the near-dup groups; injecting all three user lookups
+  fails, injecting one passes.
+- **repeat_count** — same tool invoked N times (e.g. restart 3 services →
+  `restart_service` ×3).
+- **clarification / negative_out_of_scope** — must *not* fabricate a tool call.
+- **fallback_widening** — the initially-obvious tool lacks a needed field/scope
+  (e.g. `send_email` can't attach; identity lookup has no SSN); the case encodes
+  the correct widen target.
+- **backward_compat** — unversioned request → current tool; explicit "legacy v1"
+  → deprecated tool; genuinely ambiguous → clarify.
+- **positional_bias / scope_overlap** — set `context.shuffle_catalog=true` and
+  present near-dups in random order; correct pick must be signal-driven, not
+  position-driven (DMS vs raw S3, etc.).
+
+## Registry guardrail (`registry_guardrail/submissions.json`)
+
+Feed each `invalid_submissions` entry to your intake validator; it must reject
+with the given `expected_reason` (vague/too-long description, bad id/name,
+missing field, duplicate id, bad param type, too many params, bad semver,
+dangling `replaced_by`, malformed JSON). The two `valid_submissions` must pass.
+
+## Catalog churn (`test_cases/catalog_mutations.json`)
+
+Directly answers the "what if tools change?" evaluation questions. Apply each
+`delta` to the catalog, re-run the probe, and assert the invariant:
+
+- **add** — new tool routable immediately, no regressions elsewhere.
+- **remove** — depended-on tool gone → graceful degradation, no crash.
+- **replace** — renamed tool routes by capability, not hardcoded id.
+- **upgrade (both alive)** — `PREFER_CURRENT` unless an explicit version signal;
+  never inject both versions at once.
+- **scale stress** — auto-grow to ~104 tools; accuracy and token savings must hold.
+
+## Scoring & metrics
+
+`run_benchmark.py` checks each case and prints, in plain text: a PASS/FAIL line
+per case (with token cost), a per-category tally, and a summary with overall
+accuracy and token savings (average routed tokens vs. the full-catalog dump).
+Run with `--verbose` to see the reason each failed case failed. No charts, no
+dependencies — just numbers you can read or pipe.
+
+Baseline reference: ~24% case accuracy, ~94% token savings. Beat both.

route66-0.1.1/specs/README.md

@@ -0,0 +1,110 @@
+# Route66 — Spec Index
+
+Spec-driven development for the hackathon build-out of the `tool_router`
+library (this repo) against the `router-testkit` grading contract.
+
+## How to use this directory
+
+- Each spec is **pickable independently**. If the "Dependencies" block is
+  empty or its dependencies are marked ✅, you can start.
+- Every spec ships with **concrete acceptance criteria tied to specific
+  testkit case IDs** (`TC-XYZ`). "Done" means: those cases flip PASS on the
+  harness *and* the unit tests defined in the spec pass.
+- Read the spec top-to-bottom before writing code — the Design section
+  locks names, signatures, and file paths that later specs depend on.
+
+## Ground-truth baseline (measured 2026-07-13)
+
+| Router | Cases | Token savings |
+|---|---|---|
+| Reference `baseline_router` | 7/29 (24%) | 94% |
+| POC via adapter (today) | 5/29 (17%) | 90% |
+
+Target after P0: **≥20/29 (69%)**. Target with P1: **≥24/29 (83%)**.
+
+## Priority tiers
+
+- **P0 — Ship-critical.** Without this the submission is below baseline.
+- **P1 — Category-unlock.** Turns a 0/N category into a passing one.
+- **P2 — Pitch / polish.** Doesn't move the score much, but sells the story.
+
+## Specs
+
+### P0 — Foundation cleanup (Day 1, ~half a day total)
+
+| ID | Title | Unblocks |
+|---|---|---|
+| [SPEC-001](./SPEC-001-structured-tool-metadata.md) | Structured `Tool` metadata (id, cluster, tags, deprecated, side_effects, parameters) | Every P0/P1 spec below |
+| [SPEC-002](./SPEC-002-runtime-dedup.md) | Runtime near-duplicate resolution (replace index-time collapse) | Deduplication, positional_bias |
+| [SPEC-003](./SPEC-003-route-context-and-plan-return.md) | `RouteContext` passthrough + return full plan (not top-1) | Everything downstream — every case scored on `plan` |
+
+### P0 — Core build (Days 2–4)
+
+| ID | Title | Unblocks (testkit categories) |
+|---|---|---|
+| [SPEC-004](./SPEC-004-chain-planner.md) | `ChainPlanner`: multi-step ordered plans | `multi_tool_ordered`, `cross_cluster`, `fallback_widening` |
+| [SPEC-005](./SPEC-005-policy-layer.md) | Policy layer: `PREFER_CURRENT`, `NO_TOOL_ON_OOS`, `CLARIFY_ON_DESTRUCTIVE_VAGUE` | `backward_compat`, `clarification`, `negative_out_of_scope` |
+| [SPEC-006](./SPEC-006-capability-filter.md) | Capability-based fallback widening | `fallback_widening` |
+| [SPEC-010](./SPEC-010-testkit-adapter-hardening.md) | Harden the `MyRouter` testkit adapter | Passes the actual `run_benchmark.py` scoring |
+
+### P1 — Category unlocks (Day 5)
+
+| ID | Title | Unblocks |
+|---|---|---|
+| [SPEC-007](./SPEC-007-entity-count-extractor.md) | Entity/count extractor for repeat-call plans | `repeat_count` |
+| [SPEC-008](./SPEC-008-schema-intake-validator.md) | `SchemaIntakeValidator` for registry guardrail | `registry_guardrail/submissions.json` |
+
+### P2 — Completeness & pitch (Days 6–7)
+
+| ID | Title | Value |
+|---|---|---|
+| [SPEC-009](./SPEC-009-catalog-registry.md) | `CatalogRegistry` with add/remove/replace/upgrade deltas | `catalog_mutations.json`; the "scales to 104 tools" pitch slide |
+| [SPEC-011](./SPEC-011-poc-reframing.md) | Reframe POC README + demo around what the testkit actually grades | Judge-facing story |
+
+## Dependency graph
+
+```
+SPEC-001 (Tool metadata)
+   │
+   ├── SPEC-002 (runtime dedup)
+   ├── SPEC-003 (context + plan)
+   │      │
+   │      ├── SPEC-004 (planner) ──┬── SPEC-006 (capability filter)
+   │      ├── SPEC-005 (policy)     │
+   │      └── SPEC-007 (entities) ──┘
+   │
+   ├── SPEC-008 (intake validator, independent)
+   └── SPEC-009 (catalog registry, depends on SPEC-001)
+
+SPEC-010 (adapter) — depends on all P0 specs shipping
+SPEC-011 (framing) — independent, do last
+```
+
+## Suggested pickup order (single-owner)
+
+**Day 1:** SPEC-001 → SPEC-002 → SPEC-003 (~5 hrs, cleanup)
+**Day 2:** SPEC-005 (policy layer — highest per-hour ROI)
+**Day 3:** SPEC-004 (planner)
+**Day 4:** SPEC-006 (capability filter) → SPEC-010 (adapter hardening)
+**Day 5:** SPEC-007 → SPEC-008 (parallelizable)
+**Day 6:** SPEC-009 (catalog churn)
+**Day 7:** SPEC-011 (reframing) + bulletproofing
+
+## What is *out of scope* for these specs
+
+Per the decision to keep-but-bypass POC assets that don't score against
+this testkit, the following files are **frozen**: `session.py`,
+`storage.py`, `providers.py`, `adapters.py`, and the
+`AssociationRetriever` in `retrieval.py`. Do not modify them; do not
+delete them. The specs below explicitly bypass them via the `RouteContext`
++ adapter path.
+
+## Test commands (reference)
+
+```bash
+# 1. From router-testkit/harness/:
+python run_benchmark.py --router testkit_adapter:MyRouter --verbose
+
+# 2. From :
+python3 -m pytest tests/           # spec-defined unit tests
+```

route66-0.1.1/src/route66/__init__.py

@@ -0,0 +1,24 @@
+"""route66 - Python SDK.
+
+Public entry point for the route66 package. Version is sourced from the
+installed distribution metadata so that provider-specific builds report
+the version that was actually published.
+"""
+
+from __future__ import annotations
+
+from importlib import metadata as _metadata
+
+from ._version import __version__ as _stamped_version
+
+try:
+    __version__ = _metadata.version("route66")
+except _metadata.PackageNotFoundError:  # pragma: no cover - local checkout
+    __version__ = _stamped_version
+
+__all__ = ["__version__", "hello"]
+
+
+def hello(name: str = "world") -> str:
+    """Return a friendly greeting. Placeholder SDK surface."""
+    return f"Hello from route66, {name}!"

route66-0.1.1/src/route66/__main__.py

@@ -0,0 +1,13 @@
+"""Allow `python -m route66`."""
+
+from __future__ import annotations
+
+from . import hello
+
+
+def main() -> None:
+    print(hello())
+
+
+if __name__ == "__main__":
+    main()

route66-0.1.1/src/route66/_version.py

@@ -0,0 +1,3 @@
+"""Version stamp for route66. Managed by ci/scripts/bump_version.py."""
+
+__version__ = "0.1.1"

route66-0.1.1/src/route66/py.typed

@@ -0,0 +1 @@
+# Marker file so type checkers (mypy, pyright) treat route66 as a typed package.