forgecad 0.9.13 → 0.9.15

package/dist/docs-raw/rl-environments.md

@@ -1,508 +0,0 @@
-# ForgeCAD RL Environments
-
-This is the permanent operating guide for ForgeCAD reinforcement-learning and
-AI-lab reconstruction environments. Temporary notes can record experiments, but
-this document is the stable contract for building, running, evaluating, and
-maintaining reconstruction tasks.
-
-## Product Goal
-
-ForgeCAD RL environments should let an external lab train or evaluate an agent
-that turns reference physical geometry into readable ForgeCAD source code.
-
-The loop is intentionally simple:
-
-1. ForgeCAD authors Harbor task folders directly under
-   `ai-labs/reconstruction/tasks/`.
-2. `npm run harbor:reconstruction:refresh` validates those folders and updates
-   `ai-labs/reconstruction/dataset.toml` digests.
-3. Harbor builds the sandbox and runs the agent.
-4. The hidden RewardKit verifier evaluates `/app/submission/main.forge.js`
-   after the agent exits.
-5. RewardKit writes Harbor rewards and per-criterion details; ForgeCAD-specific
-   verifier reports preserve geometry comparison evidence.
-
-Harbor is the executable contract for both RL rollouts and benchmark evals.
-There is no second task catalog, no generated sibling registry, and no local
-runner command surface to keep in sync.
-
-The external collaboration repository for SpaceXAI-facing handoff material is:
-
-```text
-https://github.com/KoStard/SpaceXAI_ForgeCAD
-```
-
-Treat that repository as coordination and distribution context. The executable
-benchmark contract is the Harbor dataset in this repo.
-
-## Current Reconstruction Suite
-
-The canonical reconstruction dataset lives in:
-
-```text
-ai-labs/reconstruction/
-```
-
-The current seed tasks are:
-
-```text
-ball-bearing
-sealing-frame
-wheel
-```
-
-The old source-file-style labels are not primary task IDs. Public benchmark data
-normalizes historical rows to the user-facing labels above.
-
-## Harbor Task Contract
-
-Each task folder includes:
-
-```text
-README.md
-instruction.md
-task.toml
-environment/
-  Dockerfile
-  forgecad-wrapper.sh
-  reference/<task-reference>.3mf
-  starter/main.forge.js
-tests/
-  test.sh
-  checks.py
-  enrich-reward.mjs
-  forgecad-reconstruction-check.mjs
-  score-calibration.mjs
-  task.json
-```
-
-The agent sees the reference asset, starter source, instructions, and the
-workspace-local `forgecad` wrapper. The verifier files under `tests/` are hidden
-until the agent exits.
-
-The final answer is always:
-
-```text
-/app/submission/main.forge.js
-```
-
-The answer must be editable parametric ForgeCAD source. It must not import,
-read, embed, copy, or encode the reference asset.
-
-The `forgecad` wrapper exposes run/render/inspect/export checks but blocks
-direct `score` and `compare` use during the episode. The hidden verifier uses
-the internal comparison path after the agent exits.
-
-Verifier execution goes through RewardKit:
-
-```bash
-uvx --from 'harbor-rewardkit==0.1.*' rewardkit /tests
-```
-
-`checks.py` exposes one programmatic criterion,
-`forgecad_reconstruction_score`, backed by the ForgeCAD guard/render/compare
-script. RewardKit writes `/logs/verifier/reward.json` and
-`/logs/verifier/reward-details.json`; `enrich-reward.mjs` preserves the
-RewardKit output while ensuring `reward.json` has numeric Harbor fields:
-`reward`, `score`, `rawCompareScore`, `guard`, and `valid`.
-
-## NPM Command Surface
-
-Keep this surface small and boring:
-
-```bash
-npm run harbor:reconstruction:refresh
-npm run harbor:reconstruction:check
-npm run harbor:reconstruction:smoke
-npm run harbor:reconstruction:pack-local
-npm run harbor:reconstruction:clean-local
-npm run benchmark:reconstruction:list
-npm run benchmark:reconstruction:import-harbor -- <job-or-trial-dir> --budget standard
-npm run benchmark:reconstruction:leaderboard
-npm run benchmark:reconstruction:report
-```
-
-`refresh` validates task layout and rewrites dataset digests. It does not
-generate task folders. If a task should change, edit the Harbor task folder
-itself.
-
-`check` runs the same validation and fails if `dataset.toml` is stale.
-
-`smoke` runs Harbor with the no-op agent on `ball-bearing`. Use it whenever the
-task image, verifier dependencies, or RewardKit contract changes.
-
-## Local Task Workflow
-
-Use this flow when adding or changing tasks inside the ForgeCAD repository.
-
-1. Add or edit the task under `ai-labs/reconstruction/tasks/<task-id>/`.
-2. Keep the task folder self-contained and Harbor-runnable.
-3. Update the task `README.md` with environment, verifier dimensions, layout,
-   and concrete Harbor run commands.
-4. Refresh the dataset manifest:
-
-   ```bash
-   npm run harbor:reconstruction:refresh
-   npm run harbor:reconstruction:check
-   ```
-
-5. List tasks through the local reporting wrapper:
-
-   ```bash
-   npm run benchmark:reconstruction:list
-   ```
-
-6. Run Harbor directly when testing the lab-facing package:
-
-   ```bash
-   npm run harbor:reconstruction:smoke
-   ```
-
-7. Import completed Harbor jobs when you need local leaderboard/report output:
-
-   ```bash
-   npm run benchmark:reconstruction:import-harbor -- \
-     "$HOME/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/smoke/<timestamp>" \
-     --budget smoke
-   ```
-
-## Direct Harbor Runs
-
-### One Task, One Agent
-
-Use `--include-task-name` to select one Harbor task. Use `--agent` for a
-built-in Harbor agent:
-
-```bash
-CODEX_AUTH_JSON_PATH="$HOME/.codex/auth.json" \
-uvx --from harbor harbor run \
-  -p ai-labs/reconstruction/tasks \
-  --include-task-name ball-bearing \
-  --agent codex \
-  --model gpt-5.5 \
-  --artifact /app/submission/main.forge.js \
-  --jobs-dir "$HOME/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/codex" \
-  --n-concurrent 1
-```
-
-Use `--agent-import-path` for a custom adapter such as the bundled Grok adapter:
-
-```bash
-PYTHONPATH="$PWD/ai-labs/reconstruction/agents" \
-GROK_AUTH_JSON="$HOME/.grok/auth.json" \
-GROK_CONFIG_TOML="$HOME/.grok/config.toml" \
-uvx --from harbor harbor run \
-  -p ai-labs/reconstruction/tasks \
-  --include-task-name ball-bearing \
-  --agent-import-path grok_harbor_agent:GrokCliAgent \
-  --model grok-build \
-  --artifact /app/submission/main.forge.js \
-  --jobs-dir "$HOME/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/grok" \
-  --n-concurrent 1
-```
-
-Current task names are `ball-bearing`, `sealing-frame`, and `wheel`.
-
-### Full Suite
-
-Codex:
-
-```bash
-CODEX_AUTH_JSON_PATH="$HOME/.codex/auth.json" \
-uvx --from harbor harbor run \
-  -p ai-labs/reconstruction/tasks \
-  --agent codex \
-  --model gpt-5.5 \
-  --jobs-dir "$HOME/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/codex" \
-  --artifact /app/submission/main.forge.js \
-  --n-concurrent 3
-```
-
-Grok:
-
-```bash
-PYTHONPATH="$PWD/ai-labs/reconstruction/agents" \
-GROK_AUTH_JSON="$HOME/.grok/auth.json" \
-GROK_CONFIG_TOML="$HOME/.grok/config.toml" \
-uvx --from harbor harbor run \
-  -p ai-labs/reconstruction/tasks \
-  --agent-import-path grok_harbor_agent:GrokCliAgent \
-  --model grok-build \
-  --jobs-dir "$HOME/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/grok" \
-  --artifact /app/submission/main.forge.js \
-  --n-concurrent 3
-```
-
-Direct Harbor CLI-agent runs need outbound internet from the task container for
-agent install/auth/model calls, so the task definitions set
-`allow_internet = true`. The verifier remains hidden until the agent exits.
-
-## Remote Harbor Runs On popos
-
-Use `popos` as a remote Harbor execution host when local machine resources are
-the bottleneck. Do not add a second benchmark runner for this. The remote
-machine should run the same Harbor commands against the same task folders.
-
-Remote prerequisites:
-
-1. `ssh popos` reaches the server.
-2. Docker, `uvx`, Node, and npm are installed on the server.
-3. A dedicated checkout exists at `~/Projects/CAD/ForgeCAD`.
-4. Agent auth lives outside the repo on the server.
-5. Run outputs stay outside the checkout under
-   `~/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/`.
-
-For normal runs, update the dedicated server checkout with git after pushing
-the branch you want to test:
-
-```bash
-branch="$(git branch --show-current)"
-git push origin "$branch"
-ssh popos "cd ~/Projects/CAD/ForgeCAD && \
-  git fetch origin && \
-  git switch '$branch' && \
-  git pull --ff-only origin '$branch'"
-```
-
-Use `rsync` only for scratch experiments where the local work is intentionally
-uncommitted. It mirrors the dirty checkout onto the server, so use it only
-against a dedicated throwaway checkout:
-
-```bash
-rsync -az --delete \
-  --exclude .git \
-  --exclude node_modules \
-  --exclude dist \
-  --exclude dist-cli \
-  --exclude dist-bundles \
-  --exclude .data \
-  --exclude 'ai-labs/reconstruction/tasks/*/environment/forgecad-package/*.tgz' \
-  ./ popos:~/Projects/CAD/ForgeCAD/
-```
-
-Run preflight validation on the server:
-
-```bash
-ssh popos 'cd ~/Projects/CAD/ForgeCAD && npm run harbor:reconstruction:check'
-```
-
-If `popos` reports `Missing script: "harbor:reconstruction:check"`, the server
-checkout is stale. Update the server checkout with git first.
-
-Run the no-op Harbor smoke on the server:
-
-```bash
-ssh popos 'cd ~/Projects/CAD/ForgeCAD && npm run harbor:reconstruction:smoke'
-```
-
-Run one task on one Codex agent remotely:
-
-```bash
-ssh popos 'cd ~/Projects/CAD/ForgeCAD && \
-CODEX_AUTH_JSON_PATH="$HOME/.codex/auth.json" \
-uvx --from harbor harbor run \
-  -p ai-labs/reconstruction/tasks \
-  --include-task-name ball-bearing \
-  --agent codex \
-  --model gpt-5.5 \
-  --artifact /app/submission/main.forge.js \
-  --jobs-dir "$HOME/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/codex" \
-  --n-concurrent 1'
-```
-
-Run one task on the bundled Grok adapter remotely:
-
-```bash
-ssh popos 'cd ~/Projects/CAD/ForgeCAD && \
-PYTHONPATH="$PWD/ai-labs/reconstruction/agents" \
-GROK_AUTH_JSON="$HOME/.grok/auth.json" \
-GROK_CONFIG_TOML="$HOME/.grok/config.toml" \
-uvx --from harbor harbor run \
-  -p ai-labs/reconstruction/tasks \
-  --include-task-name ball-bearing \
-  --agent-import-path grok_harbor_agent:GrokCliAgent \
-  --model grok-build \
-  --artifact /app/submission/main.forge.js \
-  --jobs-dir "$HOME/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/grok" \
-  --n-concurrent 1'
-```
-
-Fetch remote Harbor jobs back before importing them into the local benchmark
-ledger:
-
-```bash
-rsync -az \
-  popos:~/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/ \
-  "$HOME/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/"
-
-npm run benchmark:reconstruction:import-harbor -- \
-  "$HOME/Projects/CAD/ForgeCAD-RL-Agent-Runs/harbor/codex/<timestamp>" \
-  --budget standard
-```
-
-By default, task Dockerfiles install the pinned published ForgeCAD npm package.
-For unpublished local ForgeCAD changes, run the local package flow on the server
-checkout:
-
-```bash
-ssh popos 'cd ~/Projects/CAD/ForgeCAD && npm run harbor:reconstruction:pack-local'
-ssh popos 'cd ~/Projects/CAD/ForgeCAD && npm run harbor:reconstruction:smoke'
-ssh popos 'cd ~/Projects/CAD/ForgeCAD && npm run harbor:reconstruction:clean-local'
-```
-
-If `pack-local` fails because repo build tools are missing, run `npm install`
-once in the server checkout.
-
-There is intentionally no `popos` npm script. The host alias, checkout path,
-and auth file locations are per-developer machine configuration; the repo-owned
-commands remain the small Harbor command surface documented above.
-
-## Local Reporting Wrapper
-
-The local reporting wrapper is not a runner. It exists only to list Harbor
-tasks, import completed Harbor jobs, and build submission ledgers, leaderboards,
-and visual reports.
-
-There is no local matrix command, no host-side agent adapter layer, no separate
-task tracker, and no host-side evaluator. If an agent should run, run it
-through Harbor.
-
-## Run Data
-
-Keep run outputs outside the source checkout under:
-
-```text
-~/Projects/CAD/ForgeCAD-RL-Agent-Runs/
-  benchmark/
-  harbor/
-  research/
-  smoke/
-```
-
-Keep raw agent logs and Harbor job directories. They are evidence for prompt
-design, task packaging, tool behavior, and reward design.
-
-Important Harbor files:
-
-```text
-<jobs-dir>/<timestamp>/<task-trial>/
-  result.json
-  trial.log
-  artifacts/main.forge.js
-  verifier/reward.json
-  verifier/reward-details.json
-  verifier/report.json
-  verifier/score.json
-  verifier/submission.png
-  verifier/reference.png
-```
-
-## Result Payload Contract
-
-The Harbor verifier writes numeric reward files:
-
-| Field | Purpose |
-|---|---|
-| `reward` | Scalar in `[0, 1]` for RL. |
-| `score` | Human scale score in `[0, 100]`, normally `reward * 100`. |
-| `rawCompareScore` | Raw ForgeCAD geometry comparison score before calibration. |
-| `guard` | Numeric guardrail pass marker. |
-| `valid` | Numeric validity marker. |
-
-Harbor reward values must stay numeric because Harbor validates reward payloads
-as scalar metrics. Detailed status, aggregation, guard issues, score vectors,
-renders, and command logs live in `/logs/verifier/report.json`,
-`/logs/verifier/score.json`, `/logs/verifier/reward-details.json`, and the
-copied Harbor trial artifacts.
-
-## Scoring Principles
-
-The scalar reward is a projection, not the whole truth. Keep the score vector
-first-class.
-
-For reconstruction tasks, the geometry component should include:
-
-- Bidirectional surface coverage over multiple thresholds.
-- Tail-distance penalties such as p95/p99.
-- Feature-edge matching for sharp and boundary structure.
-- Dimension agreement.
-- Volume or occupancy overlap where reliable.
-- Hard caps for obvious shortcut failures.
-
-Avoid solving score problems by only changing a curve shape. If a two-ring
-bearing approximation scores too high, the missing signal is structural: rolling
-elements, feature edges, component count, section occupancy, or topology. Add
-the missing measurement and then calibrate the scalar projection.
-
-## Maintaining Tasks
-
-When adding or changing a task:
-
-1. Put the task under `ai-labs/reconstruction/tasks/<task-id>/`.
-2. Use a user-facing slug such as `ball-bearing`, not an opaque source asset ID.
-3. Include the full Harbor layout: prompt, task TOML, environment, reference,
-   starter, tests, task metadata, and README.
-4. Record the reference asset SHA-256 in `tests/task.json`.
-5. Keep `/app/submission/main.forge.js` as the stable deliverable.
-6. Refresh the dataset with `npm run harbor:reconstruction:refresh`.
-7. Run identity scoring: reference vs reference should score near 100.
-8. Run the starter score and record it.
-9. Run at least one known weak shortcut candidate and one stronger candidate.
-10. Confirm the stronger candidate ranks higher for the right reasons.
-11. Store local experiment artifacts under `docs/temporary/projects/...`, then
-    promote only durable conclusions here.
-
-## Maintaining Agent Adapters
-
-Agent adapters belong in Harbor, not in the local benchmark reporting wrapper.
-For each supported Harbor agent CLI, keep a short note answering:
-
-1. How to run headlessly.
-2. How to set cwd.
-3. How to set task-local home/config.
-4. How skills are discovered.
-5. How to disable global/user config.
-6. How to disable web search, subagents, plugins, MCP, and memory.
-7. What filesystem sandbox exists.
-8. How auth is supplied.
-9. How stdin is made non-interactive.
-10. How logs are captured.
-11. How the wall-clock timeout is enforced.
-
-Promote stable CLI adapter facts here when they become part of the supported
-contract.
-
-## Versioning Policy
-
-Version every surface that labs depend on:
-
-- Harbor dataset digest.
-- Task IDs.
-- `tests/task.json` schema fields.
-- Verifier payload shape.
-- Score aggregation profile.
-- Geometry comparison algorithm version.
-
-Changing the scalar reward is a benchmark change. Keep the score vector and raw
-compare metrics in the payload so old and new rewards can be compared across
-saved runs.
-
-## Merge Readiness Checklist
-
-Before merging reconstruction environment changes:
-
-1. `npm test` passes, or a narrower check is justified in the PR.
-2. `npm run harbor:reconstruction:refresh` passes.
-3. `npm run harbor:reconstruction:check` passes.
-4. `npm run benchmark:reconstruction:list` passes.
-5. `npm run harbor:reconstruction:smoke` passes when environment packaging
-   changes.
-6. Verifier reward output contains numeric `reward`, `score`,
-   `rawCompareScore`, `guard`, and `valid`.
-7. Verifier report output contains the detailed status, score vector,
-   aggregation, guard data, and artifacts.
-8. RewardKit writes `reward-details.json`.
-9. Agent logs are captured in the Harbor trial folder.
-10. Timeouts are enforced by Harbor and inside the verifier.
-11. No global skills or parent repo instructions are required.
-12. The docs index and this guide are updated when the contract changes.