@autonoma-ai/planner 0.1.22 → 0.1.23

package/README.md

@@ -41,8 +41,8 @@ inside the dashboard; the terminal is handed over only for the
 SDK-integration handoff below, and the dashboard comes back when the agent exits.
 
 Piped output, CI, and `--non-interactive` keep the plain line-based output. See
-`docs/ui-design-brief.md` for the design rationale and `docs/tui-plan.md` for the build plan;
-`pnpm ui:gallery` steps through every dashboard state with fixture data (Tab / Shift+Tab).
+`docs/ui-design-brief.md` for the design rationale; `pnpm ui:gallery` steps through every
+dashboard state with fixture data (Tab / Shift+Tab).
 Pass a past run's output directory - `pnpm ui:gallery ~/.autonoma/<slug>` - to add a scene
 backed by real files, so navigation and scrolling can be tested on real documents.
 
@@ -76,10 +76,11 @@ of a copy-paste guide, the CLI hands the whole integration to your **locally-ins
 one interactive, autonomous session - like `git commit` with no `-m` opening your editor. You watch
 it install the SDK, build the endpoint, write the factories, **generate the test-data recipe**, and
 validate each entity itself: for every entity it runs `up`, checks your database for the new rows,
-runs `down`, and checks they're gone. It drives the endpoint through the CLI's own signed client
-(`autonoma-planner sdk discover|up|down`), so its checks use the exact request signing the platform
-uses. When it reports the session complete, the CLI uploads the recipe it produced and continues to
-test generation.
+runs `down`, and checks they're gone. It finishes by seeding two instances at once, proving your
+recipe survives concurrent test runs. It drives the endpoint through the CLI's own signed client
+(`autonoma-planner sdk discover|up|down`), so its checks use the exact request signing and the exact
+recipe-token substitution the platform uses. When it reports the session complete, the CLI uploads
+the recipe it produced and continues to test generation.
 
 - `--agent <name>` - preselect the agent to hand off to (`claude` or `codex`). Omit to auto-detect;
   if both are installed you're prompted to pick.
@@ -108,8 +109,15 @@ Artifacts are written to `~/.autonoma/<project-slug>/`:
 ├── recipe.json       # scenario recipes (SDK factories); the agent generates + validates it
 ├── integration-prompt.md  # rendered SDK-integration instructions (drives the agent + manual fallback)
 └── qa-tests/         # generated test cases (markdown)
+    ├── INDEX.md      # table of contents for the suite
+    └── _invalid/     # tests that failed structural validation; never uploaded
 ```
 
+`qa-tests/INDEX.md` is written once, at the end, from the files on disk - so it always matches
+the suite beside it. Alongside the totals it names what the run could *not* deliver: features it
+walked without producing a test, and tests the review cycle removed that nothing could put back.
+Both are fixed by re-running the planner.
+
 ## Automatic upload
 
 When started from Autonoma onboarding, the CLI uploads the artifacts itself once the run finishes -
@@ -129,11 +137,72 @@ If the upload credentials are not set, the CLI just leaves the artifacts on disk
 | `AUTONOMA_GENERATION_ID` | for upload | The setup id artifacts are uploaded against. Injected by onboarding. |
 | `AUTONOMA_SHARED_SECRET` | no | Per-application secret used to sign SDK/webhook requests. Injected by onboarding. |
 | `AUTONOMA_DISTINCT_ID` | no | PostHog identity so CLI events join the signup funnel. Injected by onboarding. |
-| `DONT_TRACK` | no | Set to `1`/`true` to disable anonymous analytics. |
+| `DONT_TRACK` | no | Set to `1`/`true` to disable all telemetry - events, log shipping and session replay. |
 
 `AUTONOMA_API_TOKEN` + `AUTONOMA_GENERATION_ID` together enable automatic upload (the endpoint
 defaults to production unless `AUTONOMA_API_URL` is set).
 
+## Telemetry
+
+Three lanes, all to PostHog, all off when `DONT_TRACK=1`:
+
+- **Events** (`core/analytics.ts`) - `cli_run_started`, `cli_step_completed`, `$exception`, and friends,
+  posted to the capture endpoint.
+- **Logs** (`core/logs.ts`) - the run's narrative, shipped as OTLP records under the service name
+  `autonoma-planner`: run and step lifecycle, every agent tool call, tool errors, retries and nudges,
+  and everything the CLI prints to the user.
+- **Session replay** (`src/replay/`) - the dashboard itself, as rrweb events, played back in PostHog's
+  session-replay player.
+
+All three lanes are indexed by the same identifiers (`core/session.ts`), so one run resolves the same
+way from any of them:
+
+| Attribute | What it identifies |
+|-----------|--------------------|
+| `run_id` / `sessionId` | This CLI invocation. `sessionId` is PostHog's own grouping key, so a run's logs sit together. |
+| `generation_id` | The onboarding setup the run is fulfilling - the join back to an Autonoma record. |
+| `posthogDistinctId` | The person, when the app launched the CLI with an identity; otherwise an anonymous per-machine device id. |
+| `project_slug`, `cli_version`, `node_version` | Which project, which build, which runtime. |
+
+To read one run: filter logs by `service.name = autonoma-planner` and the `generation_id` (or `run_id`)
+you are chasing, ordered earliest-first.
+
+Log records carry **metadata only** - step names, agent and tool names, file paths, patterns, commands,
+durations, and error messages. Model prose and reasoning, prompts, and the contents of any file the
+agent read or wrote are never sent; those are what would carry a user's source code off their machine.
+Records are truncated and a single run is capped at 5000 of them, so a stuck agent loop cannot flood
+ingestion - the cap being reached is itself logged.
+
+### Session replay
+
+An interactive run is recorded and plays back in PostHog's normal session-replay player, alongside web
+recordings for the same person. It uses the run id as its `$session_id`, so a recording, its events and
+its logs all resolve to one another.
+
+`DONT_TRACK=1` turns it off along with the other two lanes - one switch, no partial opt-out.
+
+**This lane captures more than the others, and it is worth being explicit about it.** Events and logs
+are deliberately metadata-only - no prompts, no model prose, no file contents. A replay is a verbatim
+copy of the dashboard, and the dashboard renders repository paths and the contents of the files the run
+generates. Anything visible on screen is in the recording. Keystrokes are the one exception: printable
+characters are recorded as a placeholder, never the literal key, so the upload is not a transcript of
+the keyboard.
+
+How it works (`src/replay/`):
+
+- Frames come from an off-screen render of the same `<App>` the user sees, using Ink's `debug`
+  mode, which writes a complete frame with no cursor escapes. Scraping the terminal repaint stream
+  would not work, because Ink can emit either full redraws or per-line incremental updates.
+- Each frame becomes a synthetic DOM: one `<div>` per terminal row, spans for each colour run.
+- Frames are diffed row by row, so a repaint uploads only the rows that changed. A full snapshot of
+  the dashboard is tens of kilobytes; a typical repaint is a few hundred bytes.
+- Keystrokes are emitted as rrweb input events, because PostHog derives the active/inactive split
+  from interaction events alone. Without them a run of pure repaints reads as entirely idle and the
+  player's inactivity-skipping has nothing to skip to. Printable characters are recorded as a
+  placeholder rather than the literal key, so the upload is not a transcript of the keyboard.
+- Capture is rate limited to 2 fps, batched under PostHog's size limit, and capped per run. Any
+  failure is swallowed - a recording is never worth failing a run over.
+
 ## Development
 
 ```bash