stata-code 0.11.0__tar.gz → 0.12.0__tar.gz

{stata_code-0.11.0 → stata_code-0.12.0}/CHANGELOG.md

@@ -6,6 +6,87 @@ to semver-major.minor for the result schema (see `SCHEMA.md` §6).
 
 ## [Unreleased]
 
+## 0.12.0 — 2026-07-28
+
+Adds a resident session daemon so the CLI can keep data in memory across
+invocations, and fixes a VS Code regression introduced by 0.11.0's payload
+defaults.
+
+### Added
+
+#### Resident session daemon — state across CLI invocations
+
+The subprocess pool already kept one Stata worker alive per session, but a plain
+`stata-code run` owned that pool and tore it down on exit, so two consecutive CLI
+calls could not share data in memory. The new `daemon` subcommand holds the pool
+in a resident process behind a Unix socket:
+
+```bash
+stata-code run --daemon -e 'sysuse auto, clear' -e 'gen z = price/1000'
+stata-code run --daemon -e 'summarize z'      # new process, z is still loaded
+```
+
+- **`stata-code run --daemon`** routes the run through the daemon, starting it on
+  demand. `--socket` overrides the path. Graph refs are resolved back over the
+  socket, so `--graphs DIR` keeps working.
+- **`stata-code daemon start | stop | restart | status`** for explicit control;
+  `status --json` is machine-readable.
+- **Idle retirement** after 30 minutes by default (`--idle-timeout`, `0` disables)
+  so a forgotten daemon does not hold a Stata license slot.
+- **Unix socket only, never TCP** — mode 0600 inside a mode-0700 directory. The
+  command-safety guard still applies to everything the daemon runs.
+- **Requires the pystata backend.** `--backend console` is stateless batch
+  execution, so combining it with `--daemon` is a hard error rather than a silent
+  no-op.
+- Falls back to a short hashed `/tmp` socket path when the natural location would
+  exceed the platform's ~104-byte `sun_path` limit; every entry point derives the
+  same fallback, so clients still find the daemon.
+
+#### VS Code Outputs panel now lists what every run wrote
+
+The panel read only `log.files.output_paths` — the archived copies inside a run
+bundle — so it stayed empty unless the caller had opted into
+`persist_log_files`. It now also reads `result.outputs`, the files a run wrote
+in its working directory (reported on every run since 0.11.0), shown with their
+size and listed *ahead* of bundle copies, since the working-directory file is
+the one you want to open. Nodes carry an `origin` of `workdir` or `bundle`, and
+an overwritten file says so in its tooltip.
+
+### Fixed
+
+- **VS Code: r()/e() matrices displayed as `0x0` and opened empty.** 0.11.0
+  made `include_results: "scalars"` the server default, which returns each
+  matrix as a `matrix://` stub with `rows` / `cols` elided (the true shape moved
+  to `n_rows` / `n_cols`). The extension's sidebar derives its dimension label
+  from `rows.length` / `cols.length` and renders `matrix.values` directly, so
+  every matrix showed `0x0` and exported an empty table. The extension now
+  requests `include_results: "full"` — an editor is a human surface with no
+  token budget to defend, matching what the Jupyter kernel already did. The
+  argument builder moved to a `vscode`-free `runArgs.ts` so the behaviour is
+  covered by unit tests.
+- Documentation: the error taxonomy was described as "32 kinds" in `README.md`,
+  `README.en.md`, and `SCHEMA.md` while `ErrorKind` has had 34 members (the
+  English README's prose already said 34, contradicting its own table).
+- Documentation: both READMEs listed "Stata 17+" as a hard install requirement,
+  which contradicted the console backend's documented Stata 13+ support.
+- `README.md` was missing the command-line, client-setup, and command-safety
+  sections that `README.en.md` has had since 0.11.0.
+- Documentation: `SKILL.md` and the agent reference library still described the
+  pre-0.11 surface — 18 tools, 31 error kinds, no mention of matrix stubs,
+  background runs, `result.outputs`, or `error.source_file`. Counts in every doc
+  are now checked against the code.
+
+### Internal
+
+- `_collect_returns` no longer takes a `request_id` or creates `matrix://` refs.
+  Nothing passed it — neither the production path nor any test — and ref
+  creation belongs to `_project_returns`, which owns the payload budget.
+- `test_varname_not_found_emits_did_you_mean` failed once and could not be
+  reproduced. Its fuzzy match draws candidates from `dataset.variables`, so a
+  lost `auto.dta` in the shared in-process Stata degrades the suggestion for
+  reasons unrelated to the matching logic. The assertion was **split**, not
+  loosened: a precondition check now reports which of the two it was.
+
 ## 0.11.0 — 2026-07-28
 
 An agent-ergonomics release, driven by a report from an agent that used the MCP

{stata_code-0.11.0 → stata_code-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: stata-code
-Version: 0.11.0
+Version: 0.12.0
 Summary: Agent-native Stata bridge — one core, multiple frontends (MCP, Jupyter, VSCode)
 Project-URL: Homepage, https://github.com/brycewang-stanford/stata-code
 Project-URL: Repository, https://github.com/brycewang-stanford/stata-code
@@ -120,7 +120,7 @@ Then just ask:
 
 A fourth frontend, a **plain-terminal CLI** (`stata-code run` / `lint` / `setup`), gives any agent that can shell out — or a bare terminal — the same typed `RunResult`. And the core runs two backends: **pystata** (Stata 17+, in-memory sessions) or a **console backend** (Stata 13+ in batch mode, no pystata) — both returning the identical schema.
 
-**Status: v0.10 (July 2026)** — the core, MCP server, Jupyter kernel, VS Code extension, and CLI work end-to-end against Stata 18 MP; the console backend broadens coverage to Stata 13+ without pystata. The test suite covers schema, runner, console parser, MCP, kernel, notebook, run-index, subprocess-pool, command policy, linter, and VS Code modules; CI also checks linting, type safety, schema generation, package metadata, and VSIX packaging. License: **MIT**.
+**Status: v0.12 (July 2026)** — the core, MCP server, Jupyter kernel, VS Code extension, and CLI work end-to-end against Stata 18 MP; the console backend broadens coverage to Stata 13+ without pystata. v0.12 adds a resident session daemon so consecutive CLI runs share data in memory, on top of v0.11's agent-ergonomics work: bounded result payloads, background execution for multi-minute jobs, error localization inside `do`-files, automatic log-handle cleanup, and generated-file reporting — see the [changelog](CHANGELOG.md). The test suite covers schema, runner, console parser, MCP, kernel, notebook, run-index, subprocess-pool, command policy, linter, and VS Code modules; CI also checks linting, type safety, schema generation, package metadata, and VSIX packaging. License: **MIT**.
 
 Four workflows the current tree explicitly supports for end users and agents:
 
@@ -158,7 +158,9 @@ For the project's clean-room policy around AGPL/GPL Stata projects, see [LICENSE
 
 ## Install
 
-Requirements: **Stata 17+** (with `pystata` shipped by Stata) and **Python 3.10+**.
+Requirements: **Python 3.10+**, plus either **Stata 17+** (using the `pystata`
+shipped by Stata, for in-memory sessions) or **Stata 13+** (via the console
+backend, which needs no `pystata` and is stateless).
 
 ```bash
 # from PyPI
@@ -264,6 +266,54 @@ CI workflow template at
 bundles the runtime and needs no Python install. Paired with `--backend console`,
 it is a fully Python-free path to typed Stata results.
 
+### The Session Daemon (State Across Invocations)
+
+By default every `stata-code run` starts a fresh process, so in-memory data dies
+with it. The `daemon` subcommand moves the subprocess pool into a resident
+process behind a Unix socket, and consecutive `run --daemon` calls then land in
+the **same Stata session**:
+
+```bash
+stata-code run --daemon -e 'sysuse auto, clear' -e 'gen z = price/1000'
+stata-code run --daemon -e 'summarize z'       # new process, z is still there
+stata-code run --daemon -e 'regress price mpg'
+stata-code run --daemon -e 'display e(r2)'     # e() from the previous process too
+```
+
+The first `run --daemon` starts the daemon on demand — no manual step needed. You
+can also drive it explicitly:
+
+```bash
+stata-code daemon start                    # detached (--foreground to stay attached)
+stata-code daemon status                   # pid, uptime, live sessions
+stata-code daemon status --json
+stata-code daemon stop
+stata-code daemon restart
+stata-code daemon start --idle-timeout 0   # 0 disables idle retirement
+```
+
+Worth knowing:
+
+- **Session isolation is unchanged.** `--session other` is still a separate Stata
+  frame / worker inside the daemon; sessions cannot see each other's data.
+- **It retires itself after 30 idle minutes** so a forgotten daemon does not sit on
+  a Stata license slot. Tune with `--idle-timeout`; `0` disables it.
+- **Unix socket only, never TCP.** The daemon executes arbitrary Stata code, so it
+  is deliberately reachable only through a mode-0600 socket inside a mode-0700
+  directory. The [command-safety](#command-safety) guard still applies.
+- **Requires the pystata backend.** `--backend console` is stateless batch
+  execution by design, so pairing it with a daemon is an error rather than a
+  silent no-op.
+- **Long socket paths degrade gracefully.** `sun_path` caps out around 104 bytes;
+  when the natural location is longer (a deep home directory, a long
+  `XDG_RUNTIME_DIR`), the socket moves to a short `/tmp` path derived from a hash
+  of the intended one. Every entry point derives it the same way, so clients still
+  find the daemon.
+
+Plain `stata-code run`, `run --daemon`, and the MCP server all drive the same
+engine and return the same `RunResult` schema — they differ only in how long the
+Stata session lives.
+
 ### One-Command Client Setup
 
 `stata-code setup` writes the MCP server entry into a client's config — the
@@ -513,11 +563,14 @@ The extension still requires the MCP extra on your system Python (`pip install "
 
 ## Token-Economy Defaults
 
-A typical `stata_run` response is about **10x smaller** than servers that dump logs and images directly. Three design choices drive this:
+A typical `stata_run` response is about **10x smaller** than servers that dump logs and images directly. Four design choices drive this:
+
+1. **Logs return `head` + `tail` + `ref`** by default. Full logs are fetched on demand via `get_log(ref)`, or grepped in place with `search_log(ref, pattern)`. A Stata regression log can be about 6,000 tokens; `stata-code` returns about 600 by default.
+2. **Graphs return refs, not inline base64**. A 30 KB PNG can become about 50,000 base64 tokens; returning a ref avoids that unless the agent actually needs the bytes. When they are requested inline, they come back as real MCP image content blocks — viewable by a vision-capable client, rather than base64 buried in a JSON string that costs tokens and shows nothing.
+3. **Each estimation is described once.** By default (`include_results: "scalars"`) `r()` / `e()` scalars and macros are inline, while every matrix becomes a `matrix://` stub carrying just its shape. Without this, one regression ships the same numbers four times — `e(b)`, `e(V)`'s label lists, `e(beta)`, `r(table)` — on top of `results.estimation`, which already holds the typed coefficient table. A 123-term regression drops from ~57 KB to ~28 KB. Raw values stay one `get_matrix(ref)` away, and `include_results: "full"` restores the old shape.
+4. **Errors are typed**. Agents can check `err.kind == "varname_not_found"` instead of regex-parsing English logs.
 
-1. **Logs return `head` + `tail` + `ref`** by default. Full logs are fetched on demand via `get_log(ref)`. A Stata regression log can be about 6,000 tokens; `stata-code` returns about 600 by default.
-2. **Graphs return refs, not inline base64**. A 30 KB PNG can become about 50,000 base64 tokens; returning a ref avoids that unless the agent actually needs the bytes.
-3. **Errors are typed**. Agents can check `err.kind == "varname_not_found"` instead of regex-parsing English logs.
+Two further knobs matter for fixed-effect-heavy work: `include_estimation: "summary"` keeps the model-level block and drops per-term rows, and `max_coefficients` caps the table. Either way `estimation.n_coefficients` reports the model's true size and `coefficients_truncated` flags the cut, so a trimmed table is never mistaken for a smaller model.
 
 For example, a misspelled variable returns a structured error:
 
@@ -529,6 +582,7 @@ For example, a misspelled variable returns a structured error:
     "kind": "varname_not_found",
     "varname": "mpgg",
     "line": 3,
+    "source_file": null,
     "context": {
       "before": ["use auto"],
       "failing": "summarize mpgg",
@@ -536,11 +590,19 @@ For example, a misspelled variable returns a structured error:
     },
     "suggestions": [
       {"action": "Did you mean `mpg`?", "command": "describe"}
-    ]
+    ],
+    "recovery": {
+      "category": "user_code",
+      "retriable": false,
+      "needs_code_change": true,
+      "needs_user_input": false
+    }
   }
 }
 ```
 
+When the failing command lives inside a script you invoked (`do "analysis.do"`), `line` and `context` point *inside that script* and `source_file` names it — so an agent does not have to re-read the file to find the offending line.
+
 The full schema is in [SCHEMA.md](SCHEMA.md).
 
 ---
@@ -576,7 +638,7 @@ stata_code/
 | Standalone MCP | ✓ | ✓ | bundled with VS Code | — (bespoke HTTP + copy-paste) | — |
 | Jupyter kernel | ✓ | — | — | — | ✓ |
 | Unified result schema | ✓ ([SCHEMA.md](SCHEMA.md)) | per-tool | per-tool | raw log to the agent | per-tool |
-| Typed errors + suggestions | ✓ (32 kinds) | — | — | — | — |
+| Typed errors + suggestions | ✓ (34 kinds) | — | — | — | — |
 | Token-economy defaults | ✓ (log refs, graph refs) | — | — | — | — |
 | Command-safety guard | ✓ (`shell`/`erase`/`rmdir`/`!`) | ✓ (27-rule) | — | — | — |
 | Bash / plain-terminal CLI | ✓ (`stata-code run`) | ✓ | — | — | — |
@@ -590,7 +652,7 @@ stata_code/
 
 `stata-code` owns the **agent-native, typed-contract** lane: one structured
 `RunResult` schema across MCP, Jupyter, VS Code, and a plain-terminal CLI; a
-32-kind error taxonomy with recovery contracts; token-economy refs; and now a
+34-kind error taxonomy with recovery contracts; token-economy refs; and now a
 console backend (Stata 13+) plus a zero-Python binary. Editor-first tools like
 `stata-all-in-one` lead on human IDE polish and hand the agent raw log text;
 `stata-code` matches their onboarding/version reach while keeping the typed
@@ -598,6 +660,49 @@ execution contract they don't have. See
 [docs/competitive-landscape.md](docs/competitive-landscape.md) for the full
 teardown.
 
+### Versus calling the Stata CLI directly
+
+To be clear up front: **you can drive Stata from a shell with no tooling at all.**
+Most Stata installs ship a command-line executable, and for one-shot execution it
+is perfectly good — no reason to add a dependency:
+
+```bash
+/Applications/Stata/StataMP.app/Contents/MacOS/stata-mp -b do analysis.do
+printf 'sysuse auto, clear\nsummarize mpg\nexit, clear\n' | stata-mp -q
+```
+
+What's worth knowing is where that path breaks down. Both columns below drive the
+**same Stata binary and parse the same log** (`stata-code` via `--backend console`);
+the only difference is the wrapper around it:
+
+| | plain `stata-mp -b` | `stata-code run` |
+| --- | --- | --- |
+| Exit code on error | **`0`** — fails silently | `1` |
+| How you learn it failed | grep the log for `r(111);` yourself | `ok=False`, `rc=111` |
+| Error classification | none | `[varname_not_found]` (34 kinds) |
+| Recovery hint | none | "Run `describe` to list available variables" |
+| Getting a coefficient | scrape the aligned ASCII table | `e.scalars["r2"] → 0.6515312529087511` (float) |
+| State across calls | no | `run --daemon` (see below) |
+
+The first row is the one that bites: Stata's batch mode **returns exit code 0 even
+when the code errors out**. In an automated loop, an agent sees `exit 0`, moves on,
+and treats a failed run's output as valid.
+
+Rule of thumb:
+
+- **One-shot execution** ("run this do-file and show me the output") — the plain
+  CLI is fine; don't over-engineer it.
+- **Scripts / CI** (anything that branches on success) — use `stata-code run` and
+  get a trustworthy signal for near-zero cost.
+- **Iterative analysis** (many rounds, reading intermediate results, deciding the
+  next step from `e()`) — use `run --daemon` or MCP; otherwise statelessness and
+  text-scraping compound round over round.
+
+> Without `--daemon`, `stata-code run` **starts a fresh process per invocation**:
+> multiple `-e` snippets in one call share a session, but in-memory data is gone
+> once the process exits. With `--daemon` the data stays in a resident process —
+> see [the session daemon](#the-session-daemon-state-across-invocations).
+
 ---
 
 ## Roadmap
@@ -607,11 +712,12 @@ teardown.
 - v1.0 result schema ([SCHEMA.md](SCHEMA.md))
 - `pystata`-based runner with native-typed `r()`, `e()`, and matrices
 - Multi-session via Stata frames (`session_id` accepts `[A-Za-z0-9_-]+`; ids such as `model-a` are mapped to private legal frame names internally while the public id is echoed back)
+- Resident session daemon (`stata-code daemon`, `run --daemon`) so CLI invocations share one live Stata session over a mode-0600 Unix socket, with idle retirement
 - Per-line error attribution: line number, context, commands_executed
 - Graph capture: `png` / `svg` / `pdf` with ref store and source-command attribution
 - Log truncation with ref store
 - Warning extraction: 5 categories + generic notes
-- 32-kind error taxonomy with canonical suggestions
+- 34-kind error taxonomy with canonical suggestions and a machine-readable `recovery` verdict (retriable / needs-code-change / needs-user-input)
 - MCP server: 21 tools, including notebook navigation / search / atomic edits, the run-bundle index (`list_runs`), log grep (`search_log`), dataset inspection (`inspect_data`), static linting (`lint_do`), and package installation (`install_package`)
 - Command-safety guard: OS-escape / file-deletion commands (`shell`, `winexec`, `erase`, `rm`, `rmdir`, `!`) are blocked before Stata runs; configurable via `STATA_CODE_COMMAND_POLICY` / `STATA_CODE_POLICY_ALLOW` / `STATA_CODE_POLICY_BLOCK`
 - Bash / plain-terminal surface: `stata-code run` (a `.do` file, `-e` snippets, or stdin) prints the same structured `RunResult` any agent that can shell out can consume; `stata-code lint` runs the linter; `stata-code setup` writes MCP client configs
@@ -619,8 +725,12 @@ teardown.
 - Zero-Python standalone binary ([`scripts/build_standalone.py`](scripts/build_standalone.py) + CI workflow template [`packaging/standalone.github-workflow.yml`](packaging/standalone.github-workflow.yml)); with `--backend console` it is a fully Python-free path to typed results
 - One-click VS Code onboarding: the extension offers to create a workspace `.venv` and install the server (command palette: “Stata: Set Up MCP Server”)
 - Jupyter kernel: rewired to the v1.0 pipeline, kernel logos bundled
-- Matrix size cap + `get_matrix(ref)` for large matrices (>10k cells)
-- Subprocess-backed hard timeout and cancellation for the public Python API and MCP server: `timeout_ms`, `cancel(session_id)`, and MCP `cancel_session`
+- Result-payload budget: `include_results` (matrices become `matrix://` stubs by default), `include_estimation`, and `max_coefficients`, with `get_matrix(ref)` to pull raw values on demand
+- Subprocess-backed hard timeout and cancellation for the public Python API and MCP server: `timeout_ms` (now an advertised `stata_run` argument, and it budgets queueing too — contention returns `rc=-5` / `session_busy` instead of blocking), `cancel(session_id)`, and MCP `cancel_session`
+- Background execution for multi-minute jobs: `run_in_background` returns a job id; `stata_run_status` (with a bounded `wait_ms`) and `list_background_runs` poll it
+- Error localization inside `do`/`run` scripts: `error.line` + `error.context` resolve within the invoked file and `error.source_file` names it; a failed run still carries a full searchable log
+- Log-handle hygiene: handles leaked by a failed run are closed automatically (`auto_close_logs`), so an aborted script cannot make every later run in that session fail with r(604)
+- Generated-file reporting: `result.outputs` lists the tables, exports and datasets each run wrote, independent of the run-bundle options
 - Per-cell repair loop on `.ipynb` via `notebook_outline` / `notebook_get_cell` / `notebook_edit_cell` with optimistic-concurrency `expected_source` guards and `origin_cell_id` echo on `RunResult`
 - Persistent run bundles + `list_runs` query over `manifest.json` files (filter by cell / origin / session / since / ok; page with limit / offset)
 - Read-only `stata-code doctor` / `verify` diagnostics for package version,
@@ -635,7 +745,7 @@ teardown.
 
 ### Next Up
 
-- Streaming / progress for long runs (`log.complete:false`, incremental log lines) so 20-minute `boottest` / `csdid` jobs report before they finish
+- Streaming / incremental progress for long runs (`log.complete:false`, partial log lines). v0.11's `run_in_background` already unblocks the caller for 20-minute `boottest` / `csdid` jobs, but a running job still reports nothing until it finishes
 - Hard timeout / cancellation for the Jupyter kernel (move it from the direct in-process runner to the subprocess pool, or an equivalent)
 - Console backend: graph capture and richer matrix coverage (values currently materialized for the estimation matrices; state is per-call)
 - Human IDE polish to match editor-first tools: inline graph rendering + DPI export, a scalable data viewer, and an optional "attach to a running Stata" backend

{stata_code-0.11.0 → stata_code-0.12.0}/README.en.md

@@ -81,7 +81,7 @@ Then just ask:
 
 A fourth frontend, a **plain-terminal CLI** (`stata-code run` / `lint` / `setup`), gives any agent that can shell out — or a bare terminal — the same typed `RunResult`. And the core runs two backends: **pystata** (Stata 17+, in-memory sessions) or a **console backend** (Stata 13+ in batch mode, no pystata) — both returning the identical schema.
 
-**Status: v0.10 (July 2026)** — the core, MCP server, Jupyter kernel, VS Code extension, and CLI work end-to-end against Stata 18 MP; the console backend broadens coverage to Stata 13+ without pystata. The test suite covers schema, runner, console parser, MCP, kernel, notebook, run-index, subprocess-pool, command policy, linter, and VS Code modules; CI also checks linting, type safety, schema generation, package metadata, and VSIX packaging. License: **MIT**.
+**Status: v0.12 (July 2026)** — the core, MCP server, Jupyter kernel, VS Code extension, and CLI work end-to-end against Stata 18 MP; the console backend broadens coverage to Stata 13+ without pystata. v0.12 adds a resident session daemon so consecutive CLI runs share data in memory, on top of v0.11's agent-ergonomics work: bounded result payloads, background execution for multi-minute jobs, error localization inside `do`-files, automatic log-handle cleanup, and generated-file reporting — see the [changelog](CHANGELOG.md). The test suite covers schema, runner, console parser, MCP, kernel, notebook, run-index, subprocess-pool, command policy, linter, and VS Code modules; CI also checks linting, type safety, schema generation, package metadata, and VSIX packaging. License: **MIT**.
 
 Four workflows the current tree explicitly supports for end users and agents:
 
@@ -119,7 +119,9 @@ For the project's clean-room policy around AGPL/GPL Stata projects, see [LICENSE
 
 ## Install
 
-Requirements: **Stata 17+** (with `pystata` shipped by Stata) and **Python 3.10+**.
+Requirements: **Python 3.10+**, plus either **Stata 17+** (using the `pystata`
+shipped by Stata, for in-memory sessions) or **Stata 13+** (via the console
+backend, which needs no `pystata` and is stateless).
 
 ```bash
 # from PyPI
@@ -225,6 +227,54 @@ CI workflow template at
 bundles the runtime and needs no Python install. Paired with `--backend console`,
 it is a fully Python-free path to typed Stata results.
 
+### The Session Daemon (State Across Invocations)
+
+By default every `stata-code run` starts a fresh process, so in-memory data dies
+with it. The `daemon` subcommand moves the subprocess pool into a resident
+process behind a Unix socket, and consecutive `run --daemon` calls then land in
+the **same Stata session**:
+
+```bash
+stata-code run --daemon -e 'sysuse auto, clear' -e 'gen z = price/1000'
+stata-code run --daemon -e 'summarize z'       # new process, z is still there
+stata-code run --daemon -e 'regress price mpg'
+stata-code run --daemon -e 'display e(r2)'     # e() from the previous process too
+```
+
+The first `run --daemon` starts the daemon on demand — no manual step needed. You
+can also drive it explicitly:
+
+```bash
+stata-code daemon start                    # detached (--foreground to stay attached)
+stata-code daemon status                   # pid, uptime, live sessions
+stata-code daemon status --json
+stata-code daemon stop
+stata-code daemon restart
+stata-code daemon start --idle-timeout 0   # 0 disables idle retirement
+```
+
+Worth knowing:
+
+- **Session isolation is unchanged.** `--session other` is still a separate Stata
+  frame / worker inside the daemon; sessions cannot see each other's data.
+- **It retires itself after 30 idle minutes** so a forgotten daemon does not sit on
+  a Stata license slot. Tune with `--idle-timeout`; `0` disables it.
+- **Unix socket only, never TCP.** The daemon executes arbitrary Stata code, so it
+  is deliberately reachable only through a mode-0600 socket inside a mode-0700
+  directory. The [command-safety](#command-safety) guard still applies.
+- **Requires the pystata backend.** `--backend console` is stateless batch
+  execution by design, so pairing it with a daemon is an error rather than a
+  silent no-op.
+- **Long socket paths degrade gracefully.** `sun_path` caps out around 104 bytes;
+  when the natural location is longer (a deep home directory, a long
+  `XDG_RUNTIME_DIR`), the socket moves to a short `/tmp` path derived from a hash
+  of the intended one. Every entry point derives it the same way, so clients still
+  find the daemon.
+
+Plain `stata-code run`, `run --daemon`, and the MCP server all drive the same
+engine and return the same `RunResult` schema — they differ only in how long the
+Stata session lives.
+
 ### One-Command Client Setup
 
 `stata-code setup` writes the MCP server entry into a client's config — the
@@ -474,11 +524,14 @@ The extension still requires the MCP extra on your system Python (`pip install "
 
 ## Token-Economy Defaults
 
-A typical `stata_run` response is about **10x smaller** than servers that dump logs and images directly. Three design choices drive this:
+A typical `stata_run` response is about **10x smaller** than servers that dump logs and images directly. Four design choices drive this:
+
+1. **Logs return `head` + `tail` + `ref`** by default. Full logs are fetched on demand via `get_log(ref)`, or grepped in place with `search_log(ref, pattern)`. A Stata regression log can be about 6,000 tokens; `stata-code` returns about 600 by default.
+2. **Graphs return refs, not inline base64**. A 30 KB PNG can become about 50,000 base64 tokens; returning a ref avoids that unless the agent actually needs the bytes. When they are requested inline, they come back as real MCP image content blocks — viewable by a vision-capable client, rather than base64 buried in a JSON string that costs tokens and shows nothing.
+3. **Each estimation is described once.** By default (`include_results: "scalars"`) `r()` / `e()` scalars and macros are inline, while every matrix becomes a `matrix://` stub carrying just its shape. Without this, one regression ships the same numbers four times — `e(b)`, `e(V)`'s label lists, `e(beta)`, `r(table)` — on top of `results.estimation`, which already holds the typed coefficient table. A 123-term regression drops from ~57 KB to ~28 KB. Raw values stay one `get_matrix(ref)` away, and `include_results: "full"` restores the old shape.
+4. **Errors are typed**. Agents can check `err.kind == "varname_not_found"` instead of regex-parsing English logs.
 
-1. **Logs return `head` + `tail` + `ref`** by default. Full logs are fetched on demand via `get_log(ref)`. A Stata regression log can be about 6,000 tokens; `stata-code` returns about 600 by default.
-2. **Graphs return refs, not inline base64**. A 30 KB PNG can become about 50,000 base64 tokens; returning a ref avoids that unless the agent actually needs the bytes.
-3. **Errors are typed**. Agents can check `err.kind == "varname_not_found"` instead of regex-parsing English logs.
+Two further knobs matter for fixed-effect-heavy work: `include_estimation: "summary"` keeps the model-level block and drops per-term rows, and `max_coefficients` caps the table. Either way `estimation.n_coefficients` reports the model's true size and `coefficients_truncated` flags the cut, so a trimmed table is never mistaken for a smaller model.
 
 For example, a misspelled variable returns a structured error:
 
@@ -490,6 +543,7 @@ For example, a misspelled variable returns a structured error:
     "kind": "varname_not_found",
     "varname": "mpgg",
     "line": 3,
+    "source_file": null,
     "context": {
       "before": ["use auto"],
       "failing": "summarize mpgg",
@@ -497,11 +551,19 @@ For example, a misspelled variable returns a structured error:
     },
     "suggestions": [
       {"action": "Did you mean `mpg`?", "command": "describe"}
-    ]
+    ],
+    "recovery": {
+      "category": "user_code",
+      "retriable": false,
+      "needs_code_change": true,
+      "needs_user_input": false
+    }
   }
 }
 ```
 
+When the failing command lives inside a script you invoked (`do "analysis.do"`), `line` and `context` point *inside that script* and `source_file` names it — so an agent does not have to re-read the file to find the offending line.
+
 The full schema is in [SCHEMA.md](SCHEMA.md).
 
 ---
@@ -537,7 +599,7 @@ stata_code/
 | Standalone MCP | ✓ | ✓ | bundled with VS Code | — (bespoke HTTP + copy-paste) | — |
 | Jupyter kernel | ✓ | — | — | — | ✓ |
 | Unified result schema | ✓ ([SCHEMA.md](SCHEMA.md)) | per-tool | per-tool | raw log to the agent | per-tool |
-| Typed errors + suggestions | ✓ (32 kinds) | — | — | — | — |
+| Typed errors + suggestions | ✓ (34 kinds) | — | — | — | — |
 | Token-economy defaults | ✓ (log refs, graph refs) | — | — | — | — |
 | Command-safety guard | ✓ (`shell`/`erase`/`rmdir`/`!`) | ✓ (27-rule) | — | — | — |
 | Bash / plain-terminal CLI | ✓ (`stata-code run`) | ✓ | — | — | — |
@@ -551,7 +613,7 @@ stata_code/
 
 `stata-code` owns the **agent-native, typed-contract** lane: one structured
 `RunResult` schema across MCP, Jupyter, VS Code, and a plain-terminal CLI; a
-32-kind error taxonomy with recovery contracts; token-economy refs; and now a
+34-kind error taxonomy with recovery contracts; token-economy refs; and now a
 console backend (Stata 13+) plus a zero-Python binary. Editor-first tools like
 `stata-all-in-one` lead on human IDE polish and hand the agent raw log text;
 `stata-code` matches their onboarding/version reach while keeping the typed
@@ -559,6 +621,49 @@ execution contract they don't have. See
 [docs/competitive-landscape.md](docs/competitive-landscape.md) for the full
 teardown.
 
+### Versus calling the Stata CLI directly
+
+To be clear up front: **you can drive Stata from a shell with no tooling at all.**
+Most Stata installs ship a command-line executable, and for one-shot execution it
+is perfectly good — no reason to add a dependency:
+
+```bash
+/Applications/Stata/StataMP.app/Contents/MacOS/stata-mp -b do analysis.do
+printf 'sysuse auto, clear\nsummarize mpg\nexit, clear\n' | stata-mp -q
+```
+
+What's worth knowing is where that path breaks down. Both columns below drive the
+**same Stata binary and parse the same log** (`stata-code` via `--backend console`);
+the only difference is the wrapper around it:
+
+| | plain `stata-mp -b` | `stata-code run` |
+| --- | --- | --- |
+| Exit code on error | **`0`** — fails silently | `1` |
+| How you learn it failed | grep the log for `r(111);` yourself | `ok=False`, `rc=111` |
+| Error classification | none | `[varname_not_found]` (34 kinds) |
+| Recovery hint | none | "Run `describe` to list available variables" |
+| Getting a coefficient | scrape the aligned ASCII table | `e.scalars["r2"] → 0.6515312529087511` (float) |
+| State across calls | no | `run --daemon` (see below) |
+
+The first row is the one that bites: Stata's batch mode **returns exit code 0 even
+when the code errors out**. In an automated loop, an agent sees `exit 0`, moves on,
+and treats a failed run's output as valid.
+
+Rule of thumb:
+
+- **One-shot execution** ("run this do-file and show me the output") — the plain
+  CLI is fine; don't over-engineer it.
+- **Scripts / CI** (anything that branches on success) — use `stata-code run` and
+  get a trustworthy signal for near-zero cost.
+- **Iterative analysis** (many rounds, reading intermediate results, deciding the
+  next step from `e()`) — use `run --daemon` or MCP; otherwise statelessness and
+  text-scraping compound round over round.
+
+> Without `--daemon`, `stata-code run` **starts a fresh process per invocation**:
+> multiple `-e` snippets in one call share a session, but in-memory data is gone
+> once the process exits. With `--daemon` the data stays in a resident process —
+> see [the session daemon](#the-session-daemon-state-across-invocations).
+
 ---
 
 ## Roadmap
@@ -568,11 +673,12 @@ teardown.
 - v1.0 result schema ([SCHEMA.md](SCHEMA.md))
 - `pystata`-based runner with native-typed `r()`, `e()`, and matrices
 - Multi-session via Stata frames (`session_id` accepts `[A-Za-z0-9_-]+`; ids such as `model-a` are mapped to private legal frame names internally while the public id is echoed back)
+- Resident session daemon (`stata-code daemon`, `run --daemon`) so CLI invocations share one live Stata session over a mode-0600 Unix socket, with idle retirement
 - Per-line error attribution: line number, context, commands_executed
 - Graph capture: `png` / `svg` / `pdf` with ref store and source-command attribution
 - Log truncation with ref store
 - Warning extraction: 5 categories + generic notes
-- 32-kind error taxonomy with canonical suggestions
+- 34-kind error taxonomy with canonical suggestions and a machine-readable `recovery` verdict (retriable / needs-code-change / needs-user-input)
 - MCP server: 21 tools, including notebook navigation / search / atomic edits, the run-bundle index (`list_runs`), log grep (`search_log`), dataset inspection (`inspect_data`), static linting (`lint_do`), and package installation (`install_package`)
 - Command-safety guard: OS-escape / file-deletion commands (`shell`, `winexec`, `erase`, `rm`, `rmdir`, `!`) are blocked before Stata runs; configurable via `STATA_CODE_COMMAND_POLICY` / `STATA_CODE_POLICY_ALLOW` / `STATA_CODE_POLICY_BLOCK`
 - Bash / plain-terminal surface: `stata-code run` (a `.do` file, `-e` snippets, or stdin) prints the same structured `RunResult` any agent that can shell out can consume; `stata-code lint` runs the linter; `stata-code setup` writes MCP client configs
@@ -580,8 +686,12 @@ teardown.
 - Zero-Python standalone binary ([`scripts/build_standalone.py`](scripts/build_standalone.py) + CI workflow template [`packaging/standalone.github-workflow.yml`](packaging/standalone.github-workflow.yml)); with `--backend console` it is a fully Python-free path to typed results
 - One-click VS Code onboarding: the extension offers to create a workspace `.venv` and install the server (command palette: “Stata: Set Up MCP Server”)
 - Jupyter kernel: rewired to the v1.0 pipeline, kernel logos bundled
-- Matrix size cap + `get_matrix(ref)` for large matrices (>10k cells)
-- Subprocess-backed hard timeout and cancellation for the public Python API and MCP server: `timeout_ms`, `cancel(session_id)`, and MCP `cancel_session`
+- Result-payload budget: `include_results` (matrices become `matrix://` stubs by default), `include_estimation`, and `max_coefficients`, with `get_matrix(ref)` to pull raw values on demand
+- Subprocess-backed hard timeout and cancellation for the public Python API and MCP server: `timeout_ms` (now an advertised `stata_run` argument, and it budgets queueing too — contention returns `rc=-5` / `session_busy` instead of blocking), `cancel(session_id)`, and MCP `cancel_session`
+- Background execution for multi-minute jobs: `run_in_background` returns a job id; `stata_run_status` (with a bounded `wait_ms`) and `list_background_runs` poll it
+- Error localization inside `do`/`run` scripts: `error.line` + `error.context` resolve within the invoked file and `error.source_file` names it; a failed run still carries a full searchable log
+- Log-handle hygiene: handles leaked by a failed run are closed automatically (`auto_close_logs`), so an aborted script cannot make every later run in that session fail with r(604)
+- Generated-file reporting: `result.outputs` lists the tables, exports and datasets each run wrote, independent of the run-bundle options
 - Per-cell repair loop on `.ipynb` via `notebook_outline` / `notebook_get_cell` / `notebook_edit_cell` with optimistic-concurrency `expected_source` guards and `origin_cell_id` echo on `RunResult`
 - Persistent run bundles + `list_runs` query over `manifest.json` files (filter by cell / origin / session / since / ok; page with limit / offset)
 - Read-only `stata-code doctor` / `verify` diagnostics for package version,
@@ -596,7 +706,7 @@ teardown.
 
 ### Next Up
 
-- Streaming / progress for long runs (`log.complete:false`, incremental log lines) so 20-minute `boottest` / `csdid` jobs report before they finish
+- Streaming / incremental progress for long runs (`log.complete:false`, partial log lines). v0.11's `run_in_background` already unblocks the caller for 20-minute `boottest` / `csdid` jobs, but a running job still reports nothing until it finishes
 - Hard timeout / cancellation for the Jupyter kernel (move it from the direct in-process runner to the subprocess pool, or an equivalent)
 - Console backend: graph capture and richer matrix coverage (values currently materialized for the estimation matrices; state is per-call)
 - Human IDE polish to match editor-first tools: inline graph rendering + DPI export, a scalable data viewer, and an optional "attach to a running Stata" backend