stata-code 0.9.1__tar.gz → 0.12.0__tar.gz

{stata_code-0.9.1 → stata_code-0.12.0}/CHANGELOG.md

@@ -6,6 +6,273 @@ to semver-major.minor for the result schema (see `SCHEMA.md` §6).
 
 ## [Unreleased]
 
+## 0.12.0 — 2026-07-28
+
+Adds a resident session daemon so the CLI can keep data in memory across
+invocations, and fixes a VS Code regression introduced by 0.11.0's payload
+defaults.
+
+### Added
+
+#### Resident session daemon — state across CLI invocations
+
+The subprocess pool already kept one Stata worker alive per session, but a plain
+`stata-code run` owned that pool and tore it down on exit, so two consecutive CLI
+calls could not share data in memory. The new `daemon` subcommand holds the pool
+in a resident process behind a Unix socket:
+
+```bash
+stata-code run --daemon -e 'sysuse auto, clear' -e 'gen z = price/1000'
+stata-code run --daemon -e 'summarize z'      # new process, z is still loaded
+```
+
+- **`stata-code run --daemon`** routes the run through the daemon, starting it on
+  demand. `--socket` overrides the path. Graph refs are resolved back over the
+  socket, so `--graphs DIR` keeps working.
+- **`stata-code daemon start | stop | restart | status`** for explicit control;
+  `status --json` is machine-readable.
+- **Idle retirement** after 30 minutes by default (`--idle-timeout`, `0` disables)
+  so a forgotten daemon does not hold a Stata license slot.
+- **Unix socket only, never TCP** — mode 0600 inside a mode-0700 directory. The
+  command-safety guard still applies to everything the daemon runs.
+- **Requires the pystata backend.** `--backend console` is stateless batch
+  execution, so combining it with `--daemon` is a hard error rather than a silent
+  no-op.
+- Falls back to a short hashed `/tmp` socket path when the natural location would
+  exceed the platform's ~104-byte `sun_path` limit; every entry point derives the
+  same fallback, so clients still find the daemon.
+
+#### VS Code Outputs panel now lists what every run wrote
+
+The panel read only `log.files.output_paths` — the archived copies inside a run
+bundle — so it stayed empty unless the caller had opted into
+`persist_log_files`. It now also reads `result.outputs`, the files a run wrote
+in its working directory (reported on every run since 0.11.0), shown with their
+size and listed *ahead* of bundle copies, since the working-directory file is
+the one you want to open. Nodes carry an `origin` of `workdir` or `bundle`, and
+an overwritten file says so in its tooltip.
+
+### Fixed
+
+- **VS Code: r()/e() matrices displayed as `0x0` and opened empty.** 0.11.0
+  made `include_results: "scalars"` the server default, which returns each
+  matrix as a `matrix://` stub with `rows` / `cols` elided (the true shape moved
+  to `n_rows` / `n_cols`). The extension's sidebar derives its dimension label
+  from `rows.length` / `cols.length` and renders `matrix.values` directly, so
+  every matrix showed `0x0` and exported an empty table. The extension now
+  requests `include_results: "full"` — an editor is a human surface with no
+  token budget to defend, matching what the Jupyter kernel already did. The
+  argument builder moved to a `vscode`-free `runArgs.ts` so the behaviour is
+  covered by unit tests.
+- Documentation: the error taxonomy was described as "32 kinds" in `README.md`,
+  `README.en.md`, and `SCHEMA.md` while `ErrorKind` has had 34 members (the
+  English README's prose already said 34, contradicting its own table).
+- Documentation: both READMEs listed "Stata 17+" as a hard install requirement,
+  which contradicted the console backend's documented Stata 13+ support.
+- `README.md` was missing the command-line, client-setup, and command-safety
+  sections that `README.en.md` has had since 0.11.0.
+- Documentation: `SKILL.md` and the agent reference library still described the
+  pre-0.11 surface — 18 tools, 31 error kinds, no mention of matrix stubs,
+  background runs, `result.outputs`, or `error.source_file`. Counts in every doc
+  are now checked against the code.
+
+### Internal
+
+- `_collect_returns` no longer takes a `request_id` or creates `matrix://` refs.
+  Nothing passed it — neither the production path nor any test — and ref
+  creation belongs to `_project_returns`, which owns the payload budget.
+- `test_varname_not_found_emits_did_you_mean` failed once and could not be
+  reproduced. Its fuzzy match draws candidates from `dataset.variables`, so a
+  lost `auto.dta` in the shared in-process Stata degrades the suggestion for
+  reasons unrelated to the matching logic. The assertion was **split**, not
+  loosened: a precondition check now reports which of the two it was.
+
+## 0.11.0 — 2026-07-28
+
+An agent-ergonomics release, driven by a report from an agent that used the MCP
+server to write a full empirical paper. Each item below is a cost that agent
+actually paid.
+
+**Minor, not patch:** `include_results` defaults to `"scalars"`, so `r()` / `e()`
+matrices now arrive as `matrix://` stubs instead of inline values. Pass
+`include_results="full"` for the previous shape. `results.estimation` is
+unaffected — and is now *more* accurate, since two silent numeric bugs in it are
+fixed. The Jupyter kernel and the `stata-code run` CLI keep `"full"`.
+
+### Agent-ergonomics pass
+
+#### Result payloads are now bounded (and correct)
+
+- **`include_results`** (`"none" | "scalars" | "full"`, default **`"scalars"`**).
+  A single estimation used to encode the same numbers four times over — `e(b)`,
+  `e(V)`'s row/column labels, `e(beta)`, and `r(table)` — on top of
+  `results.estimation`. Under the new default, scalars and macros stay inline
+  and every matrix becomes a `matrix://` stub carrying only its shape
+  (`n_rows` / `n_cols`); values and labels remain retrievable with
+  `get_matrix(ref)`. A 123-term regression's envelope drops from ~57 KB to
+  ~28 KB. **This is a behaviour change** — pass `include_results="full"` for
+  the previous shape. The Jupyter kernel keeps `"full"`, since a notebook has
+  no token budget to defend.
+- **`include_estimation`** (`"none" | "summary" | "full"`, default `"full"`)
+  and **`max_coefficients`**. `"summary"` keeps the model-level block and
+  diagnostics but drops per-term rows — the right setting for specifications
+  dominated by `i.year i.firm` nuisance terms. `estimation.n_coefficients`
+  always reports the model's true term count and
+  `estimation.coefficients_truncated` flags any cut, so trimmed output can
+  never be mistaken for a smaller model.
+
+#### Fixed: two silent numeric-correctness bugs
+
+- **Standard errors were blanked whenever `e(V)` was returned by reference.**
+  Any model whose VCV exceeded the inline cell cap (roughly >100 terms) came
+  back with `se`, `statistic`, `p_value` and both CI bounds `null` for *every*
+  coefficient, while still reporting `source: "e_b_v"` as though inference had
+  been computed. The estimation builder now resolves `matrix://` refs, so the
+  wire representation no longer degrades the numbers. `e(b)` by reference is
+  resolved too, where it previously suppressed the estimation block entirely.
+- **`r()` did not survive to the next call.** The runner's own probes are all
+  r-class (`graph dir` before a run, `graph export` after), so the canonical
+  two-call pattern — `summarize price`, then `display r(mean)` — read back a
+  missing value. All internal housekeeping now runs inside a Stata
+  `_return hold` / `_return restore` pair. Pre-existing; the new `log query`
+  probe would have widened it to runs with `include_graphs="none"`.
+- **Stata's missing values leaked as `8.988e+307`.** `sfi` returns system
+  missing (`.`) and extended missings (`.a`–`.z`) as ordinary doubles at or
+  above `2^1023`. These reached the wire unconverted in `r()`/`e()` scalars and
+  in every matrix cell — an omitted base level's standard error arrived as a
+  number an agent would format into a table. All numerics now normalize to
+  `null`, matching what SCHEMA.md already required of scalars.
+
+#### Errors inside `do`-files are now located
+
+- `error.line` and `error.context` are resolved **inside the invoked script**.
+  A failing `do "analysis.do"` previously produced `line: null`, empty
+  before/after context, and `message: "end of do-file"` — Stata's epilogue
+  rather than the diagnosis. The transcript parser now skips structural
+  boilerplate (`end of do-file`, `--Break--`), folds `///` continuation
+  fragments back into one logical command, and resolves the failing command
+  against the script on disk. New `error.source_file` names the file that
+  `error.line` indexes into.
+- **A failed run now produces a log.** pystata raises the whole transcript as
+  its exception message and leaves stdout empty, so failures used to come back
+  with no `log.head`, no `log://` ref and nothing for `search_log` to search —
+  exactly when the transcript matters most.
+
+#### A failed run can no longer poison its session
+
+- New **`auto_close_logs`** (default `true`) closes log handles that a *failed*
+  run opened. A script aborting between `log using` and `log close` previously
+  left the handle dangling, and every subsequent run in that session died with
+  r(604) — an error about a different run, with no suggestions attached.
+  Handles opened by earlier runs are left alone; a `log_closed` warning records
+  what was closed.
+- r(604) and r(606) are now the typed `ErrorKind.LOG_STATE` with Stata's
+  official labels and rc-specific fixes (`capture log close _all` for 604,
+  `capture log close` guarding for 606), and `recovery.retriable = true`.
+
+#### Long runs no longer block the caller
+
+- **`timeout_ms` is an advertised `stata_run` argument.** It was enforced but
+  not exposed, so passing it was an input-validation error.
+- **`run_in_background`** returns a `job_id` immediately; poll with the new
+  **`stata_run_status`** tool (`wait_ms` blocks up to 60 s instead of
+  busy-polling) and enumerate with **`list_background_runs`**. Intended for
+  bootstraps, permutation tests and grouped estimation loops.
+- **Fixed: the worker lock was acquired without a deadline**, so a call queued
+  behind a long run blocked indefinitely no matter what `timeout_ms` said.
+  `timeout_ms` now budgets the whole call, queueing included, and contention
+  returns the new `rc=-5` / `error.kind="session_busy"` — distinct from a
+  timeout because nothing was submitted to Stata and the healthy worker is
+  *not* killed.
+
+#### Inline graphs are visible again
+
+- `include_graphs: "inline"` now delivers graph bytes as MCP `ImageContent`
+  blocks instead of base64 inside a JSON string. The old form was unreadable to
+  a vision-capable client and was pure token cost. `graphs[].inline` is cleared
+  from the structured body so the bytes cross the wire once, and
+  `inline_delivered` records the outcome. At most 4 images per response (the
+  rest stay reachable via `get_graph(ref)`, reported as an
+  `inline_graphs_truncated` warning); `pdf` is never sent as an image block.
+
+#### Generated files are reported
+
+- New top-level **`outputs`** lists files a run created or modified in its
+  working directory (`esttab` tables, exported graphs, saved `.dta`), with
+  `path` / `bytes` / `created`. Previously this required
+  `persist_log_files=true` *and* `origin_path` — a dependency that was not
+  obvious from `persist_generated_files` defaulting to `true`, so the field was
+  always `null` in practice. Controlled by the new `track_output_files`
+  (default `true`); working directories above 5,000 files skip detection and
+  say so with an `output_tracking_skipped` warning rather than reporting a
+  partial answer.
+
+### Added
+
+- The VS Code extension now accepts Stata's familiar `Ctrl+D` shortcut
+  (`Cmd+D` on macOS) for running the selection or current line, while retaining
+  `Cmd/Ctrl+Enter`. The extension README also documents how to customize the
+  `stataCode.runSelection` keybinding and resolve conflicts.
+
+## 0.10.0 — 2026-07-24
+
+Surface-coverage and safety additions so every Stata usage scenario — MCP,
+Jupyter, VS Code, and now a plain Bash/terminal — routes through the same typed
+engine, and so an autonomous agent loop cannot escape to the OS. Also broadens
+execution to **Stata 13+ without pystata** (console backend) and adds a
+zero-Python standalone binary and one-click VS Code onboarding. The default
+README is now Chinese (`README.md`); the English README moves to `README.en.md`.
+
+### Added
+
+- **Command-safety guard** (`stata_code.core.policy`). OS-escape / file-deletion
+  commands (`shell`, `winexec`, `erase`, `rm`, `rmdir`, and the `!` shell escape)
+  are screened out of submitted code *before* Stata runs, at both the
+  subprocess-pool boundary and the in-process runner. A blocked run returns
+  `ok=false`, `rc=-4`, `error.kind="policy_blocked"` (a new 32nd `ErrorKind`)
+  without touching Stata. Configurable from the environment so it crosses the
+  worker boundary: `STATA_CODE_COMMAND_POLICY` (`enforce` default / `warn` /
+  `off`), `STATA_CODE_POLICY_ALLOW`, `STATA_CODE_POLICY_BLOCK`. It is a guard
+  rail, not a sandbox.
+- **`stata-code run`** — execute a `.do` file, one or more `-e` snippets, or
+  stdin through the subprocess pool and print the `RunResult` (text summary or
+  `--json`), with `--session`, `--timeout-ms`, `--full-log`, and `--graphs DIR`.
+  Exit code is `0` on success, `1` on a Stata/adapter error. This is the
+  Bash / plain-terminal surface: any agent that can shell out gets the full
+  structured error loop without MCP.
+- **`stata-code setup`** — opt-in, config-mutating counterpart to `doctor`.
+  Writes the `stata-code` MCP server entry into Claude Code (`.mcp.json`),
+  Cursor (`.cursor/mcp.json`), and VS Code (`.vscode/mcp.json`), preserving other
+  servers and backing up any file it overwrites (`--dry-run`, `--python`,
+  `--json`). Codex (TOML) and Claude Desktop are emitted as copy-paste snippets.
+- **Static do-file linting** (`stata_code.core.lint`), exposed as the `lint_do`
+  MCP tool (19th tool) and `stata-code lint`. A Stata-free syntactic check —
+  unbalanced braces, a `program` / `mata` / `python` block with no `end`, a stray
+  `end`, and a dangling `///` — so an agent can catch a class of mistakes before
+  spending a run. Advisory: a clean result is not a correctness guarantee.
+- Public API: `lint_code`, `LintFinding`, `CommandPolicy`, `Violation`,
+  `policy_from_env`.
+- **Console (batch) backend** (`stata_code.core.console`, `Backend.CONSOLE`).
+  Drives the Stata command-line executable in batch mode and parses the log plus
+  a marker-delimited results dump into the same v1.0 `RunResult` — typed
+  `r()`/`e()` scalars and macros, the estimation matrices (`e(b)`/`e(V)`/
+  `r(table)`), the error taxonomy, warnings, and dataset metadata. Works with
+  **Stata 13+** and **no pystata**. Exposed as `stata_code.run_console()`,
+  `stata-code run --backend {auto,pystata,console}`, and a `doctor` check. Stateless
+  per call; graphs not captured yet. Public API adds `run_console`,
+  `console_available`, `find_stata_cli`, `ConsoleNotAvailable`.
+- **Zero-Python standalone binary** — `scripts/build_standalone.py` (PyInstaller)
+  and a CI workflow template (`packaging/standalone.github-workflow.yml`, copy to
+  `.github/workflows/`) build a self-contained `stata-code` executable per OS.
+  Paired with `--backend console` it needs no Python on the target machine.
+- **One-click VS Code onboarding** — when the MCP server is missing, the extension
+  offers to create a workspace `.venv` and install the package; also available as
+  the command-palette entry "Stata: Set Up MCP Server (create .venv)".
+- **Benchmark harness** (`benchmarks/`) — a tool-agnostic scaffold measuring
+  iterations-to-correct and tokens for typed vs. raw-log execution, with an offline
+  mock runner wired end-to-end and CI-covered.
+- Chinese quickstart ([docs/quickstart.zh.md](docs/quickstart.zh.md)).
+
 ## 0.9.1 — 2026-07-22
 
 Quality-hardening pass: correctness fixes from an adversarial code review,