esp-modkit 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 0a70a9625b2c9f54539ee7e5a561275fd9020e90996112022a40406484278805
+  data.tar.gz: 8d9342e2c02d123e7247e81198a8d01e46785b94871685716bb7898c97a76a02
+SHA512:
+  metadata.gz: 1bde927727d1388c8c41cebff7640739cfb807809e68768c33982823b982b6da0fc05832c0dd3cf289a18317f2199295dc6a8e76175246a189543636a5b966a7
+  data.tar.gz: 30c973f51eaa4693f6cb12cfecff8517a30bb58c92c49862d26a3d7dd6da6b2c200b03e6d2d870c8b77c925e64d15cd106a93d5b48496b44c738721b31f50c7e

data/CHANGELOG.md

@@ -0,0 +1,35 @@
+# Changelog
+
+All notable changes to `esp-modkit` are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/), and the project aims to
+adhere to [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+### Added
+- **Packaged as the `esp-modkit` gem.** `gem install esp-modkit` puts the
+  `esp` command on PATH — no toolchain clone required. The gem ships the
+  library, locale catalogues, generated reference docs, and the narrative
+  guides. (step 24, Half A)
+- `esp doctor` — reports Ruby/gem version, whether `tes3conv` is on PATH,
+  and whether the vanilla references index exists, so a fresh install can
+  self-check its prerequisites.
+
+### Changed
+- The gem's executable (`exe/esp`) is plain Ruby that loads via RubyGems;
+  the repo-local `bin/esp` bash launcher (rbenv + Bundler pinning) stays for
+  in-repo development only.
+
+### Removed
+- `bin/mw` — the renamed-to-`esp` deprecation shim from step 22.5. Anyone
+  still invoking `mw` must switch to `esp`. AI clients with a stale
+  `.mcp.json` should re-run `esp mcp install`.
+
+### Prerequisites
+- **tes3conv is not bundled** (it is an external Rust binary, not on
+  Homebrew). Install it from
+  https://github.com/Greatness7/tes3conv/releases or via
+  `cargo install --git https://github.com/Greatness7/tes3conv`, then run
+  `esp doctor` to confirm. The Homebrew tap installs it for you.
+- **Native extensions** (`sqlite3`, `zstd-ruby`) compile on install — a C
+  toolchain is required.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Corey Ellis
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,117 @@
+# esp — author Morrowind plugins as source code
+
+`esp` is a Ruby toolchain for authoring Morrowind plugins **without the
+Construction Set**. You write records in **JSON, Ruby, Python,
+JavaScript, or TypeScript**; `esp` builds them to an `.esp` via
+[tes3conv], manages scripts, translations, and dialogue, and lints
+against the real vanilla game data. Every operation renders as human
+text or structured JSON, so a CLI, an HTTP API, an MCP server, and an AI
+agent all drive the same pipeline.
+
+## Why — source is truth, the `.esp` is output
+
+The Construction Set is a direct-manipulation binary editor: the `.esp`
+*is* the document. `esp` inverts that. **You author intent as
+human-readable source and the `.esp` is a disposable build artifact** —
+the relationship a compiler has with its object files. You never
+hand-edit the binary; you edit source and rebuild.
+
+That one inversion buys the whole software-engineering toolchain:
+`git diff` / blame / branch / review, procedural generation, lint as a
+build step, CI, HTTP/MCP automation, and AI authoring. The cost is the
+loss of direct manipulation — spatial placement and visual browsing.
+[**ESPresso**](docs/architecture.md) is the planned AI-first GUI built on
+this backend, where you describe intent, an agent writes diffable
+source, and you review the git diff.
+
+## Quickstart
+
+Install once:
+
+```sh
+gem install esp-modkit    # puts `esp` on your PATH
+# tes3conv is a required prerequisite, not on Homebrew: download a release
+#   https://github.com/Greatness7/tes3conv/releases
+# or:  cargo install --git https://github.com/Greatness7/tes3conv
+esp doctor               # confirm Ruby, tes3conv, and the references index
+esp refs unpack          # unpack vanilla ESMs to ~/.config/esp/references/ (one-time)
+esp refs index           # build the SQLite index there (one-time)
+```
+
+Native extensions (`sqlite3`, `zstd-ruby`) compile on install, so a C
+toolchain is needed. To work from a checkout instead of the gem, clone the
+repo, `bundle install`, and put `bin/esp` on your `PATH` (the launcher pins
+this repo's Ruby + Gemfile).
+
+Then per-project — anywhere on disk:
+
+```sh
+mkdir -p ~/morrowind/MyFirstMod && cd ~/morrowind/MyFirstMod
+esp init                 # writes .esp/project.json + git init + mods/ + dist/
+esp new MyMod            # scaffold mods/MyMod/
+esp lint MyMod           # check references against vanilla data
+esp build MyMod          # -> dist/MyMod.esp (in *this* project)
+esp install MyMod                   # OpenMW: register via data= + content=
+esp install MyMod --to-data-files   # Original engine: copy into Morrowind/Data Files/
+```
+
+`esp` walks up from the current directory to find the `.esp/project.json`
+marker, so commands work from any subdir of your project tree —
+no flags needed inside a project, `--root PATH` if you're scripting
+across several, or `export ESP_PROJECT_ROOT=…` to pin one per shell.
+
+**New here? Read the [walkthrough](docs/walkthrough.md)** — one plugin
+from empty to built, with the thesis shown in practice.
+
+## Three frontends, one pipeline
+
+Every command has a `--json` mode whose payload is identical across all
+three frontends:
+
+| Frontend | Command | For |
+|---|---|---|
+| CLI | `esp <cmd>` | humans + scripts |
+| HTTP | `esp serve` | ESPresso, local automation |
+| MCP | `esp mcp serve` (`esp mcp install` to wire up) | AI clients (Claude Code / Desktop) |
+
+## Command surface
+
+```
+doctor                               check prerequisites (Ruby, tes3conv, refs index)
+init [NAME]                          new project (marker + git init + mods/ + dist/)
+new MOD [--format rb|py|js|ts]       scaffold a mod
+build [MOD] [--locale fr] [--all] [--install]   build -> dist/MOD[.locale].esp
+watch MOD                            rebuild on change
+lint MOD                             dangling-ref check (needs refs index)
+extract-scripts MOD                  hoist inline script text to files
+unpack PLUGIN [NAME]                 .esp -> mods/NAME/NAME.json
+install MOD [--copy-to PATH | --to-data-files]    OpenMW + optional file copy
+refs unpack | index | find Q         vanilla data: build + query
+i18n check MOD                       missing/orphan locale keys
+docs build | introspect              regenerate / dump docs
+serve | mcp serve                    HTTP API / MCP server
+```
+
+## Documentation
+
+- **[Walkthrough](docs/walkthrough.md)** — empty → built, end to end.
+- **[Getting started](docs/getting-started.md)** — install, paths on
+  macOS, wiring `esp` into an AI client.
+- **[Authoring guide](docs/authoring-guide.md)** — source formats,
+  `text_source`, dialogue DSL, translation, linting.
+- **[Architecture](docs/architecture.md)** — layered design, the
+  roadmap, why each piece exists.
+- **[Command reference](docs/reference/commands.md)** *(auto-generated)*
+- **[API reference](docs/reference/api/index.md)** *(auto-generated)*
+
+Regenerate the auto pages with `bin/esp docs build`; a pre-commit hook
+fails if they drift.
+
+## Development
+
+```sh
+bundle exec rake            # RuboCop + Minitest (must be green)
+bundle exec rake admin      # local quality reports (coverage, lint, stats)
+```
+
+[tes3conv]: https://github.com/Greatness7/tes3conv

data/docs/architecture.md

@@ -0,0 +1,125 @@
+# Architecture
+
+## Layered design
+
+```mermaid
+flowchart TB
+    subgraph frontends["Frontends"]
+        direction LR
+        CLI(CLI)
+        GUI(GUI — planned)
+        MCP(MCP — planned)
+        HTTP(HTTP — planned)
+    end
+
+    subgraph service["Service layer (returns structured results)"]
+        direction LR
+        Builder
+        Linter
+        ScriptExtractor
+        ReferenceIndex
+        I18n
+    end
+
+    subgraph core["Core primitives"]
+        direction LR
+        Loader
+        Preflight
+        ScriptBlob
+        Tes3conv
+        OpenmwConfig
+    end
+
+    subgraph protocol["Source-program protocol"]
+        direction LR
+        InProcess["Ruby (in-process)"]
+        Subprocess["Spawn (Python / JS / TS)"]
+    end
+
+    frontends --> service
+    service --> core
+    core --> protocol
+```
+
+Today **CLI**, **HTTP** (`esp serve`), and **MCP** (`esp mcp serve`) are
+all live. The HTTP layer is a thin WEBrick wrapper over the same
+service-layer modules — every endpoint payload matches the
+corresponding `esp <cmd> --json` shape. The MCP server exposes the same
+operations as tools over stdio JSON-RPC for AI clients. The **GUI**
+(ESPresso, a separate repo) talks to `esp serve` instead of spawning a CLI
+per call — including `POST /agent`, which streams a Claude tool-use loop
+(`Esp::Agent`) over Server-Sent Events, dispatching the same tool surface
+MCP exposes.
+
+## Why JSON is the universal contract
+
+The build pipeline:
+
+```mermaid
+flowchart LR
+    Src["source file<br/>(.json / .rb / .py / .js / .ts)"]
+    --> Loader
+    --> Records["records<br/>(Array&lt;Hash&gt;)"]
+    --> I18n["I18n.resolve!<br/>(@t: sentinels)"]
+    --> Preflight
+    --> Tmp[JSON tempfile]
+    --> tes3conv
+    --> Esp["dist/&lt;Mod&gt;.esp"]
+```
+
+Anywhere along that chain you can substitute a different producer.
+Today's authoring formats — JSON, Ruby, Python, JavaScript, TypeScript
+— all converge on the same `Array<Hash>` shape. A future language
+(Lua? Rust?) gets added by registering an interpreter command in
+`Esp::Mw::Loader::INTERPRETERS`.
+
+## Why two paths for i18n
+
+`.rb` sources can call `t("key")` directly because the Ruby loader
+runs in-process; we inject a singleton method on the eval module.
+
+Subprocess loaders (Python/JS/TS) can't call back into Ruby. They emit
+`"@t:key"` sentinel strings instead, which the build resolves
+post-load via a single recursive walk. Same `Esp::Mw::I18n` instance
+services both paths, so miss tracking is unified.
+
+JSON has no execution model at all, so it uses sentinels too.
+
+## Why a SQLite reference index
+
+Vanilla data unpacks to ~200 MB of JSON across three ESMs and ~69 k
+records. `grep` is slow and brittle (record boundaries aren't lines).
+The index is a 4-column SQLite table built once with `esp refs index`,
+queried in milliseconds for lookups by id / type / SQL `LIKE` pattern.
+The linter uses it for foreign-id resolution; future tooling will too.
+
+## Why golden tests for ZSTD blob synthesis
+
+The Script record's variables list and bytecode placeholder are
+embedded as ZSTD-framed binary blobs that tes3conv accepts only in a
+very specific layout. Getting the bytes wrong is silent: builds appear
+to succeed and OpenMW just logs warnings. The test suite sweeps every
+vanilla Script record (~1.2 k) and asserts that our synthesis +
+libzstd-backed decode round-trip byte-identically.
+
+## Self-documenting outputs
+
+Every CLI command has a `--json` mode whose payload is the same data
+the human text branch yields. `esp docs introspect` dumps the full
+command tree and module surface as one JSON object — that's the
+mechanism the markdown docs in `docs/reference/` are generated from,
+and the same mechanism the MCP server uses to register its tools.
+
+## Roadmap status
+
+**Phase A — the toolchain** (the "compiler") and **Phase B — production
+toolchain + AI backbone** are done: the full authoring/build pipeline,
+the MCP/`Operations` surface, internal tool i18n, OS-aware game-install
+integration, and a central load manifest.
+
+**Phase C — ESPresso** (the AI-first GUI, in the separate `espresso`
+repo) is in
+progress: the Tauri+SvelteKit shell and the agent workspace (chat →
+`POST /agent` SSE → `Esp::Agent`) are built; git-diff review, visual
+panels, and packaging (bundling a Ruby runtime) are next. The
+local-only `.claude/ROADMAP.md` holds the authoritative step list.

data/docs/authoring-guide.md

@@ -0,0 +1,206 @@
+# Authoring guide
+
+## Source formats
+
+Each mod lives in `mods/<Mod>/` with exactly one source file. Pick the
+language you're most comfortable with — they all produce the same ESP.
+
+| Extension | Loader | Notes |
+|---|---|---|
+| `<Mod>.json` | in-process | Array of record hashes, matches tes3conv's JSON shape |
+| `<Mod>.rb` | in-process | Last expression must be the records array; symbol keys auto-converted |
+| `<Mod>.py` | subprocess (`python3`) | Print a JSON array of records to stdout |
+| `<Mod>.js` / `<Mod>.mjs` | subprocess (`node`) | Same contract |
+| `<Mod>.ts` | subprocess (`deno run --quiet`) | Same contract |
+
+For subprocess loaders the interpreter runs with `cwd` set to the mod's
+source directory, so the script can read sibling files like
+`scripts/foo.mwscript` with plain relative paths. The script's own path
+is passed as the last command-line arg.
+
+To add another language, add an entry to `Esp::Mw::Loader::INTERPRETERS` in
+`lib/esp/mw/loader.rb`. The contract is just "print a JSON array of records
+to stdout."
+
+## Per-mod self-containment
+
+Everything that belongs to a mod — its source file, scripts, dialogue
+drafts, design docs, i18n catalogues — lives **inside that mod's
+folder**. The tool repo doesn't own user content; cross-mod files
+belong wherever the user organises them. A typical mod layout:
+
+```
+mods/<Mod>/
+├── <Mod>.rb            # or <Mod>.json, <Mod>.py, etc.
+├── design/             # docs, voice guides, narrative notes
+├── scripts/            # .mwscript files referenced via text_source
+├── i18n/               # locale catalogues (see Translation below)
+└── README.md
+```
+
+## MWScript: `text_source`
+
+Script records carry their source code in `text`. For real scripts this
+is verbose and unreadable inside a JSON array, so the build accepts an
+alternative:
+
+```json
+{
+  "type": "Script",
+  "id": "MyScript",
+  "text_source": "scripts/MyScript.mwscript"
+}
+```
+
+At build time the preflight pass reads `scripts/MyScript.mwscript`,
+parses out variable declarations, and regenerates the SCHD header and
+SCVR/SCDT blobs in the byte layout tes3conv expects. The `text` field
+is set to the file's contents; `bytecode` is a placeholder (OpenMW
+recompiles from text on load).
+
+Going the other direction: if you `esp unpack` someone else's mod, every
+script lives inline. Run `esp extract-scripts <Mod>` to hoist those out
+to `scripts/<id>.mwscript` files. The transformation is lossless —
+rebuilding produces an identical ESP.
+
+## Dialogue (DSL, Ruby sources only)
+
+Hand-writing DIAL and DialogueInfo records as JSON is the most painful
+single thing the Construction Set is good at. The Ruby DSL replaces
+the bookkeeping (per-info IDs, prev/next chains, repeated speaker
+filters) so the source reads like the dialogue itself:
+
+```ruby
+records = [
+  { type: 'Header', flags: '', file_type: 'Esp', author: 'me',
+    version: 1.3, num_objects: 0, masters: [] }
+]
+
+records.concat(
+  dialogue do
+    speaker 'Hrisskar Flat-Foot' do
+      topic 'Flat-Foot' do
+        info t('hrisskar.flat_foot.intro')
+        info t('hrisskar.flat_foot.legion'), pc_faction: 'Imperial Legion'
+        info t('hrisskar.flat_foot.warning'),
+             pc_faction: 'Imperial Legion',
+             pc_rank: 3,
+             result_script: 'StartScript AFSN_TrackerScript'
+      end
+    end
+
+    topic 'AFSN_Tracker', type: :journal do
+      info 'Quest started.', journal_index: 10
+      info 'Hrisskar agreed.', journal_index: 20
+    end
+  end
+)
+
+records
+```
+
+What the DSL does for you:
+
+- **One `Dialogue` record per `topic`**, carrying `dialogue_type`
+  (Topic / Greeting / Journal / Persuasion / Voice).
+- **`DialogueInfo` records chained in author order** via prev_id /
+  next_id. Order is filter precedence in Morrowind — first match
+  wins, so write fallbacks last.
+- **`speaker "Name" do … end`** scopes a speaker filter to every info
+  inside; an explicit `speaker:` on an info overrides.
+- **Predictable IDs**: `"<topic>_<idx>"`. Git diffs stay readable.
+- **i18n**: `t("key.path")` works inside the block — same catalogue
+  as the rest of the mod.
+
+Supported info kwargs (all optional):
+
+| Kwarg | Maps to |
+|---|---|
+| `speaker:` | `speaker_id` |
+| `race:` | `speaker_race` |
+| `class:` *(or `class_:`)* | `speaker_class` |
+| `faction:` | `speaker_faction` |
+| `cell:` | `speaker_cell` |
+| `sex:` | `data.speaker_sex` (`:any` / `:female` / `:male`) |
+| `speaker_rank:` | `data.speaker_rank` |
+| `pc_faction:` | `player_faction` |
+| `pc_rank:` | `data.player_rank` |
+| `disposition:` | `data.disposition` |
+| `journal_index:` | `data.disposition` (for Journal topics) |
+| `sound:` | `sound_path` |
+| `result_script:` | `script_text` (inline MWScript) |
+
+**Not yet covered** (deferred to a later step): the structured
+`filters` array (function calls / variable comparisons / global checks),
+`result_script_source:` (currently raises NotImplementedError — use
+inline text for now), and the Greeting 0–9 priority cascade.
+
+## Translation (i18n)
+
+Strings can be authored in two complementary ways:
+
+- **In `.rb` sources** call `t("key.path")` directly — the loader
+  exposes the helper as a singleton method on the eval module, so it
+  resolves at evaluation time.
+- **In `.json` / `.py` / `.js` / `.ts` sources** write
+  `"@t:key.path"` sentinel strings anywhere a string value is
+  expected. The build resolves them recursively post-load.
+
+Catalogues live at `mods/<Mod>/i18n/<locale>.yaml`. Keys are dot-pathed
+nested-hash, with lookup falling back to the default locale (`en`) on
+miss:
+
+```yaml
+# mods/MyMod/i18n/en.yaml
+meta:
+  description: "A friend in Seyda Neen"
+activator:
+  stump: "Hollow Stump"
+```
+
+```yaml
+# mods/MyMod/i18n/fr.yaml
+meta:
+  description: "Un ami à Seyda Neen"
+activator:
+  stump: "Souche Creuse"
+```
+
+Build with the default locale:
+
+```sh
+bin/esp build MyMod                  # -> dist/MyMod.esp     (English)
+bin/esp build MyMod --locale fr      # -> dist/MyMod.fr.esp  (French)
+```
+
+Missing keys fall back to `en` with a warning in the build logs.
+`esp i18n check MyMod` reports missing and orphan keys per locale.
+
+## Linting
+
+`esp lint <Mod>` checks Npc and scripted-object records for dangling
+references — script / race / class / faction IDs that don't exist in
+the mod itself or in any indexed vanilla ESM. Two severities:
+
+- `ERROR` — ref points at nothing the mod or vanilla defines.
+- `WARN` — ref resolves in a vanilla ESM that isn't listed in the
+  mod's `Header.masters`. OpenMW will still load it but log warnings.
+
+Non-zero exit on any error — wire it into CI.
+
+## Authoring with AI
+
+Since every command accepts `--json` and `esp docs introspect` dumps the
+full capability surface, an AI agent can:
+
+1. Call `esp docs introspect` to learn the available commands.
+2. Read records via `esp refs find <id> --show --json`.
+3. Author a mod source file (Ruby/Python/JSON/whatever).
+4. Validate via `esp lint <Mod> --json`.
+5. Build via `esp build <Mod> --json`.
+
+`esp mcp serve` exposes this loop as first-class MCP tools; see
+[getting-started](getting-started.md#wiring-esp-into-an-ai-client-mcp)
+for wiring it into an AI client, and the
+[walkthrough](walkthrough.md#same-pipeline-three-frontends) for the loop
+in action.