@dforge-core/dforge-mcp 0.1.0-rc.8 → 0.1.1

package/CHANGELOG.md

@@ -0,0 +1,132 @@
+# Changelog
+
+All notable changes to `@dforge-core/dforge-mcp`. This project uses semver-ish
+`0.1.0-rc.N` pre-release tags; the published version is set at publish time via
+the release workflow, so committed `package.json` versions are placeholders.
+
+## 0.1.0-rc.13
+
+Single source of truth: the authoring tools now validate against
+`@dforge-core/metadata` — the same registry/schema package the dForge app, SDK,
+and VS Code extension use.
+
+### Added
+- **`dforge_dbml_import` — DBML front-end (was a stub).** Parses the common DBML
+  subset (Table blocks, typed columns with `[settings]`, inline `[ref: > t.c]`
+  and top-level `Ref:` lines) into the table-spec and runs the import core. Drops
+  the source PK column (the identity trait provides `{entity}_id`) and remaps FK
+  targets to it. Both import tools accept a `module` identity for **greenfield**
+  imports (no manifest yet).
+- **Spreadsheet (.xlsx) import.** A binary `.xlsx` can't be read directly, so the
+  package ships a **pure-stdlib Python extractor** (`dforge://script/xlsx-to-model`,
+  no `pip install`) that decodes sheets → headers + sample rows as JSON. The skill
+  (`dforge://reference/excel-import`, wired into the Phase 1 on-ramp) drives it:
+  run the extractor, build a table-spec from the model, call `dforge_module_import`.
+  `.csv` is read directly (plain text).
+- **`dforge_module_import` — table-spec → entities (import core).** Takes a
+  normalized spec (tables → columns → relationships) and generates entities:
+  each column's `fieldTypeCd` is inferred from an explicit code, a source SQL
+  type, sample values, and name heuristics (email/phone/url/currency), validated
+  against the metadata registry with `dbDatatype` derived; every relationship
+  becomes the FK+Reference two-column pair. The shared transformer that DBML/SQL,
+  Excel/CSV, and hand-authored front-ends feed (Excel front-end is next). Output
+  passes `dforge_module_validate` clean.
+- **`dforge_entity_rename` / `dforge_entity_delete` — refactor-safe entity ops.**
+  Rename cascades the identity PK (`{old}_id → {new}_id`) and repoints every
+  reference (other entities' `link.entity`/`references`, view `entityCode`, role
+  rights keys, action entity, folder bindings, seed `entityCode` + PK keys);
+  delete drops the entity + its seed + manifest entry + role key + folder
+  binding + view sources. Both move/remove files via a new `deletes[]` field on
+  the tool response (apply `files` AND `deletes`); surfaces unhandled surfaces
+  (reports, translations, menus, DSL, dangling cross-entity FKs) as warnings.
+- **`dforge_entity_field_rename` — refactor-safe field rename.** Renames a field
+  and propagates the new name to everything that referenced it in one call: the
+  paired Reference column's `link.thisKey` + `references` block, same-entity
+  formula columns (`[old]` → `[new]`), data view columns and `order` arrays,
+  seed-data records, and OTHER entities' FKs targeting it (`link.otherKey` /
+  `references.to.field`). Wired into the backtrack protocol ("rename, don't
+  remove+add"); pair with `dforge_module_validate` to confirm nothing dangles.
+- **`dforge_module_validate` — offline pre-flight cross-reference check.** Loads
+  the whole module and catches the errors that previously only surfaced at
+  pack/install: dangling FK/reference targets, a missing hidden-FK column, view
+  `dataSources`/columns and menu `dataViewCode`s and role rights pointing at
+  entities/fields/actions/reports that don't exist, and entities with no Select
+  grant. Returns errors + warnings in `_validate.json`. Wired into the skill as
+  the first Phase 6 gate (run it and fix every error before packing).
+- **Field-type & column-type validation.** `dforge_entity_field_add` /
+  `dforge_entity_field_modify` and `dforge_setting_add` reject an unknown
+  `fieldTypeCd` (with a "did you mean" hint — e.g. `integer` → `number`,
+  `reference` → `lookup`) and an unknown `columnType`. Previously any string
+  passed.
+- **`dbDatatype` auto-derivation.** When a field omits `dbDatatype`, it's
+  derived from `fieldTypeCd` (currency → `numeric(18,2)`, text → `varchar`;
+  reference/formula columns get none). An explicit value is never overridden.
+- **Full entity trait set.** `dforge_entity_add` / `dforge_module_create`
+  accept the complete, metadata-validated trait list — identity, audit,
+  audit-full, soft-delete, sorting, postable, accumulation, ledger, period —
+  instead of only the two scaffolder presets. Codes expand server-side at
+  install.
+- **Test harness.** Added `vitest` (`pnpm test`) with coverage for the new
+  validation, `dbDatatype` derivation, and the trait flow.
+
+### Changed
+- **`dforge_entity_field_remove` now cascade-cleans.** It removes the field and
+  the paired Reference (when you remove its hidden FK), the `references` entry,
+  view columns + `order`, and seed-data keys — and warns about formula /
+  cross-entity dependents instead of leaving them dangling. (Moved into
+  `src/tools/refactor.ts` alongside the rename ops.)
+- **Schemas sourced from `@dforge-core/metadata`, cross-platform.** A new Node
+  script (`scripts/vendor-schemas.cjs`, runnable via `pnpm sync-schemas`) copies
+  the JSON schemas from the installed metadata package — works on Windows, and
+  runs automatically in `prepublishOnly` so every publish regenerates them from
+  the pinned metadata version (no silent drift). `vendor-resources.sh` is now a
+  Unix wrapper that delegates schemas to that script. The public
+  `resources/schemas/*` surface (MCP resources + jsdelivr) is unchanged.
+- **`vendor-resources.sh` reference sync is opt-in.** The conventions doc and
+  skill reference pull from `dForge-core` now runs only with `VENDOR_REFS=1`;
+  the default run is schemas-only and needs no `dForge-core` checkout. This
+  prevents clobbering the in-repo skill references, which are ahead of core.
+- **Dependencies.** Added `@dforge-core/metadata ^0.0.2` (bundled into the
+  single-file `dist/server.js` via tsup). Bumped `@dforge-core/dforge-cli`
+  `^0.1.2` → `^0.2.2` (builder API unchanged; `buildFolders` flat-root output
+  verified).
+
+### Skill
+- `dforge-mcp-author`: `field-types.md` + `SKILL.md` now tell the agent to
+  **omit `dbDatatype` on plain data columns** (it's derived), keeping the
+  explicit-value guidance only for FK columns (`cuid`) and size/precision
+  overrides — so the derivation is actually used, not bypassed.
+
+### Fixed
+- **Phase 0 scaffold gate no longer greps Markdown.** `dforge_module_plan`
+  validate now writes a machine-readable `docs/phase.json` marker, and the gate
+  (`dforge_module_create` + the plan `check`) **parse** it instead of searching
+  `VALIDATION.md` for a `readyToScaffold: true` substring — so reformatting /
+  casing / duplicate text in the human report can't fool the gate. Falls back to
+  the legacy substring for modules validated before the marker existed.
+- **Field rename now updates *every* formula** that references the field, not
+  just the first — replaced a reused global `RegExp` (`.test()` carries
+  `lastIndex` across calls) with literal bracket-token string ops.
+- **Validator no longer rubber-stamps cross-module typos.** A dotted entity ref
+  (`crm.product`) is validated against the manifest's declared `dependencies`
+  (or this module's own code); an undeclared/typo'd module is now an error
+  instead of being accepted. **Role-rights** entity keys use the same resolver,
+  so grants on a **system entity** (`user`, `document`, …) or a declared
+  cross-module entity no longer false-error, while unknown ones still do.
+- **xlsx extractor ignores styled-but-empty rows when sampling.** A bordered/
+  formatted cell with no value no longer counts as data, so placeholder rows
+  can't exhaust the row sample before the real data is reached. Headers-only
+  (structure-only) sheets are supported too — they yield `rows: []`.
+- **xlsx extractor is memory-bounded with no value loss.** Worksheets stream
+  (capped sample), and the shared-string table is read in a second pass that
+  loads **only the indices the sampled cells reference** — so a huge workbook
+  never loads the whole table, and there's no cap that could silently return a
+  raw index instead of the real string.
+- **`module pack` tarball-path resolution is robust.** It collects every
+  `*.dforge` candidate from stdout (quoted/spaced paths and Windows separators
+  included), **normalizes wrapping punctuation** (quotes, parens, trailing
+  commas), and picks the one that **actually exists on disk** — pack just wrote
+  it — via a single `stat` per path instead of trusting a fragile regex token.
+
+### Notes
+- Earlier `0.1.0-rc.*` releases predate this changelog.

package/README.md

@@ -1,8 +1,8 @@
 # @dforge-core/dforge-mcp
 
-MCP server for dForge module authoring. Exposes 18 composable tools and the canonical schemas so AI agents (Claude Code, Cursor, Zed, etc.) can drive the full module lifecycle — scaffold → entities → actions → views → security → install — through structured tool calls instead of free-form JSON generation.
+MCP server for dForge module authoring. Exposes 27 composable tools and the canonical schemas so AI agents (Claude Code, Cursor, Zed, etc.) can drive the full module lifecycle — scaffold → entities → actions → views → security → install — through structured tool calls instead of free-form JSON generation.
 
-Ships with a wizard Skill (`skills/dforge-mcp-author/SKILL.md`) that walks the AI through six phases with explicit backtrack support when later phases expose earlier gaps.
+Ships with a wizard Skill (`skills/dforge-mcp-author/`) that walks the AI through six phases with explicit backtrack support when later phases expose earlier gaps. The skill bundle includes 23 detailed reference files (field types, flags, traits, formulas, DSL, security, etc.) and an annotated `simple-todo` example module.
 
 **New here?** Start with **[docs/creating-modules.md](docs/creating-modules.md)** — three ways to scaffold a module (terminal CLI, VS Code sidebar, AI wizard) and when to pick each.
 
@@ -25,6 +25,8 @@ your AI editor (Claude Code / Cursor / Zed)
 
 The native binary actually talks to your tenant. The npm-CLI wrapper is just a launcher that picks the right platform binary and exec's it. **You don't need to install dforge-cli separately** — it comes along when you install dforge-mcp (or when `npx -y @dforge-core/dforge-mcp` runs cold).
 
+The server also bundles **`@dforge-core/metadata`** — the canonical, dependency-free registry of field types, traits, column types, and the JSON schemas (the same package the dForge app, SDK, and VS Code extension use). It's the source of truth for the authoring-time validation below (valid `fieldTypeCd` / `columnType` / trait codes, `dbDatatype` derivation) and for the vendored schemas. Bundled at build time, so there's nothing extra to install.
+
 If you want to use a hand-built native binary instead of the npm-shipped one, point `DFORGE_CLI_BINARY` at the executable file's absolute path:
 
 ```bash
@@ -70,7 +72,7 @@ claude mcp list
 # Should show: dforge — npx -y @dforge-core/dforge-mcp — connected
 ```
 
-Or inside a Claude Code session, type `/mcp` to see all connected servers + their tools. The 18 `dforge_*` tools should be listed.
+Or inside a Claude Code session, type `/mcp` to see all connected servers + their tools. The 27 `dforge_*` tools should be listed.
 
 ### Cursor / Zed
 
@@ -78,30 +80,43 @@ Same `command + args` config shape; check their docs for the file location. Veri
 
 ## What it exposes
 
-### Tools (18)
+### Tools (27)
 
 Grouped by typical phase in the wizard flow. All "return" tools emit `{ summary, files: { '<relPath>': '<contents>' } }`; the client decides whether to write — lets the AI preview diffs with the user before committing.
 
+Phase 0 (identity → requirements → design → validation) is orchestrated by the **`dforge_module_plan`** tool together with the `dforge-mcp-author` skill: the tool tracks progress from the on-disk `docs/` artifacts and gates scaffolding, while the AI authors those artifacts under its direction.
+
 **Module-level**
 | Tool | Behavior |
 |---|---|
-| `dforge_module_create` | New module scaffold |
+| `dforge_module_plan` | **Phase 0 orchestrator** — drives identity → requirements → design → validation, returning the next step each call; gates `dforge_module_create` until `readyToScaffold: true`. Call first in any session (`action: "check"`) |
+| `dforge_module_create` | New module scaffold (blocked until Phase 0 passes) |
+| `dforge_module_import` | Import a normalized **table-spec** (tables → columns → relationships) into an existing module as entities. Infers `fieldTypeCd` from SQL type / sample values / name (metadata-validated, `dbDatatype` derived) and builds the FK+Reference pair per relationship. Fed by DBML/SQL, Excel/CSV, or a hand-authored spec |
 | `dforge_module_inspect` | Read current module state. Full structured data is in `files["_inspect.json"]`; `summary` is one-line stats |
+| `dforge_module_validate` | Offline cross-reference check (run before pack): dangling FK targets, missing hidden-FK columns, view/menu/role refs to non-existent things, uncovered entities. Errors + warnings in `files["_validate.json"]` |
 | `dforge_module_pack` | Shells to `dforge-cli module pack`. Returns tarball path + size |
 | `dforge_module_install` | Shells to `dforge-cli module install`. Args: `pathOrTarball`, optional `tenantUrl` / `token` / `tenantCode` — fall back to `DFORGE_URL` / `DFORGE_TOKEN` env. `tenantCode` is an optional `--code` sanity check the server cross-references against the JWT |
 
 **Entities (Phase 1)**
 | Tool | Behavior |
 |---|---|
-| `dforge_entity_add` | Add an entity to an existing module |
+| `dforge_entity_add` | Add an entity to an existing module. `traits` accepts the full platform set — identity, audit, audit-full, soft-delete, sorting, postable, accumulation, ledger, period |
+| `dforge_entity_rename` | **Refactor-safe entity rename** — moves the file (see `deletes`), cascades the PK `{old}_id → {new}_id`, repoints link.entity / references / view entityCode / role keys / action entity / folder bindings / seed |
+| `dforge_entity_delete` | **Refactor-safe entity delete** — drops the file + seed (see `deletes`), manifest entry, role key, folder binding, view sources; warns on dangling cross-entity FKs |
 | `dforge_entity_field_add` | Patch a single field |
-| `dforge_entity_field_modify` | Replace a field's spec |
-| `dforge_entity_field_remove` | Drop a field (warns about dependents) |
+| `dforge_entity_field_modify` | Replace a field's spec (same name) |
+| `dforge_entity_field_rename` | **Refactor-safe rename** — propagates the new name to the paired Reference, formulas, view columns + order, seed data, and other entities' FKs |
+| `dforge_entity_field_remove` | **Refactor-safe remove** — cascade-cleans paired Reference, view columns + order, seed keys; warns on formula/cross-entity dependents |
+
+The field/entity tools validate against the `@dforge-core/metadata` registry: an unknown `fieldTypeCd` (e.g. `integer`, `reference`), `columnType`, or trait code is rejected at authoring time with a "did you mean" hint, and `dbDatatype` is auto-derived from the field type (currency → `numeric(18,2)`, text → `varchar`; reference/formula columns get none) unless you set it explicitly.
 
 **Behavior (Phase 2 — optional)**
 | Tool | Behavior |
 |---|---|
 | `dforge_action_add` | DSL script + `ui/actions.json` entry |
+| `dforge_trigger_add` | DB-event trigger in `logic/triggers.json` (entity event + optional condition → action) |
+| `dforge_job_add` | Scheduled job in `logic/jobs.json` (5-field cron + timeout + action) |
+| `dforge_webhook_add` | Outbound webhook in `logic/webhooks.json` (entity event → POST to endpoint) |
 
 **Views + reports (Phase 3)**
 | Tool | Behavior |
@@ -122,7 +137,7 @@ Grouped by typical phase in the wizard flow. All "return" tools emit `{ summary,
 | Tool | Behavior |
 |---|---|
 | `dforge_dependency_add` | Add a dep on another dForge module |
-| `dforge_dbml_import` | Stub — not implemented yet |
+| `dforge_dbml_import` | **DBML front-end** to `module_import` — parses DBML (Table blocks, typed columns, inline + top-level refs) into the table-spec, drops the source PK (identity provides `{entity}_id`), and imports. Pass `module` for a greenfield import |
 
 ### Resources (13)
 
@@ -152,27 +167,60 @@ https://cdn.jsdelivr.net/npm/@dforge-core/dforge-mcp@latest/resources/schemas/<n
 
 ## Claude Skill — the wizard
 
-`skills/dforge-mcp-author/SKILL.md` teaches Claude how to drive the tools as a six-phase wizard. **It is NOT auto-installed by `npm install` — the Skill file ships in the npm tarball but Claude Code looks for Skills in `~/.claude/skills/`, not in node_modules.** Sync it manually:
+The skill bundle lives at `skills/dforge-mcp-author/` and contains:
 
-```bash
-# from the published npm tarball
-mkdir -p ~/.claude/skills/dforge-mcp-author
-curl -fsSL https://cdn.jsdelivr.net/npm/@dforge-core/dforge-mcp@latest/skills/dforge-mcp-author/SKILL.md \
-  -o ~/.claude/skills/dforge-mcp-author/SKILL.md
+| Path | Contents |
+|---|---|
+| `SKILL.md` | Six-phase co-pilot wizard |
+| `references/*.md` | 23 detailed reference files (field types, flags, traits, formulas, DSL, security, views, menus, translations, …) |
+| `examples/simple-todo/` | Annotated reference module showing all core patterns |
+
+**It is NOT auto-installed by `npm install`** — the skill ships in the npm tarball but Claude Code looks for skills in `~/.claude/skills/`, not in `node_modules`. Sync the whole bundle manually:
 
-# or from this repo
+```bash
+# Resolve the actual latest published version from the npm registry,
+# then pin the jsdelivr URL to it. We don't use jsdelivr's `@latest`
+# alias directly — that CDN endpoint caches aggressively (6-12h lag
+# after a new publish), which silently serves stale Skill content.
+VERSION=$(npm view @dforge-core/dforge-mcp version)
+BASE="https://cdn.jsdelivr.net/npm/@dforge-core/dforge-mcp@${VERSION}/skills/dforge-mcp-author"
+
+# Wizard
 mkdir -p ~/.claude/skills/dforge-mcp-author
-curl -fsSL https://raw.githubusercontent.com/dforge-core/dforge-mcp/main/skills/dforge-mcp-author/SKILL.md \
-  -o ~/.claude/skills/dforge-mcp-author/SKILL.md
+curl -fsSL "$BASE/SKILL.md" -o ~/.claude/skills/dforge-mcp-author/SKILL.md
+
+# Reference files (23 guides — load on demand per the table in SKILL.md)
+mkdir -p ~/.claude/skills/dforge-mcp-author/references
+for f in action-dsl column-types conventions data-migration data-views \
+          field-types filters flags formulas jobs manifest menus \
+          number-sequences print-templates queries reports schema-import \
+          security settings traits translations validation-checklist; do
+  curl -fsSL "$BASE/references/${f}.md" \
+    -o ~/.claude/skills/dforge-mcp-author/references/${f}.md
+done
+
+# simple-todo example
+mkdir -p ~/.claude/skills/dforge-mcp-author/examples/simple-todo/{entities,logic/actions,ui,security,seed-data}
+for f in README.md manifest.json; do
+  curl -fsSL "$BASE/examples/simple-todo/$f" \
+    -o ~/.claude/skills/dforge-mcp-author/examples/simple-todo/$f
+done
+# … entity, ui, security, seed-data files follow the same pattern
+
+# Or, straight from GitHub main (always fresh, but pre-release content):
+# curl -fsSL https://raw.githubusercontent.com/dforge-core/dforge-mcp/main/skills/dforge-mcp-author/SKILL.md \
+#   -o ~/.claude/skills/dforge-mcp-author/SKILL.md
 ```
 
-Re-run after every dforge-mcp upgrade — the Skill version isn't checked at runtime, so a stale Skill against new tools will misroute calls.
+> **Note on CLAUDE.md:** Every module gets a `CLAUDE.md` in its root, authored during Phase 0a by the `dforge-mcp-author` skill (the AI drafts it; you write it) and kept current as later phases complete. It tells Claude Code that the directory is a dForge module, instructs it to use the `dforge-mcp-author` skill, describes the module layout, and carries a live **Module status** tracker (which phase is done) so future sessions resume accurately.
+
+Re-run after every dforge-mcp upgrade — the skill version isn't checked at runtime, so a stale skill against new tools will misroute calls.
 
 The phases:
 
 | Phase | Required? | Tools used |
 |---|---|---|
-| 0. Intake | yes | (brief written manually) |
+| 0. Identity / requirements / design / validation | yes | (docs authored directly) |
 | 1. Domain | yes | `module_create`, `entity_add`, `entity_field_*` |
 | 2. Actions | optional | `action_add` |
 | 3. Views + Reports | views yes, reports optional | `view_*`, `report_add`, `setting_add` |
@@ -188,29 +236,37 @@ Key principles encoded in the Skill: inspect-before-patch, one-at-a-time, determ
 
 ```bash
 pnpm install
-pnpm build          # tsup → dist/server.js (bundles SDK + zod + dforge-cli/templates)
+pnpm build          # tsup → dist/server.js (bundles SDK + zod + dforge-cli/templates + metadata)
 pnpm typecheck      # tsc --noEmit
+pnpm test           # vitest — validation + dbDatatype derivation + trait flow
 node dist/server.js # stdio JSON-RPC — pipe a request to smoke-test
 ```
 
 To iterate against an in-tree `dforge-cli`, temporarily pin the dep at the sibling path:
 
 ```bash
-sed -i '' 's|"@dforge-core/dforge-cli": "\^0.1.[0-9.]*"|"@dforge-core/dforge-cli": "file:../dforge-cli"|' package.json
+sed -i '' 's|"@dforge-core/dforge-cli": "\^0.2.[0-9.]*"|"@dforge-core/dforge-cli": "file:../dforge-cli"|' package.json
 rm -rf node_modules pnpm-lock.yaml && pnpm install
 # Flip back before publish — file: deps don't resolve for npm consumers.
 ```
 
 ### Refresh vendored resources
 
-When `dForge-core/docs/schemas/` or `MODULE_CONVENTIONS.md` change:
+**Schemas** (`resources/schemas/`) are copied from the installed **`@dforge-core/metadata`** package — bumping that dep is how you pick up schema changes. They're served as MCP resources via `dforge://schema/*` and, in the published tarball, via jsdelivr.
+
+```bash
+pnpm sync-schemas      # node script — cross-platform (Windows/macOS/Linux)
+```
+
+You rarely run this by hand: **`prepublishOnly` runs it automatically**, so every publish regenerates the schemas from the exact metadata version this package depends on — they can't silently drift. The schemas are committed, so consumers always have them regardless.
+
+`scripts/vendor-resources.sh` is a Unix convenience wrapper around the same Node script, and it additionally can pull the **conventions doc + skill reference files** from `dForge-core` — but only when `VENDOR_REFS=1`:
 
 ```bash
-scripts/vendor-resources.sh                              # auto-locate ../dForge-core
-DFORGE_CORE=/abs/path/to/dForge-core scripts/vendor-resources.sh
+VENDOR_REFS=1 scripts/vendor-resources.sh   # also pull conventions + skill refs from ../dForge-core
 ```
 
-Republish to update jsdelivr-served schemas + the bundled resources.
+**That ref pull is off by default on purpose:** the per-topic references under `skills/dforge-mcp-author/references/` are authored in this repo and are ahead of `dForge-core/skills/dforge-module-author/`, so an unguarded pull would clobber them. Only set `VENDOR_REFS=1` once you've confirmed core is the source of truth for those files. (This part is bash-only; on Windows use WSL/Git Bash for the ref pull — schema refresh via `pnpm sync-schemas` is native.)
 
 ### Publishing
 
@@ -223,7 +279,8 @@ scripts/publish.sh 0.1.0-rc.N --tag latest --otp <code>
 **Pre-publish checklist:**
 - [ ] `@dforge-core/dforge-cli` dep is a real version (not `file:...`)
 - [ ] `pnpm typecheck` passes
-- [ ] Smoke test stdio: `tools/list` returns 18 tools
+- [ ] `pnpm test` passes
+- [ ] Smoke test stdio: `tools/list` returns 27 tools
 - [ ] Skill updated for any new/changed tools (it's a SEPARATE artifact; users sync it manually after upgrades)
 
 ### Adding a new tool