zopia 0.3.0

package/docs/01-overview.md

@@ -0,0 +1,94 @@
+# 🧭 Overview
+
+zopia is a **type-safe OpenAPI ↔ Zod toolkit** for generating, validating, and
+transforming API schemas. It converts between the four formats an API team
+lives in — **Swagger 2.0**, **OpenAPI 3.x**, **JSON Schema**, and **Zod v4** —
+and turns any of them into a tree of **type-safe `km-api` endpoint files** that
+drop straight into a TypeScript codebase.
+
+## 🌩️ The problem
+
+API teams keep the same knowledge in many places, and every copy drifts:
+
+| 😖 Pain | 🔍 Consequence |
+| --- | --- |
+| The spec (`swagger.json`) describes the API | …but the validation code is written by hand |
+| Zod schemas validate at runtime | …but they are re-typed from the spec, field by field |
+| Components (`$ref`) keep the DRY spec | …but the generated code duplicates the same shapes |
+| Swagger 2.0 and OpenAPI 3.x differ subtly | …and both must be supported to read legacy specs |
+
+**Every copy is a source of bugs. Every re-typing is a waste of time.**
+
+## 🛠️ The solution
+
+zopia makes the spec and the code **two views of one fact**:
+
+```mermaid
+flowchart LR
+  A["📄 swagger.json<br/>(v2 / v3)"] -->|"③ openapi → api docs"| B["📂 api_docs/**<br/>(.ts · km-api · zod v4)"]
+  B -->|"④ api docs → openapi"| A
+  C["⚛️ Zod v4 schemas"] -->|"① zod → JSON Schema"| D["📐 JSON Schema"]
+  D -->|"② JSON Schema → zod"| C
+```
+
+- **③** reads a spec, resolves every `$ref`, and renders one **`index.ts` per
+  endpoint** — each built with `makeApiConfig()` from **km-api** (0.4.x), with
+  request/response/params/query/headers/cookies validated by **Zod v4** schemas.
+- **④** reads that tree back and regenerates a complete OpenAPI document — so
+  developer edits to the generated code become the new spec.
+- **① / ②** are the two primitive schema converters the other engines are built on.
+
+## 🎯 The promise
+
+> **Fast and valid developing.** A developer gets **valid, documented,
+> type-safe** endpoint code **in seconds**, from a spec they already trust —
+> and can always get the spec back.
+
+## 🧰 The stack (fixed, by standard)
+
+| 🧩 Piece | Version | Role in zopia |
+| --- | --- | --- |
+| ⚛️ [Zod](https://zod.dev/) | **v4** (`^4`) | Runtime validation + static types; built-in `z.toJSONSchema()` |
+| 🧱 [km-api](https://www.npmjs.com/package/km-api) | **0.4.x** (`^0.4` from npm) | The "make function" — `makeApiConfig()` builds every generated endpoint |
+| 🟣 [Bun](https://bun.sh/) | `≥ 1.1` | Primary runtime & toolchain (runs the package, tests, and generated code) |
+| 🧪 [Vitest](https://vitest.dev/) | latest stable | Test runner for the full scenario matrix |
+| 🔷 TypeScript | `5.9+`, `strict` | Language of the package and of every generated file |
+
+## 👥 Who is this for?
+
+- 🧑‍💻 **TypeScript backend teams** that maintain a Swagger/OpenAPI spec and
+  want the validation layer generated, not hand-written
+- 🔄 **Legacy teams** still on Swagger 2.0 who need a path to OpenAPI 3.x
+- 🤝 **Full-stack teams** that want one artifact (`api_docs/**`) shared between
+  server validation and client code
+
+## 🚫 Non-goals (Phase 1)
+
+Being explicit about what zopia **does not do** keeps the scope honest:
+
+- 🌐 It is **not a runtime** — no HTTP server, no client, no hosting of specs.
+  (km-api already provides client adapters; zopia generates the definitions.)
+- ✍️ It is **not a spec editor** — it never edits your original spec file.
+- 🧮 It does **not execute user code** except through the documented,
+  trusted-input contract of engine ④ (see [Standards → Safety](12-standards.md#-safety)).
+- 📝 **YAML input** — v0.1.0 accepted JSON only; v0.2.x lifts that limit
+  (D-16): JSON/YAML objects, JSON/YAML text, and `spec.json` / `spec.yaml` /
+  `spec.yml` all enter the same normalized model through zopia's owned
+  deterministic YAML parser.
+
+## 💎 Core principles
+
+| # | Principle | Meaning |
+| --- | --- | --- |
+| P-1 | 🎯 **Deterministic** | Same input + same options ⇒ **byte-identical** output. No timestamps, no random order, no environment leakage. |
+| P-2 | 🔒 **Lossless by design** | Every conversion is reversible; what cannot be represented in the target format is recorded (manifest + warnings), never silently dropped. |
+| P-3 | 🧼 **Pure core** | In-memory transforms stay pure; documented file reads, generated-tree writes/imports, and CLI process access live at explicit adapter boundaries. |
+| P-4 | 🛡️ **Safe** | Fail loudly with typed, actionable errors; never write outside the configured output directory; never hide lossy conversions. |
+| P-5 | 📖 **Documented** | Every public symbol has JSDoc; every rule is numbered; every decision is recorded. |
+| P-6 | 📦 **Dependency-light** | Zero bundled runtime dependencies; `zod` and `km-api` remain explicit peers (`zod` powers engines ①/② and generated schemas; km-api powers generated endpoints and their reverse imports). |
+
+## 🔗 Next
+
+- 🎯 The completed public targets → [Targets](02-targets.md)
+- 🗺️ Release phases and deferred breadth → [Roadmap](03-roadmap.md)
+- 🧩 Words you will keep seeing → [Concepts](05-concepts.md)

package/docs/02-targets.md

@@ -0,0 +1,55 @@
+# 🎯 Targets
+
+These are the **explicit, testable targets** of zopia. Every Phase 1 target maps
+to its governing document and automated release-gate coverage — see
+[Testing → Scenario matrix](11-testing.md).
+
+> ✅ **Complete** = implemented, documented, and covered by the release gate.
+> Deferred work is listed explicitly under [Roadmap → Phase 2](03-roadmap.md#-phase-2--breadth-v02x-).
+
+## 🔄 Conversion targets
+
+| # | 🎯 Target | Spec | Status |
+| --- | --- | --- | --- |
+| T-1 | **Convert Zod → JSON Schema** | [Conversions → Engine ①](06-conversions.md) | ✅ |
+| T-2 | **Convert JSON Schema → Zod** | [Conversions → Engine ②](06-conversions.md) | ✅ |
+| T-3 | **Convert OpenAPI → api docs** — Swagger 2.0 *and* OpenAPI 3.0/3.1 | [Conversions → Engine ③](06-conversions.md) | ✅ |
+| T-4 | **Convert api docs → OpenAPI** (the reverse direction) | [Conversions → Engine ④](06-conversions.md) | ✅ |
+
+## 📂 API docs targets
+
+| # | 🎯 Target | Spec | Status |
+| --- | --- | --- | --- |
+| T-5 | **Layout mode `directory`** — `api_docs/` + path segments as nested directories + a method-named directory at the last level + `index` file | [API docs format → directory mode](07-api-docs.md) | ✅ |
+| T-6 | **Layout mode `flat`** — `api_docs/` + one directory per API + method directory + `index` file | [API docs format → flat mode](07-api-docs.md#-mode--flat) | ✅ |
+| T-7 | **`index.ts` files are TypeScript** and fill **all practical content** of the endpoint (method, path, operationId, summary, description, tags, auth, content types, request, response, examples) using **`makeApiConfig()` from km-api** (the package's make function) | [API docs format → `index.ts` contract](07-api-docs.md#-the-indexts-contract) | ✅ |
+| T-8 | **Option `insertComponents: boolean`** — when `true`, components are written into `api_docs` as their own schema files; **default `false`** | [Components](08-components.md) · [Configuration](09-configuration.md) | ✅ |
+| T-9 | **Option `useComponentAsReference: boolean`** — endpoints import emitted components recursively; requires `insertComponents: true`; **default `false`** | [Components → option matrix](08-components.md) | ✅ |
+
+## 🔁 Reverse-conversion targets
+
+| # | 🎯 Target | Spec | Status |
+| --- | --- | --- | --- |
+| T-10 | **Reversible output** — the generated tree contains everything needed to regenerate the spec (paths, methods, schemas, metadata), guaranteed by the manifest | [API docs format → manifest](07-api-docs.md) | ✅ |
+| T-11 | **Round-trip stability** — `openapi → api docs → openapi` and `zod → JSON Schema → zod` converge: re-running the pipeline on its own output is a no-op (idempotent) | [Testing → round-trip tests](11-testing.md#-round-trip-property-tests) | ✅ |
+
+## 🏗️ Engineering targets
+
+| # | 🎯 Target | Spec | Status |
+| --- | --- | --- | --- |
+| T-12 | **JSDoc everywhere** — every exported function, type, constant, and class is documented (template + rules fixed) | [Standards → JSDoc standard](12-standards.md) | ✅ |
+| T-13 | **Tests in all scenarios** — Vitest suite covering the full scenario matrix (spec versions, ref graphs, modes, option combinations, zod features, edge cases), run with Bun | [Testing](11-testing.md) | ✅ |
+| T-14 | **Bun runs the project** — install, typecheck, test, coverage, CLI, and generated-TypeScript imports all work through one pinned-Bun gate | [Standards → Bun gate](12-standards.md#-bun-gate) | ✅ |
+| T-15 | **Standard documentation** — `docs/` directory with targets, roadmap, architecture, conventions; README links into it; beautiful, icon-based markdown | this document set · [Standards → Docs convention](12-standards.md#-docs-convention) | ✅ |
+| T-16 | **Quality bar** — pure, safe, clean code with descriptions; deterministic output; typed errors; no silent lossy conversions | [Standards](12-standards.md) | ✅ |
+| T-17 | **Changelog after commits** — `CHANGELOG.md` updated by every user-facing commit under `Unreleased` | [Standards → Changelog convention](12-standards.md#-changelog-convention) | ✅ |
+
+## 🗺️ Out of scope for the first phase
+
+> 📌 The reverse-conversion *capability* (T-4/T-10/T-11) **is** in the first
+> phase — it is part of the four targets above. What is deferred:
+
+- ~~📝 YAML spec input~~ → shipped in v0.2.x (D-16; `.yaml`/`.yml` files and inline YAML text reach the same normalized model as JSON)
+- 🔗 External (multi-file) `$ref`s → [Roadmap Phase 2](03-roadmap.md)
+- 🧩 Reusable **parameters / responses** as emitted components (Phase 1 emits `components.schemas` only) → [Roadmap Phase 2](03-roadmap.md)
+- ~~🖥️ `zopia validate` (spec linting)~~ → shipped in v0.3.x (S-89; [Roadmap Phase 3](03-roadmap.md)); incremental regeneration remains deferred

package/docs/03-roadmap.md

@@ -0,0 +1,205 @@
+# 🗺️ Roadmap
+
+zopia ships in phases. **Phase 0 is this documentation set** — the contracts
+are fixed before a line of code is written, so implementation never re-decides
+anything.
+
+## 🚦 Phase 0 — Documentation & standards ✅
+
+The standard documentation of the project:
+
+- 📚 `docs/` — overview, targets, roadmap, architecture, concepts, conversions,
+  api-docs format, components, configuration, usage, testing, standards
+- 🏠 `README.md` — project home with the documentation map
+- 📜 `CHANGELOG.md` — Keep-a-Changelog format + the "changelog after commits" rule
+- 🔐 `LICENSE` (MIT) · 🧹 `.gitignore`
+- 🔑 Key decisions **D-01 … D-15** recorded in [Standards](12-standards.md#-key-decisions)
+
+**Completed:** the documentation fixed the Phase 1 contracts before
+implementation; it now describes the implemented v0.1.0 behavior and deferred scope.
+
+## 🚀 Phase 1 — The four engines (v0.1.0) ✅
+
+**Released:** 2026-09-28. Everything in [Targets](02-targets.md) is marked ✅:
+
+- [x] ⚙️ Package scaffold — ESM `package.json` (Bun-first, zero direct runtime
+      dependencies; `zod` + `km-api ^0.4.1` peers from npm — D-15), strict `tsconfig`,
+      Vitest/V8, pinned Bun, and checked-in `bun.lock`
+- [x] ① **Engine 1** — `zodToJsonSchema()` on top of `z.toJSONSchema()`
+      ([rules](06-conversions.md))
+- [x] ② **Engine 2** — `jsonSchemaToZod()` recursive emitter
+      ([rules](06-conversions.md))
+- [x] ③ **Engine 3** — `openApiToApiDocs()` — normalize v2/v3 → operation IR
+      → render `directory` / `flat` trees of `index.ts` files
+      ([rules](06-conversions.md),
+      [format](07-api-docs.md))
+- [x] ④ **Engine 4** — `apiDocsToOpenApi()` — manifest-driven reverse
+      conversion to OpenAPI 3.0/3.1
+      ([rules](06-conversions.md))
+- [x] 🧱 Component options — `insertComponents`, `useComponentAsReference`,
+      nested references, and direct/mutual cycles ([rules](08-components.md))
+- [x] 📝 **YAML input** — owned deterministic YAML 1.2 core-schema parser,
+      `.yaml`/`.yml` paths and inline YAML text, anchors/aliases/merge keys,
+      stable `ZOPIA_SPEC_INVALID_YAML` ([rules](06-conversions.md#engine-③--openapi--api-docs), D-16)
+- [x] 🔗 **External `$ref`s** — same-folder references bundled inline for
+      spec file paths, byte-identical to inline twins ([rules](06-conversions.md#engine-③--openapi--api-docs), D-17)
+- [x] 📦 Manifest writer/reader — `.zopia-manifest.json` (D-06)
+- [x] ⚠️ **Warnings pipeline** — stable typed codes, exact JSON Pointer locations,
+      deterministic collection/callbacks, generated-code markers, and CLI stderr
+      reporting across engines ①–④ (R-144/R-408/D-12)
+- [x] ♻️ **Manifest staleness** — canonical source/config comparison, invalid and
+      incomplete-tree detection, source-located warnings, safe obsolete-artifact
+      pruning, and symlink-safe regeneration (R-741…R-743)
+- [x] 🔐 **Reverse security fallback** — manifest-authoritative requirements,
+      validated metadata, collision-safe deterministic bearer synthesis,
+      OpenAPI/Swagger representations, and operation-located warnings (R-656)
+- [x] 🛑 **Typed errors** — one immutable stable-code catalogue and `ZopiaError`
+      boundary across engines, manifests, generated-module imports, filesystem
+      operations, warnings, and CLI validation, with locations, hints, and causes
+      (R-141…R-143/R-404)
+- [x] 🔁 **Round-trip contract** — fixture-backed source-dialect identity across
+      Swagger 2.0 and OpenAPI 3.0/3.1, every layout/component mode, refs/cycles,
+      lossy-keyword overlays, path-item metadata, and byte-identical regeneration;
+      supported Zod ↔ JSON Schema pipelines converge after canonicalization
+      (T-10/T-11/R-409)
+- [x] ⌨️ **CLI contract** — strict `generate` / `reverse` grammar, complete option
+      mapping and help, isolated stdout/stderr channels, stable exit statuses,
+      trusted-tree disclosure, and direct/package execution ([usage](10-usage.md#-cli))
+- [x] 📸 **Golden generated-tree contract** — deliberate reproducible updates,
+      complete byte-for-byte tree comparisons in all canonical layouts, and
+      strict compilation against installed published `km-api@0.4.1`, including
+      `trace`, custom/default statuses, and extension media types (R-112/R-126)
+- [x] 📈 **Coverage gates** — V8 measures every source file; overall line/function
+      coverage is gated at 90%, branches at 85%, conversion engines at 95% lines,
+      and every source file at 80% lines, with omitted-file detection
+- [x] 🟣 **Bun gate** — pinned Bun 1.2.21, frozen `bun.lock` installation,
+      typecheck, Vitest, coverage, direct/package CLI smoke tests, and Bun-native
+      generated-TypeScript reverse imports run through one command (T-14/D-01)
+- [x] 🧪 **Vitest suite** — unit, integration, round-trip, golden-tree, JSDoc,
+      CLI, and package-release contracts run in the pinned Bun gate
+      ([testing](11-testing.md))
+- [x] 📖 **JSDoc audit** — every exported declaration and exposed interface/class
+      member has a useful summary; public callables document parameters and return
+      values; optional configuration fields state defaults; examples and links are
+      checked; and named-only exports are enforced by an AST contract suite (T-12,
+      R-131…R-135/R-1003)
+- [x] 📦 **Package/release readiness** — public npm metadata, an allowlisted
+      source archive, executable CLI, isolated packed-consumer import and CLI
+      smoke tests, and a mandatory prepublish gate (R-191…R-193)
+- [x] 📚 **Documentation status sync** — current module/test layouts, generated
+      examples, Phase 1 completion labels, and deferred boundaries agree with
+      the implementation and are guarded by a documentation contract test
+- [x] 📜 First versioned entry in `CHANGELOG.md` → **v0.1.0**
+
+### ✅ Definition of done — Phase 1 (met)
+
+The v0.1.0 release satisfies all of the following:
+
+1. 📄 `bun run test` is green — unit + integration + round-trip suites
+2. 📈 Coverage gates are met (see [Testing → Coverage gates](11-testing.md#-coverage-gates))
+3. 🔁 Round-trip: for every fixture spec, `openapi(docs(spec))` equals `spec` after canonicalization
+4. 🧩 All four engine docs sections are implemented exactly as specified (mapping tables are the contract)
+5. 📖 Every public symbol has JSDoc; `tsc --noEmit` (strict) passes
+6. 📜 `CHANGELOG.md` v0.1.0 entry exists and `README` status banner is updated
+7. ✅ **km-api dependency gate** (D-15) — `km-api@0.4.1` is published and zopia
+   uses `km-api: ^0.4.1` from npm; no Git submodule or unpublished commit remains.
+   The dependency remains installed and type-checkable in every release gate.
+
+## 🧰 Phase 2 — Breadth (v0.2.x) ✅
+
+- 📝 **YAML input** — ✅ accept `swagger.yaml` / `openapi.yaml` (D-13 lifts → D-16)
+- 🔗 **External `$ref`s** — ✅ resolve references to other files in the same folder (D-17)
+- 🧩 **Reusable parameters & responses** — ✅ emitted as their own component
+  files `components/parameters/<Name>/index.ts` and
+  `components/responses/<Name>/index.ts` with `<Name>Parameter` / `<Name>Response`
+  exports, kind barrels, manifest `kind` entries, and a reverse conversion that
+  refreshes declarations from the modules while restoring every use-site `$ref`
+  verbatim (D-18; km-api has no standalone-parameter concept, so modules hold the
+  derived schema only — name/location/`required` remain operation data)
+- 🧾 **`zopia.config.ts`** — ✅ project-level config file with working-directory
+  discovery, explicit `--config` paths, validated `{ generate, reverse }` defaults,
+  and stable precedence CLI flag > config value > built-in default (D-19);
+  CLI flags stay available
+- 📤 **OpenAPI 2.0 output** — ✅ engine ④ accepts `version: '2.0'` (CLI
+  `--version 2.0`, config `reverse.version`) and downgrades 3.x-sourced
+  manifests into Swagger 2.0 documents: `nullable` becomes `x-nullable`,
+  `requestBody` becomes `body`/`formData` parameters, `components` becomes
+  top-level `definitions`/`parameters`/`responses`, `servers` decomposes into
+  `host`/`basePath`/`schemes`, and unrepresentable 3.x features (webhooks,
+  `jsonSchemaDialect`, cookie params, `links`, multi-flow OAuth2, …) drop with
+  deterministic `ZOPIA_WARN_DIALECT_DOWNGRADE` warnings (D-20)
+- 🧪 **More JSON Schema keywords** — ✅ `propertyNames` graduates from the D-12
+  approximation to native conversion: the exact
+  `{ type: 'object', propertyNames: { type: 'string', <pattern/length constraints> }, additionalProperties: <schema> }`
+  form now converts to `z.record(key, value)` (and back) with no warning or
+  frozen overlay (D-22); every other listed keyword keeps its runtime-refinement
+  + frozen-overlay round-trip (`patternProperties`, `if/then/else`,
+  `minProperties/maxProperties`, non-native `propertyNames` forms, `contains`) (Phase 1: documented approximation + warning, D-12)
+- 🪝 **3.1 webhook endpoint generation** — ✅ `document.webhooks` operations now
+  generate real endpoint files under `webhooks/<name>/<method>/index.ts`
+  alongside path operations (D-23). The manifest records `webhooks[]` entries
+  with the same reference/overlay/security metadata as path endpoints plus
+  `webhookOrder` and a `webhooksOverlay` (item-level metadata, `x-` names, and
+  operation-less items stay verbatim), so engine ④ reassembles webhooks in exact
+  source order for 3.1 output and omits them with source-located
+  `ZOPIA_WARN_WEBHOOKS` warnings for 3.0/2.0 output. Operation-less webhook maps
+  keep the Phase 1 manifest-only behavior (no endpoint files, forward warning)
+- 👀 **Watch mode** — ✅ `zopia generate --watch` regenerates whenever the spec
+  file changes (S-88): an immediate initial run, 50 ms-coalesced re-runs that
+  serialize against in-flight generation, run errors printed to stderr while
+  watching continues, and abort/cleanup on exit. Uses `fs.watch` with the
+  existing stale-tree/prune pipeline, so spec edits refresh owned files in place
+
+## 🌌 Phase 3 — Ecosystem (v0.3+) ✅
+
+- 🧹 **`zopia validate`** — ✅ CLI command + `validateZopia()` API lint specs
+  and generated trees (S-89): specs are checked for dialect validity, broken
+  local `$ref`s, endpoint-planning failures (name collisions, cross-namespace
+  duplicate operationIds), and components unreachable from any operation
+  (transitive; Swagger `definitions` included). Generated trees are checked for
+  manifest presence/validity, a successful reverse dry-run, and km-api peer
+  drift. Findings are deterministic sorted diagnostics with stable
+  `ZOPIA_VALIDATE_*` lint codes; the CLI prints them on stdout and exits `1`
+  when any error-severity finding exists
+- ♻️ **Incremental regeneration** — ✅ unchanged generated files keep their
+  mtimes (byte-identical regeneration writes nothing); the opt-in merge-safe
+  custom layer (D-24, S-90) exports a `custom` companion namespace per
+  endpoint/webhook scaffolded once and never overwritten (`--custom`,
+  `generate.custom`, or the `custom` generate option)
+- 🔍 **Diff tool** — ✅ `zopia diff old.json new.json` + `diffOpenApiSpecs()`
+  API (S-91): semantic comparison of dialect, info, endpoints (with
+  parameter/request-body/response details), webhooks, schema components,
+  document fields, and `x-` extensions; key order is ignored; deterministic
+  `+`/`-`/`~` human-readable lines with a summary; differences are data on
+  stdout, so the CLI exits `0` for changed pairs
+- 🧩 **Presets** — ✅ split-generation layouts (S-92): `--preset multi-tag`
+  routes each operation by its primary tag, `--preset multi-server` by the
+  effective first server (repeatedly), producing one independently
+  reverse-convertible api-docs sub-tree per bucket; untagged/default-server
+  operations land in `untagged`/`https-default-server`-style buckets; nothing
+  to split → the normal single tree. Programmatic `preset` option plus
+  `generate.preset` configuration; `planPresetBuckets()` exposes the pure
+  deterministic planner (collision-safe slugs, `ZOPIA_WARN_PRESET_PRIMARY_TAG`
+  on multi-tagged operations) and results report `trees[]`
+- 🧑‍💻 **VS Code extension** — ✅ navigate spec ↔ generated code both ways
+  (S-93): manifest-driven navigation core (`loadNavigationIndex`,
+  `specToLocations`/`treeToSpecLocation`, one-pass JSON
+  `specPointersToLines`/`specPointerAtLine` cursor resolution, YAML
+  operationId fallback) plus the `zopia navigate` CLI (`--to-code` /
+  `--to-spec`) and a plain-JS extension package under `editors/vscode/`
+  resolving the workspace's own zopia install
+
+## 🧮 Versioning
+
+SemVer, strictly:
+
+| 🔖 Bump | When |
+| --- | --- |
+| `MAJOR` | A generated-file format or manifest format breaks (`zopia:manifest@1` → `@2`) |
+| `MINOR` | New option, new engine feature, new CLI flag — backward compatible |
+| `PATCH` | Bug fixes, doc corrections, mapping-table clarifications |
+
+## 🔗 Next
+
+- 🎯 The exact list of what Phase 1 builds → [Targets](02-targets.md)
+- 🏗️ How the pieces fit → [Architecture](04-architecture.md)