docsgov 0.1.0

package/README.md

@@ -0,0 +1,242 @@
+# docgov
+
+Config-driven **documentation governance**. `docgov` checks that the
+references inside your Markdown docs still point at things that actually exist —
+so docs rot loudly (a failing check) instead of silently.
+
+It runs three independent guards over your `docs/`, plus a `dedup` analysis:
+
+| Guard | Checks that… |
+| --- | --- |
+| **code** | every `{{code:path#Symbol}}` token resolves to a real symbol in your source tree (via tree-sitter) |
+| **doc**  | every Markdown link/image target exists |
+| **api**  | every `{{api: METHOD /path}}` token exists in your OpenAPI spec |
+
+A fourth command, **dedup**, flags near-duplicate documentation concepts in
+`docs/` using sentence embeddings.
+
+The package source lives at the repository root (a TypeScript/Node project).
+
+---
+
+## Install
+
+```bash
+npm install -g docgov
+docgov --version
+```
+
+Or run it without installing:
+
+```bash
+npx docgov check
+```
+
+**Requirements**
+
+- **Node.js >= 22.** docgov uses the built-in `node:sqlite` module (for the
+  `dedup` index), which requires Node 22 or newer.
+- **No system libraries needed.** The tree-sitter grammars ship as bundled
+  WebAssembly, and the `dedup` embedder runs through `@huggingface/transformers`
+  (which bundles `onnxruntime-node`). There is no CGo, no ONNX Runtime install,
+  and no native toolchain to set up.
+- `dedup` downloads its embedding model to a local cache on first use.
+
+---
+
+## Quick start
+
+1. Mark the repo root and declare what to govern by creating
+   **`.docgov/docgov.yaml`**:
+
+   ```yaml
+   code:
+     boundary: [docs/**]              # .md files scanned for {{code:…}} tokens
+     source:   [internal/**, cmd/**]  # where those symbols must live
+   doc:
+     boundary: [docs/**]              # every link/image target must exist
+   ```
+
+2. Run the check from anywhere inside the repo:
+
+   ```bash
+   docgov check
+   ```
+
+   Exit `0` = clean, `1` = violations found, `2` = config/usage error.
+
+---
+
+## Commands
+
+### `docgov check [--format text|json]`
+
+Runs the code, doc, and api guards over their configured scopes and reports
+every violation. `--format json` emits machine-readable output for CI;
+`--format text` (the default) is styled for the terminal.
+
+### `docgov dedup`
+
+Refreshes the embedding index for `docs/` and reports near-duplicate concepts.
+Exits `1` if a high-confidence duplicate group is found.
+
+### `docgov update [--version vX.Y.Z] [--check]`
+
+docgov is distributed via npm, so `update` does not self-replace the binary.
+
+- `docgov update --check` reports the current version and the latest version
+  from the npm registry (best-effort; degrades gracefully when offline).
+- `docgov update` (or with `--version vX.Y.Z`) prints the install command to run,
+  e.g. `npm install -g docgov@latest`.
+
+### Exit codes
+
+| Code | Meaning |
+| --- | --- |
+| `0` | success — no violations |
+| `1` | `check` found violations (or `dedup` found duplicates) |
+| `2` | usage or configuration error (e.g. no `.docgov/` found, bad `--format`) |
+
+These are stable, so CI can gate on them directly.
+
+---
+
+## Configuration
+
+docgov reads a **single** file: `.docgov/docgov.yaml`. The `.docgov/` directory
+is also the **repo-root marker** — docgov walks up from the working directory
+until it finds one. A missing file makes `check` exit `2`.
+
+The file has three optional top-level sections. Each has a `boundary` (which
+`.md` files to scan) and, for `code`/`api`, a `source` (where referenced
+targets must live). `doc` takes only `boundary`.
+
+```yaml
+code:                              # omit → code guard skipped
+  boundary: [docs/**]              #   .md files scanned for {{code:…}} tokens
+  source:   [internal/**, cmd/**]  #   a token's path must be under source
+
+doc:                               # omit → doc guard skipped
+  boundary: [docs/**]              #   every markdown link/image target is existence-checked
+
+api:                               # omit → api guard skipped
+  boundary: [docs/api/contract/**] #   .md files scanned for {{api:…}} tokens
+  source:   [docs/api/openapi/**]  #   .json OpenAPI specs (union of all matched)
+```
+
+**Semantics**
+
+- Glob patterns use `**` to match any depth. A trailing `/` is normalized
+  to `/**`.
+- The three scopes are **independent and overlapping**: a file may be in more
+  than one boundary, and each guard runs its own pass. There is no precedence.
+- An **omitted** section skips that guard entirely. A **present-but-empty**
+  section runs the guard with no files in scope (no violations).
+
+Full reference: [`docs/config.md`](docs/config.md) and
+[`docs/guards.md`](docs/guards.md).
+
+---
+
+## Reference syntax
+
+These tokens live in your Markdown **prose**. Tokens inside fenced code blocks
+or inline code spans are skipped.
+
+### Code references — `{{code:path#Symbol}}`
+
+```markdown
+Loading is handled by {{code:internal/config/config.go#LoadConfig}}.
+```
+
+- `path` must be under one of `code.source`, else a violation.
+- `Symbol` is resolved with tree-sitter: top-level types, functions, variables,
+  and interface members resolve. Pointer-receiver methods do **not** resolve as
+  `#Member` — cite the type instead.
+
+### API references — `{{api: METHOD /path [qualifier]}}`
+
+```markdown
+Create a team: {{api: POST /teams}} with body field {{api: POST /teams body:name}}.
+```
+
+Qualifiers: none (operation exists), `param:name`, `header:name`,
+`body:a.b.c`, `response:a.b.c`. Path templates are normalized
+(`{id}` ≡ `{teamId}`).
+
+### Doc links — standard Markdown
+
+The doc guard checks ordinary Markdown links and images:
+
+```markdown
+See [the config reference](config.md) and ![the diagram](img/flow.png).
+```
+
+External URLs (`http:`, `https:`, `mailto:`), pure fragments (`#heading`), and
+`file.md#frag` suffixes are handled for you; absolute paths and paths escaping
+the repo root are rejected.
+
+---
+
+## Folder structure
+
+A governed repository looks like this — the only required piece is
+`.docgov/docgov.yaml`; the `docs/` layout is yours to choose:
+
+```
+your-repo/
+├── .docgov/
+│   └── docgov.yaml          # the single config file + repo-root marker
+├── docs/                    # whatever structure you like
+│   ├── README.md
+│   ├── config.md
+│   └── ...
+├── internal/                # source referenced by {{code:…}} tokens
+└── cmd/
+```
+
+This repository governs **its own** `docs/` with docgov; see
+[`docs/README.md`](docs/README.md) for that layout as a worked example.
+
+---
+
+## CI usage
+
+`check` is exit-code driven, so gating a pipeline is one line:
+
+```yaml
+docs-governance:
+  image: node:22
+  script:
+    - npm install -g docgov
+    - docgov check        # exits 1 on violations → job fails
+```
+
+---
+
+## Development
+
+The TypeScript source lives under [`src/`](src/):
+
+```bash
+npm ci
+npm run typecheck   # tsc --noEmit
+npm test            # vitest run
+npm run build       # tsc + copy bundled .wasm grammars into dist/
+```
+
+`npm run build` compiles to `dist/` and copies the vendored tree-sitter
+`.wasm` grammars into `dist/codeq/grammars/` so the published package resolves
+them at runtime. The repo-root [`Makefile`](Makefile) provides thin wrappers
+(`make build`, `make test`, `make typecheck`, `make self-check`, `make clean`)
+that delegate to these npm scripts.
+
+---
+
+## Documentation
+
+- [`docs/README.md`](docs/README.md) — documentation index
+- [`docs/config.md`](docs/config.md) — full config schema
+- [`docs/guards.md`](docs/guards.md) — what each guard checks
+- [`docs/command/`](docs/command/) — per-command reference
+- [`docs/architecture/`](docs/architecture/) — system design

package/dist/apispec/apispec.js

@@ -0,0 +1,401 @@
+// Loads and queries a self-contained OpenAPI v3.1 document. It supports
+// operation existence (path-template normalised) and loose field-name set
+// queries used by the api guard passes in internal/check.
+//
+// The Go original keeps schema bodies as json.RawMessage and re-parses them at
+// each recursion step; here the document is parsed once into a plain object
+// tree (via the yaml lib, which accepts JSON as a subset) and the walks operate
+// directly on those already-parsed nodes. Where Go's json.Unmarshal would fail
+// because a blob is not the expected shape, this port type-checks and skips.
+import { readFile } from "node:fs/promises";
+import YAML from "yaml";
+import { SpecNotFoundError, SpecParseError } from "./errors.js";
+const SCHEMA_REF_PREFIX = "#/components/schemas/";
+const COMBINERS = ["allOf", "oneOf", "anyOf"];
+function isObject(v) {
+    return typeof v === "object" && v !== null && !Array.isArray(v);
+}
+function asString(v) {
+    return typeof v === "string" ? v : "";
+}
+/**
+ * Spec is a loaded, self-contained OpenAPI v3.1 document. Only the parts docgov
+ * needs are typed; schema bodies stay as generic nodes for recursive walking.
+ */
+export class Spec {
+    // paths: normalised-or-raw path -> method (lowercase) -> operation.
+    paths;
+    // components.schemas: schema name -> raw schema node.
+    schemas;
+    constructor(paths, schemas) {
+        this.paths = paths;
+        this.schemas = schemas;
+    }
+    /** Builds a Spec from an already-parsed document object. */
+    static fromDocument(doc) {
+        const paths = new Map();
+        const pathsNode = doc["paths"];
+        if (isObject(pathsNode)) {
+            for (const [specPath, methodsNode] of Object.entries(pathsNode)) {
+                if (!isObject(methodsNode)) {
+                    continue;
+                }
+                const methods = new Map();
+                for (const [method, opNode] of Object.entries(methodsNode)) {
+                    if (!isObject(opNode)) {
+                        continue;
+                    }
+                    methods.set(method.toLowerCase(), parseOperation(opNode));
+                }
+                paths.set(specPath, methods);
+            }
+        }
+        const schemas = new Map();
+        const components = doc["components"];
+        if (isObject(components)) {
+            const schemasNode = components["schemas"];
+            if (isObject(schemasNode)) {
+                for (const [name, schema] of Object.entries(schemasNode)) {
+                    schemas.set(name, schema);
+                }
+            }
+        }
+        return new Spec(paths, schemas);
+    }
+    /** Returns the operation for method+path under path-template normalisation. */
+    findOp(method, path) {
+        const want = normalizePath(path);
+        const m = method.toLowerCase();
+        for (const [specPath, methods] of this.paths) {
+            if (normalizePath(specPath) !== want) {
+                continue;
+            }
+            const op = methods.get(m);
+            if (op !== undefined) {
+                return op;
+            }
+        }
+        return undefined;
+    }
+    /** Reports whether method+path exists in the spec (path-template normalised). */
+    hasOperation(method, path) {
+        return this.findOp(method, path) !== undefined;
+    }
+    /**
+     * Returns every field name reachable in the operation's contract: parameter
+     * names ∪ requestBody schema props ∪ all response schema props, recursively
+     * through internal $ref, array items, and allOf/oneOf/anyOf. Returns undefined
+     * if the operation is absent.
+     */
+    operationFields(method, path) {
+        const op = this.findOp(method, path);
+        if (op === undefined) {
+            return undefined;
+        }
+        const out = new Set();
+        for (const p of op.parameters) {
+            if (p.name !== "") {
+                out.add(p.name);
+            }
+        }
+        const visited = new Set();
+        this.collectSchemaFields(op.requestBody, out, visited);
+        for (const resp of op.responses) {
+            this.collectSchemaFields(resp, out, visited);
+        }
+        return out;
+    }
+    /**
+     * Walks a node and collects every reachable field name into out. Handles
+     * content descent, internal $ref (cycle-guarded), properties, array items,
+     * and allOf/oneOf/anyOf.
+     */
+    collectSchemaFields(node, out, visited) {
+        if (!isObject(node)) {
+            return;
+        }
+        // content → descend each media-type entry's schema
+        const content = node["content"];
+        if (isObject(content)) {
+            for (const media of Object.values(content)) {
+                if (isObject(media) && "schema" in media) {
+                    this.collectSchemaFields(media["schema"], out, visited);
+                }
+            }
+        }
+        // $ref → "#/components/schemas/X"
+        const ref = asString(node["$ref"]);
+        if (ref.startsWith(SCHEMA_REF_PREFIX)) {
+            const name = ref.slice(SCHEMA_REF_PREFIX.length);
+            if (!visited.has(name)) {
+                visited.add(name);
+                if (this.schemas.has(name)) {
+                    this.collectSchemaFields(this.schemas.get(name), out, visited);
+                }
+            }
+        }
+        // properties → add each key, recurse into values
+        const props = node["properties"];
+        if (isObject(props)) {
+            for (const [key, val] of Object.entries(props)) {
+                out.add(key);
+                this.collectSchemaFields(val, out, visited);
+            }
+        }
+        // items → recurse (array)
+        if ("items" in node) {
+            this.collectSchemaFields(node["items"], out, visited);
+        }
+        // allOf / oneOf / anyOf → recurse into each element
+        for (const combiner of COMBINERS) {
+            const arr = node[combiner];
+            if (Array.isArray(arr)) {
+                for (const elem of arr) {
+                    this.collectSchemaFields(elem, out, visited);
+                }
+            }
+        }
+    }
+    /**
+     * Reports whether a parameter named name with in ∈ {path, query, cookie}
+     * exists on the given operation (path-template normalised).
+     */
+    hasParam(method, path, name) {
+        const op = this.findOp(method, path);
+        if (op === undefined) {
+            return false;
+        }
+        for (const p of op.parameters) {
+            if (p.name === name) {
+                if (p.in === "path" || p.in === "query" || p.in === "cookie") {
+                    return true;
+                }
+            }
+        }
+        return false;
+    }
+    /**
+     * Reports whether a parameter with in=="header" named name exists, OR any
+     * response object declares a header named name.
+     */
+    hasHeader(method, path, name) {
+        const op = this.findOp(method, path);
+        if (op === undefined) {
+            return false;
+        }
+        for (const p of op.parameters) {
+            if (p.in === "header" && p.name === name) {
+                return true;
+            }
+        }
+        for (const resp of op.responses) {
+            for (const h of responseHeaderNames(resp)) {
+                if (h === name) {
+                    return true;
+                }
+            }
+        }
+        return false;
+    }
+    /**
+     * Reports whether the path segs (e.g. ["user","address","city"]) is reachable
+     * inside the requestBody schema.
+     */
+    hasBodyField(method, path, segs) {
+        const op = this.findOp(method, path);
+        if (op === undefined) {
+            return false;
+        }
+        return this.walkContentSchemas(op.requestBody, segs);
+    }
+    /**
+     * Reports whether the path segs is reachable inside any response schema.
+     */
+    hasResponseField(method, path, segs) {
+        const op = this.findOp(method, path);
+        if (op === undefined) {
+            return false;
+        }
+        for (const resp of op.responses) {
+            if (this.walkContentSchemas(resp, segs)) {
+                return true;
+            }
+        }
+        return false;
+    }
+    /**
+     * Extracts content[*].schema from node and walks each with segs. Returns true
+     * if walkSchemaPath succeeds on any media-type schema.
+     */
+    walkContentSchemas(node, segs) {
+        if (!isObject(node)) {
+            return false;
+        }
+        const content = node["content"];
+        if (!isObject(content)) {
+            return false;
+        }
+        for (const media of Object.values(content)) {
+            if (!isObject(media) || !("schema" in media)) {
+                continue;
+            }
+            const visited = new Set();
+            if (this.walkSchemaPath(media["schema"], segs, visited)) {
+                return true;
+            }
+        }
+        return false;
+    }
+    /**
+     * Navigates a schema node following segs as a dotted path. Returns true when
+     * all segments have been consumed (target reached). Handles $ref (cycle
+     * guard), allOf/oneOf/anyOf (any branch), properties descent, and transparent
+     * array items descent (same segs, no segment consumed).
+     */
+    walkSchemaPath(node, segs, visited) {
+        if (segs.length === 0) {
+            return true;
+        }
+        if (!isObject(node)) {
+            return false;
+        }
+        // $ref → follow into component schema (cycle guard)
+        const ref = asString(node["$ref"]);
+        if (ref.startsWith(SCHEMA_REF_PREFIX)) {
+            const name = ref.slice(SCHEMA_REF_PREFIX.length);
+            if (!visited.has(name)) {
+                visited.add(name);
+                if (this.schemas.has(name)) {
+                    if (this.walkSchemaPath(this.schemas.get(name), segs, visited)) {
+                        return true;
+                    }
+                }
+            }
+        }
+        // allOf / oneOf / anyOf → true if any branch satisfies segs
+        for (const combiner of COMBINERS) {
+            const arr = node[combiner];
+            if (Array.isArray(arr)) {
+                for (const elem of arr) {
+                    if (this.walkSchemaPath(elem, segs, visited)) {
+                        return true;
+                    }
+                }
+            }
+        }
+        // properties → if segs[0] matches a property key, recurse with segs[1:]
+        const props = node["properties"];
+        const head = segs[0];
+        if (isObject(props) && head !== undefined && head in props) {
+            if (this.walkSchemaPath(props[head], segs.slice(1), visited)) {
+                return true;
+            }
+        }
+        // items → transparent array descent: descend into items with same segs
+        if ("items" in node) {
+            if (this.walkSchemaPath(node["items"], segs, visited)) {
+                return true;
+            }
+        }
+        return false;
+    }
+}
+/** Builds an Operation from a parsed operation object node. */
+function parseOperation(node) {
+    const parameters = [];
+    const paramsNode = node["parameters"];
+    if (Array.isArray(paramsNode)) {
+        for (const p of paramsNode) {
+            if (isObject(p)) {
+                parameters.push({ name: asString(p["name"]), in: asString(p["in"]) });
+            }
+        }
+    }
+    const responses = [];
+    const responsesNode = node["responses"];
+    if (isObject(responsesNode)) {
+        for (const resp of Object.values(responsesNode)) {
+            responses.push(resp);
+        }
+    }
+    return { parameters, requestBody: node["requestBody"], responses };
+}
+/**
+ * Replaces every {var} segment with {} so a doc's {id} matches the spec's
+ * {teamId}. Trailing slash is trimmed (except root "/").
+ */
+export function normalizePath(p) {
+    let b = "";
+    let inVar = false;
+    for (const r of p) {
+        if (r === "{") {
+            inVar = true;
+            b += "{}";
+        }
+        else if (r === "}") {
+            inVar = false;
+        }
+        else if (inVar) {
+            // skip var-name characters
+        }
+        else {
+            b += r;
+        }
+    }
+    let out = b;
+    if (out.length > 1) {
+        out = out.replace(/\/+$/, "");
+    }
+    return out;
+}
+/**
+ * Returns the header keys declared in a response node. The response object
+ * shape is: {"headers": {"X-Trace": {...}, ...}, ...}.
+ */
+function responseHeaderNames(node) {
+    if (!isObject(node)) {
+        return [];
+    }
+    const headers = node["headers"];
+    if (!isObject(headers)) {
+        return [];
+    }
+    return Object.keys(headers);
+}
+/**
+ * Parses spec text (YAML or JSON) into a Spec. Invalid syntax — or a document
+ * whose top level is not an object — throws SpecParseError.
+ */
+export function parse(data, specPath) {
+    let doc;
+    try {
+        doc = YAML.parse(data);
+    }
+    catch (err) {
+        const detail = err instanceof Error ? err.message : String(err);
+        throw new SpecParseError(specPath, detail);
+    }
+    if (!isObject(doc)) {
+        throw new SpecParseError(specPath, "document is not an object");
+    }
+    return Spec.fromDocument(doc);
+}
+/**
+ * Reads and parses specPath from disk. A missing file throws SpecNotFoundError;
+ * invalid YAML/JSON throws SpecParseError.
+ */
+export async function load(specPath) {
+    let data;
+    try {
+        data = await readFile(specPath, "utf8");
+    }
+    catch (err) {
+        if (typeof err === "object" &&
+            err !== null &&
+            err.code === "ENOENT") {
+            throw new SpecNotFoundError(specPath);
+        }
+        throw err;
+    }
+    return parse(data, specPath);
+}