mdkg 0.1.1 → 0.1.3

package/CHANGELOG.md

@@ -4,6 +4,110 @@ All notable changes to mdkg are documented here.
 
 This project follows a pragmatic changelog style inspired by Keep a Changelog. Versions use npm package versions.
 
+## 0.1.3 - Unreleased
+
+### Added
+
+- Added first-class SQLite access cache support using Node's built-in `node:sqlite`; no third-party SQLite package is introduced.
+- Added `.mdkg/index/mdkg.sqlite` as a rebuildable derived cache for nodes, edges, skills, capabilities, archives, bundle imports, source hashes, and schema metadata when `index.backend` is `sqlite`.
+- Added fresh init defaults for `index.backend: sqlite`, `index.sqlite_path`, `index.sqlite_commit_warning_bytes`, and `index.lock_timeout_ms`.
+- Added a shared mutation lock plus atomic writes for mdkg mutations and index writes.
+- Added SQLite transactional id reservation for numeric node/checkpoint ids in SQLite mode.
+- Added `npm run smoke:sqlite` and `npm run smoke:parallel` packed/temp-repo coverage.
+
+### Changed
+
+- Raised the required Node runtime to `>=24.15.0`.
+- Existing workspaces that are migrated from older configs remain on `index.backend: json` until they explicitly opt in to SQLite.
+- Init ignore policy now keeps JSON indexes, temp files, lock directories, WAL, SHM, and journal files ignored while allowing `.mdkg/index/mdkg.sqlite` to be committed by intentional repo policy.
+- `mdkg index` continues to write JSON compatibility indexes and also rebuilds SQLite when enabled.
+- `mdkg doctor` and `mdkg validate` now report SQLite cache health when SQLite mode is enabled.
+- README and seeded docs now state that mdkg is pre-v1 public alpha software and cache/DAL contracts may churn before v1.
+
+### Fixed
+
+- Removed the accidental self-dependency on `mdkg` from package metadata.
+- Hardened parallel `mdkg new`, checkpoint, task, work, archive, bundle import config, and index writes against naming conflicts and partial cache writes.
+
+## 0.1.2 - 2026-05-19
+
+### Added
+
+- Added `.mdkg/index/capabilities.json` as a derived JSON cache for skills, `SPEC.md`, `WORK.md`, core docs, and design docs.
+- Added read-only `mdkg capability list/search/show` commands with JSON output, kind filters, and advisory visibility filters.
+- Added workspace `visibility` metadata for capability cache filtering, defaulting to `private`.
+- Added capability cache health reporting to `mdkg doctor`.
+- Added capability-cache smoke coverage for root plus child workspace aggregation and cache auto-rebuild.
+- Added packed-package init smoke coverage for fresh base init, fresh `mdkg init --agent`, removed flag failures, repeated init idempotency, doctor/validate, upgrade dry-run parity, task creation, and pack generation.
+- Added init preflight checks for seed config parseability and unmanaged skill mirror collisions.
+- Added first-class archive sidecars under `.mdkg/archive` with `mdkg archive add/list/show/verify/compress`.
+- Added deterministic single-file ZIP cache generation for archived source and artifact files.
+- Added `type: archive` graph nodes and `archive://<archive.id>` reference validation.
+- Added `mdkg work contract/order/receipt/artifact` lifecycle helpers for semantic mirror work contracts, work orders, receipts, and artifact registration.
+- Added archive/work packed-package smoke coverage for fresh temp repositories.
+- Added `mdkg bundle create/list/show/verify` for deterministic full `.mdkg` graph snapshot bundles.
+- Added private and public bundle profiles with fail-closed public filtering for private graph and archive refs.
+- Added bundle-local generated indexes (`global.json`, `skills.json`, `capabilities.json`) inside snapshot ZIPs.
+- Added bundle unit and CLI coverage plus packed-package bundle smoke coverage.
+- Added `mdkg bundle import add/list/rm/enable/disable/verify` for read-only child graph snapshot imports.
+- Added `bundle_imports` config with explicit alias, bundle path, visibility, expected profile, source metadata, and optional staleness policy.
+- Added `.mdkg/index/imports.json` as a derived import projection and health cache.
+- Added packed-package bundle import smoke coverage.
+- Added shared visibility policy enforcement for workspace nodes, archive sidecars, and imported bundle nodes.
+- Added `mdkg pack --visibility public|internal|private` for explicit public-safe and internal-safe packs.
+- Added `mdkg archive add --visibility private|internal|public` and `mdkg archive list --visibility ...`.
+- Added packed-package visibility smoke coverage.
+- Added `receipt_status: superseded` support for committed receipt mirrors.
+- Added `mdkg work receipt new|update --receipt-status superseded` CLI parity with graph validation.
+- Added runtime-style work/order/receipt fixture coverage with input refs, requested outputs, proof refs, artifacts, and hashes.
+- Added local qid support for `mdkg work order update`, `mdkg work receipt update`, and `mdkg work artifact add`.
+- Added `archive.large_cache_warning_bytes` config and `mdkg doctor` warnings for large committed archive ZIP caches.
+
+### Changed
+
+- `mdkg index` now writes the node index, skill index, and capability cache together.
+- Generated bootstrap config now includes the default capability cache path and root workspace visibility.
+- Docs and command matrix now teach capability discovery as separate from normal graph node search.
+- Made `mdkg init --agent` the single canonical AI-agent bootstrap path.
+- `mdkg init --agent` now creates `AGENTS.md` and `CLAUDE.md` alongside `AGENT_START.md`, `llms.txt`, `CLI_COMMAND_MATRIX.md`, default skills, event log, registry, and skill mirrors.
+- Init manifests are now mode-aware: base init only claims base assets, while agent init claims the full agent bootstrap.
+- Updated generated `.mdkg/README.md` onboarding to guide already-initialized repos toward `upgrade`, `new`, `search`, `pack`, and `validate`.
+- Added first-ingestion guidance to `AGENT_START.md` for imported docs bundles.
+- Hardened `WORK_ORDER.md` and `RECEIPT.md` templates with input refs, requested outputs, constraint refs, proof refs, attestation refs, and input/output hashes.
+- Init ignore updates now ignore raw archive source copies under `.mdkg/archive/**/source/` while leaving sidecar `.md` files and ZIP caches commit-eligible.
+- `mdkg doctor` now reports archive storage hygiene warnings for stray uncompressed archive files.
+- Added default bundle config under `.mdkg/bundles` without making `mdkg index` rewrite bundles.
+- Updated docs, command matrix, and release skills with bundle creation and verification guidance.
+- Updated seeded init/upgrade skills so managed workspaces receive pre-commit archive compression and private bundle refresh guidance.
+- `list`, `search`, `show`, `pack`, and `capability` now include enabled read-only bundle imports by default.
+- Imported nodes use import-alias qids such as `child_repo:task-1` and expose original bundle/source metadata in JSON output.
+- Stale imports warn during planning reads while `mdkg bundle import verify` exits nonzero.
+- Public bundle creation now fails when public local nodes reference private or internal imported graphs.
+- Public/internal bundle imports now require public bundle profiles.
+- `mdkg validate` and `mdkg doctor` now report public/internal references to less-visible mdkg records.
+- Archive JSON receipts now include sidecar visibility.
+- Archive sidecars created from outside-repo files now redact `source_path` to `external:<basename>` instead of storing absolute local paths.
+- `mdkg validate` and `mdkg archive verify` now share strict ZIP cache integrity checks for ZIP hash, readability, payload SHA-256, and payload byte size.
+- Work mirror docs and templates now state the canonical-system boundary for production order, receipt, feedback, dispute, payment, ledger, marketplace, fulfillment, and execution state.
+- Work lifecycle packed-package smoke now proves local qid mutation, order status updates, final superseded receipts, archive verification, indexing, show, and pack.
+
+### Fixed
+
+- Fixed fresh `mdkg init --agent` leaving missing managed wrapper docs that immediately required `mdkg upgrade --apply`.
+- Fixed misleading init summaries by reporting manifest, ignore, registry, event log, core pin, and skill mirror actions.
+- Fixed late init failure UX by printing a partial-init receipt with recovery guidance.
+- Kept bundle output deterministic across repeated creates when only `.mdkg/bundles/` changes.
+- Hardened publish readiness and init smoke checks to assert seeded release skills include archive compression and bundle refresh guidance.
+- Mutating task and work update flows now reject imported qids with explicit read-only import errors.
+- Work lifecycle mutation commands now reject imported order/receipt qids with explicit read-only bundle import guidance.
+- Local graph indexing now allows edges to configured import aliases without treating them as missing local workspace nodes.
+- Public bundle checks now reuse the same fail-closed policy as public/internal pack checks.
+- `mdkg archive verify --json` now emits a verification receipt for corrupt archive ZIP caches instead of being blocked by strict index validation.
+
+### Removed
+
+- Removed `mdkg init --llm`, `mdkg init --agents`, `mdkg init --claude`, and `mdkg init --omni`; each now fails before mutation with guidance to use `mdkg init --agent`.
+
 ## 0.1.1 - 2026-05-12
 
 ### Added

package/README.md

@@ -9,11 +9,14 @@ It is built for:
 
 mdkg stays deliberately boring:
 - repo-native under `.mdkg/`
-- TypeScript + Node.js 18+
-- zero runtime dependencies
-- no sqlite, daemon, hosted index, or vector DB
+- TypeScript + Node.js `>=24.15.0`
+- zero third-party runtime dependencies
+- first-class rebuildable SQLite cache through built-in `node:sqlite`
+- no daemon, hosted index, or vector DB
 
-Current package version in source: `0.1.1`
+Current package version in source: `0.1.3`
+
+mdkg is still pre-v1 public alpha software. The public package is usable, but graph, cache, bundle, and DAL contracts may continue to change quickly while the project converges on a stable v1 surface.
 
 ## The product shape
 
@@ -43,19 +46,13 @@ bun add -g mdkg
 
 Initialize mdkg in a repo:
 
-```bash
-mdkg init --llm
-```
-
-This is the generic OSS bootstrap path. It creates `.mdkg/` and updates `.gitignore` / `.npmignore` by default. Use `--no-update-ignores` to opt out of those ignore-file updates.
-
-Optional agent-ready scaffold:
-
 ```bash
 mdkg init --agent
 ```
 
-This adds strict-node `SOUL.md` / `HUMAN.md`, seeds the three default mdkg usage skills, creates `events.jsonl`, updates the skill registry, adds core pin updates, and creates mirrored skill folders under `.agents/skills/` and `.claude/skills/`.
+This is the canonical AI-agent bootstrap path. It creates `.mdkg/`, `AGENT_START.md`, `AGENTS.md`, `CLAUDE.md`, `llms.txt`, `CLI_COMMAND_MATRIX.md`, strict-node `SOUL.md` / `HUMAN.md`, the three default mdkg usage skills, `events.jsonl`, the skill registry, core pin updates, and mirrored skill folders under `.agents/skills/` and `.claude/skills/`. It also updates `.gitignore` / `.npmignore` by default. Use `--no-update-ignores` to opt out of those ignore-file updates.
+
+For a non-agent markdown graph only, run `mdkg init`.
 
 Preview safe scaffold upgrades in an existing mdkg workspace:
 
@@ -100,14 +97,69 @@ Build deterministic context:
 ```bash
 mdkg pack task-1
 mdkg pack task-1 --profile concise --dry-run --stats
+mdkg pack task-1 --visibility public --dry-run
+```
+
+Create a full `.mdkg` graph snapshot bundle for root or child orchestration:
+
+```bash
+mdkg archive compress --all
+mdkg archive verify --json
+mdkg bundle create --profile private
+mdkg bundle verify .mdkg/bundles/private/all.mdkg.zip
+mdkg bundle list --json
 ```
 
+Bundles are explicit graph transport artifacts, separate from task context packs. Before a commit in repos that track archives or bundles, refresh compressed archive caches first, then create the private bundle so the committed graph state is self-consistent. Private bundles are the default and may be committed in private repos when configured. Public bundles require at least one selected workspace with `visibility: public` and include only public workspace content and public archive sidecars; bundle creation fails if public content points at private graph, archive, or imported bundle records.
+
+Import a child repo bundle as a read-only planning view:
+
+```bash
+mdkg bundle import add child_repo child-repo/.mdkg/bundles/private/all.mdkg.zip --source-path child-repo
+mdkg bundle import list --json
+mdkg search "child capability"
+mdkg show child_repo:work.example
+mdkg pack child_repo:work.example --dry-run --stats
+mdkg bundle import verify child_repo --json
+```
+
+Imported bundle nodes are projected under the import alias, for example `child_repo:task-1`. They are available to `list`, `search`, `show`, `pack`, and capability discovery, but remain read-only; mutate the child repo and refresh its bundle to change imported content. Stale imports warn during planning reads and fail `mdkg bundle import verify`. Public or internal imports must be backed by public bundle profiles; private imports stay private planning context.
+
 Validate before handoff or commit:
 
 ```bash
 mdkg validate
 ```
 
+Discover cached capability surfaces:
+
+```bash
+mdkg index
+mdkg capability list --kind skill --json
+mdkg capability search "image worker" --kind work --json
+mdkg capability show <id-or-qid-or-slug> --json
+```
+
+Register source and artifact files as committed archive sidecars:
+
+```bash
+mdkg archive add ./inputs/key_input_doc.pdf --id archive.key-input-doc --kind source --visibility private
+mdkg archive verify archive://archive.key-input-doc
+mdkg archive list --json
+```
+
+Create semantic mirror work contracts, orders, receipts, and artifacts:
+
+```bash
+mdkg work contract new "generate image" --id work.generate-image --agent-id agent.image-worker --kind image_generation --inputs prompt:text:required --outputs image_url:url:required
+mdkg work order new "generate image request" --id order.generate-image-1 --work-id work.generate-image --requester user://example --input-refs archive://archive.key-input-doc
+mdkg work receipt new "generate image receipt" --id receipt.generate-image-1 --work-order-id order.generate-image-1 --outcome success --receipt-status recorded
+mdkg work artifact add receipt.generate-image-1 ./outputs/image.png --id archive.generated-image --kind artifact
+```
+
+Receipt statuses are `recorded`, `verified`, `rejected`, and `superseded`.
+Update and artifact commands accept local ids or local qids; imported bundle qids are read-only and must be changed in their source workspace.
+
 Update structured task state and evidence while keeping body and narrative edits in markdown:
 
 ```bash
@@ -151,9 +203,13 @@ mdkg lives under a hidden root directory:
 - `.mdkg/work/` tasks, bugs, tests, epics, checkpoints
 - `.mdkg/templates/` templates used by `mdkg new`
 - `.mdkg/skills/` Agent Skills packages
+- `.mdkg/archive/` sidecar metadata plus deterministic compressed source/artifact caches
+- `.mdkg/bundles/` optional committed full graph snapshot bundles
+- `.mdkg/index/mdkg.sqlite` optional committed, rebuildable SQLite access cache
+- `.mdkg/index/imports.json` generated read-only bundle import cache
 - `.agents/skills/` Codex/OpenAI-facing mirrored skills
 - `.claude/skills/` Claude-facing mirrored skills
-- `.mdkg/index/` generated cache files
+- `.mdkg/index/*.json` generated JSON compatibility cache files
 
 ## Primary commands
 
@@ -166,6 +222,9 @@ These are the commands new users and agents should learn first:
 - `mdkg next`
 - `mdkg pack`
 - `mdkg skill`
+- `mdkg capability`
+- `mdkg archive`
+- `mdkg work`
 - `mdkg task`
 - `mdkg validate`
 
@@ -224,6 +283,27 @@ This repo now dogfoods three internal skills:
 
 Optional skill metadata with prefixes such as `ochatr_*` is treated as vendor extension data. Structured skill output exposes it under `extensions.ochatr` while keeping the top-level `ochatr` field as a compatibility alias introduced in 0.0.9. ochatr.ai is a pioneering adopter of this extension pattern, not the name of the base mdkg standard.
 
+## Capability cache
+
+mdkg maintains `.mdkg/index/capabilities.json` as a derived access cache for deterministic capability surfaces:
+- skills from `.mdkg/skills/**/SKILL.md`
+- `SPEC.md`
+- `WORK.md`
+- core docs
+- design docs
+
+The capability cache is not the full graph and is not source of truth. Normal tasks, epics, bugs, tests, feats, and checkpoints remain in the standard graph index. Markdown remains authoritative; deleting the cache is recoverable with `mdkg index` or by running a capability command when auto-reindex is enabled.
+
+Capability records aggregate enabled registered workspaces and include deterministic source metadata such as `workspace`, `visibility`, `kind`, `id`, `qid`, `path`, headings, refs, source hash, and `indexed_at`. Workspace `visibility` also feeds mdkg's export safety checks for public/internal packs and public bundles. This is a CLI safety layer, not secret scanning, body redaction, or a replacement for private git hosting.
+
+## Index backends and parallel safety
+
+Fresh `mdkg init` workspaces default to `index.backend: sqlite`, which writes `.mdkg/index/mdkg.sqlite` as a rebuildable access cache using Node's built-in `node:sqlite`. Existing workspaces that are migrated from older configs default to `index.backend: json` until they opt in. Markdown files, archive sidecars, bundle manifests, and config remain source of truth in both modes.
+
+`mdkg index` still writes JSON compatibility caches (`global.json`, `skills.json`, `capabilities.json`, and import projections when configured). In SQLite mode it also rebuilds the SQLite cache with nodes, edges, skills, capabilities, archive metadata, bundle imports, source hashes, and schema metadata. Deleting the SQLite file is recoverable with `mdkg index`.
+
+Mutating commands use a workspace mutation lock plus atomic writes. SQLite mode additionally reserves numeric ids in a SQLite transaction before writing Markdown so parallel `mdkg new` and checkpoint calls avoid naming conflicts. Skipped ids after failed writes are acceptable because Markdown remains canonical.
+
 ## Agent workflow files
 
 mdkg recognizes a small set of canonical agent workflow documents:
@@ -237,13 +317,30 @@ Use `mdkg new spec|work|work_order|receipt|feedback|dispute|proposal "<title>"`
 
 Relational templates contain editable placeholder refs. `spec` and `work` scaffold as validation-clean standalone docs; `work_order`, `receipt`, `feedback`, `dispute`, and `proposal` need real refs before strict `mdkg validate` passes.
 
+For executable or purchasable capability mirrors, prefer the lifecycle helpers under `mdkg work ...`. They create and update `WORK.md`, `WORK_ORDER.md`, and `RECEIPT.md` semantic mirror files only. Production order state, receipt state, feedback, disputes, payments, ledgers, marketplace inventory, fulfillment records, and execution state remain canonical outside mdkg, such as in Postgres or another application database. Do not store raw secrets, credentials, live payment state, ledger mutations, canonical marketplace state, or bulky raw payloads in these mirrors.
+
+## Archive sidecars
+
+Archive entries live under `.mdkg/archive/<archive.id>/` and are normal graph nodes with `type: archive`. `mdkg archive add` copies the source into a managed local `source/` directory, writes a frontmatter sidecar `<file>.md`, and writes a deterministic single-file ZIP cache `<file>.zip`. The original source path is left untouched.
+
+Archive sidecars support `archive://archive.example` refs from orders, receipts, artifacts, proof refs, and other workflow metadata. `artifact://...` refs remain external or runtime-managed artifact identities; `archive://...` refs name committed mdkg archive sidecars. `mdkg validate` and `mdkg archive verify` both require the sidecar contract, ZIP cache hash, readable ZIP payload, payload SHA-256, and payload byte size to match. A missing raw local source copy is non-fatal when the committed sidecar and ZIP cache are valid.
+
+When the source passed to `mdkg archive add` is inside the repo, `source_path` is repo-relative. Outside-repo sources are redacted to `external:<basename>` so sidecars do not leak absolute local paths.
+
+Archive sidecar visibility defaults to `private`. Use `mdkg archive add --visibility public` only when the sidecar metadata and ZIP cache are safe for public packs or public bundles.
+
+By default, init/upgrade ignore generated raw archive source copies with `.mdkg/archive/**/source/`; sidecar `.md` files and compressed `.zip` caches remain commit-eligible. `mdkg doctor` warns when a committed archive ZIP cache exceeds `archive.large_cache_warning_bytes` in `.mdkg/config.json` (default `26214400`; set `0` to disable). Large-cache warnings do not block archive add or validation.
+
 ## Current direction
 
 This release includes:
 - `init --agent`
-- default ignore updates with `--no-update-ignores` for `.mdkg/index/` and `.mdkg/pack/`
+- default ignore updates with `--no-update-ignores` for generated JSON index/temp/lock files, `.mdkg/pack/`, and raw archive source copies
 - root-only published init seed config
 - skills indexing and search/show/list support
+- JSON capability cache for skills, `SPEC.md`, `WORK.md`, core docs, and design docs
+- SQLite index backend for fresh workspaces using built-in `node:sqlite`
+- mutation locking and atomic writes for parallel mdkg calls
 - optional `skills: [...]` on work items
 - pack-time skill inclusion
 - latest-checkpoint resolver + index hint
@@ -252,16 +349,22 @@ This release includes:
 - agent workflow file types and semantic `mdkg new --id` support
 - product-specific skill mirrors for Codex/OpenAI and Claude
 - shared `AGENT_START.md` startup guidance
+- conservative `mdkg upgrade` with mode-aware init manifests
+- archive sidecars with deterministic ZIP caches
+- semantic mirror helpers under `mdkg work ...`
+- explicit public/internal/private visibility enforcement for packs, bundles, archives, imports, validation, and doctor diagnostics
+- strict archive ZIP payload integrity checks during validation
 
 Current direction:
-- keep the OSS story generic around `init --llm`
-- use `init --agent` for deeper AI-agent bootstrap
+- keep the OSS story generic around `mdkg init --agent`
+- use base `mdkg init` only for repos that do not want agent bootstrap assets
 - keep `pack <id>` at the center of the human/agent loop
 - use `mdkg task ...` for structured state changes and markdown edits for narrative/body content
 - make event logging guided instead of purely manual
 - dogfood real skills inside the repo
 - make skill authoring first-class through `mdkg skill`
 - make `CLI_COMMAND_MATRIX.md` the single source of truth for the live CLI surface
+- keep production execution databases canonical while mdkg stores committed semantic mirrors
 - run manual behavior audits before enforcing stronger coverage thresholds
 
 Design and decision records live in the internal graph under `.mdkg/design/`.
@@ -271,8 +374,11 @@ Design and decision records live in the internal graph under `.mdkg/design/`.
 mdkg is not a secret store.
 
 Use these defaults:
-- keep `.mdkg/index/` gitignored
+- keep generated `.mdkg/index/*.json`, temp, lock, WAL, SHM, and journal files gitignored
+- commit `.mdkg/index/mdkg.sqlite` only when the repo intentionally tracks a reasonably sized rebuildable access cache
 - keep `.mdkg/pack/` gitignored
+- keep `.mdkg/archive/**/source/` gitignored unless a repo intentionally commits raw local copies
+- commit archive sidecar `.md` metadata and deterministic `.zip` caches when they are needed for reviewable evidence
 - event logs are committed by default; ignore or delete them manually if a repo wants local-only provenance
 - do not ship `.mdkg/` into production builds or published packages
 - if an external orchestrator is writing mdkg state, keep one durable writer per run and batch commits at end-of-run or checkpoint boundaries