semantifold 0.7.0 → 0.9.0

package/docs/generated-artifact-publication.md

@@ -0,0 +1,95 @@
+# Transactional generated-artifact publication
+
+`GeneratedArtifactPublisher` persists one complete, already-validated project generation without moving filesystem behavior into a frontend or backend. It publishes all targets through one active-generation pointer; it never presents multiple file renames as project-level atomicity.
+
+## Public API
+
+```js
+import {GeneratedArtifactPublisher} from "semantifold"
+
+const publisher = new GeneratedArtifactPublisher({
+  projectId: "example-project",
+  projectRoot: "/workspace/example",
+  publicationRoot: "/workspace/example/.semantifold-output",
+  sourceRoots: ["/workspace/example/src"]
+})
+
+const published = await publisher.publish({
+  generationId: "cycle-0001",
+  targets: [{
+    id: "java-main",
+    role: "text",
+    artifactSet: generatedJava,
+    sourceProjection: "targets/java/source",
+    buildProjection: "targets/java/build",
+    validators: [async ({sourcePath, buildPath, targetId}) => {
+      // Run a caller-owned checker/compiler with exact arguments and these exact paths.
+      // Return metadata for every regular file it produced below buildPath.
+      return [{
+        path: "Main.class",
+        mediaType: "application/java-vm",
+        role: "compiler-output"
+      }]
+    }]
+  }]
+})
+```
+
+The constructor requires a stable lowercase project ID, one canonical absolute project root, one dedicated canonical publication directory strictly beneath that project root, and at least one canonical absolute source root within the project. Publication and source roots must remain disjoint both lexically and after real-path resolution. Absolute roots use the active host's native separator and reject its non-native separator alias; target-relative artifact and projection paths remain portable POSIX-style paths. The publication directory must be on a filesystem that supports atomic same-directory regular-file replacement and directory synchronization, and must not be changed concurrently by an untrusted process.
+
+Each publication request supplies a unique generation ID and a complete ordered target array. A target has its own stable ID, explicit language-neutral `text`, `binary`, or `application` role, immutable `GeneratedArtifactSet`, and disjoint normalized source/build projections. Empty, absolute, dot, dot-dot, separator-alias, duplicate, portable case-fold, nested, manifest-reserved, escaping, and symlink-traversing paths fail before reachability. The class serializes calls made through one instance; callers must ensure there is only one project publication writer across instances and processes.
+
+Validators receive one frozen object containing only `targetId`, the exact generation-scoped `sourcePath`, and the exact generation-scoped `buildPath`. The publisher does not grant a project root, publication root, command runner, or general filesystem capability. A validator may create regular files below its build path and must return ordered `{path, mediaType, role}` metadata for every such file. The publisher inventories the complete generation namespace, including all intermediate directories, and allows only the manifest plus declared source/build projections and files. Every projection root and descended directory is checked without following symbolic links before its entries are read. Undeclared siblings or directories, missing files, symlinks, special nodes, hard-linked files, duplicate reported device/inode identities, collisions, invalid metadata, callback failures, or generated-source mutation reject the candidate.
+
+## Owned layout and manifests
+
+The publisher owns these reserved entries beneath the caller-supplied root:
+
+```text
+<publication-root>/
+  active-generation.json
+  generations/
+    <generation-id>/
+      manifest.json
+      <target source projection>/...
+      <target build projection>/...
+  .semantifold-publication-journal/
+    <generation-id>.json
+```
+
+Generation directories are immutable after publication. `SemantifoldGenerationManifest` version 1 records the project and generation identities plus every ordered target ID and language-neutral role, backend target IDs, source/build projections, target metadata, generated artifacts, and validator/compiler outputs. Each generated-artifact record includes its relative path, exact UTF-8 or binary byte length, media type, role, SHA-256 hash, and complete rich/byte/synthetic provenance. Each build-artifact record includes its relative path, exact byte length, media type, language-neutral role, and SHA-256 hash.
+
+The publisher synchronizes staged files, validator outputs, the manifest, and candidate directories. It then reads and hashes every referenced file and reconstructs every `GeneratedArtifactSet` from the staged bytes plus persisted provenance. This revalidates mapping filenames, generated content, Source Map projections, byte ranges, roles, target metadata, ordering, and entry identity before the generation can become active. Manifest JSON is deterministic apart from the explicit generation identity and the deterministic input content.
+
+`active-generation.json` is a `SemantifoldActiveGeneration` version-1 record containing the project ID, generation ID, and SHA-256 manifest hash. The manifest intentionally contains no timestamps or ambient host paths.
+
+## Commit and reader state machine
+
+Publication uses this order:
+
+1. Validate identities, roots, projections, portable collisions, existing state, and the host replacement contract.
+2. Write and synchronize an ownership journal, then create one unique candidate generation.
+3. Stage, synchronize, validate, hash, and re-read every generated and build artifact plus the versioned manifest.
+4. Write and synchronize a sibling temporary active pointer.
+5. Atomically rename that regular file over `active-generation.json` in the same directory.
+6. Synchronize the publication-root directory, then clean the no-longer-needed journal and synchronize the journal directory.
+
+There is no copy, truncate, live-pointer rewrite, per-target promotion, or source/build directory rename fallback. Before step 5, the prior pointer remains authoritative. Once step 5 succeeds, the new complete generation is the visible authority; a later directory-sync failure is reported as `PUBLICATION_POINTER_FAILED` without deleting that generation. A journal-cleanup failure returns the committed result with `cleanupPending: true` and cannot roll the pointer back.
+
+Readers call `resolveActive()`. It reads the pointer exactly once, verifies the selected manifest, complete filesystem inventory, and all referenced bytes/provenance, then validates journal state without consulting the pointer again. It returns frozen `generationPath`, `manifestPath`, manifest, target ID/role plus `sourcePath`/`buildPath` values, and truthful `cleanupPending` state for a committed journal. Malformed or ambiguous journal state fails closed. A reader must retain that resolved result for its whole operation and must not re-read the active pointer between files. Old committed generations remain present because an existing reader may still hold their paths. Retention and garbage collection are deliberately outside Task 038.
+
+## Recovery
+
+The journal proves publisher ownership of a candidate and its sibling temporary pointer; it is not the commit mechanism. Before a later valid publication starts, reconciliation first validates any active pointer and selected generation. It then:
+
+- removes a journal-proven candidate and temporary pointer when another generation is active or no pointer exists;
+- retains a journal-proven candidate when the active pointer selects it, removing only its stale temporary pointer and journal; and
+- leaves every unjournaled, committed, or otherwise ambiguous generation untouched.
+
+Every candidate-directory deletion is followed by synchronization of `generations/`, and every temporary-pointer deletion is followed by synchronization of the publication directory, before the journal may be removed and its own directory synchronized. An interruption therefore cannot durably retain an orphan after durably losing its only ownership proof.
+
+Malformed pointers, manifests, hashes, journals, symlinks, special nodes, project mismatches, and ambiguous reserved entries fail closed with `SemantifoldDiagnostic`. Filesystem failures are preserved as `cause`; stable diagnostic details avoid embedding unrelated ambient environment data.
+
+The main diagnostic families are `INVALID_PUBLICATION_REQUEST`, `INVALID_PUBLICATION_ROOT`, `INVALID_PUBLICATION_PROJECTION`, `PUBLICATION_SOURCE_OVERLAP`, `PUBLICATION_PATH_COLLISION`, `PUBLICATION_PROJECTION_OVERLAP`, `PUBLICATION_PATH_ESCAPE`, `PUBLICATION_SYMLINK_TRAVERSAL`, `PUBLICATION_HARD_LINK`, `PUBLICATION_GENERATION_INVENTORY_MISMATCH`, `PUBLICATION_GENERATION_VERIFICATION_FAILED`, `PUBLICATION_UNOWNED_CONFLICT`, `PUBLICATION_ATOMIC_REPLACE_UNSUPPORTED`, `PUBLICATION_STAGE_WRITE_FAILED`, `PUBLICATION_STAGE_HASH_MISMATCH`, `PUBLICATION_VALIDATION_FAILED`, `PUBLICATION_POINTER_FAILED`, `PUBLICATION_RECOVERY_FAILED`, `MALFORMED_ACTIVE_GENERATION`, `ACTIVE_GENERATION_MISSING`, and `MALFORMED_GENERATION_MANIFEST`.
+
+Task 038 does not load a project manifest, discover sources, derive compiler arguments, start a CLI or watcher, invoke language-specific tools, delete committed generations, or implement retention. Those responsibilities remain in Tasks 039 and later.

package/docs/watch-build-pipeline.md

@@ -0,0 +1,154 @@
+# Watch, build, and target-check pipeline
+
+## Status and verified baseline
+
+This document is a roadmap contract, not an implemented CLI contract.
+
+The inspected baseline is `semantifold@0.7.0`, annotated tag `v0.7.0`, commit `8cd70c5a6c7de98df2d4a183d6555ad4f3d5ba5a`. At that revision:
+
+- JavaScript/JSDoc is a registered source frontend and Java is a registered text backend.
+- `parse`, `parseProgram`, `generateArtifactSet`, and `generateProgramArtifactSet` are public, deterministic, in-memory APIs.
+- `discoverCanonicalToolchain` finds the configured `javac` and `java` executables.
+- `runAcceptanceStages` materializes an artifact set in a temporary directory and runs caller-supplied exact argument arrays with bounded process execution.
+- A bounded direct proof parsed the 2,648-byte JSDoc fixture `spec/fixtures/compatibility/program.js`, generated `Main.java`, compiled it with `javac 25.0.4`, executed it with Java 25.0.4, and observed the canonical output `compat\nbranch-ok!\npresent\nabsent\n7\n2\n3\n5\n4\n`.
+- `package.json` has no `bin` entry. The public API does not persist generated artifacts, load a project manifest, derive compiler arguments from a target, or watch source files.
+- The only repository file-change watcher is a test helper; there is no product watch coordinator, incremental dependency graph, event coalescing policy, or long-lived compiler owner.
+
+The released baseline therefore already proves **JavaScript/JSDoc → semantic IR → Java → `javac`** as a one-shot composition. Task 038 now adds the public language-neutral `GeneratedArtifactPublisher`, immutable generation manifests, one atomic active pointer, pointer-once resolution, and journal-bounded recovery described below. Project configuration, target-owned check plans, CLI behavior, and watching remain Tasks 039 and later.
+
+## Product outcome
+
+A project can declare explicit source modules, one or more generated targets, one publisher-owned project publication root, target-relative generated/build projections, and whether each target is checked with its real toolchain. It can then run:
+
+- a planned one-shot `semantifold build`; or
+- a planned long-lived `semantifold watch`.
+
+For the first end-to-end slice, changing a JSDoc-typed JavaScript source regenerates Java and compiles the staged candidate with `javac`. A successful cycle publishes one coherent generated-source/compiler-output generation through one active-generation pointer switch. A parse, semantic, generation, tool-discovery, compiler, or publication failure reports a located diagnostic, keeps the process alive in watch mode, and leaves the previous successful generation usable. A later valid edit recovers without restarting the watcher.
+
+The architecture is target-neutral. Every current text backend eventually supplies one target-owned check plan. Every current source frontend participates through the existing parser/project APIs; the watcher never contains language syntax logic.
+
+## Architecture boundaries
+
+### Project manifest and loader
+
+A versioned, strict manifest declares ordered source modules and target entries. Source language, module identity, entry identity, target language/role, one project publication root, and each target's generated/check subpaths inside an immutable project generation are explicit. These target paths are logical projections, not independently promoted filesystem roots. Version 1 uses explicit file paths rather than introducing a glob language or package-manager resolution.
+
+The loader owns path normalization and rejects unknown fields, duplicate module/target identities, ambiguous target roles, source/publication overlap, publication roots outside the project root, colliding or nested target projections, and symlink/path traversal across the owned publication root. It returns an immutable project request and never parses language syntax.
+
+### Snapshot builder
+
+Each cycle reads a complete content snapshot of every declared source and the project manifest. File events are hints only. Before parsing, the coordinator re-reads and hashes the declared graph in stable order; it never assumes an event filename is complete or authoritative.
+
+A snapshot is immutable for the cycle. Edits arriving during parse/generation/check mark the watcher dirty and cause one immediate follow-up cycle after the current owned work closes. Cycles never overlap.
+
+### Existing semantic compiler core
+
+Frontends, semantic validation, linking, and backends remain pure with respect to the project filesystem. They consume the immutable snapshot and return complete `GeneratedArtifactSet` values. No frontend or backend starts watchers, writes files, spawns a compiler, or reads undeclared project files.
+
+### Transactional artifact publication (Task 038 implemented)
+
+`GeneratedArtifactPublisher` stages one complete project candidate as a new immutable generation beneath a dedicated publisher-owned publication directory inside the canonical project root. Every target carries an explicit language-neutral role, and its generated artifacts, mappings, and optional validator/compiler outputs live in generation-relative subtrees on the same filesystem. The publisher validates paths, mappings, hashes, hard-link identities, complete generation inventory, collisions, ownership, and manifests before touching committed state. Its public contract and caller obligations are documented in [transactional generated-artifact publication](generated-artifact-publication.md).
+
+When checking is enabled, target checks run against those staged generation subtrees. Only a fully generated and successfully checked project candidate may be published. The generation manifest is the explicit versioned authority for files the publisher created; it never authorizes mutation outside the publication root.
+
+Publication commits exactly one small active-generation pointer by writing and syncing a sibling temporary pointer, atomically replacing the prior pointer on the same filesystem, and syncing the parent directory before reporting success. A host filesystem that cannot provide atomic same-directory regular-file replacement is rejected before publication; copying, truncating, or rewriting the live pointer in place is never a fallback. Readers resolve that pointer once and consume all generated/build paths from the referenced immutable generation. Configured generated-source and compiler-output names are logical projections returned from that resolved generation; they are never independent directories, symlinks, or manifests promoted one after another. The same rule covers every target in a multi-target cycle.
+
+An interruption before the pointer replacement leaves the previous generation active; an interruption after it leaves either the old or new complete pointer durably recoverable, never a composite state. A recovery journal may finish cleanup of unpublished candidates and temporary pointer files, but it is not used to claim atomicity across multiple filesystem operations. Candidate and temporary-pointer parent directories are synchronized before the proving journal is removed and its directory synchronized. Publication never removes a committed generation, because a reader may already hold its resolved snapshot; committed-generation retention/garbage collection is a separate explicit policy. A failed candidate is removed without changing the active pointer.
+
+### Target check-plan registry
+
+The existing language registry remains authoritative for target identity, roles, declared acceptance stages, and required toolchain IDs. A planned target-check capability derives exact stage requests from a validated staged artifact set and its generation-scoped target build subtree.
+
+The generic runner owns process lifecycle, locale/timezone normalization, output capture, timeout/cancellation, and close observation. A target check plan owns filenames, exact argv, toolchain IDs, offline/cache policy, and which stages constitute a non-executing developer check. The watcher does not switch on language IDs.
+
+The initial check plan is Java/`javac`. Later tasks add current text targets in two cohorts:
+
+- interpreted/managed: PHP, Ruby, JavaScript, TypeScript, Kotlin/JVM, Python, and C# (Java is supplied by the initial slice);
+- native/project: Go, C, C++, Rust, Swift, Dart, and Zig.
+
+A watch check validates or compiles but does not execute generated application code by default. Restore/download behavior is never an implicit per-edit action; targets with project metadata use pinned, offline, isolated state and rerun prerequisite stages only when their declared inputs change.
+
+### Watch coordinator
+
+One lifecycle-aware class owns filesystem subscriptions, content snapshots, cycle state, the active checker child, reporting, and shutdown. Its state machine is:
+
+```text
+starting -> building -> ready
+                 |       |
+                 v       v
+               failed <- rebuilding
+                 |       |
+                 +-- valid edit --+
+
+any live state -- SIGINT/SIGTERM --> stopping --> stopped
+```
+
+`failed` means the last candidate failed; it does not discard the last successful output or terminate watch mode. A watcher event during `building`/`rebuilding` sets one dirty generation marker. The coordinator starts exactly one follow-up after the owned child closes. It never starts a retry while prior compiler work is still live.
+
+Native filesystem events require a bounded reconciliation strategy because Node and TypeScript both document platform-dependent watcher behavior. Version 1 watches the explicit manifest/source files and relevant parent directories, re-scans the declared graph after each hint, supports a documented polling fallback, and suppresses no-op cycles by content hash. The publication root and every generated/build projection are rejected as source roots and are never watched.
+
+### Reporting contract
+
+Human output is concise and stable. A machine-readable mode emits one JSON record per state transition with at least project identity, monotonically increasing cycle number, source snapshot hash, target ID/role, state, timing, changed paths, tool identity, exit status, and a structured diagnostic when present.
+
+Exactly one terminal cycle record is emitted per cycle. Process exit is non-zero for failed one-shot builds, but ordinary watch-cycle failures keep the watcher process alive. Startup/configuration failure exits. Signal-driven shutdown forwards cancellation to an active owned child, waits for `close`, removes subscriptions and staging state, and exits once.
+
+## Generation and failure semantics
+
+1. Initial watch startup performs a real build; it does not wait for the first edit.
+2. Event bursts coalesce into one snapshot rebuild after a bounded quiet period.
+3. Content hashes, not timestamps alone, decide whether a cycle is a no-op.
+4. Parse/generation failure never starts a target checker and never publishes.
+5. Tool discovery/check failure never publishes the staged generated source or build products.
+6. Publication failure before the active-pointer replacement leaves the previous generation active; failure after that replacement reports the new generation as committed and limits recovery to cleanup.
+7. A valid edit after failure performs a fresh complete cycle and may publish normally.
+8. Deleted or renamed declared sources are ordinary invalid snapshots until the manifest is updated; stale owned outputs are not silently treated as current.
+9. Multiple target entries are staged from one source snapshot beneath one immutable project generation. The manifest defines whether all targets must pass before the single active pointer is switched; version 1 defaults to one all-or-nothing project generation.
+10. No shell command strings, compiler-specific branches in the watcher, implicit network access, or arbitrary user hooks are introduced.
+
+## Cross-language scope
+
+“Works for all languages” means:
+
+- every registered frontend can be an explicitly declared watched source when the selected semantic features are supported;
+- every current text backend has an explicit real-toolchain check plan or a fail-loud reason it cannot be checked;
+- the same watcher/materializer/runner code handles every source/target pairing;
+- acceptance uses a bounded matrix: all frontends and all text targets are covered, with the JavaScript/JSDoc-to-Java path plus representative cross-family paths, not a quadratic promise that every semantic feature works for every pair.
+
+Binary and application targets use the same generation watcher only after explicit target policy is defined. Browser Wasm can validate/instantiate in its own lane. Android and Flutter checks are heavier platform builds and must be opt-in rather than an automatic keystroke default. iOS remains generation-only until the separately deferred Apple/Xcode acceptance work is authorized; this roadmap does not reopen Tasks 026 or 027.
+
+## Test strategy
+
+- Unit contracts cover strict manifest validation, snapshot hashing, event coalescing, dirty-during-build behavior, no-op suppression, generation manifests, single-pointer publication, recovery journals, reader snapshot resolution, and exact check-plan construction.
+- Real temporary-directory tests cover create/change/delete/rename behavior and ensure generated roots do not self-trigger.
+- Real-child lifecycle tests cover spawn failure, non-zero compiler close, late output, signal forwarding, shutdown during an active check, and no leaked child/staging directory.
+- Real `javac` acceptance proves the initial JavaScript/JSDoc-to-Java cycle, compiler failure reporting at the check boundary, an unchanged last-good active pointer, recovery after a valid edit, and source/class execution from one resolved committed generation.
+- The terminal text-target matrix invokes every declared real checker/compiler in CI and fails rather than skips when a configured tool is unavailable.
+- Focused local tests remain small; full language/toolchain matrices run only in the existing CI lanes.
+
+## Dependency order
+
+```text
+015 artifact/toolchain foundation
+        |
+        038 transactional publication
+        |
+010 ----+---- 039 project manifest + one-shot build CLI
+                   |
+                   040 generic check plans + Java/javac
+                   |                    |
+             041 managed cohort    043 watcher coordinator
+             042 native cohort          |
+                   |                    044 JS/JSDoc -> Java watch slice
+                   +--------------------+
+                              |
+                  045 terminal all-text-language matrix
+                              |
+                  046 later binary/application policies
+```
+
+Task 038 is the implemented publication foundation. Tasks 039–045 are the remaining selected text-language delivery chain, with Task 045 as its single terminal acceptance task. Task 046 is a non-blocking later extension.
+
+## Non-goals
+
+This roadmap does not implement language-server/editor protocols, incremental semantic-IR invalidation, daemon IPC, remote/distributed builds, arbitrary compiler flags, shell hooks, package installation, source discovery by glob, runtime execution on every edit, hot reload into a running application, production deployment, signing, store submission, or Apple/Xcode acceptance. It does not change semantic language support or claim all-pairs feature compatibility.

package/index.js

@@ -16,6 +16,7 @@ export {SemantifoldDiagnostic}
 export {createCapabilityAuthority} from "./src/semantic/capabilities.js"
 export {languageCapabilities, supportedLanguages} from "./src/language-registry.js"
 export {createGeneratedArtifactSet} from "./src/artifacts.js"
+export {GeneratedArtifactPublisher} from "./src/publication.js"
 export {createStdlibContractRegistry, listStdlibModules, resolveStdlibModule} from "./src/semantic/stdlib.js"
 export {createStdlibProviderRegistry, listStdlibProviders} from "./src/stdlib-providers.js"
 export {negotiateStdlibProviders} from "./src/stdlib-negotiation.js"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "semantifold",
-  "version": "0.7.0",
+  "version": "0.9.0",
   "description": "Language-neutral semantic code analysis, transformation, generation, and execution toolkit",
   "type": "module",
   "main": "build/index.js",