utterplan 0.4.1__tar.gz → 0.5.0__tar.gz

{utterplan-0.4.1 → utterplan-0.5.0}/AGENTS.md

@@ -1,19 +1,20 @@
 # UtterPlan development rules
 
 1. UtterPlan is engine-independent.
-2. `.utterplan.json` schema v1 is the canonical inter-package contract.
+2. `.utterplan.toml` schema v5 is the current inter-package contract; historical JSON schemas v1–v4 remain migration inputs only.
 3. Python objects may not contain semantic state that cannot survive file roundtrip.
 4. UtterPlan ends before G2P.
 5. Core code must not import PyKokoro, PiperSynth, kokorog2p, piperg2p, ONNX Runtime, or audio packages.
-6. Public UtterancePlan objects contain no phonemes, model token IDs, models, sessions, or audio.
+6. Public FlowPlan objects contain no phonemes, model token IDs, models, sessions, or audio.
 7. Completed plans must not retain spaCy or provider documents.
 8. Every offset has a documented coordinate space.
 9. Default planning is deterministic.
 10. Schema changes require explicit compatibility review.
 11. Repository tests must be self-contained and must not require sibling consumer checkouts.
 12. Consumer-specific compatibility testing belongs in the consumer repository.
-13. Released schema files are immutable and retained under versioned paths.
+13. Released schema files are immutable and retained under versioned paths; v5 has no historical JSON Schema resource.
 14. Every supported older schema must retain a sequential migration path to current.
 15. Migrations operate on serialized plain data and may not call parser, planner, G2P, renderer, or audio stages.
 16. Schema changes require frozen historical fixture coverage, deterministic migration tests, and package artifact checks.
 17. Package version and schema version remain independent. Migration preserves a compiled plan; replanning is a separate operation.
+18. Optional compiler traces are sidecars; they must never alter executable plan content or identity.

{utterplan-0.4.1 → utterplan-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: utterplan
-Version: 0.4.1
+Version: 0.5.0
 Summary: Engine-independent text-to-speech planning compiler and interchange format
 Author-email: Holger Nahrstaedt <nahrstaedt@gmail.com>
 License:                                  Apache License
@@ -250,10 +250,9 @@ Dynamic: license-file
 
 UtterPlan is the canonical, engine-independent semantic compiler from one SSMD
 document (or explicitly selected plain-text document) to an executable speech
-plan. It produces deterministic language runs, prepared text, segments, resolved
-pauses, portable metadata, directives, markers, and render units shared by
-independent consumers such as ttsready and Readio. It does not manage books or
-consumer workspaces, and stops before G2P, synthesis, and audio.
+plan. It produces deterministic prepared text, ordered flow units, segment-local
+semantics, pause intents, portable document metadata, and typed directives for
+independent consumers such as ttsready and Readio. It does not manage or modify book workspaces; `compile-book` reads an SSMDBook manifest only to discover selected source chapters. It stops before G2P, synthesis, and audio.
 
 ## CLI
 
@@ -285,18 +284,39 @@ utterplan compile-many examples/*.ssmd --output-dir build/plans
 
 This writes one `.utterplan.toml` per source plus an atomically refreshed `compile-report.toml`. Use `--fail-fast` to skip later inputs after a failure, `--force` to replace existing plans, or `--report PATH` to choose another report location.
 
+Compile a selected chapter range directly from an editable SSMDBook workspace:
+
+```bash
+cd "Platform Decay - Martha Wells.ssmdbook"
+
+utterplan compile-book chapters --chapters 5-17
+```
+
+The command reads the workspace manifest to discover chapters and writes independent v5 plans and a report under `utterplan/`:
+
+```text
+utterplan/
+├── compile-report.toml
+├── chapter-0005.utterplan.toml
+├── chapter-0006.utterplan.toml
+├── ...
+└── chapter-0017.utterplan.toml
+```
+
+Omit `--chapters` to compile all manifest chapters. Use comma-separated source-number selections for non-contiguous ranges, for example `utterplan compile-book chapters --chapters 1,3-5`. The book root itself is also accepted as input; see the [SSMDBook guide](docs/ssmdbook.md).
 The CLI also provides:
 
 ```bash
 utterplan --version
 utterplan validate chapter.utterplan.toml
 utterplan inspect chapter.utterplan.toml --segment 0
-utterplan inspect chapter.utterplan.toml --semantic-boundaries
-utterplan explain chapter.utterplan.toml
+utterplan inspect chapter.utterplan.toml --boundaries --tokens
+utterplan explain chapter.utterplan.toml --trace chapter.trace.toml
+utterplan inspect-trace chapter.trace.toml --preparation --boundaries
 utterplan inspect-attempt chapter.attempt.toml --issues
 ```
 
-`explain` presents the compiled plan as a human-readable speech plan, while `inspect` exposes lower-level diagnostic fields.
+`explain` presents executable flow as a human-readable speech plan. Use `compile --trace PATH` to retain optional compiler provenance; `explain --trace PATH` and `inspect-trace` read that sidecar separately from the plan.
 
 `inspect-attempt` reads the separate `utterplan.planning-attempt.v1` TOML artifact,
 not a canonical plan. Use it to review renderability issues and safe repair
@@ -318,11 +338,11 @@ ssmd migrate old.ssmd --to 0.9
 ```
 
 `utterplan migrate` explicitly imports supported historical `.utterplan.json` plans and writes current TOML. It does not migrate SSMD source.
-Normal plan loading is TOML-only: `UtterancePlan.load()` and plan-inspection commands reject JSON rather than auto-detecting it.
+Normal plan loading is TOML-only: `FlowPlan.load()` and plan-inspection commands reject JSON rather than auto-detecting it.
 
 ## Planning defaults
 
-The minimal CLI defaults are explicit: `spokenform` is the default text-preparation backend, `tts` is the default pause mode, and `spacy off` is the default linguistic-resource policy. With `spacy off`, UtterPlan uses its deterministic fallback tokenizer and analysis and does not depend on an installed spaCy model.
+The minimal CLI defaults are explicit: `spokenform` is the default text-preparation backend, `tts` is the default pause activation mode, and `spacy off` is the default linguistic-resource policy. With `spacy off`, UtterPlan uses its deterministic fallback tokenizer and analysis and does not depend on an installed spaCy model.
 
 Python `PlannerConfig` defaults to `document_format="plain"`; set it to `"ssmd"` when a Python string contains SSMD source.
 
@@ -333,7 +353,7 @@ Python `PlannerConfig.renderability_mode` also defaults to safe punctuation-only
 ## Python API
 
 ```python
-from utterplan import PlannerConfig, UtterancePlan, compile_document
+from utterplan import PlannerConfig, FlowPlan, compile_document
 
 ssmd_source = """---
 ssmd_version: "0.9"
@@ -350,7 +370,7 @@ result = compile_document(
 plan = result.plan
 assert result.trace is not None
 plan.save("example.utterplan.toml")
-assert UtterancePlan.load("example.utterplan.toml") == plan
+assert FlowPlan.load("example.utterplan.toml") == plan
 ```
 
 `compile_document` is the stable public one-document API shared by consumers. The
@@ -378,21 +398,9 @@ Callbacks run synchronously and exceptions propagate. See the [Python API guide]
 
 ## Renderer-consumer boundary
 
-Renderers consume `PlanSegment.text`, which is prepared/spoken text, and use
-`spoken_start`/`spoken_end` for spoken-text coordinates. Resolved segment
-pauses, language, directives, annotations, boundaries, semantic boundaries,
-markers, units, and
-document metadata are public plan fields. Semantic boundaries are stable
-engine-neutral opportunities for spoken-text subdivision; they are not pause
-events and carry no duration or renderer policy. Plans contain no phonemes, model
-tokens, model sessions, renderer configuration, provider documents, or audio.
-For SSMD input, `plan.annotations` preserve declared source semantics and source
-provenance, `document_metadata` preserves portable header data, and
-`segment.directives` carries effective typed semantics after scope and voice-default
-resolution. Typed directives include voice, pronunciation, prosody, emphasis, say-as,
-substitution, audio references, and extension references. Audio and extensions are
-data only. UtterPlan does not fetch media or execute extension handlers.
+Renderers consume `FlowPlan.flow` in order. Each `FlowUnit` contains local `FlowSegment` objects with already-prepared speech text, effective language, segment-local token views, semantic pause intents, directives, markers, and optional heading level. Token `start`/`end` offsets are Python-character indexes into that segment's `text`; they are never document-global coordinates. Pause intents express `none`, semantic strengths, or exact authored times and do not prescribe engine-specific duration policy.
 
+`FlowPlan.document` carries compact document metadata and `FlowPlan.linguistics` identifies the analysis provider without retaining provider documents. Compiler source, preparation maps, diagnostics, and renderability details belong only in an optional trace sidecar. Plans contain no phonemes, model token IDs, models, sessions, renderer configuration, provider documents, or audio. UtterPlan does not fetch media or execute extension handlers.
 The intended dependency direction is:
 
 ```text
@@ -407,6 +415,7 @@ renderer repository rather than UtterPlan's test suite.
 
 - [Getting started](docs/getting-started.md)
 - [CLI](docs/cli.md)
+- [SSMDBook workflows](docs/ssmdbook.md)
 - [Format and schema](docs/format.md)
 - [Consumer guide](docs/consumer-guide.md)
 - [Architecture](docs/architecture.md)
@@ -418,10 +427,9 @@ renderer repository rather than UtterPlan's test suite.
 
 ## Versions
 
-The package version is dynamically derived from Git tags by setuptools-scm. Package version and UtterPlan schema version are independent. Current `.utterplan.toml` plans use semantic schema v4; released schema v1, v2, and v3 remain immutable and supported through sequential migrations.
+The package version is dynamically derived from Git tags by setuptools-scm. Package version and UtterPlan schema version are independent. Current `.utterplan.toml` plans use semantic schema v5. Historical JSON Schemas v1–v4 remain immutable and supported through sequential migrations; v5 deliberately has no JSON Schema resource because its canonical executable contract is the validated TOML/FlowPlan model.
 
-Schema v4 adds stable `SemanticBoundary` records in spoken-text coordinates. Migration converts serialized plan data only: the registered v3-to-v4 step preserves existing evidence and derives topology where possible, but does not rerun parsing, NLP, planning, G2P, rendering, or audio processing. Unit hashes use `utterplan-unit-v3` and include relative semantic-boundary positions; older hash algorithms remain available for their historical migration paths.
-Schema v4 retains typed renderer-neutral SSMD semantics, final pass-B token facts, and per-language-run provider provenance. Plans never serialize provider documents, models, sessions, phonemes, token IDs, or audio.
+Schema v5 is a breaking redesign around ordered flow units and segment-local semantics. It preserves deterministic output and moves preparation/source provenance to an optional TOML trace sidecar, which does not affect plan identity. Migration converts serialized plan data only: v1–v4 migrate sequentially to v5 without reparsing, NLP, planning, G2P, rendering, or audio processing. The v4-to-v5 projection fails safely when historical token coordinates or pause semantics cannot be verified.
 
 ## Development
 

{utterplan-0.4.1 → utterplan-0.5.0}/README.md

@@ -7,10 +7,9 @@
 
 UtterPlan is the canonical, engine-independent semantic compiler from one SSMD
 document (or explicitly selected plain-text document) to an executable speech
-plan. It produces deterministic language runs, prepared text, segments, resolved
-pauses, portable metadata, directives, markers, and render units shared by
-independent consumers such as ttsready and Readio. It does not manage books or
-consumer workspaces, and stops before G2P, synthesis, and audio.
+plan. It produces deterministic prepared text, ordered flow units, segment-local
+semantics, pause intents, portable document metadata, and typed directives for
+independent consumers such as ttsready and Readio. It does not manage or modify book workspaces; `compile-book` reads an SSMDBook manifest only to discover selected source chapters. It stops before G2P, synthesis, and audio.
 
 ## CLI
 
@@ -42,18 +41,39 @@ utterplan compile-many examples/*.ssmd --output-dir build/plans
 
 This writes one `.utterplan.toml` per source plus an atomically refreshed `compile-report.toml`. Use `--fail-fast` to skip later inputs after a failure, `--force` to replace existing plans, or `--report PATH` to choose another report location.
 
+Compile a selected chapter range directly from an editable SSMDBook workspace:
+
+```bash
+cd "Platform Decay - Martha Wells.ssmdbook"
+
+utterplan compile-book chapters --chapters 5-17
+```
+
+The command reads the workspace manifest to discover chapters and writes independent v5 plans and a report under `utterplan/`:
+
+```text
+utterplan/
+├── compile-report.toml
+├── chapter-0005.utterplan.toml
+├── chapter-0006.utterplan.toml
+├── ...
+└── chapter-0017.utterplan.toml
+```
+
+Omit `--chapters` to compile all manifest chapters. Use comma-separated source-number selections for non-contiguous ranges, for example `utterplan compile-book chapters --chapters 1,3-5`. The book root itself is also accepted as input; see the [SSMDBook guide](docs/ssmdbook.md).
 The CLI also provides:
 
 ```bash
 utterplan --version
 utterplan validate chapter.utterplan.toml
 utterplan inspect chapter.utterplan.toml --segment 0
-utterplan inspect chapter.utterplan.toml --semantic-boundaries
-utterplan explain chapter.utterplan.toml
+utterplan inspect chapter.utterplan.toml --boundaries --tokens
+utterplan explain chapter.utterplan.toml --trace chapter.trace.toml
+utterplan inspect-trace chapter.trace.toml --preparation --boundaries
 utterplan inspect-attempt chapter.attempt.toml --issues
 ```
 
-`explain` presents the compiled plan as a human-readable speech plan, while `inspect` exposes lower-level diagnostic fields.
+`explain` presents executable flow as a human-readable speech plan. Use `compile --trace PATH` to retain optional compiler provenance; `explain --trace PATH` and `inspect-trace` read that sidecar separately from the plan.
 
 `inspect-attempt` reads the separate `utterplan.planning-attempt.v1` TOML artifact,
 not a canonical plan. Use it to review renderability issues and safe repair
@@ -75,11 +95,11 @@ ssmd migrate old.ssmd --to 0.9
 ```
 
 `utterplan migrate` explicitly imports supported historical `.utterplan.json` plans and writes current TOML. It does not migrate SSMD source.
-Normal plan loading is TOML-only: `UtterancePlan.load()` and plan-inspection commands reject JSON rather than auto-detecting it.
+Normal plan loading is TOML-only: `FlowPlan.load()` and plan-inspection commands reject JSON rather than auto-detecting it.
 
 ## Planning defaults
 
-The minimal CLI defaults are explicit: `spokenform` is the default text-preparation backend, `tts` is the default pause mode, and `spacy off` is the default linguistic-resource policy. With `spacy off`, UtterPlan uses its deterministic fallback tokenizer and analysis and does not depend on an installed spaCy model.
+The minimal CLI defaults are explicit: `spokenform` is the default text-preparation backend, `tts` is the default pause activation mode, and `spacy off` is the default linguistic-resource policy. With `spacy off`, UtterPlan uses its deterministic fallback tokenizer and analysis and does not depend on an installed spaCy model.
 
 Python `PlannerConfig` defaults to `document_format="plain"`; set it to `"ssmd"` when a Python string contains SSMD source.
 
@@ -90,7 +110,7 @@ Python `PlannerConfig.renderability_mode` also defaults to safe punctuation-only
 ## Python API
 
 ```python
-from utterplan import PlannerConfig, UtterancePlan, compile_document
+from utterplan import PlannerConfig, FlowPlan, compile_document
 
 ssmd_source = """---
 ssmd_version: "0.9"
@@ -107,7 +127,7 @@ result = compile_document(
 plan = result.plan
 assert result.trace is not None
 plan.save("example.utterplan.toml")
-assert UtterancePlan.load("example.utterplan.toml") == plan
+assert FlowPlan.load("example.utterplan.toml") == plan
 ```
 
 `compile_document` is the stable public one-document API shared by consumers. The
@@ -135,21 +155,9 @@ Callbacks run synchronously and exceptions propagate. See the [Python API guide]
 
 ## Renderer-consumer boundary
 
-Renderers consume `PlanSegment.text`, which is prepared/spoken text, and use
-`spoken_start`/`spoken_end` for spoken-text coordinates. Resolved segment
-pauses, language, directives, annotations, boundaries, semantic boundaries,
-markers, units, and
-document metadata are public plan fields. Semantic boundaries are stable
-engine-neutral opportunities for spoken-text subdivision; they are not pause
-events and carry no duration or renderer policy. Plans contain no phonemes, model
-tokens, model sessions, renderer configuration, provider documents, or audio.
-For SSMD input, `plan.annotations` preserve declared source semantics and source
-provenance, `document_metadata` preserves portable header data, and
-`segment.directives` carries effective typed semantics after scope and voice-default
-resolution. Typed directives include voice, pronunciation, prosody, emphasis, say-as,
-substitution, audio references, and extension references. Audio and extensions are
-data only. UtterPlan does not fetch media or execute extension handlers.
+Renderers consume `FlowPlan.flow` in order. Each `FlowUnit` contains local `FlowSegment` objects with already-prepared speech text, effective language, segment-local token views, semantic pause intents, directives, markers, and optional heading level. Token `start`/`end` offsets are Python-character indexes into that segment's `text`; they are never document-global coordinates. Pause intents express `none`, semantic strengths, or exact authored times and do not prescribe engine-specific duration policy.
 
+`FlowPlan.document` carries compact document metadata and `FlowPlan.linguistics` identifies the analysis provider without retaining provider documents. Compiler source, preparation maps, diagnostics, and renderability details belong only in an optional trace sidecar. Plans contain no phonemes, model token IDs, models, sessions, renderer configuration, provider documents, or audio. UtterPlan does not fetch media or execute extension handlers.
 The intended dependency direction is:
 
 ```text
@@ -164,6 +172,7 @@ renderer repository rather than UtterPlan's test suite.
 
 - [Getting started](docs/getting-started.md)
 - [CLI](docs/cli.md)
+- [SSMDBook workflows](docs/ssmdbook.md)
 - [Format and schema](docs/format.md)
 - [Consumer guide](docs/consumer-guide.md)
 - [Architecture](docs/architecture.md)
@@ -175,10 +184,9 @@ renderer repository rather than UtterPlan's test suite.
 
 ## Versions
 
-The package version is dynamically derived from Git tags by setuptools-scm. Package version and UtterPlan schema version are independent. Current `.utterplan.toml` plans use semantic schema v4; released schema v1, v2, and v3 remain immutable and supported through sequential migrations.
+The package version is dynamically derived from Git tags by setuptools-scm. Package version and UtterPlan schema version are independent. Current `.utterplan.toml` plans use semantic schema v5. Historical JSON Schemas v1–v4 remain immutable and supported through sequential migrations; v5 deliberately has no JSON Schema resource because its canonical executable contract is the validated TOML/FlowPlan model.
 
-Schema v4 adds stable `SemanticBoundary` records in spoken-text coordinates. Migration converts serialized plan data only: the registered v3-to-v4 step preserves existing evidence and derives topology where possible, but does not rerun parsing, NLP, planning, G2P, rendering, or audio processing. Unit hashes use `utterplan-unit-v3` and include relative semantic-boundary positions; older hash algorithms remain available for their historical migration paths.
-Schema v4 retains typed renderer-neutral SSMD semantics, final pass-B token facts, and per-language-run provider provenance. Plans never serialize provider documents, models, sessions, phonemes, token IDs, or audio.
+Schema v5 is a breaking redesign around ordered flow units and segment-local semantics. It preserves deterministic output and moves preparation/source provenance to an optional TOML trace sidecar, which does not affect plan identity. Migration converts serialized plan data only: v1–v4 migrate sequentially to v5 without reparsing, NLP, planning, G2P, rendering, or audio processing. The v4-to-v5 projection fails safely when historical token coordinates or pause semantics cannot be verified.
 
 ## Development
 

utterplan-0.5.0/docs/architecture.md

@@ -0,0 +1,63 @@
+# Architecture
+
+UtterPlan is an engine-independent compiler from one SSMD document (or explicitly selected plain-text document) to an executable speech plan. Consumers compile documents independently; UtterPlan does not load books or manage workspaces.
+
+```text
+one SSMD document / explicit plain text
+                 |
+                 v
+       parse -> language semantics -> spokenform preparation
+                 -> segmentation -> directive resolution
+                 -> semantic pause intent -> internal compiler plan
+                 |                                |
+                 |                                +--> optional trace sidecar
+                 v
+       FlowPlan (ordered units, local segments)
+              /                 \
+       consumer A             consumer B
+                 |
+                 v
+          G2P / renderer / audio
+```
+
+The dependency direction is from SSMD, spokenform, and phrasplit into UtterPlan, then from UtterPlan to consumer packages. UtterPlan must not depend on ttsready, Readio, engine registries, G2P packages, model runtimes, or audio packages. Book iteration belongs to consumers and conversion tools.
+
+## Compiler and public plan boundary
+
+`compile_document` returns a `CompileResult` containing the current immutable `FlowPlan`, renderer-independent diagnostics, and an optional `PreparationTrace`. `UtterancePlanner.plan` also returns `FlowPlan`. Compiler-only normalized graph state remains private. When requested, the trace retains source text/hash, coordinate maps, preparation provenance, diagnostics, and renderability evidence in a separate versioned TOML sidecar. Trace state is not renderer input and does not affect plan identity or TOML.
+
+`FlowPlan` is the current v5 executable contract: ordered `FlowUnit` objects each own ordered `FlowSegment` records. Segments include prepared text, effective language, local tokens, semantic pause intents, directives, markers, and optional heading metadata. `DocumentInfo` retains compact portable document information; `LinguisticProvenance` identifies providers without retaining their documents. UtterPlan ends before G2P. Engine/model selection, synthesis, and audio composition happen downstream.
+
+## SSMD and preparation
+
+The document parser calls SSMD with `dialect="0.9"` for every SSMD source path, including unversioned fragments selected with `document_format="ssmd"`. UtterPlan does not parse SSMD 0.8 or migrate source. Older documents must first be converted with `ssmd migrate FILE --to 0.9`. The `utterplan migrate` command imports serialized legacy plans and applies schema migrations only.
+
+SSMD header language is authoritative over a fallback. The CLI can omit `--language` when the header declares a language; plain input always needs a fallback. A fallback never forces or rewrites document semantics.
+
+Preparation creates structural and spoken representations internally. The executable FlowPlan retains only prepared segment text; it does not retain the exact source string, structural text, dense source-to-spoken maps, parser annotations, or global compiler token tables. These diagnostics/provenance survive only when an optional trace is requested. See the [coordinate-space contract](coordinate-spaces).
+
+## Pause and directive semantics
+
+Authored SSMD breaks and semantic pause strengths project to local `PauseIntent` values on segment edges. The plan does not invent numeric durations or serialize an activation policy. Application pause mode controls which intents the compiler activates; consumer-specific timing policy remains downstream. Exact authored timed breaks remain exact. Ambiguous historical pause evidence fails migration rather than being guessed.
+
+Typed directives are renderer-neutral values resolved from SSMD scopes and voice defaults. Logical voice bindings remain references, not concrete engine voices. Audio references and extension names are data only; UtterPlan does not fetch media or execute handlers.
+
+## Identity and determinism
+
+Plan identity is deterministic and renderer-independent. The v5 `plan_id` hashes executable flow and compact portable metadata rather than source text or TOML formatting. Flow-unit hashes use `utterplan-flow-v1` over local ordered segment semantics. Trace data, provider documents, model sessions, diagnostics, producer/package version, G2P output, and audio do not define executable identity. Package version is independent of explicit plan `schema_version`.
+
+## Persistence compatibility boundary
+
+Current plans persist as deterministic `.utterplan.toml` files with schema v5; `FlowPlan.load()` accepts TOML only. The v5 TOML codec validates and constructs `FlowPlan`. Schema v5 deliberately has no JSON Schema resource or current JSON plan format. Frozen JSON Schema definitions for historical schemas v1–v4 remain under versioned resources.
+
+```text
+current plan:
+.utterplan.toml -- TOML decode / schema-v5 validation --> FlowPlan --> consumer
+
+legacy saved plan:
+.utterplan.json -- explicit `utterplan migrate` --> current .utterplan.toml
+```
+
+Migration is not planning. Supported historical plans migrate through the registered sequential chain v1 -> v2 -> v3 -> v4 -> v5. Every step operates on serialized plain data and does not rerun source parsing, preparation, NLP, planning, G2P, rendering, or audio processing. The v4-to-v5 projection verifies token spans and pause evidence and fails rather than guessing. Downgrades are not supported.
+
+`compile-many` orchestrates independent source documents in order. Each successful FlowPlan is validated and atomically saved before the next source is processed; ordinary failures are reported and later inputs continue unless fail-fast is requested. Its TOML report is operational data, not a FlowPlan.

{utterplan-0.4.1 → utterplan-0.5.0}/docs/changelog.md

@@ -4,6 +4,18 @@
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-10-09
+
+### Added
+
+- Added compile-book for selecting and compiling SSMDBook chapters into independent v5 plans with operational reports
+
+### Changed
+
+- **BREAKING:** Changed plan persistence to schema v5 with deterministic TOML, sequential legacy migration, and optional trace sidecars
+
+<!-- generated by releaseledger -->
+
 ## [0.4.1] - 2026-10-07
 
 ### Added

{utterplan-0.4.1 → utterplan-0.5.0}/docs/cli.md

@@ -9,7 +9,8 @@ usage: utterplan compile [-h] [--file FILE] [--language LANGUAGE]
                          [--text-preparation {spokenform,identity}]
                          [--pause-mode {tts,manual,auto}]
                          [--spacy {auto,off,sm,md,lg,trf}] [-o OUTPUT]
-                         [--force] [--stdout] [--renderability {strict,repair}]
+                         [--trace TRACE] [--force] [--stdout]
+                         [--renderability {strict,repair}]
                          [text ...]
 ```
 
@@ -65,6 +66,7 @@ Without `--output`, compile writes the pretty TOML plan to stdout. With
 `--force` is supplied. Add `--stdout` with an output path to write the file and
 also emit the same TOML plan to stdout.
 
+`--trace PATH` writes optional compiler provenance to a separate TOML sidecar. The trace is not embedded in, or hashed as part of, the executable plan. Keep and distribute the sidecar only when source text, preparation maps, diagnostics, or renderability evidence must be retained.
 Human status messages are written to stderr, never mixed into TOML stdout:
 
 ```bash
@@ -85,10 +87,35 @@ utterplan compile-many chapters/*.ssmd --output-dir build/plans \
   --language en-us --report build/compile-report.toml
 ```
 
-Source names determine output names: `chapter.ssmd`, `chapter.ssmd.md`, and `chapter.md` each map to `chapter.utterplan.toml`. Duplicate output names and a report path that collides with a plan output are rejected before any writes. Existing plans are protected unless `--force` is supplied. By default the command writes an atomically refreshed `compile-report.toml` in the output directory; the report is operational TOML, not a plan accepted by `UtterancePlan.load()`.
+Source names determine output names: `chapter.ssmd`, `chapter.ssmd.md`, and `chapter.md` each map to `chapter.utterplan.toml`. Duplicate output names and a report path that collides with a plan output are rejected before any writes. Existing plans are protected unless `--force` is supplied. By default the command writes an atomically refreshed `compile-report.toml` in the output directory; the report is operational TOML, not a plan accepted by `FlowPlan.load()`.
 
 Shared planning options include `--language`, `--input-format auto|plain|ssmd`, `--unit`, `--text-preparation`, `--pause-mode`, `--spacy`, and `--renderability`. Progress, repair notices, actionable errors, and the final counts are written to stderr. Exit status is 0 if every input succeeds, 1 if any input fails, and 2 for usage errors such as output collisions. `--fail-fast` skips later requests after the first failure but preserves all completed outputs and updates the report.
 
+## Compile chapters from an SSMDBook workspace
+
+`compile-book` discovers selected SSMD chapters from an editable `.ssmdbook` manifest and compiles each chapter independently through the batch compiler. Pass either the book directory or its canonical `chapters/` directory:
+
+```bash
+utterplan compile-book novel.ssmdbook --chapters 5-17
+utterplan compile-book novel.ssmdbook/chapters --chapters 1,3-5
+```
+
+The workflow used from inside the workspace is:
+
+```bash
+cd "Platform Decay - Martha Wells.ssmdbook"
+
+utterplan compile-book chapters --chapters 5-17
+```
+
+`--chapters` selects manifest `source_number` values, not filenames or manifest indexes. It accepts `all` (the default), one number, inclusive ranges, or comma-separated combinations such as `1,3-5,9`. Duplicate selections do not duplicate work, and selected chapters always follow manifest array order. A selected missing source number or malformed selector is rejected before compilation.
+
+By default, plans and the operational report are written to `<book-root>/utterplan/`. Use `--output-dir PATH` to override the destination. One canonical v5 `.utterplan.toml` plan is written per selected chapter, plus `compile-report.toml`. Existing plans are protected unless `--force` is given; `--fail-fast` skips later chapters after the first failure. `--report PATH` overrides the report location. Duplicate output names, report/plan collisions, and writes into the source `chapters/` directory or over `manifest.json` are rejected before writing.
+
+Shared planning options are `--language`/`--lang`, `--unit`, `--text-preparation`, `--pause-mode`, `--renderability`, and `--spacy`. Chapter documents are SSMD 0.9 by contract, so `compile-book` does not accept `--input-format`. The manifest is read-only: edited chapter content is compiled as it currently exists even if its recorded digest is stale, and UtterPlan never refreshes the manifest or chapter hashes. Only selected chapter files are opened, so an unrelated broken chapter outside the selection does not block the request.
+
+For the full editable-workspace behavior and examples, see the [SSMDBook guide](ssmdbook.md).
+
 ## Explain a plan
 
 Explain an existing compiled plan as a human-readable speech narrative:
@@ -98,7 +125,7 @@ utterplan explain chapter.utterplan.toml
 utterplan explain chapter.utterplan.toml --details
 ```
 
-The default output shows prepared wording, render units, ordered segments, languages, resolved pauses, headings, SSMD version/title/document language, effective typed directives, metadata, and warning/error codes with source locations. Add `--details` for IDs, offsets, provenance, hashes, plan identity, and token analysis beneath each segment. Use `inspect --tokens` for a compact token/provenance view.
+The default output explains executable flow: prepared segment text, unit order, effective languages, semantic pause intents, headings, markers, and typed directives. Compiler source diagnostics and preparation provenance are intentionally absent from the plan unless a trace sidecar was saved. Add `--details` for local token offsets, flow hashes, plan identity, and trace preparation mappings. Pass `--trace PATH` to attach a matching sidecar; the TOML trace can also be read independently with `inspect-trace`.
 
 ## Inspect a persisted planning attempt
 
@@ -116,7 +143,7 @@ The default view summarizes status, renderability, and candidate size. `--issues
 shows source context and conservative repair assessments; segment and unit selectors
 accept an ID or zero-based index. `--json` emits the complete artifact as JSON for
 inspection, but the persisted attempt itself remains TOML. A blocked candidate is an
-inspect-only draft, not a canonical plan, and is rejected by `UtterancePlan.load()`.
+inspect-only draft, not a canonical plan, and is rejected by `FlowPlan.load()`.
 
 ## Planning controls
 
@@ -142,12 +169,13 @@ utterplan validate chapter.ssmd.md
 utterplan validate plain.txt --input-format plain --language en-us
 utterplan inspect chapter.utterplan.toml --segment 0
 utterplan inspect chapter.utterplan.toml --unit 0 --boundaries --tokens
-utterplan inspect chapter.utterplan.toml --preparation
-utterplan inspect chapter.utterplan.toml --semantic-boundaries
+utterplan compile chapter.ssmd.md -o chapter.utterplan.toml --trace chapter.trace.toml
+utterplan explain chapter.utterplan.toml --trace chapter.trace.toml
+utterplan inspect-trace chapter.trace.toml --preparation --boundaries
 utterplan inspect-attempt chapter.attempt.toml --issues
 ```
 
-`inspect --preparation` reports the preparation backend and version, structural and spoken text lengths, replacement count, and each replacement's structural and spoken ranges and text. This is the supported human-facing preparation diagnostic; raw coordinate lookup tables are intentionally absent from the persisted TOML plan.
+`inspect-trace --preparation` reports source/prepared text lengths, maps, preparation changes, and repair/diagnostic evidence from the optional trace. `inspect-trace --boundaries` shows compiler-only boundary events and semantic candidates. These fields are intentionally absent from the executable plan TOML.
 
 ## Migrate a saved plan
 
@@ -167,11 +195,7 @@ route and reports source schema, target schema, and whether migration is require
 without writing a file. A future schema version is rejected rather than guessed
 or downgraded.
 
-Schema migration is a separate operation from SSMD source migration. UtterPlan
-preserves released schemas v1, v2, and v3 and migrates supported plans through the
-registered chain to current schema v4. The v3-to-v4 step does not rerun parsing,
-NLP, or planning. Use `ssmd migrate FILE --to 0.9` for older SSMD source
-documents.
+Schema migration is separate from SSMD source migration. Frozen JSON schemas v1–v4 migrate through the registered sequential chain to current schema v5. The v4-to-v5 projection does not rerun parsing, text preparation, NLP, or planning, and refuses ambiguous token coordinates or pause meanings rather than guessing. Use `ssmd migrate FILE --to 0.9` for older SSMD source documents.
 
 For a saved TOML plan, `validate` verifies the current semantic schema and plan
 invariants; JSON plan paths are rejected outside `migrate`. For an SSMD or

utterplan-0.5.0/docs/consumer-guide.md

@@ -0,0 +1,92 @@
+# Renderer consumer guide
+
+UtterPlan ends at a semantic planning boundary. A renderer consumes the public `FlowPlan` and begins G2P after planning. It does not need a TOML round trip when planner and renderer run in the same process.
+
+For persistence across processes, save a current plan as `.utterplan.toml` and load it with `FlowPlan.load()`. `to_toml()` / `from_toml()` provide text round-tripping, and `plan_id` identifies executable semantics rather than TOML formatting. Batch reports and optional compiler traces are operational/provenance artifacts, not plans.
+
+UtterPlan accepts SSMD source syntax at version 0.9 only. Convert older source with `ssmd migrate FILE --to 0.9`; `utterplan migrate` is for importing serialized legacy plans.
+
+## Importing legacy plans
+
+Historical JSON schemas v1–v4 remain immutable and supported through sequential migrations to schema v5. Import explicitly:
+
+```bash
+utterplan migrate old.utterplan.json -o current.utterplan.toml
+```
+
+Migration operates on serialized plain data and does not reparse, reprepare, rerun NLP, or replan. The v4-to-v5 projection fails safely if token ownership/spans or authored pause semantics cannot be verified without guessing. Recompiling source is a separate operation. Normal `FlowPlan.load()` and plan-inspection commands reject JSON.
+
+## Planning defaults
+
+The default text-preparation backend is `spokenform`, pause activation mode is `tts`, and the default CLI linguistic-resource policy is `spacy off`. Deterministic fallback tokenization and analysis do not require an installed spaCy model. `spacy auto` is opt-in; it may expose richer tokenization, POS, lemmas, and tags, but provider documents remain internal.
+
+In Python, `PlannerConfig.document_format` defaults to `"plain"`; select `"ssmd"` for SSMD source. `PauseConfig.mode` controls whether eligible semantic pause intents are activated by the compiler. The executable plan carries semantic intent (or an exact authored timed pause), not numeric durations invented from an activation policy. A renderer may interpret supported intents according to its own capabilities.
+
+## Consumer contract
+
+`FlowPlan.flow` is the render order. Each `FlowUnit` owns its ordered `FlowSegment` records; no ID join is required. A segment provides:
+
+- `text`: prepared speech text for this segment.
+- `language`: effective language, defaulting from `FlowPlan.language` when omitted on the wire.
+- `tokens`: local token views with optional lemma, POS, tag, and morphology.
+- `pause_before` and `pause_after`: semantic pause intents such as `none`, `sentence`, `parenthetical`, or an exact authored time.
+- `directives`: typed voice, pronunciation, prosody, emphasis, say-as, substitution, audio, and extension data.
+- `markers` and optional `heading` metadata already localized to the owning segment.
+
+Token offsets are half-open Python-character ranges into the owning `segment.text`. Do not use them to slice document text or add offsets from another segment. Consumers render flow units and segments in their stored order.
+
+```python
+from utterplan import FlowPlan
+
+plan = FlowPlan.load("chapter.utterplan.toml")
+plan.validate()
+
+for unit in plan.flow:
+    for segment in unit.segments:
+        token_views = [
+            (segment.text[token.start : token.end], token.lemma, token.pos, token.tag, token.morph)
+            for token in segment.tokens
+        ]
+        render_speech(
+            text=segment.text,
+            language=segment.language,
+            tokens=token_views,
+            pause_before=segment.pause_before,
+            pause_after=segment.pause_after,
+            directives=segment.directives,
+        )
+```
+
+`FlowPlan.document` retains compact document information such as format, title, SSMD version, and portable semantics. `FlowPlan.linguistics` identifies analysis provider/model provenance without retaining provider state. Source text, source locations, compiler diagnostics, preparation mappings, and renderability details are intentionally not executable-plan fields.
+
+## Optional compiler trace
+
+When source-level evidence must survive a process boundary, request a separate trace sidecar:
+
+```bash
+utterplan compile chapter.ssmd.md -o chapter.utterplan.toml --trace chapter.trace.toml
+utterplan explain chapter.utterplan.toml --trace chapter.trace.toml
+utterplan inspect-trace chapter.trace.toml --preparation --boundaries
+```
+
+The trace contains source text/hash, structural/prepared text, coordinate maps, preparation changes, diagnostics, and renderability evidence. It is not renderer input, does not affect `FlowPlan.plan_id`, and does not change the resulting plan TOML. See the [coordinate-space contract](coordinate-spaces) for every trace offset's coordinate space.
+
+## Directives and media
+
+Voice bindings and directive voice references are logical names, not backend voice IDs; consumers translate them to engine-specific resources. UtterPlan resolves SSMD scopes and defaults but does not choose engine settings or recompute pause policy.
+
+An audio directive is already attached to its owning segment. Its `src` is opaque data for an external resolver; UtterPlan does not select or invoke one. For an audio-bearing segment, `segment.text` is an optional spoken fallback. Identical source URIs on distinct segments are separate timeline occurrences and must not be deduplicated as playback events. Extension references are also data only; UtterPlan does not execute handlers.
+
+## Stable renderer input
+
+A completed plan is immutable consumer input. Consumers may derive data for G2P/rendering, but must not rewrite planning decisions or mutate the plan:
+
+```python
+before = plan.to_toml()
+plan_id = plan.plan_id
+consume_plan(plan)
+assert plan.to_toml() == before
+assert plan.plan_id == plan_id
+```
+
+Plans contain no phonemes, model token IDs, model sessions, audio, renderer configuration, or model-derived timings. Acoustic retries and renderer-level randomness remain outside UtterPlan.

utterplan-0.5.0/docs/coordinate-spaces.md

@@ -0,0 +1,22 @@
+# Coordinate spaces
+
+Schema v5 keeps the executable plan compact: token coordinates are local to each `FlowSegment.text`; document-global source and preparation coordinates live only in an optional `PreparationTrace` sidecar.
+
+## Executable plan coordinates
+
+`FlowSegment.text` is the prepared speech text for that segment. Each `TokenView.start` and `TokenView.end` is a half-open Python-character offset into that segment's text. Slice a token with `segment.text[token.start:token.end]`. Do not add a unit, segment, document, or compiler offset to these values. Token membership is stored locally in `segment.tokens`; there are no plan-global token indices or lookup joins.
+
+Pauses, directives, markers, and optional heading levels are attached directly to their owning flow-segment edge or segment. They have no independent text coordinate. Flow-unit and segment order is the render order.
+
+## Optional trace coordinates
+
+Trace coordinates use Python string character boundaries (Unicode code points), not UTF-8 bytes or grapheme clusters:
+
+- `source_text` is the exact caller input, including SSMD front matter and markup.
+- `source_spans` maps original-source `[source_start, source_end)` ranges to parsed structural `[structural_start, structural_end)` ranges.
+- `structural_text` is SSMD-clean structural text before written-to-spoken preparation.
+- `spoken_text` is the prepared text used to construct executable segments.
+- `structural_to_spoken` maps structural character boundaries to spoken boundaries; `spoken_to_structural` maps spoken boundaries back to structural boundaries.
+- `PreparationTraceUnit.source_start` / `source_end` address structural text. `PreparationChange.source_start` / `source_end` use structural coordinates; `output_start` / `output_end` use prepared spoken coordinates.
+
+Diagnostic source offsets use the exact original-source coordinate space. Diagnostic `line` and `column` values are 1-based; columns count Python characters from the start of a line. The trace retains all these maps and source evidence. The executable FlowPlan intentionally does not.