flexdoc 0.3.0__tar.gz → 0.4.0__tar.gz

flexdoc-0.4.0/.github/release-request/notes.md

@@ -0,0 +1,32 @@
+## What's Changed
+
+This release adds the portable reference and annotation layer from the 2026-07 TextRef design cycle plus cross-language logical word metrics, finalized under an independent senior release review ([review and breaking-change catalog](https://github.com/jlevy/flexdoc/blob/main/docs/project/review/senior-engineering-review-v0.4-release-2026-07.md); PRs #18, #20, #21). Because it changes documented behavior on a pre-1.0 library, it bumps the minor version (per the pre-1.0 rule in `docs/publishing.md`). If you depend on an unbounded `flexdoc>=0.3.0`, review the breaking changes below or pin `<0.4` before upgrading.
+
+### New Features
+
+**Native TextRef integration**
+
+Strict `DocRef`/`TextRef` values identify whole documents, spans, points, and semantic sections with canonical JSON and reversible `textref:0.1` URIs. `FlexDoc.references()` binds a document locator to one source snapshot: it maps every locatable public value (`for_target`, `for_span`, `for_point`, `for_section`), resolves references with typed outcomes on independent document/source-validation/selector axes (resolution never silently picks a duplicate), retrieves structured line-window context, and renders deterministic annotation views for humans and LLMs. Span quote evidence is configurable per context or per span; compact exact-less spans stay bound to one source hash.
+
+**Consumer-owned annotations**
+
+`TextAnnotation` and the one-document `AnnotationSet` sidecar round-trip strict JSON and safe YAML, and every sidecar entry is validated through the complete TextRef evidence contract at parse time. `DocGraph/v0.2` embeds a matching sidecar after verifying its document and source hash against the snapshot, and bounds-checks embedded positions. Committed JSON Schemas ship for both formats in the wheel.
+
+**Cross-language logical word metrics**
+
+`TextUnit.words` now measures normalized word-equivalent volume across natural language, CJK text, source code, URLs, and punctuation-dense content. `TextUnit.raw_words` and `raw_word_count()` keep literal whitespace counting, and `logical_word_count()` exposes the dependency-free primitive. See the [logical-word definition and validation](https://gist.github.com/jlevy/0d6d87885f6d85f31440e58b8cfce663).
+
+### Breaking Changes
+
+Made cleanly with no compatibility aliases, given the pre-1.0 status. Full catalog with rationales and migrations: [review doc §4](https://github.com/jlevy/flexdoc/blob/main/docs/project/review/senior-engineering-review-v0.4-release-2026-07.md).
+
+- **`FlexDoc.graph()` requires a document locator** — `doc.graph()` becomes `doc.graph(document="path/or/id.md")`; `build_doc_graph()` and the debug helpers `doc_graph_yaml()`/`dump_views()` take the same argument. Every serialized graph is now self-identifying.
+- **DocGraph wire contract v0.1 → v0.2** — `schema` is `"DocGraph/v0.2"`; `source.document` is required; the unqualified `source.sha256` field is replaced by algorithm-qualified `source.source_hash` (shared with TextRef); models are strict (unknown fields rejected, frozen, strict types) and validate node references, span/text consistency, and annotation bounds.
+- **`TextUnit.words` is logical** — it matches raw counts for ordinary non-wide prose but differs for wide/fullwidth scripts, URLs, code, and symbolic content; use `TextUnit.raw_words` for the previous whitespace-split behavior. Ripples through `size()`, `size_summary()`, `section_size_tree()`, and debug reports.
+- **Token estimates scale logical words** — `estimate_tokens(text, tokens_per_logical_word=1.6)` replaces the `chars_per_token` parameter, and `TOKENS_PER_LOGICAL_WORD` replaces `CHARS_PER_TOKEN`; estimates change numerically even at defaults.
+- **`format_read_time()` takes logical word counts** — same signature; the default rate corresponds to roughly 450 CJK characters per minute under the default wide-character weight.
+- **Pinned outputs change** — regenerate any golden DocGraph/report snapshots (schema string, `source` block, and `words` fields).
+
+### Full Changelog
+
+https://github.com/jlevy/flexdoc/compare/v0.3.0...v0.4.0

flexdoc-0.4.0/.github/release-request/request.json

@@ -0,0 +1,3 @@
+{
+  "tag": "v0.4.0"
+}

{flexdoc-0.3.0 → flexdoc-0.4.0}/.tbd/config.yml

@@ -32,6 +32,7 @@ docs_cache:
     shortcuts/system/skill-baseline.md: internal:shortcuts/system/skill-baseline.md
     shortcuts/system/skill-brief.md: internal:shortcuts/system/skill-brief.md
     shortcuts/system/skill-minimal.md: internal:shortcuts/system/skill-minimal.md
+    shortcuts/standard/address-pr-review.md: internal:shortcuts/standard/address-pr-review.md
     shortcuts/standard/agent-handoff.md: internal:shortcuts/standard/agent-handoff.md
     shortcuts/standard/checkout-third-party-repo.md: internal:shortcuts/standard/checkout-third-party-repo.md
     shortcuts/standard/code-cleanup-all.md: internal:shortcuts/standard/code-cleanup-all.md
@@ -51,6 +52,7 @@ docs_cache:
     shortcuts/standard/new-shortcut.md: internal:shortcuts/standard/new-shortcut.md
     shortcuts/standard/new-validation-plan.md: internal:shortcuts/standard/new-validation-plan.md
     shortcuts/standard/plan-implementation-with-beads.md: internal:shortcuts/standard/plan-implementation-with-beads.md
+    shortcuts/standard/pr-review-workflows.md: internal:shortcuts/standard/pr-review-workflows.md
     shortcuts/standard/precommit-process.md: internal:shortcuts/standard/precommit-process.md
     shortcuts/standard/review-code-python.md: internal:shortcuts/standard/review-code-python.md
     shortcuts/standard/review-code-typescript.md: internal:shortcuts/standard/review-code-typescript.md

{flexdoc-0.3.0 → flexdoc-0.4.0}/CHANGELOG.md

@@ -4,6 +4,59 @@ All notable changes to flexdoc are documented here.
 This project uses [semantic versioning](https://semver.org/); while pre-1.0, breaking
 changes bump the **minor** version (see `docs/publishing.md`).
 
+## 0.4.0 (2026-07-20)
+
+This is an intentional pre-1.0 API break; per the pre-1.0 rule it is a minor release,
+not a 0.3.x patch. The release-gate review, validation evidence, and full
+breaking-change catalog with migrations are in
+`docs/project/review/senior-engineering-review-v0.4-release-2026-07.md`.
+
+### Added
+
+- **Native TextRef integration.** Strict `DocRef`/`TextRef` values now identify whole
+  documents, spans, points, and semantic sections with canonical JSON and reversible
+  `textref:0.1` URIs. `FlexDoc.references()` maps every locatable public document value,
+  resolves references with typed outcomes, retrieves structured source context, and
+  renders deterministic human/LLM-readable annotation views. Span quote evidence is
+  optional: callers can select a context-wide size policy or override individual spans,
+  while compact exact-less spans remain safely bound to one source hash.
+- **Consumer-owned TextRef annotations.** `TextAnnotation` and the one-document
+  `AnnotationSet` sidecar round-trip strict JSON and safe YAML. `DocGraph/v0.2` carries
+  required document identity and source hash in every graph, with optional typed
+  annotations in the same schema. The protocol and renderer add no dependencies.
+- **Cross-language logical word metrics.** `TextUnit.words` now measures normalized
+  word-equivalent volume across natural language, CJK text, source code, URLs, and other
+  punctuation-dense content. `TextUnit.raw_words` and `raw_word_count()` preserve
+  literal whitespace-delimited counting, while `logical_word_count()` exposes the
+  dependency-free normalized primitive.
+
+### Changed
+
+- **DocGraph is v0.2 and self-identifying.** `FlexDoc.graph()` and `build_doc_graph()`
+  now require a consumer-owned `document` locator and always return `DocGraph/v0.2`;
+  annotations optionally populate that same model, and the debug helpers
+  `doc_graph_yaml()`/`dump_views()` take the same `document` argument. In the
+  serialized graph, `source.document` is required and the unqualified `source.sha256`
+  field is replaced by `source.source_hash`, whose algorithm-qualified value is shared
+  with TextRef. All DocGraph models are now strict (unknown fields rejected, strict
+  types, frozen instances) and validate node-reference integrity, span/text
+  consistency, and annotation bounds.
+
+- **`TextUnit.words` changes from raw to logical semantics.** It matches the raw count
+  for ordinary non-wide prose averaging 3–6 characters per word, but differs for
+  wide/fullwidth scripts, longer or shorter average word lengths, URLs, code, and other
+  symbolic content. Callers requiring the exact previous whitespace-split behavior
+  must use `TextUnit.raw_words`; there is no separate `TextUnit.logical_words` member.
+  Size summaries, section-tree defaults, and debug-report `words` fields use the logical
+  measure, and aggregate counts are rounded only after the full text is measured.
+- **Approximate token estimates now scale logical words.** `estimate_tokens()` uses
+  `TOKENS_PER_LOGICAL_WORD` (default `1.6`) instead of the former
+  `CHARS_PER_TOKEN`/`chars_per_token` API. This is still a model-family heuristic; use
+  the target provider's tokenizer for exact counts or hard context limits.
+- **Reading-time guidance is language-robust.** Pass logical word counts to
+  `format_read_time()`; the default rate corresponds to roughly 450 CJK characters per
+  minute under the default wide-character weight.
+
 ## 0.3.0 (2026-07-11)
 
 Fixes from the 2026-07 pre-promotion design review
@@ -102,7 +155,7 @@ Remaining pre-1.0 design decisions and future mechanisms are collected in
 ## 0.2.0 (2026-06-14)
 
 Correctness fixes and a completed inline/heading/link surface for the document-metrics
-use case (`docs/project/specs/active/plan-2026-06-13-metrics-use-case.md`, issue #6). As
+use case (`docs/project/specs/done/plan-2026-06-13-metrics-use-case.md`, issue #6). As
 a preview-stage library this takes the cleanest shape with no compatibility shims; the
 API additions below include breaking signature changes (see **Changed**).
 

flexdoc-0.4.0/PKG-INFO

@@ -0,0 +1,282 @@
+Metadata-Version: 2.4
+Name: flexdoc
+Version: 0.4.0
+Summary: Document model for powerful text parsing and processing in Markdown
+Project-URL: Repository, https://github.com/jlevy/flexdoc
+Project-URL: Homepage, https://github.com/jlevy/flexdoc
+Project-URL: Documentation, https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md
+Project-URL: Changelog, https://github.com/jlevy/flexdoc/blob/main/CHANGELOG.md
+Author-email: Joshua Levy <joshua@cal.berkeley.edu>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: annotations,document-model,llm,markdown,nlp,parsing,sentences,spans,text-processing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Classifier: Typing :: Typed
+Requires-Python: <4.0,>=3.11
+Requires-Dist: cydifflib>=1.2.0
+Requires-Dist: flowmark>=0.7.1
+Requires-Dist: frontmatter-format>=0.3.0
+Requires-Dist: funlog>=0.2.1
+Requires-Dist: marko>=2.2.2
+Requires-Dist: prettyfmt>=0.3.0
+Requires-Dist: pydantic>=2.13.4
+Requires-Dist: regex>=2024.11.6
+Requires-Dist: selectolax>=0.3.32
+Requires-Dist: strif>=3.1.0
+Requires-Dist: typing-extensions>=4.12.2
+Description-Content-Type: text/markdown
+
+# FlexDoc
+
+[![PyPI version](https://img.shields.io/pypi/v/flexdoc)](https://pypi.org/project/flexdoc/)
+[![CI](https://github.com/jlevy/flexdoc/actions/workflows/ci.yml/badge.svg)](https://github.com/jlevy/flexdoc/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/jlevy/flexdoc/blob/main/LICENSE)
+[![Python versions](https://img.shields.io/pypi/pyversions/flexdoc)](https://pypi.org/project/flexdoc/)
+
+A Markdown parser gives you a block AST but no sentences, sizes, or exact source
+offsets. An NLP toolkit gives you sentences but no Markdown structure.
+If you need to know which sentence is in which section, how many words (or LLM tokens) a
+section holds, or exactly where a link sits in the original text, you end up gluing
+tools together; the glue breaks on the first edit.
+
+FlexDoc builds a single source-grounded model and exposes its structure as layers over
+one shared coordinate space: exact `[start, end)` offsets into one retained, normalized
+source string. The **Markdown layer** (blocks, inline elements, typed attributes), the
+**textual layer** (paragraphs, sentences, word tokens), and the **document layer**
+(heading hierarchy, table of contents) are independent parses of the same text, so
+cross-cutting questions are simple offset queries.
+On top of the model sit portable formats: `DocGraph` serializes any slice as
+language-neutral JSON, and `TextRef` makes spans, points, sections, and whole documents
+durable references that resolve back to exact source.
+
+FlexDoc is a standalone library.
+[chopdiff](https://github.com/jlevy/chopdiff) builds its diff-filtering and
+windowed-transform layer on top of flexdoc.
+
+## Installation
+
+```shell
+uv add flexdoc
+# or: pip install flexdoc
+```
+
+## Status
+
+**Beta** (0.4.x). The core model is established, but the later-stage mechanisms in the
+[stabilization roadmap](https://github.com/jlevy/flexdoc/blob/main/docs/project/specs/active/plan-2026-07-09-flexdoc-stabilization-roadmap.md)
+remain open. Breaking changes before 1.0 bump the minor version, so pin a minor version.
+See the [changelog](https://github.com/jlevy/flexdoc/blob/main/CHANGELOG.md).
+
+## Usage
+
+### Parse and Query
+
+The primary entry point is `FlexDoc`. Parsing any input never raises: malformed
+Markdown degrades deterministically and visibly. Every located unit carries an exact
+`[start, end)` span into the normalized `source_text`, and `reassemble()` round-trips
+the model back to normalized Markdown.
+
+```python
+from flexdoc import FlexDoc, NodeKind, TextUnit
+
+doc = FlexDoc.from_text("# Introduction\n\nSee [docs](https://example.com/docs).\n")
+
+# Section hierarchy with rolled-up sizes:
+print(doc.section_size_tree(units=(TextUnit.words, TextUnit.sentences)))
+# # Introduction  (8 words, 2 sentences)
+
+# Sizes at every grain, including approximate LLM tokens:
+print(doc.size_summary())
+# 53 bytes (3 lines, 2 paras, 2 sents, 8 words, ~13 tok)
+
+# One query primitive (collect()) spans all layers:
+link = doc.collect(kinds={NodeKind.link})[0]
+print(link.attrs["url"], link.source_span)
+# https://example.com/docs (20, 52)
+```
+
+FlexDoc delegates Markdown parsing to [marko](https://github.com/frostming/marko)
+(CommonMark with GFM tables and footnotes) via
+[flowmark](https://github.com/jlevy/flowmark), and adds sentence segmentation, the
+section hierarchy, the flat node table, offset-grounded queries, references,
+serialization, and span anchoring on top.
+
+### Reference and Annotate: TextRef
+
+Continuing the example, any public value maps to a
+[`TextRef`](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#11-textref-spanref-and-annotations):
+a portable reference carrying quote evidence and a source hash, with canonical JSON and
+a reversible `textref:0.1` URI projection.
+
+```python
+refs = doc.references(document="guide.md")
+ref = refs.for_target(link)
+
+print(ref.to_uri())
+# textref:0.1?doc=guide.md&hash=sha256%3A0e91...0f86&type=span
+#     &exact=%5Bdocs%5D%28https%3A%2F%2Fexample.com%2Fdocs%29&prefix=...&start=20
+
+res = refs.resolve(ref)
+print(res.resolved, res.method.value, (res.span.start, res.span.end))
+# True source_position (20, 52)
+```
+
+Annotations attach consumer-owned content to TextRef targets, and `AnnotationSet` is
+their one-document JSON/YAML sidecar form:
+
+```python
+from flexdoc import AnnotationSet, TextAnnotation, TextBody
+
+note = TextAnnotation(id="n1", target=ref, motivations=["commenting"],
+                      body=TextBody(type="text", value="Check this link."))
+sidecar = AnnotationSet.from_annotations([note])
+
+print(sidecar.to_yaml())
+# format: text-annotations/0.1
+# document: guide.md
+# source_hash: sha256:0e91...0f86
+# annotations:
+# - id: n1
+#   ...
+```
+
+### Serialize: DocGraph
+
+[`DocGraph`](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#10-docgraph-the-serialized-projection)
+is the language-neutral serialized projection: one source-anchored object carrying
+document identity, a source hash shared with TextRef, the node table, derived views,
+and (optionally) an embedded annotation sidecar. JSON is the wire form; YAML is the
+human and golden-test form.
+
+```python
+graph = doc.graph(document="guide.md", annotations=sidecar)
+
+print(graph.to_yaml())
+# schema: DocGraph/v0.2
+# source:
+#   format: markdown
+#   offset_unit: unicode_code_points
+#   document: guide.md
+#   source_hash: sha256:0e91...0f86
+# nodes:
+# - id: n0001
+#   kind: heading
+#   layer: markdown
+#   source_span:
+#     start: 0
+#     end: 14
+#   attrs:
+#     level: 1
+# ... (more nodes, then views and the embedded annotations)
+```
+
+### Word and Token Metrics
+
+`TextUnit.words` is a logical-word measure: it matches a whitespace count for ordinary
+non-wide prose averaging 3–6 characters per word, but normalizes wide/fullwidth scripts,
+long identifiers and URLs, short-token sequences, and punctuation-dense code or
+Markdown. Use `TextUnit.raw_words` for a literal whitespace-delimited count. See the
+[logical-word definition and validation](https://gist.github.com/jlevy/0d6d87885f6d85f31440e58b8cfce663)
+for the rationale, reference algorithm, and multilingual examples.
+
+### Module Map
+
+The full public surfaces live in the submodules:
+
+- `flexdoc.docs`: `FlexDoc`, `Paragraph`, `Sentence`, `Section`, `Block`, `BlockType`,
+  the node table, `collect()`, `DocGraph`, `TextRef`, annotation values, `SpanRef`, and
+  source-linked render/report helpers.
+- `flexdoc.docs.wordtoks`, `flexdoc.docs.search_tokens`,
+  `flexdoc.docs.token_diffs`, and `flexdoc.docs.token_mapping`: lower-level token,
+  search, diff, and mapping utilities that are not promoted by `flexdoc.docs`.
+- `flexdoc.html`: html-in-md, html/plaintext conversion, HTML tag helpers, the content
+  extractor, and timestamp extraction.
+- `flexdoc.util`: raw and cross-language logical word counts, read-time estimation, and
+  approximate token-count estimation.
+
+### Worked Examples
+
+See [usage.md](https://github.com/jlevy/flexdoc/blob/main/docs/usage.md) for the main
+workflows. Each worked example is a runnable script (run with
+`uv run python examples/<name>.py` from a checkout) demonstrating one workflow
+end to end:
+
+- [`doc_structure.py`](https://github.com/jlevy/flexdoc/blob/main/examples/doc_structure.py)
+  covers section hierarchy, size rollups, offset lookups, and the block tree.
+- [`normalized_form.py`](https://github.com/jlevy/flexdoc/blob/main/examples/normalized_form.py)
+  covers block-type tallies, list-density invariance, and per-section links.
+- [`backfill_timestamps.py`](https://github.com/jlevy/flexdoc/blob/main/examples/backfill_timestamps.py)
+  aligns an edited transcript to its timestamped source via token mapping.
+- [`textref_workflows.py`](https://github.com/jlevy/flexdoc/blob/main/examples/textref_workflows.py)
+  composes extraction provenance, context retrieval, citations, annotations, and edit
+  targets.
+
+## Design
+
+The design of record is
+[flexdoc-spec.md](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md),
+which motivates each decision and defines the invariants the tests pin. The ideas that
+carry the model:
+
+- **One coordinate space.** Every layer indexes the same retained, normalized source
+  string with `[start, end)` Unicode code-point offsets, so cross-layer questions reduce
+  to offset queries
+  ([spec §2](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#2-principles-and-goals)).
+- **Layers are independent parses.** Markdown structure, prose sentences, and the
+  heading hierarchy are separate projections of the same text; no single tree has to be
+  authoritative
+  ([spec §4](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#4-core-types-nodes-and-offsets)).
+- **Parsing never raises.** Malformed Markdown degrades deterministically and visibly,
+  pinned by the golden-test corpus
+  ([spec §13](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#13-invariants-and-non-goals)).
+- **References carry evidence, and resolution never guesses.** Positions are trusted
+  only under a matched source hash; quotes and context corroborate; a duplicated quote
+  resolves to a typed ambiguous outcome, never a silent first match
+  ([spec §11](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#11-textref-spanref-and-annotations)).
+  `SpanRef` is the lightweight quote anchor underneath.
+- **Serialized contracts are strict and versioned.** `DocGraph/v0.2` and `textref/0.1`
+  reject unknown fields, and both ship committed JSON Schemas:
+  [doc_graph_schema.json](https://github.com/jlevy/flexdoc/blob/main/src/flexdoc/docs/doc_graph_schema.json)
+  and
+  [text_ref_schema.json](https://github.com/jlevy/flexdoc/blob/main/src/flexdoc/docs/text_ref_schema.json)
+  ([spec §10](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#10-docgraph-the-serialized-projection)).
+
+## Project Docs
+
+For users:
+
+- Design of record (the full spec):
+  [flexdoc-spec.md](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md).
+- Usage guide: [usage.md](https://github.com/jlevy/flexdoc/blob/main/docs/usage.md).
+- Installing uv and Python:
+  [installation.md](https://github.com/jlevy/flexdoc/blob/main/docs/installation.md).
+
+For contributors:
+
+- Development workflows:
+  [development.md](https://github.com/jlevy/flexdoc/blob/main/docs/development.md).
+- Publishing to PyPI:
+  [publishing.md](https://github.com/jlevy/flexdoc/blob/main/docs/publishing.md).
+- Dependency policy:
+  [SUPPLY-CHAIN-SECURITY.md](https://github.com/jlevy/flexdoc/blob/main/SUPPLY-CHAIN-SECURITY.md).
+- Design history and plans (active and done):
+  [docs/project/specs/](https://github.com/jlevy/flexdoc/tree/main/docs/project/specs).
+
+* * *
+
+*This project was built from
+[simple-modern-uv](https://github.com/jlevy/simple-modern-uv).*
+
+<!-- This document follows common-doc-guidelines.md.
+See github.com/jlevy/practical-prose and review guidelines before editing.
+-->

flexdoc-0.4.0/README.md

@@ -0,0 +1,244 @@
+# FlexDoc
+
+[![PyPI version](https://img.shields.io/pypi/v/flexdoc)](https://pypi.org/project/flexdoc/)
+[![CI](https://github.com/jlevy/flexdoc/actions/workflows/ci.yml/badge.svg)](https://github.com/jlevy/flexdoc/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/jlevy/flexdoc/blob/main/LICENSE)
+[![Python versions](https://img.shields.io/pypi/pyversions/flexdoc)](https://pypi.org/project/flexdoc/)
+
+A Markdown parser gives you a block AST but no sentences, sizes, or exact source
+offsets. An NLP toolkit gives you sentences but no Markdown structure.
+If you need to know which sentence is in which section, how many words (or LLM tokens) a
+section holds, or exactly where a link sits in the original text, you end up gluing
+tools together; the glue breaks on the first edit.
+
+FlexDoc builds a single source-grounded model and exposes its structure as layers over
+one shared coordinate space: exact `[start, end)` offsets into one retained, normalized
+source string. The **Markdown layer** (blocks, inline elements, typed attributes), the
+**textual layer** (paragraphs, sentences, word tokens), and the **document layer**
+(heading hierarchy, table of contents) are independent parses of the same text, so
+cross-cutting questions are simple offset queries.
+On top of the model sit portable formats: `DocGraph` serializes any slice as
+language-neutral JSON, and `TextRef` makes spans, points, sections, and whole documents
+durable references that resolve back to exact source.
+
+FlexDoc is a standalone library.
+[chopdiff](https://github.com/jlevy/chopdiff) builds its diff-filtering and
+windowed-transform layer on top of flexdoc.
+
+## Installation
+
+```shell
+uv add flexdoc
+# or: pip install flexdoc
+```
+
+## Status
+
+**Beta** (0.4.x). The core model is established, but the later-stage mechanisms in the
+[stabilization roadmap](https://github.com/jlevy/flexdoc/blob/main/docs/project/specs/active/plan-2026-07-09-flexdoc-stabilization-roadmap.md)
+remain open. Breaking changes before 1.0 bump the minor version, so pin a minor version.
+See the [changelog](https://github.com/jlevy/flexdoc/blob/main/CHANGELOG.md).
+
+## Usage
+
+### Parse and Query
+
+The primary entry point is `FlexDoc`. Parsing any input never raises: malformed
+Markdown degrades deterministically and visibly. Every located unit carries an exact
+`[start, end)` span into the normalized `source_text`, and `reassemble()` round-trips
+the model back to normalized Markdown.
+
+```python
+from flexdoc import FlexDoc, NodeKind, TextUnit
+
+doc = FlexDoc.from_text("# Introduction\n\nSee [docs](https://example.com/docs).\n")
+
+# Section hierarchy with rolled-up sizes:
+print(doc.section_size_tree(units=(TextUnit.words, TextUnit.sentences)))
+# # Introduction  (8 words, 2 sentences)
+
+# Sizes at every grain, including approximate LLM tokens:
+print(doc.size_summary())
+# 53 bytes (3 lines, 2 paras, 2 sents, 8 words, ~13 tok)
+
+# One query primitive (collect()) spans all layers:
+link = doc.collect(kinds={NodeKind.link})[0]
+print(link.attrs["url"], link.source_span)
+# https://example.com/docs (20, 52)
+```
+
+FlexDoc delegates Markdown parsing to [marko](https://github.com/frostming/marko)
+(CommonMark with GFM tables and footnotes) via
+[flowmark](https://github.com/jlevy/flowmark), and adds sentence segmentation, the
+section hierarchy, the flat node table, offset-grounded queries, references,
+serialization, and span anchoring on top.
+
+### Reference and Annotate: TextRef
+
+Continuing the example, any public value maps to a
+[`TextRef`](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#11-textref-spanref-and-annotations):
+a portable reference carrying quote evidence and a source hash, with canonical JSON and
+a reversible `textref:0.1` URI projection.
+
+```python
+refs = doc.references(document="guide.md")
+ref = refs.for_target(link)
+
+print(ref.to_uri())
+# textref:0.1?doc=guide.md&hash=sha256%3A0e91...0f86&type=span
+#     &exact=%5Bdocs%5D%28https%3A%2F%2Fexample.com%2Fdocs%29&prefix=...&start=20
+
+res = refs.resolve(ref)
+print(res.resolved, res.method.value, (res.span.start, res.span.end))
+# True source_position (20, 52)
+```
+
+Annotations attach consumer-owned content to TextRef targets, and `AnnotationSet` is
+their one-document JSON/YAML sidecar form:
+
+```python
+from flexdoc import AnnotationSet, TextAnnotation, TextBody
+
+note = TextAnnotation(id="n1", target=ref, motivations=["commenting"],
+                      body=TextBody(type="text", value="Check this link."))
+sidecar = AnnotationSet.from_annotations([note])
+
+print(sidecar.to_yaml())
+# format: text-annotations/0.1
+# document: guide.md
+# source_hash: sha256:0e91...0f86
+# annotations:
+# - id: n1
+#   ...
+```
+
+### Serialize: DocGraph
+
+[`DocGraph`](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#10-docgraph-the-serialized-projection)
+is the language-neutral serialized projection: one source-anchored object carrying
+document identity, a source hash shared with TextRef, the node table, derived views,
+and (optionally) an embedded annotation sidecar. JSON is the wire form; YAML is the
+human and golden-test form.
+
+```python
+graph = doc.graph(document="guide.md", annotations=sidecar)
+
+print(graph.to_yaml())
+# schema: DocGraph/v0.2
+# source:
+#   format: markdown
+#   offset_unit: unicode_code_points
+#   document: guide.md
+#   source_hash: sha256:0e91...0f86
+# nodes:
+# - id: n0001
+#   kind: heading
+#   layer: markdown
+#   source_span:
+#     start: 0
+#     end: 14
+#   attrs:
+#     level: 1
+# ... (more nodes, then views and the embedded annotations)
+```
+
+### Word and Token Metrics
+
+`TextUnit.words` is a logical-word measure: it matches a whitespace count for ordinary
+non-wide prose averaging 3–6 characters per word, but normalizes wide/fullwidth scripts,
+long identifiers and URLs, short-token sequences, and punctuation-dense code or
+Markdown. Use `TextUnit.raw_words` for a literal whitespace-delimited count. See the
+[logical-word definition and validation](https://gist.github.com/jlevy/0d6d87885f6d85f31440e58b8cfce663)
+for the rationale, reference algorithm, and multilingual examples.
+
+### Module Map
+
+The full public surfaces live in the submodules:
+
+- `flexdoc.docs`: `FlexDoc`, `Paragraph`, `Sentence`, `Section`, `Block`, `BlockType`,
+  the node table, `collect()`, `DocGraph`, `TextRef`, annotation values, `SpanRef`, and
+  source-linked render/report helpers.
+- `flexdoc.docs.wordtoks`, `flexdoc.docs.search_tokens`,
+  `flexdoc.docs.token_diffs`, and `flexdoc.docs.token_mapping`: lower-level token,
+  search, diff, and mapping utilities that are not promoted by `flexdoc.docs`.
+- `flexdoc.html`: html-in-md, html/plaintext conversion, HTML tag helpers, the content
+  extractor, and timestamp extraction.
+- `flexdoc.util`: raw and cross-language logical word counts, read-time estimation, and
+  approximate token-count estimation.
+
+### Worked Examples
+
+See [usage.md](https://github.com/jlevy/flexdoc/blob/main/docs/usage.md) for the main
+workflows. Each worked example is a runnable script (run with
+`uv run python examples/<name>.py` from a checkout) demonstrating one workflow
+end to end:
+
+- [`doc_structure.py`](https://github.com/jlevy/flexdoc/blob/main/examples/doc_structure.py)
+  covers section hierarchy, size rollups, offset lookups, and the block tree.
+- [`normalized_form.py`](https://github.com/jlevy/flexdoc/blob/main/examples/normalized_form.py)
+  covers block-type tallies, list-density invariance, and per-section links.
+- [`backfill_timestamps.py`](https://github.com/jlevy/flexdoc/blob/main/examples/backfill_timestamps.py)
+  aligns an edited transcript to its timestamped source via token mapping.
+- [`textref_workflows.py`](https://github.com/jlevy/flexdoc/blob/main/examples/textref_workflows.py)
+  composes extraction provenance, context retrieval, citations, annotations, and edit
+  targets.
+
+## Design
+
+The design of record is
+[flexdoc-spec.md](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md),
+which motivates each decision and defines the invariants the tests pin. The ideas that
+carry the model:
+
+- **One coordinate space.** Every layer indexes the same retained, normalized source
+  string with `[start, end)` Unicode code-point offsets, so cross-layer questions reduce
+  to offset queries
+  ([spec §2](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#2-principles-and-goals)).
+- **Layers are independent parses.** Markdown structure, prose sentences, and the
+  heading hierarchy are separate projections of the same text; no single tree has to be
+  authoritative
+  ([spec §4](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#4-core-types-nodes-and-offsets)).
+- **Parsing never raises.** Malformed Markdown degrades deterministically and visibly,
+  pinned by the golden-test corpus
+  ([spec §13](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#13-invariants-and-non-goals)).
+- **References carry evidence, and resolution never guesses.** Positions are trusted
+  only under a matched source hash; quotes and context corroborate; a duplicated quote
+  resolves to a typed ambiguous outcome, never a silent first match
+  ([spec §11](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#11-textref-spanref-and-annotations)).
+  `SpanRef` is the lightweight quote anchor underneath.
+- **Serialized contracts are strict and versioned.** `DocGraph/v0.2` and `textref/0.1`
+  reject unknown fields, and both ship committed JSON Schemas:
+  [doc_graph_schema.json](https://github.com/jlevy/flexdoc/blob/main/src/flexdoc/docs/doc_graph_schema.json)
+  and
+  [text_ref_schema.json](https://github.com/jlevy/flexdoc/blob/main/src/flexdoc/docs/text_ref_schema.json)
+  ([spec §10](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md#10-docgraph-the-serialized-projection)).
+
+## Project Docs
+
+For users:
+
+- Design of record (the full spec):
+  [flexdoc-spec.md](https://github.com/jlevy/flexdoc/blob/main/docs/flexdoc-spec.md).
+- Usage guide: [usage.md](https://github.com/jlevy/flexdoc/blob/main/docs/usage.md).
+- Installing uv and Python:
+  [installation.md](https://github.com/jlevy/flexdoc/blob/main/docs/installation.md).
+
+For contributors:
+
+- Development workflows:
+  [development.md](https://github.com/jlevy/flexdoc/blob/main/docs/development.md).
+- Publishing to PyPI:
+  [publishing.md](https://github.com/jlevy/flexdoc/blob/main/docs/publishing.md).
+- Dependency policy:
+  [SUPPLY-CHAIN-SECURITY.md](https://github.com/jlevy/flexdoc/blob/main/SUPPLY-CHAIN-SECURITY.md).
+- Design history and plans (active and done):
+  [docs/project/specs/](https://github.com/jlevy/flexdoc/tree/main/docs/project/specs).
+
+* * *
+
+*This project was built from
+[simple-modern-uv](https://github.com/jlevy/simple-modern-uv).*
+
+<!-- This document follows common-doc-guidelines.md.
+See github.com/jlevy/practical-prose and review guidelines before editing.
+-->