flexdoc 0.1.0__tar.gz → 0.2.0__tar.gz

flexdoc-0.2.0/CHANGELOG.md

@@ -0,0 +1,135 @@
+# Changelog
+
+All notable changes to flexdoc are documented here.
+This project uses [semantic versioning](https://semver.org/); while pre-1.0, breaking
+changes bump the **minor** version (see `docs/publishing.md`).
+
+## 0.2.0 (2026-06-14)
+
+Correctness fixes and a completed inline/heading/link surface for the document-metrics
+use case (`docs/project/specs/active/plan-2026-06-13-metrics-use-case.md`, issue #6).
+As a preview-stage library this takes the cleanest shape with no compatibility shims;
+the API additions below include breaking signature changes (see **Changed**).
+
+### Fixed
+
+- **`node_table()` / `collect()` / `graph()` no longer raise on valid Markdown.** Inline
+  elements were discovered over the whole source and parented by start offset, so backtick
+  pairing across a block boundary (an empty fence next to inline backticks) produced an
+  inline span escaping its parent block and raised a layer-nesting error. Inline discovery
+  is now scoped per leaf content block, so an inline node can never straddle a block
+  boundary; links/images/definitions are parented by full containment.
+- **`sections()` / `toc()` recover every heading `blocks()` finds, and own their content
+  correctly.** Headings were re-derived from the blank-line paragraph view, dropping tight
+  headings and headings preceded by a non-blank line (e.g. an HTML-comment marker), and
+  section content was bucketed from that same view, so a heading glued to its body lost the
+  body. Sections now derive entirely from the structural block tree — the heading set *and*
+  each section's own content (`own_paragraphs()` / `blocks()` / sizes) come from the section's
+  source region — so tight and marker-preceded headings own exactly their content.
+- **Section spans nest correctly** even when a blank-line paragraph straddles a later
+  heading (e.g. an embedded `---` block marko reads as a setext heading): each section
+  spans from its heading to the next same-or-higher heading (trimmed), which nests by
+  construction. Byte-identical to the prior span for well-formed documents.
+- **Reference-definition nodes attach to their block.** A `link_ref_def` span included the
+  line's trailing newline and escaped the containing paragraph, leaving the node unparented
+  so a block-scoped `collect()` missed it; spans are now trimmed like every structural block.
+
+### Added
+
+- **Heading metadata on the structural block**: `Block.heading_info` (`HeadingInfo` with
+  parser-authoritative `level` and `title`) and the `Block.heading_level` convenience;
+  `HeadingInfo` is exported from `flexdoc.docs`. The node table reads heading level from it.
+- **Typed link forms**: `LinkForm` (`inline` / `reference` / `autolink` / `bare_url` /
+  `image` / `reference_definition`) and `Link.link_form`. `FlexDoc.links(link_forms=…)` selects any
+  forms (default: navigable links only), and `FlexDoc.images()` is a convenience for image
+  access. Reference definitions (`[id]: url`) are surfaced as `NodeKind.link_ref_def` nodes
+  and via `links(link_forms={LinkForm.reference_definition})`.
+- **`FlexDoc.prose_text()`**: prose-only text for editorial linting and prose metrics —
+  prose blocks (paragraphs/headings, and table cells when `include_tables=True`) with inline
+  code and footnote refs dropped, links/images replaced by their text/alt, inline-HTML tags
+  dropped (wrapped text kept), and heading/blockquote/list markers and reference-definition
+  lines stripped; from verbatim source slices (line wrapping preserved, never reflowed) so
+  spacing like a spaced em-dash is kept exactly.
+- **`FlexDoc.block_at_offset()`**: the innermost structural `Block` containing an offset
+  (the structural counterpart of `paragraph_at_offset`; the name, freed in 0.1.0, now
+  correctly returns a `Block`).
+- **Test-suite hardening**: adversarial corpus documents (`inline_pathology`,
+  `heading_edges`, `link_taxonomy`); cross-projection invariants tying `toc()` to the
+  heading blocks, inline nesting on the query surface, and link-form accounting; and a
+  dogfood test that parses every Markdown file in the repo and asserts the invariants. See
+  the spec's "Why These Bugs Escaped the Tests" analysis.
+
+### Changed
+
+These are breaking, made cleanly (no aliases) given the preview status:
+
+- **`Link` gains a required `link_form: LinkForm` field.** Direct `Link(...)` construction must
+  pass it.
+- **`block_links()` returns all link-like constructs** (navigable links, images, and
+  reference definitions), each with a `link_form`; previously it returned navigable links only.
+  `FlexDoc.links()` filters to navigable links by default, so its default result is
+  unchanged.
+- **`collect()` returns inline-kind nodes without `recursive=True`.** An inline-kind
+  request (e.g. `collect(kinds={NodeKind.link})`) now widens the candidate set instead of
+  silently returning `[]` — matching the documented behavior.
+
+## 0.1.0 (2026-06-12)
+
+First release.
+
+### Added
+
+- **Initial flexdoc package**, extracted from
+  [chopdiff](https://github.com/jlevy/chopdiff) as its own standalone distribution.
+  This is the document/markdown layer — `FlexDoc`, paragraphs/sentences, the block tree
+  and block types, sections, the node table, `collect()`, `DocGraph`, `SpanRef`, token
+  diffs/mappings, word tokenization, html-in-md, and read-time/token estimation — with
+  no dependency on chopdiff’s diff and windowed-transform machinery.
+
+  The import roots are `flexdoc.docs`, `flexdoc.html`, and `flexdoc.util`. Parse
+  behavior is unchanged from the `flexdoc.*` modules that previously shipped inside the
+  chopdiff wheel; this release packages them independently.
+  See `docs/project/specs/active/plan-2026-06-11-flexdoc-extraction.md`.
+
+- **A deliberate root API**: the working set is importable from the package root —
+  `FlexDoc`, `DocGraph`, `Detail`, `SpanRef`, `BlockType`, `NodeKind`, `Layer`,
+  `TextUnit` — designed against the known downstream users and pinned by contract
+  tests. The render helpers for source-linked HTML (`render_node_attrs`,
+  `wrap_with_node_attrs`, `parse_source_span_attr`) are public in `flexdoc.docs`.
+
+- **DocGraph paragraph view**: `Views.paragraphs` joins `toc`/`blocks`/`links`/
+  `sentences` in the serialized projection.
+
+### Changed (relative to the modules as shipped in chopdiff)
+
+The first standalone release also refines the API surface (the pre-publish design
+review, `docs/project/review/senior-engineering-review-flexdoc-standalone-2026-06.md`);
+these are intentional hard cuts with no compatibility aliases:
+
+- **`TextDoc` is renamed `FlexDoc`** — the package’s single entry point, named for the
+  model it carries (all layered projections hang off it).
+  It is importable from the package root: `from flexdoc import FlexDoc`. The module is
+  `flexdoc.docs.flex_doc` (was `chopdiff.docs.text_doc`), and the design of record is
+  now `docs/flexdoc-spec.md` (was `textdoc-spec.md`).
+- **`collect()` is fully keyword-only and the deprecated aliases are gone**: use
+  `subtree_of=` (was `scope=`, previously also positional) and `within=` (was
+  `contains=`).
+- **Editing-view methods are named in paragraph terms**, so “block” always means the
+  structural layer: `FlexDoc.paragraph_at_offset` (was `block_at_offset`),
+  `FlexDoc.iter_paragraphs` (was `iter_blocks`), `Section.own_paragraphs`/
+  `subtree_paragraphs` (were `own_blocks`/`subtree_blocks`).
+- **The export surface is settled**: `flexdoc.docs` now exports
+  `CodeInfo`/`TableInfo`/`ListInfo`, `resolve`/`resolve_and_update`,
+  `parse_blocks`/`walk_blocks`/`block_type_for`, and `DEFAULT_INCLUDE`; `flexdoc.html`
+  exports `html_p`, `html_tag`, `escape_attribute`, `tag_wrapper`, and
+  `identity_wrapper`. Link extraction is public as `flexdoc.docs.links.block_links`.
+- **`Node.attrs` values are JSON-typed** (`AttrValue`), validated at `DocGraph`
+  serialization, and node-id assignment order is pinned and tested for cross-language
+  ports; layer nesting guarantees (`LAYER_NESTING`) are enforced at node-table build.
+- Internally, the former `text_doc.py` was split into `flex_doc.py`, `paragraphs.py`,
+  `links.py`, and `sections.py` (package imports from `flexdoc.docs` are unaffected),
+  and `sections()` is now cached like the other derived views.
+
+Migration from chopdiff in one pass: `chopdiff.docs.TextDoc` → `flexdoc.FlexDoc` (or
+`flexdoc.docs.FlexDoc`), `chopdiff.docs.*` → `flexdoc.docs.*`, plus the method renames
+above.

{flexdoc-0.1.0 → flexdoc-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: flexdoc
-Version: 0.1.0
+Version: 0.2.0
 Summary: Document model for powerful text parsing and processing in Markdown
 Project-URL: Repository, https://github.com/jlevy/flexdoc
 Author-email: Joshua Levy <joshua@cal.berkeley.edu>

{flexdoc-0.1.0 → flexdoc-0.2.0}/docs/flexdoc-spec.md

@@ -365,13 +365,20 @@ is the reassembled working text, and the invariant holds against that.
   lazily-cached projections — `blocks()` (§6), `base_blocks()` (§6), `sections()` (§7),
   `links()` (§8), `node_table()` (§4.3), `collect()` (§9), and `graph()` (§10).
 - **Offset inversion:** `paragraph_at_offset(o)` and `sentence_at_offset(o)` map an
-  absolute offset back to the editing-view unit containing it (or `None` for offsets in
-  inter-unit whitespace or outside the document).
-  Structural blocks are addressed by their own spans or via `collect(overlaps=...)`.
+  absolute offset back to the editing-view unit containing it, and `block_at_offset(o)` to
+  the innermost structural `Block` (each `None` for offsets in inter-unit whitespace or
+  outside the document). Structural blocks are otherwise addressed by their own spans or via
+  `collect(overlaps=...)`.
 - **Frontmatter:** `FlexDoc.frontmatter` is the verbatim leading YAML block or `None`
   (§3).
 - **Sizing:** `size(unit)` and `size_summary()` measure the document in any `TextUnit`
   (see Terminology), including the approximate LLM `tokens` estimate.
+- **Prose projection:** `prose_text(include_tables=False)` returns prose-only text for
+  editorial linting and metrics — prose-bearing blocks (paragraphs/headings, and table cells
+  when `include_tables=True`) with inline code/footnote refs dropped, links/images reduced to
+  their text/alt, inline-HTML tags removed (wrapped text kept), and heading/blockquote/list
+  markers and reference-definition lines stripped. Slices are verbatim (line wrapping
+  preserved, never reflowed), so editorial spacing (e.g. a spaced em-dash) survives.
 
 ### 4.3 Nodes, kinds, layers, and the node table
 
@@ -696,12 +703,20 @@ Inline elements (links, code spans, images, inline HTML, footnote references,
 `section`/`sentence` associations, so block↔inline relationships are node edges, and
 “links in section 3” is a scoped `collect(kinds={link})`.
 
-- `Link(text, url, title, span)`: identity from `flowmark.markdown_ast.extract_links`
-  (reference links resolved, escapes honored, autolinks/images handled), which carries
-  no span by design. flexdoc recovers each exact `[start, end)` by reconciling the
+- `Link(text, url, title, span, link_form)`: identity from `flowmark.markdown_ast.extract_links`
+  (reference links resolved, escapes honored), an AST walk for images, and marko's
+  `LinkRefDef` elements for definitions. Each carries a `LinkForm` discriminator — `inline`,
+  `reference`, `autolink`, `bare_url`, `image`, or `reference_definition` — so consumers count
+  by form without heuristics. flexdoc recovers each exact `[start, end)` by reconciling the
   ordered identities with the name-tagged atomic spans from
-  `flowmark.atomic_spans.iter_atomic_spans` (`markdown_link` / `autolink` / `bare_url`);
-  reference links keep identity but no exact span.
+  `flowmark.atomic_spans.iter_atomic_spans` (and a trimmed `block_span` for definitions); an
+  identity that cannot be located keeps its identity with `span=None`.
+- **`links()` returns navigable links only by default** (`TRUE_LINK_FORMS`: `inline`,
+  `reference`, `autolink`, `bare_url`); `links(link_forms=…)` selects any forms and `images()` is
+  the convenience for `LinkForm.image`. **Reference definitions** (`[id]: url`) are surfaced
+  as `NodeKind.link_ref_def` nodes — parented to their containing block, so a block-scoped
+  `collect()` finds them — and via `links(link_forms={LinkForm.reference_definition})`, never the
+  default `links()` (a definition is not a link occurrence).
 - `link → sentence` via `sentence_at_offset(link.span[0])`.
 - **`footnote_ref`**: a footnote reference `[^label]` is a first-class inline node
   (`NodeKind.footnote_ref`) carrying its `label` in `attrs` and an exact span, collected

{flexdoc-0.1.0 → flexdoc-0.2.0}/docs/project/specs/active/plan-2026-06-11-flexdoc-extraction.md

@@ -422,8 +422,12 @@ mechanical) in the same already-breaking release; the migration note is one pass
   2026-06-12 (pypi.org returns 404 for `flexdoc`); re-check at publish time.
   Resolve the flexdoc-spec §13 name collision (Stage 2.5 / Open Questions).
   Configure the PyPI Trusted Publisher for `jlevy/flexdoc` (`docs/publishing.md`).
-- [ ] Tag and publish `flexdoc 0.1.0` (its own version line) via `publish.yml`.
-  Publishing is irreversible; it is the maintainer’s call to trigger.
+- [x] Tagged and published `flexdoc 0.1.0` (2026-06-12): release-notes PR merged, GitHub
+  release v0.1.0 created, `publish.yml` succeeded on its first trusted-publishing run,
+  and the package was verified live on PyPI (wheel + sdist) with a clean-venv install
+  exercising the root API. Note: the install check must run outside the repo, since the
+  project's own cool-off excludes the just-published version.
+
 
 ### Step 5 — rewire chopdiff to the external flexdoc (pending; the breaking release)