lexidown 0.1.0__tar.gz

lexidown-0.1.0/DEVELOPMENT.md

@@ -0,0 +1,179 @@
+# Development
+
+## Setup and checks
+
+Use Python 3.10 or newer, a C compiler, and Python development headers:
+
+```sh
+python -m venv .venv
+# Activate .venv using the command for your shell.
+python -m pip install -e '.[dev]'
+python -m unittest discover -s tests -v
+ruff check .
+ruff format --check .
+```
+
+To apply Ruff fixes and formatting:
+
+```sh
+ruff check --fix .
+ruff format .
+```
+
+The configuration in `pyproject.toml` uses 88-column formatting, double quotes,
+isort import ordering, and common checks for errors, outdated syntax, likely
+bugs, and simpler Python constructs. Vendored C sources are excluded. Ruff
+checks Python files; Cython sources are checked by the Cython compiler. Reinstall
+the editable package after changing `.pyx`, `.pxi`, `.pxd`, or bundled C files.
+
+The reusable [lint workflow](.github/workflows/lint.yml) runs Ruff lint and
+formatting checks and is called by the wheel workflow on pushes and pull requests.
+
+## Dependencies
+
+| Dependency | Purpose |
+| --- | --- |
+| Runtime Python packages | None; the wheel includes the native engine and parser |
+| Cython and setuptools | Build the extension from source |
+| html5lib | Test-only parser for independent DOM inputs |
+| Ruff | Python linting, import sorting, and formatting |
+| Node.js and the root npm dev dependencies | Optional JavaScript comparisons |
+| `benchmarks/requirements.txt` | Optional competing converters for benchmarks |
+| build, Twine, and cibuildwheel | Distribution builds and validation |
+
+`html5lib` deliberately constructs DOM inputs independently of Lexidown for
+compatibility and non-mutation tests. Its `six` and `webencodings` dependencies
+are test dependencies too. Lexidown never imports these during normal use.
+`selectolax` is not needed: the Lexbor C sources are already bundled in `vendor/`.
+
+## Compatibility tests
+
+The Python suite runs without Node.js. It includes all 147 upstream HTML fixtures
+in both string and DOM modes, plus native parser, serialization, callback,
+mutation, and optimization regressions.
+
+The golden fixtures in `tests/fixtures/turndown.json` were extracted from
+[Turndown's test document at aa84dfa](https://github.com/mixmark-io/turndown/blob/aa84dfa3e2361edea8c43acbfc2b7a9363494bfd/test/index.html).
+Their upstream MIT attribution is retained in `THIRD_PARTY_NOTICES.md`.
+
+For live comparisons against JavaScript, install the optional reference packages:
+
+```sh
+npm ci
+python scripts/differential.py
+```
+
+`package-lock.json` pins Turndown 7.2.4 and Domino 2.2.0. The differential suite
+compares outputs for option combinations, Unicode, malformed HTML, kept HTML,
+plugins, and repeated conversions. Updating the compatibility target requires
+reviewing the fixtures and differential results along with parser changes.
+
+## Benchmarks
+
+Install the npm reference with `npm ci`. With saved HTML snapshots present, run:
+
+```sh
+python scripts/benchmark.py --rounds 3 --samples 7 --warmups 5
+```
+
+This compares Lexidown and JavaScript across eight pages and 13 option profiles.
+For the additional Python converters:
+
+```sh
+python -m pip install -r benchmarks/requirements.txt
+python -m scripts.compare_converters --rounds 3 --samples 7 --warmups 5
+```
+
+Use a separate environment for competitor packages if they are not needed for
+other development work. They are not part of Lexidown's development extra.
+
+`benchmarks/sources.json` records source URLs and SHA-256 hashes. HTML snapshots
+live in `benchmarks/inputs/` and are excluded from version control. The scripts
+verify these hashes before measuring. Downloading a page again can change the
+corpus; record new hashes and rerun the comparison instead of mixing results.
+
+Each round uses fresh worker processes, shuffled page order, warmups, and serial
+timed conversions with normal garbage collection. Timing includes parsing,
+service construction, and Markdown generation. Startup, imports, file reads,
+and output hashing are excluded. Converted output is never cached.
+
+The scripts record exact output matches, sample spread, versions, and source,
+binary, and input hashes. See [the Turndown report](benchmarks/REPORT.md) and
+[the converter comparison](benchmarks/COMPARISON.md) for results and methodology.
+Treat speed as workload-specific; different converters implement different
+output conventions.
+
+## Implementation
+
+Lexbor parses HTML in C. Lexidown's own Cython code converts that DOM to Markdown.
+Cython generates C at build time, and `setup.py` compiles it with the bundled
+parser into one extension, `lexidown._native`.
+
+1. The parser constructs a native DOM. Cython node wrappers retain its owning
+   document; supplied Python DOMs are cloned into this representation.
+2. The engine normalizes whitespace and collects text boundaries and blank-node
+   metadata. Cached node names and list positions avoid repeated scans; mutations
+   invalidate affected caches.
+3. An explicit traversal stack applies the Turndown rules. Sibling output is
+   joined in chunks to avoid repeatedly copying accumulated Markdown. The engine
+   also implements kept-HTML serialization, JavaScript whitespace behavior, and
+   UTF-16 heading lengths.
+
+Options and rule dictionaries remain Python objects. Built-in rules use compiled
+calls, while Python callbacks and method overrides run within the same engine.
+There is no alternate parser or converter fallback.
+
+| File | Responsibility |
+| --- | --- |
+| [`_native.pyx`](src/lexidown/_native.pyx) | Service, rule dispatch, and traversal |
+| [`_native_dom.pxi`](src/lexidown/_native_dom.pxi) | DOM, parsing, serialization, and whitespace |
+| [`_native_rules.pxi`](src/lexidown/_native_rules.pxi) | Built-in Markdown rules |
+| [`_lexbor.pxd`](src/lexidown/_lexbor.pxd) | Lexbor C declarations |
+| [`setup.py`](setup.py) | Combined native build |
+
+The bundled Lexbor 3.1.0 has local C patches to match Domino's older parsing
+behavior. They cover legacy select/menuitem handling and quirks involving
+textarea, tables, processing instructions, and foreign content. Exact changes,
+upstream versions, and attribution are documented in
+[`vendor/lexbor/README.md`](vendor/lexbor/README.md).
+
+## Distribution builds
+
+Build a source archive and local wheel into an empty output directory:
+
+```sh
+python -m pip install build==1.6.0 twine==7.0.0
+python -m build --outdir dist/release
+python -m twine check --strict dist/release/*
+```
+
+Local wheels target the build machine. Use the
+[wheel workflow](.github/workflows/wheels.yml) for portable distributions. It
+builds a source archive, builds wheels from that archive, validates metadata,
+and runs the full suite against each installed wheel outside the source tree.
+`scripts/check_wheel.py` also verifies that the compiled extension and required
+license notices are installed. Wheel tests need html5lib, but no Node.js.
+
+The workflow configures standard CPython 3.10–3.14 for these targets:
+
+| Platform | Architectures | Build baseline |
+| --- | --- | --- |
+| Linux with glibc | x86-64, ARM64 | manylinux, glibc 2.28 |
+| Linux with musl | x86-64, ARM64 | musllinux, musl 1.2 |
+| macOS | Intel x86-64, Apple Silicon ARM64 | macOS 11 |
+| Windows | x64 | AMD64 |
+
+There are 35 configured wheel builds. Each Python version needs its own wheel;
+the extension does not use the stable `abi3` ABI. Windows ARM64, 32-bit systems,
+PyPy, and free-threaded CPython are outside this matrix. A platform is validated
+by its own successful build and installed-wheel tests.
+
+## Licensing
+
+The conversion code retains Turndown's MIT license and attribution. Lexbor's
+Apache-2.0 license, NOTICE, BSD-2-Clause notices, and local modification notices
+must be preserved. The package metadata describes these component obligations
+with `MIT AND Apache-2.0 AND BSD-2-Clause`.
+
+See [LICENSE](LICENSE), [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md), and the
+[Apache redistribution terms](https://www.apache.org/licenses/LICENSE-2.0).

lexidown-0.1.0/LICENSE

@@ -0,0 +1,24 @@
+MIT License
+
+Copyright (c) 2026 EvickaStudio
+Copyright (c) 2017 Dom Christie
+Copyright (c) 2014 Luc Thevenard <lucthevenard@gmail.com>
+Parser adaptations: Copyright (c) 2006-2013 James Graham and other contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

lexidown-0.1.0/MANIFEST.in

@@ -0,0 +1,7 @@
+include DEVELOPMENT.md package.json package-lock.json
+recursive-include tests *.py *.json
+recursive-include examples *.py
+recursive-include scripts *.py *.cjs
+recursive-include benchmarks *.md *.json *.txt
+recursive-include src/lexidown *.pyx *.pxi *.pxd
+recursive-include vendor/lexbor *.c *.h LICENSE BSD-LICENSE NOTICE version README.md

lexidown-0.1.0/PKG-INFO

@@ -0,0 +1,259 @@
+Metadata-Version: 2.4
+Name: lexidown
+Version: 0.1.0
+Summary: Fast native HTML-to-Markdown for Python, compatible with Turndown
+License-Expression: MIT AND Apache-2.0 AND BSD-2-Clause
+Keywords: html,markdown,turndown,lexbor
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Cython
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: vendor/lexbor/LICENSE
+License-File: vendor/lexbor/NOTICE
+License-File: vendor/lexbor/BSD-LICENSE
+License-File: THIRD_PARTY_NOTICES.md
+Provides-Extra: dev
+Requires-Dist: Cython==3.3.0; extra == "dev"
+Requires-Dist: html5lib==1.1; extra == "dev"
+Requires-Dist: ruff==0.16.6; extra == "dev"
+Dynamic: license-file
+
+# Lexidown
+
+**Fast native HTML-to-Markdown for Python, compatible with Turndown.**
+
+Lexidown brings [Turndown](https://github.com/mixmark-io/turndown)'s conversion
+behavior, options, and extensible service API to Python. Its Cython conversion
+engine and bundled Lexbor C parser require no runtime dependencies.
+
+## Performance
+
+Default settings, eight complete HTML pages, CPython 3.14.7 and Node.js 22.23.2
+on an Intel Core i7-1260P:
+
+| Converter | Implementation | Speed vs JS Turndown | Exact JS output |
+| --- | --- | ---: | ---: |
+| JavaScript Turndown | JavaScript + Domino | 1.00× | 8/8 |
+| **Lexidown** | **Cython + Lexbor** | **2.94×** | **8/8** |
+| [html2text](https://pypi.org/project/html2text/) | Python | 1.16× | 0/8 |
+| [markdownify](https://pypi.org/project/markdownify/) | Python + BeautifulSoup | 0.61× | 0/8 |
+| [html-to-markdown](https://pypi.org/project/html-to-markdown/) | Rust | 4.46× | 0/8 |
+| [fast-h2m](https://pypi.org/project/fast-h2m/) | Rust | 3.72× | 0/8 |
+
+Speed is the geometric mean of JavaScript time divided by converter time;
+above 1 means faster. Each page has 21 timed conversions across three fresh
+processes. Results depend on the input, and different output is not a quality
+score. See the [full comparison](benchmarks/COMPARISON.md) for per-page timings,
+versions, methodology, and additional configurations.
+
+Lexidown also matches Turndown byte for byte across all **104 page/option
+combinations** in the [compatibility benchmark](benchmarks/REPORT.md).
+
+## Installation
+
+Requires Python 3.10 or newer.
+
+```sh
+python -m pip install lexidown
+```
+
+Platform wheels contain the native conversion engine and Lexbor. Installing a
+matching wheel requires no compiler, Cython, Node.js, or separate HTML parser.
+For platforms without a matching wheel, see [building from source](#building-from-source).
+
+## Usage
+
+```python
+from lexidown import TurndownService
+
+service = TurndownService({"headingStyle": "atx"})
+markdown = service.turndown(
+    "<h1>Hello world!</h1><p>Made with <strong>Python</strong>.</p>"
+)
+assert markdown == "# Hello world!\n\nMade with **Python**."
+```
+
+To discard scripts and styles along with their contents:
+
+```python
+service = TurndownService().remove(["script", "style"])
+assert service.turndown("<style>p { color: red }</style><p>Hello</p>") == "Hello"
+```
+
+Like Turndown, the default conversion retains the text inside unsupported tags.
+Inline styling attributes do not appear in converted Markdown.
+
+For a runnable example comparing default output with custom formatting, HTML
+cleanup, and a strikethrough rule, see [examples/formatting.py](examples/formatting.py).
+After installing Lexidown, run it from the project directory:
+
+```sh
+python examples/formatting.py
+```
+
+## Options
+
+Pass a dictionary to `TurndownService`. Option names and defaults match
+Turndown 7.2.4.
+
+| Option | Default | Alternatives |
+| --- | --- | --- |
+| `headingStyle` | `"setext"` | `"atx"` |
+| `hr` | `"* * *"` | Thematic break text |
+| `bulletListMarker` | `"*"` | `"-"`, `"+"` |
+| `codeBlockStyle` | `"indented"` | `"fenced"` |
+| `fence` | `"```"` | `"~~~"` |
+| `emDelimiter` | `"_"` | `"*"` |
+| `strongDelimiter` | `"**"` | `"__"` |
+| `linkStyle` | `"inlined"` | `"referenced"` |
+| `linkReferenceStyle` | `"full"` | `"collapsed"`, `"shortcut"` |
+| `br` | `"  "` | Text inserted before a line-break newline |
+| `preformattedCode` | `False` | `True` |
+
+### Advanced options
+
+`blankReplacement`, `keepReplacement`, and `defaultReplacement` accept callbacks
+with `(content, node, options)` arguments to customize blank elements, kept HTML,
+and elements without a matching rule. `options` is a dictionary. Callbacks may
+omit unused trailing arguments.
+
+The `rules` option accepts an ordered dictionary of rules to replace the
+built-in rule set.
+
+## Methods
+
+### `turndown(html_or_node)`
+
+Convert an HTML string or DOM node to a Markdown string. A service can be reused
+for multiple conversions.
+
+### `addRule(key, rule)`
+
+Register a conversion rule. `add_rule` is also available as a Python-style alias.
+
+```python
+service = TurndownService().addRule(
+    "strikethrough",
+    {
+        "filter": ["del", "s", "strike"],
+        "replacement": lambda content: "~~" + content + "~~",
+    },
+)
+assert service.turndown("<del>old</del>") == "~~old~~"
+```
+
+### `keep(filter)`
+
+Preserve matching elements as HTML:
+
+```python
+service = TurndownService().keep("ins")
+assert service.turndown("<ins>new</ins>") == "<ins>new</ins>"
+```
+
+### `remove(filter)`
+
+Discard matching elements and their contents. `keep` and `remove` accept the
+same filters as rules. Added and built-in rules take precedence over both;
+use `addRule` to override an element that already has a conversion rule.
+
+### `use(plugin)`
+
+Apply a plugin function, or a list of plugin functions. Plugins receive the
+service and can register rules or change options:
+
+```python
+def strikethrough(service):
+    service.addRule(
+        "strikethrough",
+        {"filter": "del", "replacement": lambda content: "~~" + content + "~~"},
+    )
+
+
+service = TurndownService().use(strikethrough)
+assert service.turndown("<del>old</del>") == "~~old~~"
+```
+
+`addRule`, `keep`, `remove`, and `use` return the service for chaining.
+JavaScript plugins need to be translated into Python functions.
+
+## Extending with rules
+
+A rule is a dictionary containing:
+
+- `filter`: a lowercase tag name, a list of tag names, or a callback taking
+  `(node, options)` and returning whether the rule matches.
+- `replacement`: a callback taking `(content, node, options)` and returning
+  Markdown. `content` is the converted content of the element's children.
+- `append` (optional): a callback taking `options` and returning text to append
+  after conversion, useful for reference definitions. Use a closure or bound
+  method to maintain rule state.
+
+Callbacks may omit unused trailing arguments. Rules are selected in this order:
+blank rules, added rules (newest first), built-in rules, keep rules, remove rules,
+then the default rule.
+
+Rule callbacks receive DOM nodes with Turndown's camelCase properties, including
+`nodeName`, `nodeType`, `textContent`, `outerHTML`, `parentNode`, `childNodes`,
+`children`, `firstChild`, `nextSibling`, `previousSibling`, and `getAttribute`.
+Conversion also exposes `isBlock`, `isCode`, `isBlank`, and
+`flankingWhitespace` (a dictionary).
+
+### Escaping
+
+Text is escaped to prevent Markdown syntax from changing its meaning. Override
+`service.escape` to customize this behavior. Text inside code bypasses escaping.
+
+## DOM input
+
+In addition to HTML strings, `turndown()` accepts native Lexidown DOM nodes and
+`xml.dom.minidom` element, document, and fragment nodes. DOM inputs are cloned
+before whitespace normalization, leaving the input tree unchanged.
+
+```python
+from xml.dom.minidom import parseString
+
+from lexidown import TurndownService
+
+# minidom parses XML; use well-formed markup when constructing a DOM this way.
+document = parseString("<p>Hello <strong>world</strong></p>")
+assert TurndownService().turndown(document.documentElement) == "Hello **world**"
+```
+
+## Building from source
+
+From a source checkout or unpacked source distribution:
+
+```sh
+python -m pip install .
+```
+
+A C compiler and Python development headers are required. The build installs
+Cython automatically and compiles the included Lexbor sources. No parser sources
+need to be downloaded separately.
+
+See [Development](DEVELOPMENT.md) for tests, Ruff formatting, benchmarks,
+implementation details, and wheel builds.
+
+## Development details
+
+Turndown was ported to Python and optimized using **Codex + GPT-6 Astra Ultra**.
+
+## License
+
+Lexidown's conversion code is [MIT licensed](LICENSE), with Turndown's original
+credits retained. Bundled Lexbor is Apache-2.0 licensed and includes BSD-2-Clause
+components. Their licenses and notices are included in source distributions and
+wheels; the combined package expression is `MIT AND Apache-2.0 AND BSD-2-Clause`.
+
+See [third-party notices](THIRD_PARTY_NOTICES.md) and
+[Lexbor's provenance and modifications](vendor/lexbor/README.md).