opendockit 0.1.0__tar.gz

opendockit-0.1.0/.gitignore

@@ -0,0 +1,24 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.venv/
+
+# Tools
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+
+# Editors / OS
+.idea/
+.vscode/
+.DS_Store
+
+# Local only: sample documents and backups never go into git
+data/
+.backup/
+.env

opendockit-0.1.0/CHANGELOG.md

@@ -0,0 +1,27 @@
+# Changelog
+
+All notable changes to this project are documented here.
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-11-12
+
+First public release.
+
+### Added
+
+- `convert`, `convert_file` and `aconvert`: PDF, DOCX, PPTX, XLSX, XLS, Excel 2003 XML, CSV, HWP, HWPX, HTML/MHTML,
+  email (`.eml`, mbox), TXT, Markdown and image files to Markdown.
+- `ConversionResult` with per-page / slide / sheet sections, metadata, warnings, extracted images and a character count.
+- `ConvertOptions` with a character budget (`max_chars`) that stops conversion instead of truncating its output.
+- `IsolatedConverter`: conversion in disposable worker processes with memory and time limits.
+- `describe_images` and `OpenAIImageDescriber` (optional `openai` extra) to describe pictures and transcribe scanned pages;
+  `OpenAIImageDescriber.check()` to fail fast on a missing extra or API key.
+- `opendockit` command-line tool.
+- `examples/`: sample documents for every format with their converted output.
+- Optional `heic` extra for HEIC/HEIF images (kept out of the default install because its wheels bundle GPL/LGPL codecs).
+
+[Unreleased]: https://github.com/hyeonsang010716/OpenDocKit/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/hyeonsang010716/OpenDocKit/releases/tag/v0.1.0

opendockit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 OpenDocKit contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

opendockit-0.1.0/PKG-INFO

@@ -0,0 +1,205 @@
+Metadata-Version: 2.5
+Name: opendockit
+Version: 0.1.0
+Summary: Convert PDF, Office, HWP/HWPX, HTML, email, spreadsheets and images into clean, LLM-ready Markdown.
+Project-URL: Homepage, https://github.com/hyeonsang010716/OpenDocKit
+Project-URL: Documentation, https://github.com/hyeonsang010716/OpenDocKit/tree/main/docs
+Project-URL: Issues, https://github.com/hyeonsang010716/OpenDocKit/issues
+Project-URL: Changelog, https://github.com/hyeonsang010716/OpenDocKit/blob/main/CHANGELOG.md
+Author: OpenDocKit contributors
+License-Expression: MIT
+License-File: LICENSE
+Keywords: document-conversion,docx,hwp,hwpx,llm,markdown,pdf,pptx,rag,xlsx
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Office/Business :: Office Suites
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: beautifulsoup4!=4.14.0,>=4.13
+Requires-Dist: charset-normalizer>=3.5
+Requires-Dist: lxml>=6.0
+Requires-Dist: olefile>=0.47
+Requires-Dist: openpyxl>=3.1.4
+Requires-Dist: pdfminer-six>=20251230
+Requires-Dist: pdfplumber>=0.11.9
+Requires-Dist: pillow>=11.3
+Requires-Dist: pypdfium2>=4.30
+Requires-Dist: python-docx>=1.1
+Requires-Dist: python-pptx>=1.0
+Requires-Dist: xlrd>=2.0.1
+Provides-Extra: heic
+Requires-Dist: pillow-heif>=0.16; extra == 'heic'
+Provides-Extra: openai
+Requires-Dist: httpx>=0.27; extra == 'openai'
+Requires-Dist: langchain-core>=1.0; extra == 'openai'
+Requires-Dist: langchain-openai>=1.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+<h1 align="center">OpenDocKit</h1>
+
+<p align="center">
+  <b>Turn any document into clean, LLM-ready Markdown.</b><br>
+  PDF · DOCX · PPTX · XLSX · HWP/HWPX · HTML · email · CSV · images — one function, no GPU, MIT licensed.
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/opendockit/"><img alt="PyPI" src="https://img.shields.io/pypi/v/opendockit"></a>
+  <a href="https://pypi.org/project/opendockit/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/opendockit"></a>
+  <a href="https://github.com/hyeonsang010716/OpenDocKit/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/hyeonsang010716/OpenDocKit/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="https://github.com/hyeonsang010716/OpenDocKit/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue"></a>
+</p>
+
+---
+
+```python
+from opendockit import convert_file
+
+result = convert_file("quarterly-report.hwp")
+
+result.text        # Markdown: headings, tables, lists, footnotes, LaTeX equations
+result.sections    # per page / slide / sheet — for chunking and citations ("page 3")
+result.warnings    # what was skipped or approximated, in plain sentences
+```
+
+## Why OpenDocKit
+
+Most converters handle the happy path. OpenDocKit is built for the files people actually upload.
+
+- **Structure, not just text.** Merged table cells, multi-column PDFs, borderless tables, nested lists, footnotes,
+  speaker notes, native charts as data tables, SmartArt as nested lists, Word / PowerPoint / Hancom equations as LaTeX.
+- **First-class Korean documents.** A from-scratch HWP 5.x parser, HWPX, CP949 detection, NFC normalization, and the
+  form-heavy tables Korean offices produce. Most general-purpose converters cannot read these at all.
+- **Detects by content, not extension.** An HTML table saved as `.xls`, a CSV with a title block, an MHTML page named
+  `.doc` — each is read as what it really is, and a warning tells you.
+- **Safe on untrusted uploads.** Zip bombs, runaway merged-cell counts, recursive PDF objects and multi-gigabyte images
+  are bounded. `IsolatedConverter` runs every file in a disposable worker process with memory and time limits.
+- **Images in place.** Pictures, charts and scanned pages are extracted, filtered, and — if you plug in a vision
+  model — described right where they appeared in the document.
+- **A hard character budget.** `max_chars` *stops* work instead of truncating afterwards, so a file that will be
+  rejected anyway costs neither parsing time nor vision-model calls.
+- **Permissive by default.** MIT licensed; everything a plain `pip install opendockit` brings in is MIT / BSD /
+  Apache-style (no PyMuPDF, no pyhwp), with no ML models and no GPU. The only copyleft code is behind the opt-in
+  `heic` extra.
+
+## Installation
+
+```bash
+pip install opendockit                # conversion only — never touches the network
+pip install "opendockit[openai]"      # + image description with a vision model
+pip install "opendockit[heic]"        # + HEIC/HEIF images (pulls in GPL/LGPL-licensed codec libraries)
+```
+
+Requires Python 3.12 or newer on Linux or macOS. Windows is not supported yet for HTML and email
+([why](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/limitations.md#platforms)).
+
+## Quick start
+
+### Convert
+
+```python
+from opendockit import convert, convert_file
+
+result = convert_file("report.pdf")              # from a path
+result = convert(data, "report.pdf")             # from bytes (the filename is only a hint)
+
+print(result.text)
+print(result.char_count, result.exceeded)        # size of the output, and whether the limit stopped it
+```
+
+`convert` is synchronous, offline and deterministic. Pictures are collected in `result.images` and the text keeps an
+`[Image]` placeholder where each one was.
+
+### Describe images with a vision model
+
+```python
+from opendockit import OpenAIImageDescriber, aconvert
+
+result = await aconvert(data, "report.pdf", describer=OpenAIImageDescriber())   # reads OPENAI_API_KEY
+```
+
+Charts become data tables, diagrams become lists, scanned pages are transcribed, and decorative images are dropped.
+`OpenAIImageDescriber(model)` accepts any LangChain chat model, and anything with an
+`async describe(image) -> str` method works as a describer. See [docs/images.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/images.md).
+
+### Serve it
+
+```python
+from opendockit import ConversionError, IsolatedConverter
+
+async with IsolatedConverter(workers=2) as converter:      # 2 GB and 2 minutes per file by default
+    try:
+        result = await converter.convert(data, filename)
+    except ConversionError as exc:
+        print(exc.code, exc)                               # stable code + a message safe to show users
+```
+
+One bad file ends one worker; your server and the next upload carry on. See [docs/server.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/server.md).
+
+### Command line
+
+```bash
+opendockit report.pdf                 # Markdown on stdout, warnings on stderr
+opendockit report.pdf --json          # sections, metadata, warnings, images
+opendockit report.pdf --count         # just the character count
+opendockit scan.pdf --describe        # describe images with a vision model (uses OPENAI_API_KEY)
+```
+
+## Supported formats
+
+| Format | Sections | Highlights |
+| --- | --- | --- |
+| PDF | page | reading order across columns, ruled and borderless tables, form field values, headings from font size, repeated header/footer removal, scanned pages via vision model |
+| DOCX | document | heading styles, list numbering, merged cells, charts → tables, SmartArt, footnotes/endnotes, equations → LaTeX, comments |
+| PPTX | slide | on-screen order, bullet levels, tables, charts → tables, SmartArt, equations, speaker notes |
+| XLSX · XLS · Excel 2003 XML · CSV | sheet | several tables per sheet, merged cells, number and date formats, charts, cell comments |
+| HWP · HWPX | document | outline headings, numbering, merged cells, captions, footnotes, equations → LaTeX |
+| HTML · MHTML | document | tables with rowspan/colspan, layout tables unwrapped, MathML → LaTeX, hidden elements removed |
+| Email (`.eml`, mbox) | document | headers, HTML or plain body, forwarded messages, calendar invites |
+| TXT · Markdown | document | encoding detection (UTF-8, CP949, UTF-16, Windows-1252) |
+| Images | document / page | JPEG, PNG, WebP, GIF, BMP, TIFF, AVIF, and HEIC with the `heic` extra — described by a vision model |
+
+Not supported: legacy `.doc`/`.ppt`, RTF, ODF. The full list of what each format restores is in
+[docs/formats.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/formats.md); the honest list of what it cannot do is in [docs/limitations.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/limitations.md).
+
+## Examples
+
+[`examples/`](https://github.com/hyeonsang010716/OpenDocKit/tree/main/examples/) has one sample document per format and the Markdown OpenDocKit produces from each — with and
+without a vision model — so you can judge the output before installing anything.
+
+## Documentation
+
+| | |
+| --- | --- |
+| [Formats](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/formats.md) | What is extracted from each format and how it is rendered |
+| [Options](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/options.md) | `ConvertOptions`, exceptions and error codes |
+| [Images](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/images.md) | Extraction, filtering, vision models, custom describers |
+| [Running on a server](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/server.md) | `IsolatedConverter`, resource limits, the character budget |
+| [Integration guide](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/integration-guide.md) | What an upload UI should tell its users |
+| [Design](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/design.md) | Why it is built this way, and how it was verified |
+| [Limitations](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/limitations.md) | Known gaps, stated plainly |
+| [Decision records](https://github.com/hyeonsang010716/OpenDocKit/tree/main/docs/decisions/) | Every deliberate trade-off, with the test that pins it |
+
+## Contributing
+
+Contributions are welcome — bug reports with a minimal reproducing document are especially valuable.
+Start with [CONTRIBUTING.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/CONTRIBUTING.md).
+
+```bash
+git clone https://github.com/hyeonsang010716/OpenDocKit.git && cd OpenDocKit
+uv sync
+uv run pytest -n auto -m "not timing"   # the bulk of the suite, in parallel
+uv run pytest -m timing                  # tests that measure time, one at a time
+uv run ruff check
+```
+
+## License
+
+[MIT](https://github.com/hyeonsang010716/OpenDocKit/blob/main/LICENSE)

opendockit-0.1.0/README.md

@@ -0,0 +1,160 @@
+<h1 align="center">OpenDocKit</h1>
+
+<p align="center">
+  <b>Turn any document into clean, LLM-ready Markdown.</b><br>
+  PDF · DOCX · PPTX · XLSX · HWP/HWPX · HTML · email · CSV · images — one function, no GPU, MIT licensed.
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/opendockit/"><img alt="PyPI" src="https://img.shields.io/pypi/v/opendockit"></a>
+  <a href="https://pypi.org/project/opendockit/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/opendockit"></a>
+  <a href="https://github.com/hyeonsang010716/OpenDocKit/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/hyeonsang010716/OpenDocKit/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="https://github.com/hyeonsang010716/OpenDocKit/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue"></a>
+</p>
+
+---
+
+```python
+from opendockit import convert_file
+
+result = convert_file("quarterly-report.hwp")
+
+result.text        # Markdown: headings, tables, lists, footnotes, LaTeX equations
+result.sections    # per page / slide / sheet — for chunking and citations ("page 3")
+result.warnings    # what was skipped or approximated, in plain sentences
+```
+
+## Why OpenDocKit
+
+Most converters handle the happy path. OpenDocKit is built for the files people actually upload.
+
+- **Structure, not just text.** Merged table cells, multi-column PDFs, borderless tables, nested lists, footnotes,
+  speaker notes, native charts as data tables, SmartArt as nested lists, Word / PowerPoint / Hancom equations as LaTeX.
+- **First-class Korean documents.** A from-scratch HWP 5.x parser, HWPX, CP949 detection, NFC normalization, and the
+  form-heavy tables Korean offices produce. Most general-purpose converters cannot read these at all.
+- **Detects by content, not extension.** An HTML table saved as `.xls`, a CSV with a title block, an MHTML page named
+  `.doc` — each is read as what it really is, and a warning tells you.
+- **Safe on untrusted uploads.** Zip bombs, runaway merged-cell counts, recursive PDF objects and multi-gigabyte images
+  are bounded. `IsolatedConverter` runs every file in a disposable worker process with memory and time limits.
+- **Images in place.** Pictures, charts and scanned pages are extracted, filtered, and — if you plug in a vision
+  model — described right where they appeared in the document.
+- **A hard character budget.** `max_chars` *stops* work instead of truncating afterwards, so a file that will be
+  rejected anyway costs neither parsing time nor vision-model calls.
+- **Permissive by default.** MIT licensed; everything a plain `pip install opendockit` brings in is MIT / BSD /
+  Apache-style (no PyMuPDF, no pyhwp), with no ML models and no GPU. The only copyleft code is behind the opt-in
+  `heic` extra.
+
+## Installation
+
+```bash
+pip install opendockit                # conversion only — never touches the network
+pip install "opendockit[openai]"      # + image description with a vision model
+pip install "opendockit[heic]"        # + HEIC/HEIF images (pulls in GPL/LGPL-licensed codec libraries)
+```
+
+Requires Python 3.12 or newer on Linux or macOS. Windows is not supported yet for HTML and email
+([why](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/limitations.md#platforms)).
+
+## Quick start
+
+### Convert
+
+```python
+from opendockit import convert, convert_file
+
+result = convert_file("report.pdf")              # from a path
+result = convert(data, "report.pdf")             # from bytes (the filename is only a hint)
+
+print(result.text)
+print(result.char_count, result.exceeded)        # size of the output, and whether the limit stopped it
+```
+
+`convert` is synchronous, offline and deterministic. Pictures are collected in `result.images` and the text keeps an
+`[Image]` placeholder where each one was.
+
+### Describe images with a vision model
+
+```python
+from opendockit import OpenAIImageDescriber, aconvert
+
+result = await aconvert(data, "report.pdf", describer=OpenAIImageDescriber())   # reads OPENAI_API_KEY
+```
+
+Charts become data tables, diagrams become lists, scanned pages are transcribed, and decorative images are dropped.
+`OpenAIImageDescriber(model)` accepts any LangChain chat model, and anything with an
+`async describe(image) -> str` method works as a describer. See [docs/images.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/images.md).
+
+### Serve it
+
+```python
+from opendockit import ConversionError, IsolatedConverter
+
+async with IsolatedConverter(workers=2) as converter:      # 2 GB and 2 minutes per file by default
+    try:
+        result = await converter.convert(data, filename)
+    except ConversionError as exc:
+        print(exc.code, exc)                               # stable code + a message safe to show users
+```
+
+One bad file ends one worker; your server and the next upload carry on. See [docs/server.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/server.md).
+
+### Command line
+
+```bash
+opendockit report.pdf                 # Markdown on stdout, warnings on stderr
+opendockit report.pdf --json          # sections, metadata, warnings, images
+opendockit report.pdf --count         # just the character count
+opendockit scan.pdf --describe        # describe images with a vision model (uses OPENAI_API_KEY)
+```
+
+## Supported formats
+
+| Format | Sections | Highlights |
+| --- | --- | --- |
+| PDF | page | reading order across columns, ruled and borderless tables, form field values, headings from font size, repeated header/footer removal, scanned pages via vision model |
+| DOCX | document | heading styles, list numbering, merged cells, charts → tables, SmartArt, footnotes/endnotes, equations → LaTeX, comments |
+| PPTX | slide | on-screen order, bullet levels, tables, charts → tables, SmartArt, equations, speaker notes |
+| XLSX · XLS · Excel 2003 XML · CSV | sheet | several tables per sheet, merged cells, number and date formats, charts, cell comments |
+| HWP · HWPX | document | outline headings, numbering, merged cells, captions, footnotes, equations → LaTeX |
+| HTML · MHTML | document | tables with rowspan/colspan, layout tables unwrapped, MathML → LaTeX, hidden elements removed |
+| Email (`.eml`, mbox) | document | headers, HTML or plain body, forwarded messages, calendar invites |
+| TXT · Markdown | document | encoding detection (UTF-8, CP949, UTF-16, Windows-1252) |
+| Images | document / page | JPEG, PNG, WebP, GIF, BMP, TIFF, AVIF, and HEIC with the `heic` extra — described by a vision model |
+
+Not supported: legacy `.doc`/`.ppt`, RTF, ODF. The full list of what each format restores is in
+[docs/formats.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/formats.md); the honest list of what it cannot do is in [docs/limitations.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/limitations.md).
+
+## Examples
+
+[`examples/`](https://github.com/hyeonsang010716/OpenDocKit/tree/main/examples/) has one sample document per format and the Markdown OpenDocKit produces from each — with and
+without a vision model — so you can judge the output before installing anything.
+
+## Documentation
+
+| | |
+| --- | --- |
+| [Formats](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/formats.md) | What is extracted from each format and how it is rendered |
+| [Options](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/options.md) | `ConvertOptions`, exceptions and error codes |
+| [Images](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/images.md) | Extraction, filtering, vision models, custom describers |
+| [Running on a server](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/server.md) | `IsolatedConverter`, resource limits, the character budget |
+| [Integration guide](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/integration-guide.md) | What an upload UI should tell its users |
+| [Design](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/design.md) | Why it is built this way, and how it was verified |
+| [Limitations](https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/limitations.md) | Known gaps, stated plainly |
+| [Decision records](https://github.com/hyeonsang010716/OpenDocKit/tree/main/docs/decisions/) | Every deliberate trade-off, with the test that pins it |
+
+## Contributing
+
+Contributions are welcome — bug reports with a minimal reproducing document are especially valuable.
+Start with [CONTRIBUTING.md](https://github.com/hyeonsang010716/OpenDocKit/blob/main/CONTRIBUTING.md).
+
+```bash
+git clone https://github.com/hyeonsang010716/OpenDocKit.git && cd OpenDocKit
+uv sync
+uv run pytest -n auto -m "not timing"   # the bulk of the suite, in parallel
+uv run pytest -m timing                  # tests that measure time, one at a time
+uv run ruff check
+```
+
+## License
+
+[MIT](https://github.com/hyeonsang010716/OpenDocKit/blob/main/LICENSE)

opendockit-0.1.0/docs/decisions/core.md

@@ -0,0 +1,130 @@
+# Core decisions
+
+Guide for agents and developers working on `src/opendockit/`: package structure, dependency direction and the decisions shared by every format. Usage, options and limits are in the [README](../../README.md), [options.md](../options.md) and [limitations.md](../limitations.md).
+
+## Structure and dependency direction
+
+```
+src/opendockit/
+├── __init__.py      # public API (convert, aconvert, IsolatedConverter, describe_images, OpenAIImageDescriber, ConvertOptions, exceptions)
+├── cli.py           # command line: opendockit <file> [--json] [--describe]
+├── converter.py     # size check → detect → extractor → clean_text → image filtering → render (fill placeholders, assembly, character count, limit verdict)
+├── isolated.py      # isolated conversion for servers (IsolatedConverter): a new worker process per file, memory and time limits, a crash fails only that file
+├── image.py         # image decoding, downscaling, PNG/JPEG re-encoding, image-file magic bytes
+├── describe.py      # vision-model descriptions (LangChain ChatOpenAI). The only module that uses the network
+├── detect.py        # magic-byte detection. The extension is used only to tell the text family apart (TXT, MD, CSV, HTML) (decisions in detect.md)
+├── markdown.py      # clean_text · render_table · TableGrid · join_blocks — shared by every format
+├── models.py
+├── exceptions.py
+└── extractors/      # per format `(bytes, ConvertOptions) -> Extraction`. Registered in EXTRACTORS in __init__.py
+    ├── _shared.py   # Strict OOXML detection (`is_strict_package`, DOCX/PPTX), OOXML charts and SmartArt (PPTX/DOCX/HWPX), Office equations (OMML → LaTeX, DOCX/PPTX), automatic number strings, paragraph heads (`paragraph_head`), ruby text and list levels inside a cell (DOCX/HWP/HWPX), chart tables inside a cell (`embedded_table`), Hangul (HWP) bullet indent levels (`BulletNesting`, HWP/HWPX), symbol-font mapping (`symbol_font_text` — including ordinary letters in Symbol), character limit inside tables (`TableBudget`, DOCX/HWP/HWPX), DrawingML `a:br` line breaks (`drawing_paragraph_lines`, PPTX/DOCX/XLSX/charts), comment lines (`comment_line`, DOCX/PPTX/HWPX/Excel)
+    ├── _equation.py # Hancom equation script → LaTeX (HWP/HWPX). The original script when it cannot be converted
+    ├── docx.py
+    ├── html.py
+    ├── hwp.py
+    ├── hwpx.py
+    ├── image.py
+    ├── pdf.py
+    ├── pptx.py
+    ├── spreadsheet.py
+    └── text.py
+
+tests/               # unit tests; sample builders are in tests/builder.py
+docs/decisions/      # deliberate decisions per format (file list in "Deliberate decisions" below)
+```
+
+For use on a server see [server.md](../server.md) (conversion must go through `IsolatedConverter`).
+
+- `extractors/*` imports only `markdown`, `models` and `exceptions`. Extractors share with each other only the decoding functions of `text` (including MIME header blocks and unpacking MIME mail) and `extractors/_shared.py` / `_equation.py`. Warning filtering is `models.quiet_warnings` (spreadsheet, image, html). Exceptions: DOCX `w:altChunk` uses `html.extract_html`, `is_mhtml` and `text.extract_text` as they are (altChunk row in [office.md](office.md)). The image extractor uses `image.split_tiff_pages` (multi-page TIFF, image section of [extractors.md](extractors.md)).
+- An extractor registers images **as the original bytes** with `Extraction.add_image()` and puts the placeholder it gets back (`⟦IMG:<key>:n⟧`) at the right place in the body.
+  Decoding, size filtering, de-duplication and limits happen in one place only, `converter._prepare_images`.
+- `convert` does not use the network (deterministic, usable in a process pool). Vision-model calls live only in `describe.py`, and once descriptions are filled in
+  `converter.render` assembles again from `Section.template`. A failed description is a warning, not an exception.
+- The describer builds `langchain_openai.ChatOpenAI` itself (key from `OPENAI_API_KEY`, default model `describe.DEFAULT_MODEL`) or takes a LangChain chat model from the caller.
+- Extractors produce **sections only**. NFC normalization, whitespace cleanup, section heads (`## Slide N`) and the limit verdict are done by `converter` — not in extractors.
+  An extractor reports the character count with `Extraction.count()` for each unit it reads (PDF page, Excel/CSV row, PPTX slide, DOCX/HWP/HWPX top-level paragraph or table — large tables per cell through `TableBudget`)
+  and stops reading when it gets the exceeded signal (True). TXT and MD are read in one go; for HTML only large documents get their front part read first.
+- Tables must be built with `render_table`/`TableGrid`, so that the merged-cell rules (Markdown repeats vertically and keeps the first cell horizontally; a table that still has horizontal spans becomes HTML)
+  do not differ per format. A format that has span information passes the span counts to `TableGrid` (it does not repeat values itself). Exception: Excel spreads merges over the cells and is always Markdown ([spreadsheet-tables.md](spreadsheet-tables.md)).
+- User-facing text (exception messages, `warnings`) is English, fixed wording, and never contains file content or parser error text
+  (the cause is chained through `__cause__`). `warnings` carry no more than the sheet name or page number needed to say where.
+
+## Adding a new format
+
+1. Add a value to `models.DocumentFormat` → 2. content detection in `detect.py` → 3. write `extractors/<format>.py` and register it in `EXTRACTORS` →
+4. a sample builder in `tests/builder.py` and `tests/test_<format>.py` → 5. update the README table.
+
+## Deliberate decisions (do not revert)
+
+Only the common decisions (detection, limits, pictures, assembly, description) are here. The rest:
+
+- [extractors.md](extractors.md) — per-format decisions (PDF, PPTX, DOCX, HWP/HWPX, spreadsheets, HTML, text, mail, images)
+- DOCX / PPTX
+  - [office.md](office.md) — DOCX and PPTX
+  - [office-lists.md](office-lists.md) — numbering and lists
+  - [office-math.md](office-math.md) — equations
+  - [office-charts.md](office-charts.md) — chart value tables, axes, units
+  - [office-limits.md](office-limits.md) — processing time
+- PDF
+  - [pdf.md](pdf.md) — PDF
+  - [pdf-flow.md](pdf-flow.md) — reading order, two columns, labels beside a table, superscripts and subscripts
+  - [pdf-rows.md](pdf-rows.md) — joining fragments of the same row, pulling up pushed-down lines
+  - [pdf-tables.md](pdf-tables.md) — splitting table cells, header rows of continued tables, tables inside tables, checkboxes
+  - [pdf-borderless.md](pdf-borderless.md) — wide-cell rows of borderless tables, the empty top-left cell, narrow gaps between cells
+  - [pdf-limits.md](pdf-limits.md) — resource limits, page boxes, attachments
+  - [pdf-images.md](pdf-images.md) — pictures drawn in strips, pictures inside table cells, photo grids
+  - [pdf-fonts.md](pdf-fonts.md) — fonts without widths, symbol-font checkboxes, CMaps of built-in encodings, lower bound on fragment character width
+  - [pdf-cells.md](pdf-cells.md) — joining lines of text in ruled-table cells, filtering highlighter marks, tables made of cell shading
+- HWP / HWPX
+  - [hwp.md](hwp.md)
+- Excel / CSV
+  - [spreadsheet.md](spreadsheet.md) — spreadsheets
+  - [csv.md](csv.md) — CSV
+  - [spreadsheet-tables.md](spreadsheet-tables.md) — splitting tables, side-by-side tables, summary cells, filling merges, the first column of the column limit
+  - [spreadsheet-cells.md](spreadsheet-cells.md) — error cell values, pictures in cells, form controls, sheet SmartArt, default-password XLS, cell comments, the cell of a floating picture, time formats, parts the converter reads itself
+- Text family
+  - [encoding.md](encoding.md) — encoding detection
+  - [mail.md](mail.md) — mail
+  - [html.md](html.md) — HTML blocks (aside, quotes, checklists, item/value tables), exceptions for hidden elements, lazy-loading placeholder pictures, MHTML frames, inline SVG labels, `data:` URI pictures in MD
+- [images.md](images.md) — images, picture descriptions (the describer), placeholders
+- [detect.md](detect.md) — format detection (magic bytes, ZIP, OLE, telling the text family apart, MHTML, mail, extension warnings)
+
+| Decision | Why | Pinned by |
+| --- | --- | --- |
+| Markdown tables repeat a vertically merged value on every row | Each row carries by itself which category its values belong to (no information loss) | `test_markdown.py::TestTableGrid` |
+| Only a table that still has horizontally spanning cells after empty rows and columns are removed becomes an HTML table (rowspan/colspan, th in the first row, one row per line). A full-width merge of the title row at the top alone does not switch it. Cell text and the picture descriptions of that cell stay as written and only table tags are neutralized with `&lt;` (`markdown.neutralize_table_tags`) — the result is not rendered on screen but fed to an LLM, so escaping everything is noise. Applies to the formats that pass span counts to `TableGrid` (HWP, HWPX, DOCX, PPTX, HTML with a single header row) | A Markdown table cannot draw a horizontal merge, so nobody could tell what the empty cell in '\| 합계 \| \| 4개 과제 \|' (Total \| \| 4 tasks) was. A table with only vertical merges loses nothing through repetition, so it is not switched (existing output kept). Keeps a `</td>` in cell text from being confused with a cell boundary. In 295 real documents all 312 HTML tables had matching cell counts; 142 limit verdicts × 3, 0 violations | `test_markdown.py::TestMergedTableAsHtml`, `test_converter.py::TestAssembly::test_picture_in_an_html_table_cell_keeps_its_text` |
+| A table inside a table (HWP, HWPX, DOCX; for HTML see [extractors.md](extractors.md)) is not flattened: it is handed to the outer `TableGrid.embed`, which leaves a placeholder (private-use characters) in the cell text. The outer table becomes HTML (a one-line HTML table inside the cell); a one-column box becomes paragraphs + a standalone table. While rendering text that goes somewhere other than cell text, such as a header inside a cell, the position is marked as one that cannot take it (None) and the table is `flatten`ed ('cell / cell'). A table inside a footnote or endnote is `_shared.note_table` ('cell / cell; cell / cell'). Copies made for merged cells drop picture and table placeholders (`merged_copy`). The caption of an inner table stays with the table wherever it is | 'cell / cell' lost the row and column structure. A placeholder that leaks into a footnote is never resolved and is erased, so the content disappears. A table in a merged cell came out once per row. The inner table's caption was dropped, so a picture in the caption cost a description and then vanished | `test_markdown.py::TestMergedTableAsHtml` (4 nested cases), `test_hwp.py::TestNestedTables` (including footnotes), `test_quality.py::test_docx_nested_table_keeps_its_rows_and_columns`, `test_hwp.py::TestTableInsideFootnote`, `test_office_symbols_fields_notes.py::TestDocxTableInsideFootnote` |
+| PyMuPDF and pyhwp are not used | AGPL | — |
+| clean_text removes first, then NFC | In the other order the characters on both sides of a removed one combined anew and NFC broke (real PDF) | `test_markdown.py::test_result_is_nfc_even_when_removed_character_separated_combining_mark` |
+| Private-use bullets of Symbol/Wingdings (U+F0B7 etc.) are not removed but turned into Unicode symbols | The list markers of a Word PDF disappeared | `test_quality.py::test_symbol_font_bullets_become_unicode_bullets` |
+| Only Latin ligatures (U+FB00–FB06) are expanded (`fi`→`fi`). NFKC is not used | 'first' in a typeset PDF became a different word from 'first' (964 places in one book PDF). NFKC would also change ①, ㈜ and ² in Korean documents | `test_markdown.py::TestCleanText::test_expands_latin_ligatures_but_keeps_other_compatibility_characters` |
+| `TableGrid` limits span counts and builds the grid only from rows and columns where cells are placed. Spans are clipped at render time to the table's real extent (the last row/column where a cell was placed or that was announced with `reach`), and the number of rows dropped by the grid cell limit is reported through `omitted_rows()` so the extractor warns (`_shared.omitted_rows_warning`, HTML included). A cell that arrives without an address (`place_next`, HTML) looks for a free position starting at the column placed last in that row | Trusting span counts and cell addresses lets a few KB of input use seconds and gigabytes. It is blocked in one common place, not per format. rowSpan=5000 in a 2-row table made 4,999 fake rows, and gridSpan=10000 in the top cell left only 499 of 1,000 rows (no warning). Searching from column 0 took 56 seconds for one row of 50,000 cells | `test_markdown.py::TestTableGridLimits`, `test_office_robustness.py::TestSpansPastTheTable`, `test_hwp_robustness.py::TestSpansPastTheTable`, `test_core_html.py::TestRowsLeftOutByTheGridLimit`, `test_markdown.py::TestTableGridLimits::test_cells_placed_in_order_on_a_long_row_take_linear_time` |
+| The character limit (`max_chars`) is a criterion for stopping, not for cutting. Once exceeded, the result comes back with `exceeded=True` and the remaining pages, rows and pictures are not processed | Spending page analysis, picture decoding and vision-model calls on a document that will be rejected anyway is waste (130,000-row Excel 7.6 s → 0.4 s; 40 scanned pages 40 calls → 7). The former `max_output_chars` (cutting) and `truncated` were replaced by this | `test_converter.py::test_exceeding_the_character_limit_sets_exceeded_and_cuts_the_text`, `test_pdf.py::test_stops_reading_pages_once_the_character_limit_is_exceeded`, `test_image.py::test_description_stops_once_the_character_limit_is_exceeded` |
+| `exceeded` must be exactly "character count of the full conversion > limit". Mid-way verdicts use only values that are never larger than the real one: during extraction `markdown.solid_length` (excluding whitespace, placeholders and characters that will be removed; NFC), during description the body assembled with every remaining picture erased including its alt text (`describe._floor_length`; when there is clearly room, `converter.image_length_bound` skips the assembly). `render` does not inherit `result.exceeded` and decides again every time | Counting high blocks documents at or under the limit as exceeded. Counting by raw length made 4 documents with N = L come out exceeded, and judging by the body length at an intermediate point made 48 documents come out exceeded whose pictures were later erased as decorative and so shrank | `test_converter.py::TestLimitIsExact`, `test_image.py::test_placeholders_of_images_that_turn_out_decorative_do_not_count_against_the_limit`, `test_retry_does_not_count_already_described_images_twice`, `TestDecorativePictureWithAltText::test_limit_is_exact_when_a_picture_with_alt_text_gets_any_answer` |
+| `char_count` includes picture descriptions and `render` counts again every time. The total of a batch (several files) is added up by the caller — this library works per single file | It is the criterion the front end uses to limit uploads | `test_image.py::test_char_count_includes_image_descriptions_once_described` |
+| When the limit is exceeded, `sections[].text` is cut at the same point as `text` (`_cut_sections`; sections after the cut have empty text). The section list and `template` stay as they are | Only `text` was cut and the sections kept everything, so chunking and source attribution received content beyond the limit. Erasing the originals would shrink the character count on re-assembly and flip `exceeded` | `test_converter.py::TestAuditedAssemblyGaps::test_sections_are_cut_where_the_text_is_cut` and 2 more |
+| In documents that are not Markdown (anything but TXT/MD) an unpaired last ``` line is neutralized as `\```` (`clean_text`). Paired fences stay code blocks | A ``` in a single paragraph turned every section after it into a code block and whitespace cleanup stopped | `test_markdown.py::TestUnclosedCodeFence`, `test_converter.py::TestAuditedAssemblyGaps::test_stray_code_fence_in_a_docx_does_not_swallow_the_rest` |
+| A line holding a picture placeholder is a table row (folded to one line) only when it starts and ends with `\|` (including MD `\|a\|b\|`) or has the shape `<tr>…</tr>`. Known accepted gap: a paragraph of the form `\|…\|` is taken for a table row | A picture description was folded to one line and glued onto a paragraph that starts with `\|` or `<tr>`. Only lines ending in ` \|` were looked at, so a `\|a\|b\|` table broke at the picture | `test_converter.py::TestAuditedAssemblyGaps::test_paragraph_that_starts_like_a_table_row_gets_a_picture_block`, `test_core_text_and_mail.py::TestPictureInACompactMarkdownTable` |
+| Superscripts and subscripts become `10^6`, `10^{-3}`, `H_2O` in one place, `_shared.format_script`. Only scripts of at most 10 characters that start with a letter, digit or sign are marked; footnote marks, symbols, long text and ordinal suffixes after a number (21st) stay as they are. Scripts containing Hangul, Han characters or kana are not marked (`1,000원`, same rule as PDF). Sources: HTML sup/sub, DOCX w:vertAlign and LibreOffice w:position + small w:sz (at most 0.8× the paragraph size), PPTX baseline, HWPX hh:supscript/subscript, HWP character-shape attribute bits 15 and 16, XLSX rich text (full load only), XLS font escapement, `<Sup>`/`<Sub>` of Excel 2003 XML (`spreadsheet._xml_rich_text`) | Glued on as plain text, 10⁶ read as 106. Marking `[1]`, `†` and `: 125` as well produces hundreds of noise marks per wiki document. The HWP script bits (taken from hwplib) were confirmed on real files against the HWPX twin of the same document (hh:supscript) | `test_text_html.py::test_html_superscript_and_subscript_are_marked`, `test_html_superscript_with_hangul_or_hanja_is_left_as_plain_text`, `test_html_ordinal_suffix_after_a_number_is_not_marked_as_superscript`, `test_office.py::TestSuperscriptAndTextBoxes` (including 2 LibreOffice position cases), `test_hwp.py::TestSuperscript`, `test_spreadsheet.py::TestSuperscriptInCells`, `test_core_spreadsheet.py::TestXmlSpreadsheetScripts` |
+| Private-use characters are removed in the supplementary planes too (U+F0000–10FFFD) (`markdown._PRIVATE_USE`). Mapping Hancom font ordinals (U+F02B1–) → ① is not done for lack of evidence | Characters of Hancom-only fonts stayed as meaningless tokens | `test_markdown.py::TestCleanText::test_removes_supplementary_private_use_characters` |
+| PDF superscripts and subscripts are inferred from position (`pdf._scripted_text`): characters at most 0.8× the line's ordinary character size whose baseline is off by at least 0.2 upward or 0.1 downward (relative to the character size). Only marks are inserted into pdfminer's line text; lines whose character order differs from the line text, table cells and pages extracted the simple way are left alone. Runs containing Hangul or Han characters are not marked | A PDF has no script information. Rebuilding the line text changed the order and spacing of overlapping accents. A raised unit in a designed document became '12,345^{백만원}'. Of 2,377 marked lines in 20 public papers, a sample of 45 lines had 0 misjudgments | `test_pdf.py::TestSuperscriptByPosition` |
+| `convert` takes bytes, bytearray and memoryview; anything else is a `TypeError` (pointing to convert_file). A ConvertOptions in the file-name position, a file name that is not a str and an options value that is not ConvertOptions are also a `TypeError` before conversion (convert, aconvert, convert_file; `converter._check_arguments`) | Path strings and memoryviews were rejected as a 'corrupted file'. `convert(data, ConvertOptions(...))` raised TypeError only after running the whole conversion (222 s) | `test_converter.py::TestInputTypes`, `test_core_mbox_markdown_and_api.py::TestArgumentsInTheWrongPlace` |
+| `ConvertOptions.__post_init__` rejects values below the limit minimums (`models._OPTION_MINIMUMS`) with a `ValueError` (English). 0 for `max_chars`, `max_images`, `min_image_side` and `pdf_time_budget` means 'allow none'; `max_pages`, `max_sheet_cols`, `max_file_bytes` and the like must be at least 1. The type is checked too (`models._checked_number`, `TypeError`): None only for `max_chars`, `max_pages`, `max_images` and `pdf_time_budget`; a whole-number float for an integer limit (`5e5`) is accepted and converted to int; bool, strings and fractions are rejected. NaN and infinity are a `ValueError` (no time limit is `pdf_time_budget=None`) | `max_chars=-5` cut the last five characters and became 'exceeded', and `max_images=-1` dropped only the last picture. `max_sheet_cols=None` rejected an Excel file as 'corrupted' and `max_chars=2.5` was a TypeError in the middle of conversion. `pdf_render_scale=nan` passed the minimum comparison and a healthy PDF became 'corrupted' | `test_converter.py::TestLimits::test_negative_or_meaningless_zero_limits_are_rejected_when_options_are_made`, `test_limit_of_the_wrong_type_is_rejected_when_options_are_made`, `test_whole_number_float_limit_becomes_an_integer`, `test_non_finite_number_is_rejected_when_options_are_made` |
+| The single-cell top row of a two-column table is a table title (a paragraph above the table; `markdown._title_of`, common to `render_table` and `TableGrid` HTML) only when just the left cell is filled, the row below has non-numeric text in both cells and there is a numeric cell within the 5 rows below that. HTML, which merges multi-row headers itself, uses the same test through `markdown.table_title`. A ruled table in a PDF that continues from the previous page uses `render_table(merged_titles_only=True)` — a title only when the remaining cells are merged cells (None). Known accepted gap: the title row of a two-column table without numeric cells stays a header row | Titles were taken out only for 3 or more columns, so '부서별 인원 현황 \| ' (Headcount by department) became the header row and '부서 \| 인원' (Department \| Headcount) a data row (XLSX merged title, CSV, HTML colspan=2). A '구분 \| ' header row and two-column tables with text only stay as they are | `test_markdown.py::TestTwoColumnTitleRow`, `TestTitleRowOnlyFromMergedCells`, `test_text_html.py::TestTitleRowInsideMultiRowHeader::test_two_column_table_with_a_spanning_title_row` |
+| `clean_text` does not collapse consecutive blank lines inside paired fences (```) (TXT/MD and every other format alike). After an unpaired last fence it does. Assembly of a section with pictures (`converter._render_template` → `_collapse_blank_lines`) skips the inside of fences by the same pairing | Two blank lines between functions in an MD code block became one. Assembly collapsed the blank lines of the whole section again, so the same happened only in sections with pictures | `test_markdown.py::TestBlankLinesInsideCodeFences`, `test_core_image_and_assembly.py::TestBlankLinesInCodeNextToAPicture` |
+| Warning filtering inside the library uses `models.quiet_warnings` instead of `warnings.catch_warnings` (narrow category and module; inserted under a lock with a use count and only that entry removed at the end). Known accepted gap: while the block is open, warnings of the same category and module from other threads are ignored too, and overlap with a `catch_warnings` outside the library cannot be prevented (Python 3.12) | `catch_warnings` is not thread-safe, so after overlapping XLSX conversions the ignore filter for openpyxl's UserWarning stayed in the process for good. Serializing whole conversions with a lock would make concurrent conversions sequential | `test_core_spreadsheet.py::TestOpenpyxlWarnings::test_overlapping_conversions_on_threads_leave_the_filters_as_they_were` |
+| On a server, convert with `IsolatedConverter`: each worker gets its own `ProcessPoolExecutor(max_workers=1, max_tasks_per_child=1)`, started with forkserver (this package preloaded; spawn when unavailable). On a timeout, `BrokenProcessPool` or a cancelled request the worker is killed directly through the private `_processes` and the pool is rebuilt. On shutdown, `shutdown(wait=True)` is awaited in the default thread pool, and until it returns the pool's workers keep being killed (`isolated._shut_down`): the manager thread starts a replacement whenever a worker exits, so one can appear after the first kill. Memory or time overruns and a dead worker are `DocumentTooLargeError` (`TOO_COMPLEX_MESSAGE`) | Run on a server thread, a memory bomb or a crashing C extension takes the whole server down, and a thread cannot be stopped. With fork, the child can hang in a server that runs DB and asyncio threads. Without preloading a new worker takes 0.2 s, with it 5 ms, so a new worker per file gives the memory back. With several files in one pool, killing one fails the others too. `shutdown(wait=False)` emptied `_processes` while the management thread was refilling workers and that thread died with a TypeError (3 times per 15 tests). Known accepted gap: the forkserver preload list is one per process, so it overwrites another code's list A worker killed between writing its result and releasing the result queue's write lock left the lock held; a replacement worker then blocked on it while announcing its exit, and `aclose()` never returned (a hung run about once in four when the whole suite ran in parallel in a Linux container). | `test_isolated.py` (the memory limit only on Linux — macOS does not accept `RLIMIT_AS`, so it is skipped), `test_isolated.py::TestIsolatedConverter::test_stopping_does_not_wait_for_a_worker_blocked_on_the_result_queue` |
+
+## Commands
+
+Run from the repository root.
+
+```bash
+uv sync                                               # install dependencies
+uv run pytest                                         # unit tests
+uv run pytest -n auto -m "not timing"                 # everything but the timing tests, in parallel
+uv run pytest -m timing                               # tests that measure time, one at a time
+uv run ruff check
+uv run opendockit <file> [--json] [--describe]        # --describe calls the vision model (needs OPENAI_API_KEY, costs money)
+# To check the shape of a conversion, convert a real document with the CLI above (do not commit personal or copyrighted documents)
+```

opendockit-0.1.0/docs/decisions/csv.md

@@ -0,0 +1,19 @@
+# CSV decisions
+
+Deliberate decisions of the CSV extractor (`spreadsheet.extract_csv`) about delimiters, quotes and line verdicts. Split off from [spreadsheet.md](spreadsheet.md);
+the rule is the same: do not revert the decisions in this table. To change one, change its pinning test first and write the reason in the same row. Value formatting is,
+together with Excel, in [spreadsheet.md](spreadsheet.md); splitting tables within a sheet is in [spreadsheet-tables.md](spreadsheet-tables.md).
+
+## CSV
+
+| Decision | Why | Pinned by |
+| --- | --- | --- |
+| A row on which the csv module fails is read again with a quote-aware split (`_quoted_record`; only dialects with the doubled-quote rule and no escape character); when the quote closes, only that row is used and the csv module resumes from the next line (no warning). Only when the quote never closes before the end of the file is the rest, from that line on, split by delimiter alone (warning). `csv.field_size_limit` is not raised | An unclosed quote swallowed everything after it into one cell, and once that passed the limit (131,072 characters) the whole file was rejected. The limit is a process-global setting. The delimiter-only split used instead also cut healthy cells over the limit (long body text) at every comma, so the column limit dropped the rest and the author cell (348 KB → 2,410 characters, false warning) | `test_office.py::TestCsvWithBrokenQuotes` |
+| CSV uses all of the Sniffer's delimiter, quotechar and skipinitialspace (the single-quote and comma verdicts are changed only under the conditions of the rows below) | `, "서울시, 강남구"` was split into two cells | `test_spreadsheet.py::TestCsvDialect` |
+| A single quote in CSV is accepted as the quote character only when every cell containing it is wrapped whole (`_wraps_whole_fields`) | The Sniffer picked `'` because of a single `'목표'`, and the cell `'24년 상반기` swallowed the delimiter. `'홍길동','1,000'` is still read | `test_spreadsheet.py::TestCsvDialect::test_apostrophe_inside_a_field_is_not_a_quote_character`, `test_single_quote_as_quote_character` |
+| Even when the Sniffer picked the comma (or failed), if the number of cells split by tab, semicolon or pipe is the same on every line of the sample (2 or more cells; in the order tab → semicolon → pipe), that delimiter is used (`_steadier_delimiter`). For a single-line file, tab only. When double quotes wrap cells whole under the comma but not under the candidate, the comma stays. A file whose cell count differs per line is changed only for TSV that omits trailing empty cells (row below) and the following case, otherwise the Sniffer's choice stands: when there are two or more lines with the maximum candidate cell count (3 or more cells), they are the majority, and from the first of them (the header) on the comma cell count is uneven, that candidate is used (`_delimiter_of_most_lines`; leading title/meta lines and short lines are ignored). When the comma cell count is even from the header on, the `;`/`\|` are enumerations inside values, so comma. Known accepted gap: the `.tsv` extension is not passed to the extractor and so is not used, and an unquoted comma CSV whose every line contains the same number of semicolons or pipes is split by those | With the same number of commas per line the Sniffer picked the comma, so `서울\t1,234\t5,678` became `서울 1 \| 234 5 \| 678`, and decimal commas (`1,5`) and word enumerations were split into two cells. Title/meta lines of bank and ERP exports (`Umsatzanzeige;…`, `고객 목록`) or a single line omitting trailing empty cells made the cell count uneven, and `-1.234,56` was split into `-1.234 \| 56` (the earlier decision 'when the cell count differs per line, keep the Sniffer's choice' was changed with the owner's approval) | `test_spreadsheet.py::TestCsvDialect::test_delimiter_with_a_steady_column_count_wins_over_commas_inside_values`, `test_tab_delimiter_is_found_in_a_file_longer_than_the_sample`, `test_comma_file_with_other_delimiters_inside_values_stays_comma_delimited`, `TestCsvDelimiterOfMostLines` |
+| TSV whose tab count differs per line because trailing empty cells are omitted: when the Sniffer picked the comma, the comma cell count is uneven, the first line has a tab and every line containing a comma has a tab (2 or more lines with tabs; only when double quotes wrap cells whole by tab), it is tab (`_tab_on_every_comma_line`) | The amount `1,100,000` in Excel/ERP TSV exports was split at the thousands separators and the columns shifted. A comma CSV has lines with commas but no tab, or no tab in the header | `test_spreadsheet.py::TestCsvDialect::test_tab_file_whose_rows_omit_trailing_empty_cells_is_tab_delimited`, `test_comma_file_with_some_tabs_stays_comma_delimited` |
+| A CSV line with more delimiters than the column limit is cut before parsing to the column limit + one 'overflow marker' cell. Lines containing the quote character and lines inside an open quoted cell (decided by the parity of the quote count) are not cut (Known accepted gap: such lines stay large). When the first line is only `sep=X`, that delimiter is used and the line removed (the Sniffer is restricted to that delimiter) | A single line with 22.5 million commas (45 MB) built every cell object: 5.2 s, 1.55 GB (→ 0.4 s). Excel's `sep=;` marker line became the header and was not used for the delimiter verdict | `test_spreadsheet.py::TestCsvLimitsAndHints` |
+| When a whole CSV cell is `="…"` (the Excel idiom for keeping leading zeros), it is unwrapped to the text inside (`""` → `"`). Other formulas stay as they are | `="00123"` came out as is | `test_spreadsheet.py::TestCsvExcelTextFormula` |
+| Only when every backslash in the sample (first 8,192 characters) is in front of a character that MySQL `INTO OUTFILE` escapes (`"`, `\`, `N`, `0`, `n`, `t`, `r`, `b`, `Z`, the delimiter, a line break), there is a place where `\"` is followed by a character that is not the end of a cell, and the sample read with `\"`→`""` and `\\`→`\` has the same cell count on every line with double quotes wrapping cells whole, is the file read that way (`_escapes_quotes_with_backslashes`). `\N` (NULL) and the like stay literal. Known accepted gap: when `\"` occurs only after the sample it is not recognized, and a file mixing both rules (`"Is it \"ok\"?"` and a path ending `"c:\"`) is read with the doubled-quote rule as before | Read only with the doubled-quote rule, `"27\" 모니터"` became `27\ 모니터"`. Reading an ordinary CSV with other backslashes (`C:\temp`, `\d+`) as escapes loses characters | `test_core_spreadsheet.py::TestCsvBackslashEscapes` |
+| Even with `.csv`/`.tsv`, when the very start after BOM and whitespace is an HTML document tag (`<!doctype html`, `<html`, `<table`, `<head`, `<body`, `<style`, `<meta`) and the first 64 KB contain a table tag (`<table`, `<tr`, `<td`) together with its closing tag, the file is read as HTML with an extension warning (`detect._is_html_table_export`). A real CSV whose first cell starts with a tag (`<b>굵게</b>,…`, `<table>,<tr>` without closing table tags) is CSV. `.txt` stays text (among text formats the extension decides) | When an 'Excel download' serves an HTML table as `.csv`, the tags became cell text and `1,000` was split into two cells. Of 2,035 real CSV/TSV files from systems, 0 changed to HTML (owner's approval, conservative) | `test_core_text_and_mail.py::TestHtmlTableServedAsCsv` |