xl2jsonl 0.1.0__tar.gz

xl2jsonl-0.1.0/.gitignore

@@ -0,0 +1,22 @@
+.venv
+.pytest_cache
+__pycache__
+*.pyc
+*.pyo
+*.pyd
+*.db
+*.sqlite3
+*.log
+.DS_Store
+.idea/
+.vscode/
+*.egg-info/
+dist/
+build/
+*.egg
+*.tar.gz
+*.whl
+.env
+.env.*
+*.env
+.claude

xl2jsonl-0.1.0/CLAUDE.md

@@ -0,0 +1,61 @@
+# xl2jsonl
+
+Convert Excel (.xlsx, .xls, .xlsb) and CSV (.csv, .tsv) files to JSONL.
+
+## Quick Reference
+
+- **Language**: Python 3.10+
+- **Package manager**: uv
+- **Entry point**: `src/xl2jsonl/cli.py` (Click CLI), `src/xl2jsonl/__init__.py` (Python API)
+- **Test runner**: `uv run python -m pytest` (pytest not directly on PATH, use `python -m`)
+- **Lint**: `uv run ruff check src/ tests/`
+
+## Project Structure
+
+```
+src/xl2jsonl/
+  __init__.py    - Public API: convert(), iter_records()
+  cli.py         - Click CLI entry point
+  loader.py      - Multi-format loader: Excel (calamine + openpyxl), CSV/TSV (stdlib csv)
+  chunker.py     - Header detection, row-to-dict conversion, cell normalization
+  writer.py      - JSONL streaming output via orjson
+  models.py      - Data classes: SheetData, RowRecord, Metadata, CellValue
+  exceptions.py  - Xl2JsonlError, LoaderError, EmptySheetError, NoHeaderError
+tests/
+  conftest.py    - Programmatic xlsx fixture generators (openpyxl)
+  test_*.py      - Unit and integration tests
+```
+
+## Architecture
+
+1. **Loader** (`loader.py`): Routes by file extension. Excel (.xlsx/.xls/.xlsb) read via calamine; .xlsx gets openpyxl merge detection in `auto` mode. CSV/TSV read via stdlib `csv` module with basic type inference (int, float, bool).
+2. **Chunker** (`chunker.py`): Detects table regions, auto-detects headers, converts rows to key-value dicts.
+   - **Table region detection**: For sheets with multiple column groups (side-by-side tables separated by empty columns), splits into independent regions. Each column group is further split into row blocks by empty row gaps. Single-column-group sheets use the whole sheet as one region (backward compatible).
+   - **Header detection**: First row where 50%+ non-empty cells are strings AND values are sufficiently distinct (skips merged title rows).
+3. **Writer** (`writer.py`): Streams `{content, metadata}` JSONL records via orjson.
+
+## Key Design Decisions
+
+- **Multi-table sheets**: Column groups are detected using count-based occupancy (a column needs data in 10%+ of rows to count as occupied, filtering out noise from titles/notes that span across table boundaries). Groups are split by 1+ empty columns. Row blocks within each group are split by 1+ empty rows.
+- **Merged title detection**: Header auto-detection skips rows where most non-empty cells share the same value (resolved merged title/description rows).
+- **Multi-row headers**: When a header candidate has duplicate values (merged category row), scans subsequent rows for one with better column coverage.
+- **Format support**: Merge resolution only works for .xlsx (openpyxl limitation). For .xls/.xlsb, calamine-only (no merge detection). CSV/TSV loaded via stdlib csv with type inference.
+- `--header-row` (0-based) CLI flag overrides auto-detection when heuristics fail.
+- Duplicate headers from merged columns get `_2`, `_3` suffixes.
+- Empty leading rows are skipped during header detection.
+- Regions with no detected header are silently skipped (logged at DEBUG level).
+
+## Running
+
+```bash
+uv run xl2jsonl input.xlsx -o output.jsonl    # CLI
+uv run python -m pytest -v                     # Tests (all 39)
+uv run python -m pytest tests/test_chunker.py  # Single file
+uv run ruff check src/ tests/                  # Lint
+```
+
+## Testing Conventions
+
+- Fixtures in `conftest.py` generate xlsx files programmatically using openpyxl
+- No test data files checked in — everything is generated at test time
+- Tests cover: loader engines, merge resolution, header detection, cell normalization, CLI flags, end-to-end xlsx→jsonl

xl2jsonl-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Abhijit Choudhury
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

xl2jsonl-0.1.0/PKG-INFO

@@ -0,0 +1,290 @@
+Metadata-Version: 2.4
+Name: xl2jsonl
+Version: 0.1.0
+Summary: Convert Excel files to JSONL with robust merged-cell handling
+Project-URL: Homepage, https://github.com/abhijit-choudhury/xl2jsonl
+Project-URL: Repository, https://github.com/abhijit-choudhury/xl2jsonl
+Project-URL: Issues, https://github.com/abhijit-choudhury/xl2jsonl/issues
+Author: Abhijit Choudhury
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: File Formats
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Requires-Dist: click>=8.0
+Requires-Dist: openpyxl>=3.1.0
+Requires-Dist: orjson>=3.9
+Requires-Dist: python-calamine>=0.2.0
+Description-Content-Type: text/markdown
+
+# xl2jsonl
+
+Convert Excel (`.xlsx`, `.xls`, `.xlsb`) and CSV (`.csv`, `.tsv`) files to JSONL with robust handling of merged cells, multiple sheets, and mixed data types. Each output line is a JSON object containing row data as key-value pairs plus metadata for traceability.
+
+## Why xl2jsonl?
+
+Excel spreadsheets in the wild are messy — merged headers spanning multiple columns, merged row labels, empty leading rows, inconsistent types. Most conversion tools break on these. xl2jsonl handles them correctly by using a dual-engine approach: **calamine** (Rust-backed) for speed on clean sheets, with automatic fallback to **openpyxl** when merged cells are detected.
+
+## Output Format
+
+Each line in the output JSONL file is a self-contained JSON object:
+
+```json
+{"content": {"Name": "Alice", "Age": 30, "Department": "Engineering"}, "metadata": {"filename": "staff.xlsx", "sheet_name": "Employees", "sheet_number": 1, "row_number": 2}}
+{"content": {"Name": "Bob", "Age": 25, "Department": "Design"}, "metadata": {"filename": "staff.xlsx", "sheet_name": "Employees", "sheet_number": 1, "row_number": 3}}
+```
+
+| Field | Description |
+|---|---|
+| `content` | Key-value pairs mapping column headers to cell values for that row |
+| `metadata.filename` | Original Excel filename |
+| `metadata.sheet_name` | Name of the worksheet |
+| `metadata.sheet_number` | 1-based sheet index |
+| `metadata.row_number` | 1-based row number in the original Excel file |
+
+## Features
+
+- **Merged cell resolution** — Merged columns, rows, and blocks are filled so every cell in a merged range gets the top-left cell's value
+- **Duplicate header handling** — Merged headers that produce duplicates get `_2`, `_3` suffixes (e.g., `Revenue`, `Revenue_2`, `Revenue_3`)
+- **Auto header detection** — Skips empty leading rows, merged title rows, and finds the first row where the majority of non-empty cells are strings with sufficient diversity
+- **Multi-table sheet support** — Automatically detects side-by-side tables (separated by empty columns) and stacked tables (separated by empty rows) within a single sheet, processing each independently
+- **Multi-sheet support** — Processes all sheets by default, or filter by name/index
+- **Mixed type handling** — Dates and datetimes convert to ISO 8601 strings, strings are stripped, numbers and booleans pass through, `None` stays `None`
+- **Empty row skipping** — Empty rows are skipped by default (configurable)
+- **Short row padding** — Rows shorter than the header are padded with `None`
+- **Streaming output** — JSONL is written one record at a time via `orjson` for low memory overhead
+- **Dual engine** — calamine for speed on clean sheets, openpyxl fallback for merged cells (configurable)
+
+## Dependencies
+
+| Package | Purpose |
+|---|---|
+| [python-calamine](https://pypi.org/project/python-calamine/) | Fast Rust-backed Excel reader — supports .xlsx, .xls, .xlsb |
+| [openpyxl](https://pypi.org/project/openpyxl/) | Merged cell detection and resolution (.xlsx only) |
+| [click](https://pypi.org/project/click/) | CLI framework |
+| [orjson](https://pypi.org/project/orjson/) | Fast JSON serialization with native date handling |
+
+Dev dependencies: `pytest`, `pytest-cov`, `ruff`
+
+## Setup
+
+Requires **Python 3.10+** and **[uv](https://docs.astral.sh/uv/)**.
+
+```bash
+# Clone and install
+git clone <repo-url>
+cd excel-chunker
+uv sync
+```
+
+This creates a virtual environment in `.venv/` and installs all dependencies including dev tools.
+
+## Usage
+
+### CLI
+
+```bash
+# Convert all sheets (output defaults to <input>.jsonl)
+uv run xl2jsonl data.xlsx
+
+# CSV and TSV files
+uv run xl2jsonl report.csv -o output.jsonl
+uv run xl2jsonl data.tsv -o output.jsonl
+
+# Legacy Excel format
+uv run xl2jsonl legacy.xls -o output.jsonl
+
+# Specify output file
+uv run xl2jsonl data.xlsx -o output.jsonl
+
+# Select specific sheets (by name or 0-based index, repeatable)
+uv run xl2jsonl data.xlsx -s "Revenue" -s "Costs"
+uv run xl2jsonl data.xlsx -s 0 -s 2
+
+# Force a specific engine
+uv run xl2jsonl data.xlsx --engine openpyxl
+uv run xl2jsonl data.xlsx --engine calamine
+
+# Set header row explicitly (0-based index)
+uv run xl2jsonl data.xlsx --header-row 2
+
+# Keep empty rows in output
+uv run xl2jsonl data.xlsx --keep-empty
+
+# Verbose logging (-v for INFO, -vv for DEBUG)
+uv run xl2jsonl data.xlsx -vv
+```
+
+Full CLI help:
+
+```
+Usage: xl2jsonl [OPTIONS] INPUT_FILE
+
+  Convert an Excel (.xlsx, .xls, .xlsb) or CSV (.csv, .tsv) file to JSONL.
+
+Options:
+  -o, --output PATH              Output JSONL file. Defaults to <input>.jsonl
+  -s, --sheet TEXT               Sheet name or 0-based index. Repeatable.
+  --engine [auto|calamine|openpyxl]
+                                 Reading engine (default: auto)
+  --header-row INTEGER           0-based header row index. Default: auto-detect.
+  --skip-empty / --keep-empty    Skip empty rows (default: skip)
+  -v, --verbose                  -v for INFO, -vv for DEBUG
+  --help                         Show this message and exit.
+```
+
+### Python API
+
+```python
+import xl2jsonl
+
+# Convert to JSONL file — returns row count
+count = xl2jsonl.convert("data.xlsx", "output.jsonl")
+print(f"Wrote {count} records")
+
+# Convert and get results in memory — returns list of dicts
+records = xl2jsonl.convert("data.xlsx")
+for r in records:
+    print(r["content"], r["metadata"])
+
+# Lazy iterator for large files
+for record in xl2jsonl.iter_records("data.xlsx", sheets=["Sheet1"]):
+    print(record.data)        # dict of header -> value
+    print(record.metadata)    # Metadata(filename, sheet_name, sheet_number, row_number)
+
+# All options
+count = xl2jsonl.convert(
+    "data.xlsx",
+    "output.jsonl",
+    sheets=["Revenue", 2],       # filter by name or index
+    engine="auto",               # "auto" | "calamine" | "openpyxl"
+    header_row=None,             # None for auto-detect, or 0-based int
+    skip_empty_rows=True,        # skip rows where all cells are empty
+)
+```
+
+## How Merged Cells Are Handled
+
+### Merged column headers
+
+If a header cell spans columns B through D with value "Revenue":
+
+| | A | B | C | D |
+|---|---|---|---|---|
+| **Header** | ID | Revenue (merged B:D) | | Profit |
+| **Row 1** | 1 | 100 | 200 | 50 |
+
+After resolution, the headers become `["ID", "Revenue", "Revenue_2", "Revenue_3", "Profit"]`, and each data cell maps to its respective deduplicated header.
+
+### Merged row labels
+
+If a cell in column A spans rows 4-5 with value "Total":
+
+| | A | B |
+|---|---|---|
+| **Row 4** | Total (merged A4:A5) | 250 |
+| **Row 5** | | 300 |
+
+After resolution, both rows get `"Total"` as their value for that column.
+
+### Merged blocks
+
+A 2x3 merged region is filled entirely with the top-left cell's value.
+
+### Multi-table sheets
+
+Sheets with multiple tables laid out side-by-side or stacked vertically are automatically detected and split into independent regions:
+
+| | A | B | C | | E | F | G |
+|---|---|---|---|---|---|---|---|
+| **Row 1** | Account: | | ACME Corp | | *Note: ...* | | |
+| | | | | | | | |
+| **Row 3** | **Name** | **Score** | **Grade** | | **Region** | **Revenue** | **Growth** |
+| **Row 4** | Alice | 95 | A | | EMEA | 1.2M | 5% |
+| **Row 5** | Bob | 88 | B | | APAC | 800K | 12% |
+
+This produces two separate tables: one with headers `Name, Score, Grade` and another with `Region, Revenue, Growth`. The metadata row and note are detected as separate regions and skipped (no valid header found).
+
+**How it works:**
+1. Columns are grouped by occupancy — a column needs data in a meaningful number of rows (10%+ of the busiest column) to count, filtering out noise from titles/notes that span across table boundaries
+2. Groups are split where 1+ empty columns separate them
+3. Within each column group, row blocks are split by empty row gaps
+4. Each block gets independent header detection and processing
+5. Blocks with no detected header are silently skipped
+
+### Engine selection (`auto` mode)
+
+1. All sheets are read with calamine (fast)
+2. Each sheet is probed for merged cell ranges via openpyxl
+3. Sheets with merges are re-read through openpyxl with full merge resolution
+4. Sheets without merges use the calamine data as-is
+
+This gives you the speed of calamine on clean sheets while correctly handling merges where they exist.
+
+## Project Structure
+
+```
+excel-chunker/
+├── pyproject.toml              # Package config, dependencies, CLI entry point
+├── src/xl2jsonl/
+│   ├── __init__.py             # Public API: convert(), iter_records()
+│   ├── models.py               # Data classes: SheetData, RowRecord, Metadata
+│   ├── exceptions.py           # Xl2JsonlError, LoaderError, EmptySheetError, NoHeaderError
+│   ├── loader.py               # Dual-engine Excel reader + merge resolution
+│   ├── chunker.py              # Header detection, row-to-dict, cell normalization
+│   ├── writer.py               # JSONL streaming output via orjson
+│   └── cli.py                  # Click CLI entry point
+└── tests/
+    ├── conftest.py             # Programmatic xlsx fixture generators
+    ├── test_loader.py          # Loader unit tests (engines, merges, sheet selection)
+    ├── test_chunker.py         # Chunker unit tests (headers, normalization, edge cases)
+    ├── test_writer.py          # Writer unit tests (JSONL format, streaming)
+    ├── test_cli.py             # CLI tests via Click CliRunner
+    └── test_integration.py     # End-to-end tests (xlsx -> jsonl -> verify)
+```
+
+## Running Tests
+
+```bash
+# Run all tests
+uv run pytest
+
+# Verbose output
+uv run pytest -v
+
+# With coverage
+uv run pytest --cov=xl2jsonl
+
+# Run a specific test file
+uv run pytest tests/test_loader.py
+```
+
+## Supported Formats
+
+| Format | Extension | Merge Resolution | Notes |
+|---|---|---|---|
+| Excel (modern) | `.xlsx` | Full (openpyxl) | Default engine: calamine with openpyxl fallback for merges |
+| Excel (legacy) | `.xls` | No | Read via calamine only |
+| Excel Binary | `.xlsb` | No | Read via calamine only |
+| CSV | `.csv` | N/A | Basic type inference (int, float, bool) |
+| TSV | `.tsv` | N/A | Tab-delimited, same type inference |
+
+## Limitations
+
+- Merged cell resolution is only available for `.xlsx` files (openpyxl limitation) — `.xls` and `.xlsb` are read via calamine without merge detection
+- The entire sheet grid is materialized in memory during loading (Excel files are not row-streamable); the chunker and writer stages stream lazily
+- Header auto-detection uses heuristics (string majority, value diversity, column coverage) — use `--header-row` to override if it guesses wrong
+- Formulas are evaluated to their cached values (`data_only=True` in openpyxl) — if a file was never opened in Excel after formula changes, values may be stale
+- CSV type inference is best-effort: numeric strings become int/float, "true"/"false" become booleans, everything else stays as strings
+
+## License
+
+MIT

xl2jsonl-0.1.0/README.md

@@ -0,0 +1,263 @@
+# xl2jsonl
+
+Convert Excel (`.xlsx`, `.xls`, `.xlsb`) and CSV (`.csv`, `.tsv`) files to JSONL with robust handling of merged cells, multiple sheets, and mixed data types. Each output line is a JSON object containing row data as key-value pairs plus metadata for traceability.
+
+## Why xl2jsonl?
+
+Excel spreadsheets in the wild are messy — merged headers spanning multiple columns, merged row labels, empty leading rows, inconsistent types. Most conversion tools break on these. xl2jsonl handles them correctly by using a dual-engine approach: **calamine** (Rust-backed) for speed on clean sheets, with automatic fallback to **openpyxl** when merged cells are detected.
+
+## Output Format
+
+Each line in the output JSONL file is a self-contained JSON object:
+
+```json
+{"content": {"Name": "Alice", "Age": 30, "Department": "Engineering"}, "metadata": {"filename": "staff.xlsx", "sheet_name": "Employees", "sheet_number": 1, "row_number": 2}}
+{"content": {"Name": "Bob", "Age": 25, "Department": "Design"}, "metadata": {"filename": "staff.xlsx", "sheet_name": "Employees", "sheet_number": 1, "row_number": 3}}
+```
+
+| Field | Description |
+|---|---|
+| `content` | Key-value pairs mapping column headers to cell values for that row |
+| `metadata.filename` | Original Excel filename |
+| `metadata.sheet_name` | Name of the worksheet |
+| `metadata.sheet_number` | 1-based sheet index |
+| `metadata.row_number` | 1-based row number in the original Excel file |
+
+## Features
+
+- **Merged cell resolution** — Merged columns, rows, and blocks are filled so every cell in a merged range gets the top-left cell's value
+- **Duplicate header handling** — Merged headers that produce duplicates get `_2`, `_3` suffixes (e.g., `Revenue`, `Revenue_2`, `Revenue_3`)
+- **Auto header detection** — Skips empty leading rows, merged title rows, and finds the first row where the majority of non-empty cells are strings with sufficient diversity
+- **Multi-table sheet support** — Automatically detects side-by-side tables (separated by empty columns) and stacked tables (separated by empty rows) within a single sheet, processing each independently
+- **Multi-sheet support** — Processes all sheets by default, or filter by name/index
+- **Mixed type handling** — Dates and datetimes convert to ISO 8601 strings, strings are stripped, numbers and booleans pass through, `None` stays `None`
+- **Empty row skipping** — Empty rows are skipped by default (configurable)
+- **Short row padding** — Rows shorter than the header are padded with `None`
+- **Streaming output** — JSONL is written one record at a time via `orjson` for low memory overhead
+- **Dual engine** — calamine for speed on clean sheets, openpyxl fallback for merged cells (configurable)
+
+## Dependencies
+
+| Package | Purpose |
+|---|---|
+| [python-calamine](https://pypi.org/project/python-calamine/) | Fast Rust-backed Excel reader — supports .xlsx, .xls, .xlsb |
+| [openpyxl](https://pypi.org/project/openpyxl/) | Merged cell detection and resolution (.xlsx only) |
+| [click](https://pypi.org/project/click/) | CLI framework |
+| [orjson](https://pypi.org/project/orjson/) | Fast JSON serialization with native date handling |
+
+Dev dependencies: `pytest`, `pytest-cov`, `ruff`
+
+## Setup
+
+Requires **Python 3.10+** and **[uv](https://docs.astral.sh/uv/)**.
+
+```bash
+# Clone and install
+git clone <repo-url>
+cd excel-chunker
+uv sync
+```
+
+This creates a virtual environment in `.venv/` and installs all dependencies including dev tools.
+
+## Usage
+
+### CLI
+
+```bash
+# Convert all sheets (output defaults to <input>.jsonl)
+uv run xl2jsonl data.xlsx
+
+# CSV and TSV files
+uv run xl2jsonl report.csv -o output.jsonl
+uv run xl2jsonl data.tsv -o output.jsonl
+
+# Legacy Excel format
+uv run xl2jsonl legacy.xls -o output.jsonl
+
+# Specify output file
+uv run xl2jsonl data.xlsx -o output.jsonl
+
+# Select specific sheets (by name or 0-based index, repeatable)
+uv run xl2jsonl data.xlsx -s "Revenue" -s "Costs"
+uv run xl2jsonl data.xlsx -s 0 -s 2
+
+# Force a specific engine
+uv run xl2jsonl data.xlsx --engine openpyxl
+uv run xl2jsonl data.xlsx --engine calamine
+
+# Set header row explicitly (0-based index)
+uv run xl2jsonl data.xlsx --header-row 2
+
+# Keep empty rows in output
+uv run xl2jsonl data.xlsx --keep-empty
+
+# Verbose logging (-v for INFO, -vv for DEBUG)
+uv run xl2jsonl data.xlsx -vv
+```
+
+Full CLI help:
+
+```
+Usage: xl2jsonl [OPTIONS] INPUT_FILE
+
+  Convert an Excel (.xlsx, .xls, .xlsb) or CSV (.csv, .tsv) file to JSONL.
+
+Options:
+  -o, --output PATH              Output JSONL file. Defaults to <input>.jsonl
+  -s, --sheet TEXT               Sheet name or 0-based index. Repeatable.
+  --engine [auto|calamine|openpyxl]
+                                 Reading engine (default: auto)
+  --header-row INTEGER           0-based header row index. Default: auto-detect.
+  --skip-empty / --keep-empty    Skip empty rows (default: skip)
+  -v, --verbose                  -v for INFO, -vv for DEBUG
+  --help                         Show this message and exit.
+```
+
+### Python API
+
+```python
+import xl2jsonl
+
+# Convert to JSONL file — returns row count
+count = xl2jsonl.convert("data.xlsx", "output.jsonl")
+print(f"Wrote {count} records")
+
+# Convert and get results in memory — returns list of dicts
+records = xl2jsonl.convert("data.xlsx")
+for r in records:
+    print(r["content"], r["metadata"])
+
+# Lazy iterator for large files
+for record in xl2jsonl.iter_records("data.xlsx", sheets=["Sheet1"]):
+    print(record.data)        # dict of header -> value
+    print(record.metadata)    # Metadata(filename, sheet_name, sheet_number, row_number)
+
+# All options
+count = xl2jsonl.convert(
+    "data.xlsx",
+    "output.jsonl",
+    sheets=["Revenue", 2],       # filter by name or index
+    engine="auto",               # "auto" | "calamine" | "openpyxl"
+    header_row=None,             # None for auto-detect, or 0-based int
+    skip_empty_rows=True,        # skip rows where all cells are empty
+)
+```
+
+## How Merged Cells Are Handled
+
+### Merged column headers
+
+If a header cell spans columns B through D with value "Revenue":
+
+| | A | B | C | D |
+|---|---|---|---|---|
+| **Header** | ID | Revenue (merged B:D) | | Profit |
+| **Row 1** | 1 | 100 | 200 | 50 |
+
+After resolution, the headers become `["ID", "Revenue", "Revenue_2", "Revenue_3", "Profit"]`, and each data cell maps to its respective deduplicated header.
+
+### Merged row labels
+
+If a cell in column A spans rows 4-5 with value "Total":
+
+| | A | B |
+|---|---|---|
+| **Row 4** | Total (merged A4:A5) | 250 |
+| **Row 5** | | 300 |
+
+After resolution, both rows get `"Total"` as their value for that column.
+
+### Merged blocks
+
+A 2x3 merged region is filled entirely with the top-left cell's value.
+
+### Multi-table sheets
+
+Sheets with multiple tables laid out side-by-side or stacked vertically are automatically detected and split into independent regions:
+
+| | A | B | C | | E | F | G |
+|---|---|---|---|---|---|---|---|
+| **Row 1** | Account: | | ACME Corp | | *Note: ...* | | |
+| | | | | | | | |
+| **Row 3** | **Name** | **Score** | **Grade** | | **Region** | **Revenue** | **Growth** |
+| **Row 4** | Alice | 95 | A | | EMEA | 1.2M | 5% |
+| **Row 5** | Bob | 88 | B | | APAC | 800K | 12% |
+
+This produces two separate tables: one with headers `Name, Score, Grade` and another with `Region, Revenue, Growth`. The metadata row and note are detected as separate regions and skipped (no valid header found).
+
+**How it works:**
+1. Columns are grouped by occupancy — a column needs data in a meaningful number of rows (10%+ of the busiest column) to count, filtering out noise from titles/notes that span across table boundaries
+2. Groups are split where 1+ empty columns separate them
+3. Within each column group, row blocks are split by empty row gaps
+4. Each block gets independent header detection and processing
+5. Blocks with no detected header are silently skipped
+
+### Engine selection (`auto` mode)
+
+1. All sheets are read with calamine (fast)
+2. Each sheet is probed for merged cell ranges via openpyxl
+3. Sheets with merges are re-read through openpyxl with full merge resolution
+4. Sheets without merges use the calamine data as-is
+
+This gives you the speed of calamine on clean sheets while correctly handling merges where they exist.
+
+## Project Structure
+
+```
+excel-chunker/
+├── pyproject.toml              # Package config, dependencies, CLI entry point
+├── src/xl2jsonl/
+│   ├── __init__.py             # Public API: convert(), iter_records()
+│   ├── models.py               # Data classes: SheetData, RowRecord, Metadata
+│   ├── exceptions.py           # Xl2JsonlError, LoaderError, EmptySheetError, NoHeaderError
+│   ├── loader.py               # Dual-engine Excel reader + merge resolution
+│   ├── chunker.py              # Header detection, row-to-dict, cell normalization
+│   ├── writer.py               # JSONL streaming output via orjson
+│   └── cli.py                  # Click CLI entry point
+└── tests/
+    ├── conftest.py             # Programmatic xlsx fixture generators
+    ├── test_loader.py          # Loader unit tests (engines, merges, sheet selection)
+    ├── test_chunker.py         # Chunker unit tests (headers, normalization, edge cases)
+    ├── test_writer.py          # Writer unit tests (JSONL format, streaming)
+    ├── test_cli.py             # CLI tests via Click CliRunner
+    └── test_integration.py     # End-to-end tests (xlsx -> jsonl -> verify)
+```
+
+## Running Tests
+
+```bash
+# Run all tests
+uv run pytest
+
+# Verbose output
+uv run pytest -v
+
+# With coverage
+uv run pytest --cov=xl2jsonl
+
+# Run a specific test file
+uv run pytest tests/test_loader.py
+```
+
+## Supported Formats
+
+| Format | Extension | Merge Resolution | Notes |
+|---|---|---|---|
+| Excel (modern) | `.xlsx` | Full (openpyxl) | Default engine: calamine with openpyxl fallback for merges |
+| Excel (legacy) | `.xls` | No | Read via calamine only |
+| Excel Binary | `.xlsb` | No | Read via calamine only |
+| CSV | `.csv` | N/A | Basic type inference (int, float, bool) |
+| TSV | `.tsv` | N/A | Tab-delimited, same type inference |
+
+## Limitations
+
+- Merged cell resolution is only available for `.xlsx` files (openpyxl limitation) — `.xls` and `.xlsb` are read via calamine without merge detection
+- The entire sheet grid is materialized in memory during loading (Excel files are not row-streamable); the chunker and writer stages stream lazily
+- Header auto-detection uses heuristics (string majority, value diversity, column coverage) — use `--header-row` to override if it guesses wrong
+- Formulas are evaluated to their cached values (`data_only=True` in openpyxl) — if a file was never opened in Excel after formula changes, values may be stale
+- CSV type inference is best-effort: numeric strings become int/float, "true"/"false" become booleans, everything else stays as strings
+
+## License
+
+MIT

xl2jsonl-0.1.0/output.jsonl

@@ -0,0 +1 @@
+{"content":{"Version":14.0,"Risk Module":"01. Schedule & Delay","Category":"Schedule & Delay","Risk Question":"(S&D01) Bid Subject To Prior Sale?","Requirements Response":"","Requirements Excerpt":"","Requirement Response Analysis":"","Requirement Excerpt Analysis":"","Author":"","SME Correct Respones":"","SME Correct Context":"","Result":"","Failure Reason/ Comments":""},"metadata":{"filename":"data.xlsx","sheet_name":"Sample 1","sheet_number":1,"row_number":3}}