mtsv 0.1.0__tar.gz

mtsv-0.1.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 Demos Ra
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

mtsv-0.1.0/MANIFEST.in

@@ -0,0 +1 @@
+prune tests

mtsv-0.1.0/PKG-INFO

@@ -0,0 +1,122 @@
+Metadata-Version: 2.4
+Name: mtsv
+Version: 0.1.0
+Summary: Python implementation of Multi-Sheet Tab-Separated Values (MTSV)
+Author-email: Demos Ra <demos_ra@hotmail.com>
+License-Expression: MIT
+Project-URL: source, https://github.com/demos-ra/mtsv
+Project-URL: issues, https://github.com/demos-ra/mtsv/issues
+Keywords: tsv,tabular data,media type
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: arrow
+Requires-Dist: pyarrow>=16; extra == "arrow"
+Dynamic: license-file
+
+# MTSV for Python
+
+A parser and a generator for Multi-Sheet Tab-Separated Values (MTSV), with
+integrations for spreadsheets (ODS) and data tools (Apache Arrow). The
+version is the `version` field of `pyproject.toml`.
+
+* [Specification](https://github.com/demos-ra/mtsv-spec)
+* [Repository](https://github.com/demos-ra/mtsv)
+
+## Install
+
+From the repository. A Python that an operating system manages does not
+accept packages directly, so install into a virtual environment. Run these
+from the root of the repository, one folder above this one:
+
+```
+python3 -m venv .venv
+.venv/bin/pip install ./python
+```
+
+For the Arrow integration, which also installs pyarrow:
+
+```
+.venv/bin/pip install "./python[arrow]"
+```
+
+## Read and write MTSV
+
+Sheets are a list of dictionaries with `"sheet name"`, `"header"`, and
+`"records"`, the same shape as the
+[conformance results](https://github.com/demos-ra/mtsv/blob/main/conformance/README.md).
+
+```python
+import mtsv
+
+with open("book.mtsv", "rb") as file:
+    sheets = mtsv.load(file)
+
+with open("book.mtsv", "wb") as file:
+    mtsv.dump(sheets, file)
+```
+
+`loads` and `dumps` work on strings.
+
+## Spreadsheets (ODS)
+
+```python
+from mtsv.integrations import ods
+
+with open("book.ods", "wb") as file:
+    ods.dump(sheets, file)
+
+with open("book.ods", "rb") as file:
+    sheets = ods.load(file)
+```
+
+From the command line:
+
+```
+python -m mtsv.integrations.ods book.mtsv book.ods
+python -m mtsv.integrations.ods book.ods book.mtsv
+```
+
+## Data tools (Apache Arrow)
+
+```python
+from mtsv.integrations import arrow
+
+tables = arrow.to_arrow(sheets)
+sheets = arrow.from_arrow(tables)
+```
+
+`tables` is a list of `(sheet name, pyarrow.Table)` pairs.
+
+## What is left behind
+
+MTSV holds sheets, names, rows, and text. Going out to ODS or Arrow keeps all
+of it, with two ODS exceptions: characters that XML 1.0 does not allow raise
+`ValueError`, and empty rows and columns at the edge of a sheet do not come
+back from ODS, so a sheet of only empty fields comes back as an empty sheet.
+Coming back in, anything else (formatting, formulas, types, missing values)
+raises `ValueError` by default. To confirm and leave it behind, pass
+`errors="ignore"`, or `--errors ignore` on the command line. Text that MTSV
+cannot hold, such as a tab or line break inside a value, always raises
+`ValueError`. So does an Arrow column whose values cannot be text at all, such
+as binary, a list, or a struct.
+
+## Layout
+
+| Path                     | Contents                                    |
+|--------------------------|---------------------------------------------|
+| `src/mtsv/`              | the interface, the parser, and the generator |
+| `src/mtsv/integrations/` | one module per target standard              |
+| `tests/`                 | the test suite, run against the install     |
+
+## Test
+
+From the root of the repository:
+
+```
+.venv/bin/python -m unittest discover -s python/tests
+```
+
+## License
+
+[MIT](https://github.com/demos-ra/mtsv/blob/main/LICENSE)

mtsv-0.1.0/README.md

@@ -0,0 +1,106 @@
+# MTSV for Python
+
+A parser and a generator for Multi-Sheet Tab-Separated Values (MTSV), with
+integrations for spreadsheets (ODS) and data tools (Apache Arrow). The
+version is the `version` field of `pyproject.toml`.
+
+* [Specification](https://github.com/demos-ra/mtsv-spec)
+* [Repository](https://github.com/demos-ra/mtsv)
+
+## Install
+
+From the repository. A Python that an operating system manages does not
+accept packages directly, so install into a virtual environment. Run these
+from the root of the repository, one folder above this one:
+
+```
+python3 -m venv .venv
+.venv/bin/pip install ./python
+```
+
+For the Arrow integration, which also installs pyarrow:
+
+```
+.venv/bin/pip install "./python[arrow]"
+```
+
+## Read and write MTSV
+
+Sheets are a list of dictionaries with `"sheet name"`, `"header"`, and
+`"records"`, the same shape as the
+[conformance results](https://github.com/demos-ra/mtsv/blob/main/conformance/README.md).
+
+```python
+import mtsv
+
+with open("book.mtsv", "rb") as file:
+    sheets = mtsv.load(file)
+
+with open("book.mtsv", "wb") as file:
+    mtsv.dump(sheets, file)
+```
+
+`loads` and `dumps` work on strings.
+
+## Spreadsheets (ODS)
+
+```python
+from mtsv.integrations import ods
+
+with open("book.ods", "wb") as file:
+    ods.dump(sheets, file)
+
+with open("book.ods", "rb") as file:
+    sheets = ods.load(file)
+```
+
+From the command line:
+
+```
+python -m mtsv.integrations.ods book.mtsv book.ods
+python -m mtsv.integrations.ods book.ods book.mtsv
+```
+
+## Data tools (Apache Arrow)
+
+```python
+from mtsv.integrations import arrow
+
+tables = arrow.to_arrow(sheets)
+sheets = arrow.from_arrow(tables)
+```
+
+`tables` is a list of `(sheet name, pyarrow.Table)` pairs.
+
+## What is left behind
+
+MTSV holds sheets, names, rows, and text. Going out to ODS or Arrow keeps all
+of it, with two ODS exceptions: characters that XML 1.0 does not allow raise
+`ValueError`, and empty rows and columns at the edge of a sheet do not come
+back from ODS, so a sheet of only empty fields comes back as an empty sheet.
+Coming back in, anything else (formatting, formulas, types, missing values)
+raises `ValueError` by default. To confirm and leave it behind, pass
+`errors="ignore"`, or `--errors ignore` on the command line. Text that MTSV
+cannot hold, such as a tab or line break inside a value, always raises
+`ValueError`. So does an Arrow column whose values cannot be text at all, such
+as binary, a list, or a struct.
+
+## Layout
+
+| Path                     | Contents                                    |
+|--------------------------|---------------------------------------------|
+| `src/mtsv/`              | the interface, the parser, and the generator |
+| `src/mtsv/integrations/` | one module per target standard              |
+| `tests/`                 | the test suite, run against the install     |
+
+## Test
+
+From the root of the repository:
+
+```
+.venv/bin/python -m unittest discover -s python/tests
+```
+
+## License
+
+[MIT](https://github.com/demos-ra/mtsv/blob/main/LICENSE)

mtsv-0.1.0/pyproject.toml

@@ -0,0 +1,21 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "mtsv"
+version = "0.1.0"
+description = "Python implementation of Multi-Sheet Tab-Separated Values (MTSV)"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Demos Ra", email = "demos_ra@hotmail.com" }]
+keywords = ["tsv", "tabular data", "media type"]
+
+[project.urls]
+source = "https://github.com/demos-ra/mtsv"
+issues = "https://github.com/demos-ra/mtsv/issues"
+
+[project.optional-dependencies]
+arrow = ["pyarrow>=16"]

mtsv-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

mtsv-0.1.0/src/mtsv/__init__.py

@@ -0,0 +1,52 @@
+"""Multi-Sheet Tab-Separated Values (MTSV), draft-demosra-mtsv-00.
+
+Functions:
+dump -- write MTSV sheets to a binary file, encoded as UTF-8
+dumps -- return MTSV sheets as a string
+load -- read MTSV sheets from a binary file, decoded as UTF-8
+loads -- read MTSV sheets from a string
+
+Exceptions:
+MTSVDecodeError -- raised for text that is not an MTSV file
+
+Subpackages:
+integrations -- conversions between MTSV and other standards
+"""
+
+__all__ = ["dump", "dumps", "load", "loads", "MTSVDecodeError"]
+
+from typing import Any, BinaryIO
+
+from mtsv import _generator, _parser
+from mtsv._parser import MTSVDecodeError
+
+
+def dump(obj: list[dict[str, Any]], fp: BinaryIO) -> None:
+    """Write MTSV sheets to a binary file object, encoded as UTF-8."""
+    fp.write(dumps(obj).encode("utf-8"))
+
+
+def dumps(obj: list[dict[str, Any]]) -> str:
+    """Return MTSV sheets as an MTSV string."""
+    return _generator.mtsv_file(obj)
+
+
+def load(fp: BinaryIO, /) -> list[dict[str, Any]]:
+    """Read MTSV sheets from a binary file object, decoded as UTF-8."""
+    b = fp.read()
+    try:
+        s = b.decode("utf-8")
+    except AttributeError:
+        raise TypeError(
+            "File must be opened in binary mode,"
+            " e.g. use `open('foo.mtsv', 'rb')`"
+        ) from None
+    return loads(s)
+
+
+def loads(s: str, /) -> list[dict[str, Any]]:
+    """Read MTSV sheets from an MTSV string."""
+    if not isinstance(s, str):
+        raise TypeError(f"Expected str object, not '{type(s).__qualname__}'")
+    _, sheets = _parser.mtsv_file(s, 0)
+    return sheets

mtsv-0.1.0/src/mtsv/_generator.py

@@ -0,0 +1,106 @@
+"""Generate MTSV text, following draft-demosra-mtsv-00, Section 6."""
+
+from typing import Any
+
+HTAB = chr(0x09)
+LF = chr(0x0A)
+FF = chr(0x0C)
+SIGNATURE = chr(0xFEFF)
+
+
+def mtsv_file(sheets: list[dict[str, Any]]) -> str:
+    """Generate: mtsv-file = unnamed-sheet *named-sheet."""
+    parts: list[str] = []
+    for index, sheet in enumerate(sheets):
+        if sheet["sheet name"] is None:
+            parts.append(unnamed_sheet(sheet, index))
+        else:
+            parts.append(named_sheet(sheet))
+    return "".join(parts)
+
+
+def unnamed_sheet(sheet: dict[str, Any], index: int) -> str:
+    """Generate: unnamed-sheet = sheet-body."""
+    if index != 0 or sheet["header"] is None:
+        raise ValueError(
+            "an MTSV file has an unnamed sheet only if the file contains"
+            " at least one line before the first FF"
+        )
+    if sheet["header"] and sheet["header"][0].startswith(SIGNATURE):
+        raise ValueError(
+            "the first field of the unnamed sheet cannot begin with U+FEFF,"
+            " because a parser treats that character as an encoding"
+            " signature"
+        )
+    return sheet_body(sheet)
+
+
+def named_sheet(sheet: dict[str, Any]) -> str:
+    """Generate: named-sheet = FF sheet-name eol sheet-body."""
+    return FF + sheet_name(sheet["sheet name"]) + eol() + sheet_body(sheet)
+
+
+def sheet_body(sheet: dict[str, Any]) -> str:
+    """Generate: sheet-body = [header *record]."""
+    header_fields = sheet["header"]
+    if header_fields is None:
+        if sheet["records"]:
+            raise ValueError(
+                "a sheet with no lines is an empty sheet;"
+                " it has neither a header nor records"
+            )
+        return ""
+    lines = [header(header_fields)]
+    for record_fields in sheet["records"]:
+        if len(record_fields) != len(header_fields):
+            raise ValueError(
+                "each record in a sheet must have the same number of fields"
+                " as the header of that sheet"
+            )
+        lines.append(record(record_fields))
+    return "".join(lines)
+
+
+def header(fields: list[str]) -> str:
+    """Generate: header = record."""
+    return record(fields)
+
+
+def record(fields: list[str]) -> str:
+    """Generate: record = field *(HTAB field) eol."""
+    if not fields:
+        raise ValueError(
+            "a record must match: record = field *(HTAB field) eol"
+        )
+    return HTAB.join(field(value) for value in fields) + eol()
+
+
+def field(value: str) -> str:
+    """Generate: field = *field-char."""
+    if not all(field_char(char) for char in value):
+        raise ValueError(
+            "a field or sheet name that contains HT, LF, FF, or CR"
+            " cannot be represented in MTSV"
+        )
+    return value
+
+
+def sheet_name(value: str) -> str:
+    """Generate: sheet-name = *field-char."""
+    if not all(field_char(char) for char in value):
+        raise ValueError(
+            "a field or sheet name that contains HT, LF, FF, or CR"
+            " cannot be represented in MTSV"
+        )
+    return value
+
+
+def field_char(char: str) -> bool:
+    """Match: field-char = %x00-08 / %x0B / %x0E-10FFFF."""
+    code = ord(char)
+    return 0x00 <= code <= 0x08 or code == 0x0B or 0x0E <= code <= 0x10FFFF
+
+
+def eol() -> str:
+    """Generate: eol = LF / CRLF, writing LF."""
+    return LF

mtsv-0.1.0/src/mtsv/_parser.py

@@ -0,0 +1,147 @@
+"""Parse MTSV text, following draft-demosra-mtsv-00, Section 5."""
+
+from typing import Any
+
+HTAB = chr(0x09)
+LF = chr(0x0A)
+FF = chr(0x0C)
+CR = chr(0x0D)
+CRLF = CR + LF
+SIGNATURE = chr(0xFEFF)
+
+
+class MTSVDecodeError(ValueError):
+    """Subclass of ValueError with the following additional properties:
+
+    msg: The unformatted error message
+    doc: The MTSV document being parsed
+    pos: The start index of doc where parsing failed
+    lineno: The line corresponding to pos
+    colno: The column corresponding to pos
+    """
+
+    def __init__(self, msg: str, doc: str, pos: int) -> None:
+        """Store msg, doc and pos, and the line and column of pos."""
+        lineno = doc.count(LF, 0, pos) + 1
+        colno = pos - doc.rfind(LF, 0, pos)
+        super().__init__(f"{msg}: line {lineno} column {colno} (char {pos})")
+        self.msg = msg
+        self.doc = doc
+        self.pos = pos
+        self.lineno = lineno
+        self.colno = colno
+
+    def __reduce__(self) -> tuple[type, tuple[str, str, int]]:
+        """Return how to rebuild this error when it is unpickled."""
+        return self.__class__, (self.msg, self.doc, self.pos)
+
+
+def mtsv_file(src: str, pos: int) -> tuple[int, list[dict[str, Any]]]:
+    """Parse: mtsv-file = unnamed-sheet *named-sheet.
+
+    Skip a U+FEFF at the start of the file, which is an encoding
+    signature (Section 5).
+    """
+    if pos == 0 and src.startswith(SIGNATURE):
+        pos = 1
+    pos, sheet = unnamed_sheet(src, pos)
+    sheets: list[dict[str, Any]] = [] if sheet is None else [sheet]
+    while pos < len(src):
+        pos, sheet = named_sheet(src, pos)
+        sheets.append(sheet)
+    return pos, sheets
+
+
+def unnamed_sheet(src: str, pos: int) -> tuple[int, dict[str, Any] | None]:
+    """Parse: unnamed-sheet = sheet-body."""
+    pos, (header_fields, records) = sheet_body(src, pos)
+    if header_fields is None:
+        return pos, None
+    return pos, {
+        "sheet name": None,
+        "header": header_fields,
+        "records": records,
+    }
+
+
+def named_sheet(src: str, pos: int) -> tuple[int, dict[str, Any]]:
+    """Parse: named-sheet = FF sheet-name eol sheet-body."""
+    if not src.startswith(FF, pos):
+        raise MTSVDecodeError("expected FF", src, pos)
+    pos, name = sheet_name(src, pos + 1)
+    pos, _ = eol(src, pos)
+    pos, (header_fields, records) = sheet_body(src, pos)
+    return pos, {
+        "sheet name": name,
+        "header": header_fields,
+        "records": records,
+    }
+
+
+def sheet_body(
+    src: str, pos: int
+) -> tuple[int, tuple[list[str] | None, list[list[str]]]]:
+    """Parse: sheet-body = [header *record]."""
+    if pos == len(src) or src.startswith(FF, pos):
+        return pos, (None, [])
+    pos, header_fields = header(src, pos)
+    records: list[list[str]] = []
+    while pos < len(src) and not src.startswith(FF, pos):
+        start = pos
+        pos, record_fields = record(src, pos)
+        if len(record_fields) != len(header_fields):
+            raise MTSVDecodeError(
+                "each record in a sheet must have the same number of fields"
+                " as the header of that sheet",
+                src,
+                start,
+            )
+        records.append(record_fields)
+    return pos, (header_fields, records)
+
+
+def header(src: str, pos: int) -> tuple[int, list[str]]:
+    """Parse: header = record."""
+    return record(src, pos)
+
+
+def record(src: str, pos: int) -> tuple[int, list[str]]:
+    """Parse: record = field *(HTAB field) eol."""
+    pos, value = field(src, pos)
+    fields = [value]
+    while src.startswith(HTAB, pos):
+        pos, value = field(src, pos + 1)
+        fields.append(value)
+    pos, _ = eol(src, pos)
+    return pos, fields
+
+
+def field(src: str, pos: int) -> tuple[int, str]:
+    """Parse: field = *field-char."""
+    start = pos
+    while pos < len(src) and field_char(src[pos]):
+        pos += 1
+    return pos, src[start:pos]
+
+
+def sheet_name(src: str, pos: int) -> tuple[int, str]:
+    """Parse: sheet-name = *field-char."""
+    start = pos
+    while pos < len(src) and field_char(src[pos]):
+        pos += 1
+    return pos, src[start:pos]
+
+
+def field_char(char: str) -> bool:
+    """Match: field-char = %x00-08 / %x0B / %x0E-10FFFF."""
+    code = ord(char)
+    return 0x00 <= code <= 0x08 or code == 0x0B or 0x0E <= code <= 0x10FFFF
+
+
+def eol(src: str, pos: int) -> tuple[int, str]:
+    """Parse: eol = LF / CRLF."""
+    if src.startswith(LF, pos):
+        return pos + 1, LF
+    if src.startswith(CRLF, pos):
+        return pos + 2, CRLF
+    raise MTSVDecodeError("expected LF or CRLF", src, pos)

mtsv-0.1.0/src/mtsv/integrations/arrow.py

@@ -0,0 +1,147 @@
+"""Convert between MTSV sheets and Apache Arrow tables.
+
+The module exports from_arrow and to_arrow.
+"""
+
+__all__ = ["from_arrow", "to_arrow"]
+
+from typing import Any
+
+import pyarrow as pa
+import pyarrow.compute as pc
+
+import mtsv
+
+# The types whose slots hold one scalar, and so have a text form.
+_SCALAR = (
+    pa.types.is_null,
+    pa.types.is_boolean,
+    pa.types.is_integer,
+    pa.types.is_floating,
+    pa.types.is_decimal,
+    pa.types.is_date,
+    pa.types.is_time,
+    pa.types.is_timestamp,
+    pa.types.is_interval,
+    pa.types.is_duration,
+)
+
+# The layouts that encode a value type, rather than types of their own.
+_ENCODINGS = (pa.types.is_dictionary, pa.types.is_run_end_encoded)
+
+
+def from_arrow(
+    tables: list[tuple[str | None, pa.Table]],
+    /,
+    errors: str = "strict",
+) -> list[dict[str, Any]]:
+    """Create MTSV sheets from (sheet name, Arrow table) pairs.
+
+    With errors="strict", raise ValueError if anything outside MTSV
+    would be left behind. With errors="ignore", leave it behind.
+    """
+    if errors not in ("strict", "ignore"):
+        raise LookupError(f"unknown error handler name {errors!r}")
+    extras: set[str] = set()
+    sheets = [_sheet(name, table, extras) for name, table in tables]
+    if errors == "strict" and extras:
+        raise ValueError(
+            "these would be left behind: " + ", ".join(sorted(extras))
+        )
+    mtsv.dumps(sheets)
+    return sheets
+
+
+def to_arrow(
+    obj: list[dict[str, Any]], /
+) -> list[tuple[str | None, pa.Table]]:
+    """Create (sheet name, Arrow table) pairs from MTSV sheets."""
+    mtsv.dumps(obj)
+    return [(sheet["sheet name"], _table(sheet)) for sheet in obj]
+
+
+def _sheet(
+    name: str | None, table: pa.Table, extras: set[str]
+) -> dict[str, Any]:
+    """Create a sheet from a table, recording anything outside MTSV."""
+    if table.schema.metadata:
+        extras.add("table metadata")
+    if table.num_columns == 0:
+        if table.num_rows:
+            extras.add("rows of a table without columns")
+        return {"sheet name": name, "header": None, "records": []}
+    columns = [
+        _values(field, column, extras)
+        for field, column in zip(table.schema, table.columns)
+    ]
+    return {
+        "sheet name": name,
+        "header": table.column_names,
+        "records": [list(fields) for fields in zip(*columns)],
+    }
+
+
+def _values(
+    field: pa.Field, column: pa.ChunkedArray, extras: set[str]
+) -> list[str]:
+    """Read a column as text, recording anything outside MTSV."""
+    if field.metadata:
+        extras.add(f"metadata of column {field.name!r}")
+    if pa.types.is_dictionary(field.type) and field.type.ordered:
+        extras.add(f"ordering of column {field.name!r}")
+    data_type = _value_type(field.type)
+    if not _text_type(data_type):
+        if not any(is_scalar(data_type) for is_scalar in _SCALAR):
+            raise ValueError(
+                f"column {field.name!r} of type {field.type}"
+                " cannot be represented in MTSV"
+            )
+        extras.add(f"type {field.type} of column {field.name!r}")
+        column = _cast(column)
+    if column.null_count:
+        extras.add(f"missing values in column {field.name!r}")
+    return ["" if value is None else value for value in column.to_pylist()]
+
+
+def _value_type(data_type: pa.DataType) -> pa.DataType:
+    """Look through the dictionary and run-end encodings.
+
+    Any array can be dictionary-encoded or run-end encoded, so an
+    encoding carries the values of its value type.
+    """
+    while any(is_encoding(data_type) for is_encoding in _ENCODINGS):
+        data_type = data_type.value_type
+    return data_type
+
+
+def _text_type(data_type: pa.DataType) -> bool:
+    """Match the Arrow text types: Utf8, Large Utf8, Utf8 View."""
+    return (
+        pa.types.is_string(data_type)
+        or pa.types.is_large_string(data_type)
+        or pa.types.is_string_view(data_type)
+    )
+
+
+def _cast(column: pa.ChunkedArray) -> pa.ChunkedArray:
+    """Cast a column to text, in Python where Arrow has no cast."""
+    try:
+        return pc.cast(column, pa.string())
+    except (pa.ArrowInvalid, pa.ArrowNotImplementedError):
+        values = [
+            None if value is None else str(value)
+            for value in column.to_pylist()
+        ]
+        return pa.chunked_array([pa.array(values, pa.string())])
+
+
+def _table(sheet: dict[str, Any]) -> pa.Table:
+    """Create a table whose columns are the header fields, as strings."""
+    header_fields = sheet["header"]
+    if header_fields is None:
+        return pa.Table.from_arrays([], names=[])
+    columns = [
+        pa.array([fields[index] for fields in sheet["records"]], pa.string())
+        for index in range(len(header_fields))
+    ]
+    return pa.Table.from_arrays(columns, names=header_fields)

mtsv-0.1.0/src/mtsv/integrations/ods.py

@@ -0,0 +1,479 @@
+"""Convert between MTSV sheets and OpenDocument spreadsheets, ODF 1.3.
+
+Functions:
+dump -- write MTSV sheets to a binary file as an ODF spreadsheet
+load -- read MTSV sheets from a binary ODF spreadsheet file
+main -- convert a .mtsv file to .ods, or an .ods file to .mtsv
+"""
+
+__all__ = ["dump", "load", "main"]
+
+import argparse
+import io
+import re
+import zipfile
+from collections.abc import Iterator
+from itertools import groupby
+from pathlib import Path
+from typing import Any, BinaryIO
+from xml.etree import ElementTree
+from xml.sax.saxutils import escape, quoteattr
+
+import mtsv
+
+_MEDIA_TYPE = "application/vnd.oasis.opendocument.spreadsheet"
+_PACKAGE_FILES = ("mimetype", "META-INF/manifest.xml", "content.xml")
+_MANIFEST = "urn:oasis:names:tc:opendocument:xmlns:manifest:1.0"
+_OFFICE = "urn:oasis:names:tc:opendocument:xmlns:office:1.0"
+_TABLE = "urn:oasis:names:tc:opendocument:xmlns:table:1.0"
+_TEXT = "urn:oasis:names:tc:opendocument:xmlns:text:1.0"
+_PREFIXES = {
+    _MANIFEST: "manifest",
+    _OFFICE: "office",
+    _TABLE: "table",
+    _TEXT: "text",
+}
+
+_DOCUMENT_CONTENT = "{" + _OFFICE + "}document-content"
+_VERSION = "{" + _OFFICE + "}version"
+_BODY = "{" + _OFFICE + "}body"
+_SPREADSHEET = "{" + _OFFICE + "}spreadsheet"
+_VALUE_TYPE = "{" + _OFFICE + "}value-type"
+_STRING_VALUE = "{" + _OFFICE + "}string-value"
+_TABLE_TABLE = "{" + _TABLE + "}table"
+_NAME = "{" + _TABLE + "}name"
+_COLUMN = "{" + _TABLE + "}table-column"
+_COLUMNS = "{" + _TABLE + "}table-columns"
+_ROW = "{" + _TABLE + "}table-row"
+_ROWS = "{" + _TABLE + "}table-rows"
+_WRAPPERS = (
+    "{" + _TABLE + "}table-header-rows",
+    "{" + _TABLE + "}table-row-group",
+    "{" + _TABLE + "}table-header-columns",
+    "{" + _TABLE + "}table-column-group",
+)
+_CELL = "{" + _TABLE + "}table-cell"
+_COVERED_CELL = "{" + _TABLE + "}covered-table-cell"
+_COLUMNS_REPEATED = "{" + _TABLE + "}number-columns-repeated"
+_ROWS_REPEATED = "{" + _TABLE + "}number-rows-repeated"
+_P = "{" + _TEXT + "}p"
+_H = "{" + _TEXT + "}h"
+_S = "{" + _TEXT + "}s"
+_C = "{" + _TEXT + "}c"
+_TAB = "{" + _TEXT + "}tab"
+_LINE_BREAK = "{" + _TEXT + "}line-break"
+_RUBY = "{" + _TEXT + "}ruby"
+_RUBY_BASE = "{" + _TEXT + "}ruby-base"
+_PARAGRAPH_CONTENT = (
+    "{" + _TEXT + "}a",
+    "{" + _TEXT + "}meta",
+    "{" + _TEXT + "}meta-field",
+    "{" + _TEXT + "}span",
+)
+
+_HTAB = chr(0x09)
+_LF = chr(0x0A)
+_CR = chr(0x0D)
+_SPACE = chr(0x20)
+_SPACES = re.compile(_SPACE + "+")
+_XML_SPACE = _SPACE + _HTAB + _CR + _LF
+_DIGITS = re.compile("[0-9]+")
+
+
+def dump(obj: list[dict[str, Any]], fp: BinaryIO) -> None:
+    """Write MTSV sheets to a binary file as an ODF spreadsheet.
+
+    Raise ValueError if the sheets are not MTSV, or if a field or sheet
+    name holds a character that XML 1.0 does not allow.
+    """
+    mtsv.dumps(obj)
+    for sheet in obj:
+        values = [sheet["sheet name"] or ""]
+        for fields in [sheet["header"] or [], *sheet["records"]]:
+            values.extend(fields)
+        for value in values:
+            if not all(_xml_char(char) for char in value):
+                raise ValueError(
+                    "a field or sheet name that contains a character not"
+                    " allowed in XML 1.0 cannot be represented in ODS"
+                )
+    with zipfile.ZipFile(fp, "w") as package:
+        package.writestr(
+            zipfile.ZipInfo("mimetype"), _MEDIA_TYPE, zipfile.ZIP_STORED
+        )
+        package.writestr(
+            zipfile.ZipInfo("META-INF/manifest.xml"),
+            _manifest(),
+            zipfile.ZIP_DEFLATED,
+        )
+        package.writestr(
+            zipfile.ZipInfo("content.xml"),
+            _document_content(obj),
+            zipfile.ZIP_DEFLATED,
+        )
+
+
+def load(fp: BinaryIO, /, errors: str = "strict") -> list[dict[str, Any]]:
+    """Read MTSV sheets from a binary ODF spreadsheet file.
+
+    With errors="strict", raise ValueError if anything outside MTSV
+    would be left behind. With errors="ignore", leave it behind. Raise
+    ValueError for a file that is not an ODF spreadsheet.
+    """
+    if errors not in ("strict", "ignore"):
+        raise LookupError(f"unknown error handler name {errors!r}")
+    extras: set[str] = set()
+    try:
+        with zipfile.ZipFile(fp) as package:
+            for name in package.namelist():
+                if name not in _PACKAGE_FILES:
+                    extras.add(name)
+            root = ElementTree.fromstring(package.read("content.xml"))
+    except (zipfile.BadZipFile, KeyError, ElementTree.ParseError) as error:
+        raise ValueError("the file is not an ODF package") from error
+    sheets = _spreadsheet(root, extras)
+    if errors == "strict" and extras:
+        raise ValueError(
+            "these would be left behind: " + ", ".join(sorted(extras))
+        )
+    mtsv.dumps(sheets)
+    return sheets
+
+
+def main(argv: list[str] | None = None) -> None:
+    """Convert a .mtsv file to .ods, or an .ods file to .mtsv."""
+    parser = argparse.ArgumentParser(
+        prog="python -m mtsv.integrations.ods",
+        description="Convert a .mtsv file to .ods, or an .ods file to .mtsv.",
+    )
+    parser.add_argument("input", type=Path)
+    parser.add_argument("output", type=Path)
+    parser.add_argument(
+        "--errors", choices=["strict", "ignore"], default="strict"
+    )
+    args = parser.parse_args(argv)
+    suffixes = (args.input.suffix, args.output.suffix)
+    buffer = io.BytesIO()
+    try:
+        if suffixes == (".mtsv", ".ods"):
+            with args.input.open("rb") as source:
+                dump(mtsv.load(source), buffer)
+        elif suffixes == (".ods", ".mtsv"):
+            with args.input.open("rb") as source:
+                mtsv.dump(load(source, args.errors), buffer)
+        else:
+            parser.error("convert a .mtsv file to .ods, or .ods to .mtsv")
+    except ValueError as error:
+        raise SystemExit(error)
+    args.output.write_bytes(buffer.getvalue())
+
+
+def _manifest() -> str:
+    """Generate META-INF/manifest.xml, per ODF 1.3 Part 2, 3.2."""
+    return (
+        "<?xml version='1.0' encoding='UTF-8'?>"
+        f"<manifest:manifest xmlns:manifest='{_MANIFEST}'"
+        " manifest:version='1.3'>"
+        "<manifest:file-entry manifest:full-path='/'"
+        f" manifest:media-type='{_MEDIA_TYPE}'/>"
+        "<manifest:file-entry manifest:full-path='content.xml'"
+        " manifest:media-type='text/xml'/>"
+        "</manifest:manifest>"
+    )
+
+
+def _document_content(sheets: list[dict[str, Any]]) -> str:
+    """Generate content.xml with one table:table per sheet."""
+    tables = "".join(_table(sheet) for sheet in sheets)
+    return (
+        "<?xml version='1.0' encoding='UTF-8'?>"
+        "<office:document-content"
+        f" xmlns:office='{_OFFICE}'"
+        f" xmlns:table='{_TABLE}'"
+        f" xmlns:text='{_TEXT}'"
+        " office:version='1.3'>"
+        f"<office:body><office:spreadsheet>{tables}"
+        "</office:spreadsheet></office:body>"
+        "</office:document-content>"
+    )
+
+
+def _table(sheet: dict[str, Any]) -> str:
+    """Generate table:table; an empty sheet gets one empty cell."""
+    name = sheet["sheet name"]
+    attribute = "" if name is None else " table:name=" + quoteattr(name)
+    if sheet["header"] is None:
+        rows = [[""]]
+    else:
+        rows = [sheet["header"], *sheet["records"]]
+    column = (
+        "<table:table-column"
+        f" table:number-columns-repeated='{len(rows[0])}'/>"
+    )
+    cells = "".join(_row(fields) for fields in rows)
+    return f"<table:table{attribute}>{column}{cells}</table:table>"
+
+
+def _row(fields: list[str]) -> str:
+    """Generate table:table-row."""
+    cells = "".join(_cell(value) for value in fields)
+    return f"<table:table-row>{cells}</table:table-row>"
+
+
+def _cell(value: str) -> str:
+    """Generate table:table-cell; an empty field is an empty cell."""
+    if not value:
+        return "<table:table-cell/>"
+    return (
+        "<table:table-cell office:value-type='string'>"
+        f"<text:p>{_paragraph(value)}</text:p>"
+        "</table:table-cell>"
+    )
+
+
+def _paragraph(value: str) -> str:
+    """Generate text:p content, with spaces marked per ODF 6.1.2-6.1.3.
+
+    A single space between other characters stays a space; every other
+    space is written with text:s, so collapsing cannot remove it.
+    """
+    groups = [
+        (is_space, "".join(chars))
+        for is_space, chars in groupby(value, lambda char: char == _SPACE)
+    ]
+    parts = []
+    for index, (is_space, chars) in enumerate(groups):
+        if not is_space:
+            parts.append(escape(chars))
+            continue
+        count = len(chars)
+        if 0 < index < len(groups) - 1:
+            parts.append(_SPACE)
+            count -= 1
+        if count == 1:
+            parts.append("<text:s/>")
+        elif count > 1:
+            parts.append(f"<text:s text:c='{count}'/>")
+    return "".join(parts)
+
+
+def _xml_char(char: str) -> bool:
+    """Match XML 1.0 Char.
+
+    Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD]
+             | [#x10000-#x10FFFF]
+    """
+    code = ord(char)
+    return (
+        code in (0x09, 0x0A, 0x0D)
+        or 0x20 <= code <= 0xD7FF
+        or 0xE000 <= code <= 0xFFFD
+        or 0x10000 <= code <= 0x10FFFF
+    )
+
+
+def _prefixed(name: str) -> str:
+    """Return {namespace}local as prefix:local for known prefixes."""
+    if not name.startswith("{"):
+        return name
+    namespace, local = name[1:].split("}", 1)
+    prefix = _PREFIXES.get(namespace)
+    return name if prefix is None else f"{prefix}:{local}"
+
+
+def _note_attributes(
+    element: ElementTree.Element, allowed: tuple[str, ...], extras: set[str]
+) -> None:
+    """Record each attribute outside the MTSV mapping as left behind."""
+    for key in element.attrib:
+        if key not in allowed:
+            extras.add(_prefixed(key))
+
+
+def _count(value: str, minimum: int) -> int:
+    """Read a positiveInteger (minimum 1) or nonNegativeInteger (0)."""
+    digits = value.strip(_XML_SPACE)
+    if not _DIGITS.fullmatch(digits) or int(digits) < minimum:
+        raise ValueError(f"not a valid ODF count: {value!r}")
+    return int(digits)
+
+
+def _spreadsheet(
+    root: ElementTree.Element, extras: set[str]
+) -> list[dict[str, Any]]:
+    """Parse office:document-content into MTSV sheets."""
+    if root.tag != _DOCUMENT_CONTENT:
+        raise ValueError("content.xml is not office:document-content")
+    _note_attributes(root, (_VERSION,), extras)
+    spreadsheet = None
+    for child in root:
+        if child.tag == _BODY:
+            spreadsheet = child.find(_SPREADSHEET)
+        else:
+            extras.add(_prefixed(child.tag))
+    if spreadsheet is None:
+        raise ValueError("the document is not an office:spreadsheet")
+    _note_attributes(spreadsheet, (), extras)
+    sheets = []
+    for child in spreadsheet:
+        if child.tag != _TABLE_TABLE:
+            extras.add(_prefixed(child.tag))
+            continue
+        sheets.append(_sheet(child, extras))
+    return sheets
+
+
+def _sheet(element: ElementTree.Element, extras: set[str]) -> dict[str, Any]:
+    """Parse table:table into a sheet, reading only its used area."""
+    _note_attributes(element, (_NAME,), extras)
+    name = element.get(_NAME)
+    lines: list[list[str]] = []
+    pending = 0
+    for count, values in _rows(element, extras):
+        if values:
+            lines.extend([] for _ in range(pending))
+            pending = 0
+            lines.extend(list(values) for _ in range(count))
+        else:
+            pending += count
+    if not lines:
+        return {"sheet name": name, "header": None, "records": []}
+    width = max(len(values) for values in lines)
+    padded = [values + [""] * (width - len(values)) for values in lines]
+    return {"sheet name": name, "header": padded[0], "records": padded[1:]}
+
+
+def _rows(
+    element: ElementTree.Element, extras: set[str]
+) -> Iterator[tuple[int, list[str]]]:
+    """Yield (repeat count, values) per row, reading into wrappers."""
+    for child in element:
+        if child.tag == _ROW:
+            _note_attributes(child, (_ROWS_REPEATED,), extras)
+            count = _count(child.get(_ROWS_REPEATED, "1"), 1)
+            yield count, _cells(child, extras)
+        elif child.tag == _COLUMN:
+            _note_attributes(child, (_COLUMNS_REPEATED,), extras)
+            _count(child.get(_COLUMNS_REPEATED, "1"), 1)
+        elif child.tag in (_ROWS, _COLUMNS):
+            yield from _rows(child, extras)
+        elif child.tag in _WRAPPERS:
+            extras.add(_prefixed(child.tag))
+            _note_attributes(child, (), extras)
+            yield from _rows(child, extras)
+        else:
+            extras.add(_prefixed(child.tag))
+
+
+def _cells(row: ElementTree.Element, extras: set[str]) -> list[str]:
+    """Read a row's cell texts, without its trailing empty cells."""
+    values: list[str] = []
+    pending = 0
+    for child in row:
+        if child.tag == _CELL:
+            text = _cell_value(child, extras)
+        elif child.tag == _COVERED_CELL:
+            extras.add(_prefixed(child.tag))
+            text = ""
+        else:
+            extras.add(_prefixed(child.tag))
+            continue
+        count = _count(child.get(_COLUMNS_REPEATED, "1"), 1)
+        if text:
+            values.extend([""] * pending)
+            pending = 0
+            values.extend([text] * count)
+        else:
+            pending += count
+    return values
+
+
+def _cell_value(cell: ElementTree.Element, extras: set[str]) -> str:
+    """Read a cell's text, from office:string-value when it is given."""
+    is_string = cell.get(_VALUE_TYPE) == "string"
+    for key in cell.attrib:
+        if key == _COLUMNS_REPEATED:
+            continue
+        if is_string and key in (_VALUE_TYPE, _STRING_VALUE):
+            continue
+        extras.add(_prefixed(key))
+    text = _cell_text(cell, extras)
+    value = cell.get(_STRING_VALUE)
+    if not is_string or value is None:
+        return text
+    if text and text != value:
+        extras.add(_prefixed(_P))
+    return value
+
+
+def _cell_text(cell: ElementTree.Element, extras: set[str]) -> str:
+    """Read a cell's paragraphs, joined by line breaks."""
+    paragraphs = []
+    for child in cell:
+        if child.tag in (_P, _H):
+            if child.tag == _H:
+                extras.add(_prefixed(child.tag))
+            _note_attributes(child, (), extras)
+            paragraphs.append(_paragraph_text(child, extras))
+        else:
+            extras.add(_prefixed(child.tag))
+    return _LF.join(paragraphs)
+
+
+def _paragraph_text(paragraph: ElementTree.Element, extras: set[str]) -> str:
+    """Read a paragraph per the white space algorithm, ODF 6.1.2."""
+    tokens: list[tuple[bool, str]] = []
+    _collect(paragraph, tokens, extras)
+    merged: list[tuple[bool, str]] = []
+    for kept, text in tokens:
+        if not kept and merged and not merged[-1][0]:
+            merged[-1] = (False, merged[-1][1] + text)
+        else:
+            merged.append((kept, text))
+    if merged and not merged[0][0]:
+        merged[0] = (False, merged[0][1].lstrip(_SPACE))
+    if merged and not merged[-1][0]:
+        merged[-1] = (False, merged[-1][1].rstrip(_SPACE))
+    return "".join(
+        text if kept else _SPACES.sub(_SPACE, text) for kept, text in merged
+    )
+
+
+def _collect(
+    element: ElementTree.Element,
+    tokens: list[tuple[bool, str]],
+    extras: set[str],
+) -> None:
+    """Flatten paragraph content per steps 1 to 4 of ODF 1.3, 6.1.2."""
+    if element.text:
+        tokens.append((False, _to_spaces(element.text)))
+    for child in element:
+        if child.tag == _S:
+            tokens.append((True, _SPACE * _count(child.get(_C, "1"), 0)))
+        elif child.tag == _TAB:
+            tokens.append((True, _HTAB))
+        elif child.tag == _LINE_BREAK:
+            tokens.append((True, _LF))
+        elif child.tag == _RUBY:
+            extras.add(_prefixed(child.tag))
+            base = child.find(_RUBY_BASE)
+            if base is not None:
+                _collect(base, tokens, extras)
+        elif child.tag in _PARAGRAPH_CONTENT:
+            extras.add(_prefixed(child.tag))
+            _collect(child, tokens, extras)
+        else:
+            extras.add(_prefixed(child.tag))
+        if child.tail:
+            tokens.append((False, _to_spaces(child.tail)))
+
+
+def _to_spaces(text: str) -> str:
+    """Replace HT, CR, and LF with a space, step 4 of ODF 1.3, 6.1.2."""
+    text = text.replace(_HTAB, _SPACE)
+    text = text.replace(_CR, _SPACE)
+    return text.replace(_LF, _SPACE)
+
+
+if __name__ == "__main__":
+    main()

mtsv-0.1.0/src/mtsv.egg-info/PKG-INFO

@@ -0,0 +1,122 @@
+Metadata-Version: 2.4
+Name: mtsv
+Version: 0.1.0
+Summary: Python implementation of Multi-Sheet Tab-Separated Values (MTSV)
+Author-email: Demos Ra <demos_ra@hotmail.com>
+License-Expression: MIT
+Project-URL: source, https://github.com/demos-ra/mtsv
+Project-URL: issues, https://github.com/demos-ra/mtsv/issues
+Keywords: tsv,tabular data,media type
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: arrow
+Requires-Dist: pyarrow>=16; extra == "arrow"
+Dynamic: license-file
+
+# MTSV for Python
+
+A parser and a generator for Multi-Sheet Tab-Separated Values (MTSV), with
+integrations for spreadsheets (ODS) and data tools (Apache Arrow). The
+version is the `version` field of `pyproject.toml`.
+
+* [Specification](https://github.com/demos-ra/mtsv-spec)
+* [Repository](https://github.com/demos-ra/mtsv)
+
+## Install
+
+From the repository. A Python that an operating system manages does not
+accept packages directly, so install into a virtual environment. Run these
+from the root of the repository, one folder above this one:
+
+```
+python3 -m venv .venv
+.venv/bin/pip install ./python
+```
+
+For the Arrow integration, which also installs pyarrow:
+
+```
+.venv/bin/pip install "./python[arrow]"
+```
+
+## Read and write MTSV
+
+Sheets are a list of dictionaries with `"sheet name"`, `"header"`, and
+`"records"`, the same shape as the
+[conformance results](https://github.com/demos-ra/mtsv/blob/main/conformance/README.md).
+
+```python
+import mtsv
+
+with open("book.mtsv", "rb") as file:
+    sheets = mtsv.load(file)
+
+with open("book.mtsv", "wb") as file:
+    mtsv.dump(sheets, file)
+```
+
+`loads` and `dumps` work on strings.
+
+## Spreadsheets (ODS)
+
+```python
+from mtsv.integrations import ods
+
+with open("book.ods", "wb") as file:
+    ods.dump(sheets, file)
+
+with open("book.ods", "rb") as file:
+    sheets = ods.load(file)
+```
+
+From the command line:
+
+```
+python -m mtsv.integrations.ods book.mtsv book.ods
+python -m mtsv.integrations.ods book.ods book.mtsv
+```
+
+## Data tools (Apache Arrow)
+
+```python
+from mtsv.integrations import arrow
+
+tables = arrow.to_arrow(sheets)
+sheets = arrow.from_arrow(tables)
+```
+
+`tables` is a list of `(sheet name, pyarrow.Table)` pairs.
+
+## What is left behind
+
+MTSV holds sheets, names, rows, and text. Going out to ODS or Arrow keeps all
+of it, with two ODS exceptions: characters that XML 1.0 does not allow raise
+`ValueError`, and empty rows and columns at the edge of a sheet do not come
+back from ODS, so a sheet of only empty fields comes back as an empty sheet.
+Coming back in, anything else (formatting, formulas, types, missing values)
+raises `ValueError` by default. To confirm and leave it behind, pass
+`errors="ignore"`, or `--errors ignore` on the command line. Text that MTSV
+cannot hold, such as a tab or line break inside a value, always raises
+`ValueError`. So does an Arrow column whose values cannot be text at all, such
+as binary, a list, or a struct.
+
+## Layout
+
+| Path                     | Contents                                    |
+|--------------------------|---------------------------------------------|
+| `src/mtsv/`              | the interface, the parser, and the generator |
+| `src/mtsv/integrations/` | one module per target standard              |
+| `tests/`                 | the test suite, run against the install     |
+
+## Test
+
+From the root of the repository:
+
+```
+.venv/bin/python -m unittest discover -s python/tests
+```
+
+## License
+
+[MIT](https://github.com/demos-ra/mtsv/blob/main/LICENSE)

mtsv-0.1.0/src/mtsv.egg-info/SOURCES.txt

@@ -0,0 +1,16 @@
+LICENSE
+MANIFEST.in
+README.md
+pyproject.toml
+src/mtsv/__init__.py
+src/mtsv/_generator.py
+src/mtsv/_parser.py
+src/mtsv/py.typed
+src/mtsv.egg-info/PKG-INFO
+src/mtsv.egg-info/SOURCES.txt
+src/mtsv.egg-info/dependency_links.txt
+src/mtsv.egg-info/requires.txt
+src/mtsv.egg-info/top_level.txt
+src/mtsv/integrations/__init__.py
+src/mtsv/integrations/arrow.py
+src/mtsv/integrations/ods.py

mtsv-0.1.0/src/mtsv.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

mtsv-0.1.0/src/mtsv.egg-info/requires.txt

@@ -0,0 +1,3 @@
+
+[arrow]
+pyarrow>=16

mtsv-0.1.0/src/mtsv.egg-info/top_level.txt

@@ -0,0 +1 @@
+mtsv