pqax 0.1.0__tar.gz

pqax-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 SQLBI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pqax-0.1.0/PKG-INFO

@@ -0,0 +1,190 @@
+Metadata-Version: 2.4
+Name: pqax
+Version: 0.1.0
+Summary: Read, validate, and write PQAX files: the footprint of Parquet tables, for analysis without access to the data.
+Author: SQLBI
+License-Expression: MIT
+Project-URL: Homepage, https://pqax.org
+Project-URL: Repository, https://github.com/pqax-org/pqax
+Keywords: pqax,parquet,delta,lakehouse,footprint,vertipaq
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Database
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: pandas
+Requires-Dist: pandas; extra == "pandas"
+Provides-Extra: polars
+Requires-Dist: polars>=1.0; extra == "polars"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: jsonschema>=4.18; extra == "dev"
+Dynamic: license-file
+
+# pqax for Python
+
+`pqax` reads, validates, and writes PQAX files: the footprint of Parquet tables, as a zip of JSON tables. The format is defined by the [serialization](https://github.com/pqax-org/pqax/blob/main/docs/serialization.md) and the [object model](https://github.com/pqax-org/pqax/blob/main/docs/object-model.md), and the validation rules by the [conformance kit](https://github.com/pqax-org/pqax/blob/main/conformance/rules.md).
+
+| | |
+|---|---|
+| Version | 0.1.0 |
+| Format | Reads every version 0.x. Writes version 0.1. |
+| Python | 3.10 or later |
+| Dependencies | None. `to_dataframe` uses pandas, or polars 1.0 or later, when one is installed. |
+
+## Install
+
+```bash
+pip install pqax                       # the package and the pqax command
+pip install "pqax[pandas]"             # with pandas, for to_dataframe
+pip install "pqax[polars]"             # with polars, for to_dataframe
+```
+
+From the root of the repository:
+
+```bash
+pip install -e src/python              # the package and the pqax command
+pip install -e "src/python[pandas]"    # with pandas, for to_dataframe
+pip install -e "src/python[dev]"       # with pytest and jsonschema, for the tests
+```
+
+Python versions: 3.10 or later. The tests run on 3.10 and 3.14, each without extras and with both pandas and polars. The package has no dependency: the `pandas` and `polars` extras only install the library that `to_dataframe` uses.
+
+## Read a file
+
+```python
+import pqax
+
+with pqax.open("footprint.pqax") as doc:
+    print(doc.manifest["formatVersion"], doc.part_names)
+    tables = doc.summary["tables"]               # read when the file opens
+    for table_id, size in zip(tables["id"], tables.get("fileSize", [])):
+        print(table_id, size)
+    chunks = doc.table("columnChunks")           # read now, from all its parts, and kept
+    print(doc.row_count("columnChunks"), sum(chunks["compressedSize"]))
+    for row in doc.rows("files"):                # one dictionary for each row
+        print(row["id"], row["fileSize"])
+```
+
+A table is a dictionary of fields, and a field is a list with one value for each row, as in the file. The keys keep the names of the file, such as `fileCount`. A value that the file does not have is `None`.
+
+- Opening a file reads the zip directory, `manifest.json`, and `summary.json`. A detail part is read the first time a table that it holds is asked for.
+- `doc.table(name)` concatenates the parts of a table in the order of the manifest. A field that a part leaves out is `None` in the rows of that part. A table that the file does not have is `{}`.
+- A field that has no value in any row is not in the dictionary: use `table.get("field")`.
+- A value whose JSON type is not the type of its field, such as a string where the model says integer, reads as `None`. `doc.raw(name)` returns the table as parsed, with that value and with the members of a table that are not arrays.
+- Fields and tables that this version does not define are returned like the others. Members of the manifest that it does not define stay in `doc.manifest`.
+
+A PQAX file embedded in a VPAX file:
+
+```python
+with pqax.open_embedded("model.vpax") as doc:    # the part Footprint.pqax, stored or deflated
+    print(doc.embedded_part, doc.embedded_compression)
+```
+
+The part is read into memory, and the VPAX file is closed before `open_embedded` returns.
+
+## Validate a file
+
+```python
+result = pqax.validate("footprint.pqax")         # or pqax.validate_embedded("model.vpax")
+print(result.is_valid, result.checks)
+for issue in result.issues:
+    print(issue.code, issue.message, issue.part, issue.table, issue.field)
+```
+
+The validator reports every issue that it finds, with the codes of section 4.1 of the [contracts](https://github.com/pqax-org/pqax/blob/main/docs/contracts.md): `container.*`, `json.*`, `manifest.*`, `part.*`, `table.*`, `depth.*`, `totals.*`, and, for `validate_embedded`, `embedded.partMissing`. A rule that cannot be evaluated because an earlier one failed is skipped. A file that cannot be opened raises `OSError`. A file that is not a zip archive is an issue, `container.manifestMissing`, and so is a VPAX file given to `validate`.
+
+## Data frames
+
+```python
+frame = pqax.to_dataframe(doc.table("columnChunks"))             # pandas when installed, else polars
+frame = pqax.to_dataframe(doc.summary["tables"], library="polars")
+```
+
+With pandas, an integer or boolean column with `None` values gets a nullable type, so it does not turn into floats. Lists and objects stay as Python objects. Timestamps stay strings; `pqax.parse_timestamp` turns one into a `datetime`. The rows of `doc.rows(name)` give the same frame as `doc.table(name)`.
+
+`to_dataframe` needs polars 1.0 or later, and raises `ImportError` with an earlier version, which loses the values of a column that mixes types.
+
+## Write a file
+
+```python
+from datetime import datetime, timezone
+
+builder = pqax.Builder({"collectorApp": "my-tool", "level": 2, "startedAt": datetime.now(timezone.utc)})
+source = builder.add_source(kind="Storage folder")
+table = builder.add_table(source=source, name="sales", format="Delta", detailDepth="Files")
+file_id = builder.add_file(table=table, path="part-0001.parquet", fileSize=47820)
+builder.update("tables", table, fileCount=1, fileSize=47820)
+builder.write("footprint.pqax")
+```
+
+| Method | What it does |
+|---|---|
+| `add_source(**fields)`, `add_table`, `add_column`, and one for each of the 21 tables | Adds a row. For a table with an `id`, it keeps a positive id that the row brings, assigns the next free id when the row has none (greater than every id so far in the table, so from 1), and returns the id. For a table without, it returns the position of the row, from 0. The plural names, such as `add_files`, are the same methods. |
+| `add(table, **fields)` | Adds a row to any table, also one that this version does not define, with any field |
+| `extend(table, columns, detail=None)` | Adds many rows from a columnar table. `detail` places a table that the format does not define. |
+| `update(table, id_or_position, **fields)` | Sets fields of a row already added, such as totals that are known after the detail |
+| `write(path_or_file)` | Writes the file. It raises `ValueError`, naming the table and the id, when a table has an id in more than one row. |
+| `Builder.from_document(doc)` | A builder with every table of a document, to rewrite a file |
+
+The fields are named as in the file. An `add_` method refuses a field that its table does not have, unless the name has a producer's prefix and an underscore, as in `acme_note`. A field set to `None` has no value. A `datetime` becomes a timestamp in UTC with three decimals; a naive `datetime` is taken as UTC.
+
+`write` lays out the file as the serialization says: `manifest.json` first, then `summary.json`, always present, then one part for each detail table, named after it, as `files.json`. A table with more than `rows_per_part` rows, 250,000 by default and at most, continues in parts numbered from 1, as `columnChunks.1.json`. A field with no value in any row of a part is left out of that part, and so is a table with no rows. The JSON is compact UTF-8 with no byte order mark, the entries are deflated at level 9 by default (`compression_level`), and every entry has the same date, so the same footprint gives the same bytes. A path is written through a temporary file.
+
+`from_document` keeps the ids, the collection, the obfuscation fields, the other members of the manifest, and the fields and tables that this version does not define. It loses what the reader does not keep: the entries that the manifest does not list, the members of a table that are not arrays, and the members of an element of `parts` other than `name` and `tables`. It writes format version 0.1, and its own part names.
+
+## Command line
+
+```bash
+python -m pqax check footprint.pqax      # exit code 0 when valid, 1 when not, 2 on a usage or I/O error
+python -m pqax info footprint.pqax       # format, version, collection, parts with their tables and rows
+python -m pqax info model.vpax           # a VPAX file: the PQAX file that it embeds as Footprint.pqax
+```
+
+`check` prints one line for each issue, its code first, then a summary line. Both commands take `--part NAME` for another embedded part. The installed package also has a `pqax` command.
+
+## API
+
+| Name | What it is |
+|---|---|
+| `open(path_or_file)` | A `PqaxDocument`. The source is a path, the bytes of a file, or a binary file object, which is left open. |
+| `open_embedded(path_or_file, part_name="Footprint.pqax")` | A `PqaxDocument` of the PQAX file that a package holds as a part |
+| `PqaxDocument` | `manifest`, `summary`, `row_count(name)`, `table(name)`, `rows(name)`, `raw(name, part=None)`, `table_names`, `part_names`, `format_version`, `collection`, `embedded_part`, `embedded_compression`, `close()`, and the `with` statement |
+| `validate(path_or_file)`, `validate_embedded(path_or_file, part_name)` | A `ValidationResult`: `is_valid`, `checks`, `issues`, `codes` |
+| `Issue` | `code`, `message`, `part`, `table`, `field` |
+| `to_dataframe(table, library=None)` | A pandas or polars data frame |
+| `Builder`, `BuilderOptions` | The writer |
+| `format_timestamp(datetime)`, `parse_timestamp(text)` | Timestamps in the form of the format, `2026-10-05T08:15:30.000Z` |
+| `TABLES`, `SUMMARY_TABLES`, `DETAIL_TABLES`, `FIELDS`, `REQUIRED`, `REFERENCES` | The names of the format |
+
+Errors:
+
+| Error | When |
+|---|---|
+| `ManifestMissingError` | The archive has no `manifest.json` |
+| `FormatError` | `format` is not `pqax` |
+| `UnknownVersionError` | The first number of `formatVersion` is not 0 |
+| `JsonError` | A part that is read is not valid JSON |
+| `PartMissingError` | A detail part that the manifest lists is not in the archive, when its table is read (`manifest.partMissing`), or the embedded part is not in the package (`embedded.partMissing`) |
+| `ContainerError` | The file is not a zip archive, or an entry cannot be read: encrypted, compressed with another method, or corrupt |
+
+All derive from `PqaxError`, whose `code` is the validation code that the error matches, when there is one. A reader refuses a file for the first four, when it opens it, and `open_embedded` also for a missing embedded part. Opening also fails when the manifest or the summary cannot be read from the zip, for example because the entry is encrypted. A `summary.json` that the manifest lists and the archive does not have gives empty summary tables, and `validate` reports it. A detail part that is broken fails only the read of its tables.
+
+## Limits
+
+- The reader keeps a table in memory once it is read. A part of 250,000 column chunks takes about 0.15 s to parse.
+- A part that holds more than 1 GB of JSON text is not read, and is reported or raised as such.
+- The rows of a part are the length of its longest array, not what the manifest states, so that a file that states more rows than it holds cannot make the reader allocate them. In a valid file the two agree.
+- The reader replaces a value of another JSON type with `None` only at the level of the field. An element of a list, or a member of a nested object, is returned as the file has it.
+- `container.zip64` looks at the central directory and its locator, as the kit says. A ZIP64 field in a local header alone is not reported.
+- No pandas or polars type is chosen for timestamps, lists, maps, or nested objects.
+- On Python 3.10, `parse_timestamp` reads fewer of the other forms of ISO 8601, as `datetime.fromisoformat` does there: a fraction of seconds with other than three or six digits, a comma before the fraction, an offset without a colon, and the basic form, such as `20261005T081530Z`, raise `ValueError`. The form of the format reads on every version.

pqax-0.1.0/README.md

@@ -0,0 +1,158 @@
+# pqax for Python
+
+`pqax` reads, validates, and writes PQAX files: the footprint of Parquet tables, as a zip of JSON tables. The format is defined by the [serialization](https://github.com/pqax-org/pqax/blob/main/docs/serialization.md) and the [object model](https://github.com/pqax-org/pqax/blob/main/docs/object-model.md), and the validation rules by the [conformance kit](https://github.com/pqax-org/pqax/blob/main/conformance/rules.md).
+
+| | |
+|---|---|
+| Version | 0.1.0 |
+| Format | Reads every version 0.x. Writes version 0.1. |
+| Python | 3.10 or later |
+| Dependencies | None. `to_dataframe` uses pandas, or polars 1.0 or later, when one is installed. |
+
+## Install
+
+```bash
+pip install pqax                       # the package and the pqax command
+pip install "pqax[pandas]"             # with pandas, for to_dataframe
+pip install "pqax[polars]"             # with polars, for to_dataframe
+```
+
+From the root of the repository:
+
+```bash
+pip install -e src/python              # the package and the pqax command
+pip install -e "src/python[pandas]"    # with pandas, for to_dataframe
+pip install -e "src/python[dev]"       # with pytest and jsonschema, for the tests
+```
+
+Python versions: 3.10 or later. The tests run on 3.10 and 3.14, each without extras and with both pandas and polars. The package has no dependency: the `pandas` and `polars` extras only install the library that `to_dataframe` uses.
+
+## Read a file
+
+```python
+import pqax
+
+with pqax.open("footprint.pqax") as doc:
+    print(doc.manifest["formatVersion"], doc.part_names)
+    tables = doc.summary["tables"]               # read when the file opens
+    for table_id, size in zip(tables["id"], tables.get("fileSize", [])):
+        print(table_id, size)
+    chunks = doc.table("columnChunks")           # read now, from all its parts, and kept
+    print(doc.row_count("columnChunks"), sum(chunks["compressedSize"]))
+    for row in doc.rows("files"):                # one dictionary for each row
+        print(row["id"], row["fileSize"])
+```
+
+A table is a dictionary of fields, and a field is a list with one value for each row, as in the file. The keys keep the names of the file, such as `fileCount`. A value that the file does not have is `None`.
+
+- Opening a file reads the zip directory, `manifest.json`, and `summary.json`. A detail part is read the first time a table that it holds is asked for.
+- `doc.table(name)` concatenates the parts of a table in the order of the manifest. A field that a part leaves out is `None` in the rows of that part. A table that the file does not have is `{}`.
+- A field that has no value in any row is not in the dictionary: use `table.get("field")`.
+- A value whose JSON type is not the type of its field, such as a string where the model says integer, reads as `None`. `doc.raw(name)` returns the table as parsed, with that value and with the members of a table that are not arrays.
+- Fields and tables that this version does not define are returned like the others. Members of the manifest that it does not define stay in `doc.manifest`.
+
+A PQAX file embedded in a VPAX file:
+
+```python
+with pqax.open_embedded("model.vpax") as doc:    # the part Footprint.pqax, stored or deflated
+    print(doc.embedded_part, doc.embedded_compression)
+```
+
+The part is read into memory, and the VPAX file is closed before `open_embedded` returns.
+
+## Validate a file
+
+```python
+result = pqax.validate("footprint.pqax")         # or pqax.validate_embedded("model.vpax")
+print(result.is_valid, result.checks)
+for issue in result.issues:
+    print(issue.code, issue.message, issue.part, issue.table, issue.field)
+```
+
+The validator reports every issue that it finds, with the codes of section 4.1 of the [contracts](https://github.com/pqax-org/pqax/blob/main/docs/contracts.md): `container.*`, `json.*`, `manifest.*`, `part.*`, `table.*`, `depth.*`, `totals.*`, and, for `validate_embedded`, `embedded.partMissing`. A rule that cannot be evaluated because an earlier one failed is skipped. A file that cannot be opened raises `OSError`. A file that is not a zip archive is an issue, `container.manifestMissing`, and so is a VPAX file given to `validate`.
+
+## Data frames
+
+```python
+frame = pqax.to_dataframe(doc.table("columnChunks"))             # pandas when installed, else polars
+frame = pqax.to_dataframe(doc.summary["tables"], library="polars")
+```
+
+With pandas, an integer or boolean column with `None` values gets a nullable type, so it does not turn into floats. Lists and objects stay as Python objects. Timestamps stay strings; `pqax.parse_timestamp` turns one into a `datetime`. The rows of `doc.rows(name)` give the same frame as `doc.table(name)`.
+
+`to_dataframe` needs polars 1.0 or later, and raises `ImportError` with an earlier version, which loses the values of a column that mixes types.
+
+## Write a file
+
+```python
+from datetime import datetime, timezone
+
+builder = pqax.Builder({"collectorApp": "my-tool", "level": 2, "startedAt": datetime.now(timezone.utc)})
+source = builder.add_source(kind="Storage folder")
+table = builder.add_table(source=source, name="sales", format="Delta", detailDepth="Files")
+file_id = builder.add_file(table=table, path="part-0001.parquet", fileSize=47820)
+builder.update("tables", table, fileCount=1, fileSize=47820)
+builder.write("footprint.pqax")
+```
+
+| Method | What it does |
+|---|---|
+| `add_source(**fields)`, `add_table`, `add_column`, and one for each of the 21 tables | Adds a row. For a table with an `id`, it keeps a positive id that the row brings, assigns the next free id when the row has none (greater than every id so far in the table, so from 1), and returns the id. For a table without, it returns the position of the row, from 0. The plural names, such as `add_files`, are the same methods. |
+| `add(table, **fields)` | Adds a row to any table, also one that this version does not define, with any field |
+| `extend(table, columns, detail=None)` | Adds many rows from a columnar table. `detail` places a table that the format does not define. |
+| `update(table, id_or_position, **fields)` | Sets fields of a row already added, such as totals that are known after the detail |
+| `write(path_or_file)` | Writes the file. It raises `ValueError`, naming the table and the id, when a table has an id in more than one row. |
+| `Builder.from_document(doc)` | A builder with every table of a document, to rewrite a file |
+
+The fields are named as in the file. An `add_` method refuses a field that its table does not have, unless the name has a producer's prefix and an underscore, as in `acme_note`. A field set to `None` has no value. A `datetime` becomes a timestamp in UTC with three decimals; a naive `datetime` is taken as UTC.
+
+`write` lays out the file as the serialization says: `manifest.json` first, then `summary.json`, always present, then one part for each detail table, named after it, as `files.json`. A table with more than `rows_per_part` rows, 250,000 by default and at most, continues in parts numbered from 1, as `columnChunks.1.json`. A field with no value in any row of a part is left out of that part, and so is a table with no rows. The JSON is compact UTF-8 with no byte order mark, the entries are deflated at level 9 by default (`compression_level`), and every entry has the same date, so the same footprint gives the same bytes. A path is written through a temporary file.
+
+`from_document` keeps the ids, the collection, the obfuscation fields, the other members of the manifest, and the fields and tables that this version does not define. It loses what the reader does not keep: the entries that the manifest does not list, the members of a table that are not arrays, and the members of an element of `parts` other than `name` and `tables`. It writes format version 0.1, and its own part names.
+
+## Command line
+
+```bash
+python -m pqax check footprint.pqax      # exit code 0 when valid, 1 when not, 2 on a usage or I/O error
+python -m pqax info footprint.pqax       # format, version, collection, parts with their tables and rows
+python -m pqax info model.vpax           # a VPAX file: the PQAX file that it embeds as Footprint.pqax
+```
+
+`check` prints one line for each issue, its code first, then a summary line. Both commands take `--part NAME` for another embedded part. The installed package also has a `pqax` command.
+
+## API
+
+| Name | What it is |
+|---|---|
+| `open(path_or_file)` | A `PqaxDocument`. The source is a path, the bytes of a file, or a binary file object, which is left open. |
+| `open_embedded(path_or_file, part_name="Footprint.pqax")` | A `PqaxDocument` of the PQAX file that a package holds as a part |
+| `PqaxDocument` | `manifest`, `summary`, `row_count(name)`, `table(name)`, `rows(name)`, `raw(name, part=None)`, `table_names`, `part_names`, `format_version`, `collection`, `embedded_part`, `embedded_compression`, `close()`, and the `with` statement |
+| `validate(path_or_file)`, `validate_embedded(path_or_file, part_name)` | A `ValidationResult`: `is_valid`, `checks`, `issues`, `codes` |
+| `Issue` | `code`, `message`, `part`, `table`, `field` |
+| `to_dataframe(table, library=None)` | A pandas or polars data frame |
+| `Builder`, `BuilderOptions` | The writer |
+| `format_timestamp(datetime)`, `parse_timestamp(text)` | Timestamps in the form of the format, `2026-10-05T08:15:30.000Z` |
+| `TABLES`, `SUMMARY_TABLES`, `DETAIL_TABLES`, `FIELDS`, `REQUIRED`, `REFERENCES` | The names of the format |
+
+Errors:
+
+| Error | When |
+|---|---|
+| `ManifestMissingError` | The archive has no `manifest.json` |
+| `FormatError` | `format` is not `pqax` |
+| `UnknownVersionError` | The first number of `formatVersion` is not 0 |
+| `JsonError` | A part that is read is not valid JSON |
+| `PartMissingError` | A detail part that the manifest lists is not in the archive, when its table is read (`manifest.partMissing`), or the embedded part is not in the package (`embedded.partMissing`) |
+| `ContainerError` | The file is not a zip archive, or an entry cannot be read: encrypted, compressed with another method, or corrupt |
+
+All derive from `PqaxError`, whose `code` is the validation code that the error matches, when there is one. A reader refuses a file for the first four, when it opens it, and `open_embedded` also for a missing embedded part. Opening also fails when the manifest or the summary cannot be read from the zip, for example because the entry is encrypted. A `summary.json` that the manifest lists and the archive does not have gives empty summary tables, and `validate` reports it. A detail part that is broken fails only the read of its tables.
+
+## Limits
+
+- The reader keeps a table in memory once it is read. A part of 250,000 column chunks takes about 0.15 s to parse.
+- A part that holds more than 1 GB of JSON text is not read, and is reported or raised as such.
+- The rows of a part are the length of its longest array, not what the manifest states, so that a file that states more rows than it holds cannot make the reader allocate them. In a valid file the two agree.
+- The reader replaces a value of another JSON type with `None` only at the level of the field. An element of a list, or a member of a nested object, is returned as the file has it.
+- `container.zip64` looks at the central directory and its locator, as the kit says. A ZIP64 field in a local header alone is not reported.
+- No pandas or polars type is chosen for timestamps, lists, maps, or nested objects.
+- On Python 3.10, `parse_timestamp` reads fewer of the other forms of ISO 8601, as `datetime.fromisoformat` does there: a fraction of seconds with other than three or six digits, a comma before the fraction, an offset without a colon, and the basic form, such as `20261005T081530Z`, raise `ValueError`. The form of the format reads on every version.

pqax-0.1.0/pqax/__init__.py

@@ -0,0 +1,34 @@
+"""Read, validate, and write PQAX files: the footprint of Parquet tables, as a zip of JSON tables.
+
+    import pqax
+
+    with pqax.open("footprint.pqax") as doc:
+        tables = doc.summary["tables"]                  # every summary table is read when the file opens
+        chunks = doc.table("columnChunks")              # a detail table is read when it is asked for
+        print(sum(chunks.get("compressedSize", [])))
+
+    result = pqax.validate("footprint.pqax")            # every issue, with the codes of the conformance kit
+    print(result.is_valid, [issue.code for issue in result.issues])
+
+The package needs only the standard library. to_dataframe uses pandas or polars when one is installed.
+"""
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+from ._errors import (
+    ContainerError, FormatError, JsonError, ManifestMissingError, PartMissingError, PqaxError, UnknownVersionError,
+)
+from ._json import format_timestamp, parse_timestamp
+from ._schema import DETAIL_TABLES, FIELDS, FORMAT_VERSION, REFERENCES, REQUIRED, ROWS_PER_PART, SUMMARY_TABLES, TABLES
+from .builder import Builder, BuilderOptions
+from .dataframe import to_dataframe
+from .reader import PqaxDocument, open, open_embedded
+from .validator import Issue, ValidationResult, validate, validate_embedded
+
+__all__ = [
+    "Builder", "BuilderOptions", "ContainerError", "DETAIL_TABLES", "FIELDS", "FORMAT_VERSION", "FormatError",
+    "Issue", "JsonError", "ManifestMissingError", "PartMissingError", "PqaxDocument", "PqaxError", "REFERENCES",
+    "REQUIRED", "ROWS_PER_PART", "SUMMARY_TABLES", "TABLES", "UnknownVersionError", "ValidationResult",
+    "format_timestamp", "open", "open_embedded", "parse_timestamp", "to_dataframe", "validate", "validate_embedded",
+]

pqax-0.1.0/pqax/__main__.py

@@ -0,0 +1,133 @@
+"""The command line: python -m pqax check <file>, python -m pqax info <file>.
+
+check exits with 0 when the file is valid, 1 when it is not, and 2 on a usage error or a file that cannot be read.
+Both commands also take a VPAX file, and then work on the PQAX file that it embeds as Footprint.pqax.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import zipfile
+from typing import Any, List, Optional
+
+from . import __version__
+from ._container import find_entry, open_archive
+from ._errors import ContainerError, PqaxError
+from ._schema import EMBEDDED_PART, FORMAT_VERSION, MANIFEST_PART, SUMMARY_PART
+from .reader import open as open_document, open_embedded
+from .validator import validate, validate_embedded
+
+
+def _embedded_part(path: str, part: Optional[str]) -> Optional[str]:
+    """The part to read when the file is a package: the one asked for, or Footprint.pqax when there is no manifest."""
+    if part:
+        return part
+    try:
+        archive, fp, owned = open_archive(path)
+    except ContainerError:
+        return None
+    try:
+        if MANIFEST_PART in archive.namelist():
+            return None
+        info = find_entry(archive, EMBEDDED_PART)
+        return info.filename if info is not None else None
+    finally:
+        archive.close()
+        if owned:
+            fp.close()
+
+
+def _check(path: str, part: Optional[str]) -> int:
+    embedded = _embedded_part(path, part)
+    result = validate_embedded(path, embedded) if embedded else validate(path)
+    if embedded:
+        print(f"{path}: the PQAX file embedded as {embedded}")
+    for issue in result.issues:
+        print(issue)
+    verdict = "valid" if result.is_valid else f"invalid, {len(result.issues):,} issue(s)"
+    print(f"{path}: {verdict}, {result.checks:,} checks")
+    return 0 if result.is_valid else 1
+
+
+def _text(value: Any) -> str:
+    return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False)
+
+
+def _info(path: str, part: Optional[str]) -> int:
+    embedded = _embedded_part(path, part)
+    with (open_embedded(path, embedded) if embedded else open_document(path)) as document:
+        manifest = document.manifest
+        lines: List[str] = [f"file: {path}"]
+        if document.embedded_part:
+            lines.append(f"embedded part: {document.embedded_part}, {document.embedded_compression}")
+        lines.append(f"format: {_text(manifest.get('format'))}")
+        lines.append(f"formatVersion: {_text(manifest.get('formatVersion'))}")
+        collection = manifest.get("collection")
+        if isinstance(collection, dict):
+            lines.append("collection: " + ", ".join(f"{name} {_text(value)}" for name, value in collection.items()))
+        for member in ("obfuscatorDictionaryId", "obfuscatorLib", "obfuscatorLibVersion"):
+            if member in manifest:
+                lines.append(f"{member}: {_text(manifest[member])}")
+        others = [name for name in manifest if name not in
+                  ("format", "formatVersion", "collection", "parts", "obfuscatorDictionaryId", "obfuscatorLib",
+                   "obfuscatorLibVersion")]
+        if others:
+            lines.append("other members: " + ", ".join(others))
+        parts = manifest.get("parts") if isinstance(manifest.get("parts"), list) else []
+        entries = {info.filename: info for info in document._archive.infolist()}
+        lines.append(f"parts: {len(parts)}")
+        width = max((len(str(item.get("name"))) for item in parts if isinstance(item, dict)), default=0)
+        for item in parts:
+            if not isinstance(item, dict):
+                continue
+            name = str(item.get("name"))
+            tables = item.get("tables") if isinstance(item.get("tables"), dict) else {}
+            rows = ", ".join(f"{table} {count:,}" if type(count) is int else f"{table} {_text(count)}"
+                             for table, count in tables.items())
+            entry = entries.get(name)
+            size = (f"{entry.compress_size:>10,} bytes in the zip, {entry.file_size:>11,} of JSON"
+                    if entry is not None else "missing from the archive")
+            lines.append(f"  {name:<{width}}  {size}  {rows}")
+        total = {name: document.row_count(name) for name in document.table_names}
+        if total:
+            lines.append("rows: " + ", ".join(f"{name} {count:,}" for name, count in total.items()))
+        if SUMMARY_PART not in document.part_names:
+            lines.append("the manifest lists no summary.json")
+    print("\n".join(lines))
+    return 0
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    """Runs the command line and returns the exit code."""
+    for stream in (sys.stdout, sys.stderr):
+        try:
+            stream.reconfigure(errors="backslashreplace")  # type: ignore[union-attr]
+        except (AttributeError, ValueError):
+            pass
+    parser = argparse.ArgumentParser(prog="pqax", description="Check and describe PQAX files.")
+    parser.add_argument("--version", action="version",
+                        version=f"pqax {__version__}, reads format 0.x, writes format {FORMAT_VERSION}")
+    commands = parser.add_subparsers(dest="command", metavar="command")
+    commands.required = True
+    check = commands.add_parser("check", help="validate a file; exit code 0 when valid, 1 when not")
+    info = commands.add_parser("info", help="print the manifest: format, version, collection, and parts")
+    for command in (check, info):
+        command.add_argument("file", help="a .pqax file, or a .vpax file that embeds one")
+        command.add_argument("--part", help=f"the embedded part of a package (default: {EMBEDDED_PART} when the file "
+                                            f"has no manifest.json)")
+    arguments = parser.parse_args(argv)
+    try:
+        if arguments.command == "check":
+            return _check(arguments.file, arguments.part)
+        return _info(arguments.file, arguments.part)
+    except OSError as error:
+        print(f"pqax: {error}", file=sys.stderr)
+        return 2
+    except (PqaxError, zipfile.BadZipFile) as error:
+        print(f"pqax: {error}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())