tabalyst 0.4.4__tar.gz → 0.5.0__tar.gz

{tabalyst-0.4.4/src/tabalyst.egg-info → tabalyst-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tabalyst
-Version: 0.4.4
+Version: 0.5.0
 Summary: Open-source toolkit for understanding and working with unfamiliar data.
 Author: Gregory Borelli
 License-Expression: MIT
@@ -44,7 +44,8 @@ with structured data.
 Its first tool is **Tabalyst Report**. The current CSV implementation,
 **Tabalyst CSV Report**, analyzes a CSV file and produces both a structured JSON
 profile and a self-contained interactive HTML report.
-**Tabalyst Scan** describes CSV and JSON files in a scan document, and
+**Tabalyst Scan** describes CSV, JSON and JSONL files in a scan document,
+**Tabalyst Inspect** finds how to read a JSON or JSONL file, and
 **Tabalyst Sample** creates smaller CSV files.
 
 Tabalyst is in beta. Its interfaces may still change while
@@ -70,6 +71,7 @@ changes often: update it before each new test.
 | Several files, automatic names | `tabalyst report *.csv` | Beside each source |
 | Several files, one directory | `tabalyst report *.csv -d reports/` | The directory `reports/` |
 | One JSON file | `tabalyst report data.json` | `data.report.html` beside the source |
+| One JSONL file | `tabalyst report events.jsonl` | `events.report.html` beside the source |
 | From a scan document | `tabalyst report --scan data.scan.json` | `data.html` beside the scan |
 
 The simplest command keeps the source filename:
@@ -109,16 +111,23 @@ tabalyst report customers.csv -o customer-analysis.html
 
 ### JSON files
 
-A JSON file gets a report too, named `<stem>.report.html` so its profile
-`<stem>.report.json` never replaces the source. Each collection of records,
-such as the `customers` array of `{"customers": [...]}`, is a dataset of the
-report, and nested fields are columns named by their path, such as
-`address.city`:
+A JSON or JSONL file gets a report too, named `<stem>.report.html` so its
+profile `<stem>.report.json` never replaces the source. The report analyzes one
+collection of records, such as the `customers` array of
+`{"customers": [...]}`, chosen by Tabalyst Inspect, and nested fields are
+columns named by their path, such as `address.city`:
 
 ```console
 tabalyst report orders.json
+tabalyst report events.jsonl
 ```
 
+In a JSONL file (`.jsonl` or `.ndjson`), each line is a record. A line that is
+not valid JSON or not an object is excluded and counted, and the report is
+partial. When a JSON file holds several arrays that are equally plausible,
+Tabalyst stops and lists them instead of choosing: see
+[Tabalyst Inspect](#tabalyst-inspect).
+
 ### Multiple files
 
 Report several CSV files at once:
@@ -236,16 +245,46 @@ Sampling reads CSV records as a stream. Random and stratified sampling keep
 only the requested sample, plus stratum counts, in memory. Existing outputs
 require `--force`, and an input file is never overwritten.
 
+## Tabalyst Inspect
+
+A JSON file can hold several arrays. **Tabalyst Inspect** reads a JSON, JSONL
+or NDJSON file once, finds which array holds the records and writes its answer
+in `<source>-inspect.json` beside the source:
+
+```console
+tabalyst inspect orders.json
+```
+
+```text
+Inspect: orders.json-inspect.json
+Selection: $.customers[] (the only eligible collection)
+```
+
+The last section of that file, `config`, holds the rules that `tabalyst scan`
+and `tabalyst report` apply to the source: the collection
+(`structure.dataset_path`), the flatten settings, the array mode and the error
+policy. Edit it, then scan or report as usual. Running `tabalyst inspect` again
+refreshes the detection and keeps your `config`; `--reset-config` replaces it.
+
+Inspect is optional: `tabalyst scan` and `tabalyst report` inspect a JSON file
+themselves when it has no Inspect file. When several arrays are equally
+plausible, or none holds objects, they stop with exit code `2` before analyzing
+anything and list the candidates; set `config.structure.dataset_path` in the
+Inspect file, or pass `--collection`. See
+[Inspect JSON and JSONL files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/inspect-json-files.md)
+and the [Inspect format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/inspect-format.md).
+
 ## Tabalyst Scan
 
-Describe every field of a CSV or JSON file in one JSON document:
+Describe every field of a CSV, JSON or JSONL file in one JSON document:
 
 ```console
 tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
+tabalyst scan events.jsonl
 ```
 
-For a CSV, the default scan is a reusable `scan.json` under Tabalyst's local
+Without `-o` or `-d`, the scan is a reusable `scan.json` under Tabalyst's local
 storage directory (or `TABALYST_HOME`). The default workflow does not build a
 DuckDB database. `-o` or `-d` writes
 a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
@@ -260,10 +299,11 @@ interrupted scan never leaves a partial file. Files of 16 MiB or more are
 analyzed by several worker processes, with the same result; `--workers`
 chooses their number, `--workers 1` keeps one process.
 
-`tabalyst report customers.csv` reuses the verified stored scan when the CSV
-content and requested scan settings are current. It creates a scan when none
-exists and atomically replaces a stale one. The source check uses size,
-modification time and SHA-256. Existing DuckDB projects from 0.4.3 are left
+`tabalyst report customers.csv` reuses the verified stored scan when the
+source content and requested scan settings are current, and so does a report on
+a JSON or JSONL file. It creates a scan when none exists and atomically replaces
+a stale one. The source check uses the SHA-256 of the whole content, and a scan
+written by another version of Tabalyst is replaced. Existing DuckDB projects from 0.4.3 are left
 untouched.
 
 Build the report from a standalone scan document without reading the source again:
@@ -336,14 +376,21 @@ sample = tabalyst.sample_csv(
 
 scan = tabalyst.scan("orders.json")
 scans = tabalyst.generate_scans(["data/*.json"], output_dir="scans")
+
+inspection = tabalyst.inspect("orders.json")
+inspections = tabalyst.generate_inspections(["data/*.json", "logs/*.jsonl"])
 reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 ```
 
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
-`scan()` returns a scan result without writing anything; `generate_scans()`
-writes one standalone `.scan.json` document per source by default. Pass
-`project_storage=True` to use the command's scan-only storage for CSV files. `from_scan=True`
+`scan()` returns a scan result without writing anything, at the level of the
+engine: it does not read an Inspect file. `generate_scans()` writes one
+standalone `.scan.json` document per source by default, and applies Inspect to
+JSON and JSONL sources. Pass `project_storage=True` to use the command's
+scan-only storage. `inspect()` writes the Inspect file of one JSON or JSONL
+source and returns its path and document; `generate_inspections()` does it for
+several sources. `from_scan=True`
 builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
@@ -367,7 +414,8 @@ Configuration files are strict JSON. A minimal file is:
 ```
 
 Analysis settings, CSV reading included, go in the `scan` object, shared by
-`tabalyst report` and `tabalyst scan`. Top-level settings shape the report
+`tabalyst report`, `tabalyst scan` and `tabalyst inspect`. The `config` of an
+Inspect file outranks the `scan` object for the source it sits beside. Top-level settings shape the report
 presentation, and `csv` configures `tabalyst sample`. Explicit CLI or Python
 arguments override the configuration file, which overrides Tabalyst defaults.
 No configuration file is loaded unless it is passed with `--config`. See the
@@ -384,7 +432,9 @@ Reports, samples, and scans read files as a stream, with memory bounded by
 configurable limits, including duplicate-row detection
 (`scan.limits.max_tracked_records`). Reports and scans show the share of the file read;
 throughput estimates, recursive directory input, and parallel batch execution
-will require later work.
+will require later work. Inspect reads a JSON or JSONL source once from start to
+end, with little memory; on the synthetic benchmarks it took roughly a tenth of the
+time of a scan, a ratio that depends on the data.
 
 ## Examples and development
 
@@ -402,6 +452,7 @@ Additional documentation:
 
 - [Architecture](https://github.com/loribel-labs/tabalyst/blob/main/docs/dev/architecture.md)
 - [Configuration](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/configuration.md)
+- [Inspect JSON and JSONL files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/inspect-json-files.md)
 - [Release procedure](https://github.com/loribel-labs/tabalyst/blob/main/RELEASING.md)
 
 Please report defects and feature requests through the

{tabalyst-0.4.4 → tabalyst-0.5.0}/README.md

@@ -8,7 +8,8 @@ with structured data.
 Its first tool is **Tabalyst Report**. The current CSV implementation,
 **Tabalyst CSV Report**, analyzes a CSV file and produces both a structured JSON
 profile and a self-contained interactive HTML report.
-**Tabalyst Scan** describes CSV and JSON files in a scan document, and
+**Tabalyst Scan** describes CSV, JSON and JSONL files in a scan document,
+**Tabalyst Inspect** finds how to read a JSON or JSONL file, and
 **Tabalyst Sample** creates smaller CSV files.
 
 Tabalyst is in beta. Its interfaces may still change while
@@ -34,6 +35,7 @@ changes often: update it before each new test.
 | Several files, automatic names | `tabalyst report *.csv` | Beside each source |
 | Several files, one directory | `tabalyst report *.csv -d reports/` | The directory `reports/` |
 | One JSON file | `tabalyst report data.json` | `data.report.html` beside the source |
+| One JSONL file | `tabalyst report events.jsonl` | `events.report.html` beside the source |
 | From a scan document | `tabalyst report --scan data.scan.json` | `data.html` beside the scan |
 
 The simplest command keeps the source filename:
@@ -73,16 +75,23 @@ tabalyst report customers.csv -o customer-analysis.html
 
 ### JSON files
 
-A JSON file gets a report too, named `<stem>.report.html` so its profile
-`<stem>.report.json` never replaces the source. Each collection of records,
-such as the `customers` array of `{"customers": [...]}`, is a dataset of the
-report, and nested fields are columns named by their path, such as
-`address.city`:
+A JSON or JSONL file gets a report too, named `<stem>.report.html` so its
+profile `<stem>.report.json` never replaces the source. The report analyzes one
+collection of records, such as the `customers` array of
+`{"customers": [...]}`, chosen by Tabalyst Inspect, and nested fields are
+columns named by their path, such as `address.city`:
 
 ```console
 tabalyst report orders.json
+tabalyst report events.jsonl
 ```
 
+In a JSONL file (`.jsonl` or `.ndjson`), each line is a record. A line that is
+not valid JSON or not an object is excluded and counted, and the report is
+partial. When a JSON file holds several arrays that are equally plausible,
+Tabalyst stops and lists them instead of choosing: see
+[Tabalyst Inspect](#tabalyst-inspect).
+
 ### Multiple files
 
 Report several CSV files at once:
@@ -200,16 +209,46 @@ Sampling reads CSV records as a stream. Random and stratified sampling keep
 only the requested sample, plus stratum counts, in memory. Existing outputs
 require `--force`, and an input file is never overwritten.
 
+## Tabalyst Inspect
+
+A JSON file can hold several arrays. **Tabalyst Inspect** reads a JSON, JSONL
+or NDJSON file once, finds which array holds the records and writes its answer
+in `<source>-inspect.json` beside the source:
+
+```console
+tabalyst inspect orders.json
+```
+
+```text
+Inspect: orders.json-inspect.json
+Selection: $.customers[] (the only eligible collection)
+```
+
+The last section of that file, `config`, holds the rules that `tabalyst scan`
+and `tabalyst report` apply to the source: the collection
+(`structure.dataset_path`), the flatten settings, the array mode and the error
+policy. Edit it, then scan or report as usual. Running `tabalyst inspect` again
+refreshes the detection and keeps your `config`; `--reset-config` replaces it.
+
+Inspect is optional: `tabalyst scan` and `tabalyst report` inspect a JSON file
+themselves when it has no Inspect file. When several arrays are equally
+plausible, or none holds objects, they stop with exit code `2` before analyzing
+anything and list the candidates; set `config.structure.dataset_path` in the
+Inspect file, or pass `--collection`. See
+[Inspect JSON and JSONL files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/inspect-json-files.md)
+and the [Inspect format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/inspect-format.md).
+
 ## Tabalyst Scan
 
-Describe every field of a CSV or JSON file in one JSON document:
+Describe every field of a CSV, JSON or JSONL file in one JSON document:
 
 ```console
 tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
+tabalyst scan events.jsonl
 ```
 
-For a CSV, the default scan is a reusable `scan.json` under Tabalyst's local
+Without `-o` or `-d`, the scan is a reusable `scan.json` under Tabalyst's local
 storage directory (or `TABALYST_HOME`). The default workflow does not build a
 DuckDB database. `-o` or `-d` writes
 a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
@@ -224,10 +263,11 @@ interrupted scan never leaves a partial file. Files of 16 MiB or more are
 analyzed by several worker processes, with the same result; `--workers`
 chooses their number, `--workers 1` keeps one process.
 
-`tabalyst report customers.csv` reuses the verified stored scan when the CSV
-content and requested scan settings are current. It creates a scan when none
-exists and atomically replaces a stale one. The source check uses size,
-modification time and SHA-256. Existing DuckDB projects from 0.4.3 are left
+`tabalyst report customers.csv` reuses the verified stored scan when the
+source content and requested scan settings are current, and so does a report on
+a JSON or JSONL file. It creates a scan when none exists and atomically replaces
+a stale one. The source check uses the SHA-256 of the whole content, and a scan
+written by another version of Tabalyst is replaced. Existing DuckDB projects from 0.4.3 are left
 untouched.
 
 Build the report from a standalone scan document without reading the source again:
@@ -300,14 +340,21 @@ sample = tabalyst.sample_csv(
 
 scan = tabalyst.scan("orders.json")
 scans = tabalyst.generate_scans(["data/*.json"], output_dir="scans")
+
+inspection = tabalyst.inspect("orders.json")
+inspections = tabalyst.generate_inspections(["data/*.json", "logs/*.jsonl"])
 reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 ```
 
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
-`scan()` returns a scan result without writing anything; `generate_scans()`
-writes one standalone `.scan.json` document per source by default. Pass
-`project_storage=True` to use the command's scan-only storage for CSV files. `from_scan=True`
+`scan()` returns a scan result without writing anything, at the level of the
+engine: it does not read an Inspect file. `generate_scans()` writes one
+standalone `.scan.json` document per source by default, and applies Inspect to
+JSON and JSONL sources. Pass `project_storage=True` to use the command's
+scan-only storage. `inspect()` writes the Inspect file of one JSON or JSONL
+source and returns its path and document; `generate_inspections()` does it for
+several sources. `from_scan=True`
 builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
@@ -331,7 +378,8 @@ Configuration files are strict JSON. A minimal file is:
 ```
 
 Analysis settings, CSV reading included, go in the `scan` object, shared by
-`tabalyst report` and `tabalyst scan`. Top-level settings shape the report
+`tabalyst report`, `tabalyst scan` and `tabalyst inspect`. The `config` of an
+Inspect file outranks the `scan` object for the source it sits beside. Top-level settings shape the report
 presentation, and `csv` configures `tabalyst sample`. Explicit CLI or Python
 arguments override the configuration file, which overrides Tabalyst defaults.
 No configuration file is loaded unless it is passed with `--config`. See the
@@ -348,7 +396,9 @@ Reports, samples, and scans read files as a stream, with memory bounded by
 configurable limits, including duplicate-row detection
 (`scan.limits.max_tracked_records`). Reports and scans show the share of the file read;
 throughput estimates, recursive directory input, and parallel batch execution
-will require later work.
+will require later work. Inspect reads a JSON or JSONL source once from start to
+end, with little memory; on the synthetic benchmarks it took roughly a tenth of the
+time of a scan, a ratio that depends on the data.
 
 ## Examples and development
 
@@ -366,6 +416,7 @@ Additional documentation:
 
 - [Architecture](https://github.com/loribel-labs/tabalyst/blob/main/docs/dev/architecture.md)
 - [Configuration](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/configuration.md)
+- [Inspect JSON and JSONL files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/inspect-json-files.md)
 - [Release procedure](https://github.com/loribel-labs/tabalyst/blob/main/RELEASING.md)
 
 Please report defects and feature requests through the

{tabalyst-0.4.4 → tabalyst-0.5.0}/examples/README.md

@@ -9,7 +9,8 @@ examples/
 |-- input/
 |   |-- basic.csv
 |   |-- insurance-customers.csv
-|   `-- orders.json
+|   |-- orders.json
+|   `-- web-events.jsonl
 `-- output/
     |-- basic/
     |   |-- report.html
@@ -20,7 +21,11 @@ examples/
     |   |-- report.json
     |   |-- report/ (34 standalone column pages with --details)
     |   `-- executions.json
-    `-- orders/
+    |-- orders/
+    |   |-- report.html
+    |   |-- report.json
+    |   `-- executions.json
+    `-- web-events/
         |-- report.html
         |-- report.json
         `-- executions.json
@@ -30,8 +35,14 @@ examples/
 synthetic customer and contract records across 34 columns. Its people, contact
 details, and contracts are fictional; email addresses use reserved `.example`
 domains. `orders.json` is a synthetic shop export: 60 customers with nested
-addresses, tags and orders, whose report has two datasets, `$` and
-`$.customers[]`.
+addresses, tags and orders. Its report has one dataset, `$.customers[]`, the only
+collection of arrays of objects that Tabalyst Inspect finds in it.
+`web-events.jsonl` is a synthetic web analytics log in JSON Lines: 300 events with
+nested `user` and `device` objects, tags arrays, null values and an optional
+field that appears late, plus three bad lines (a truncated record, a line that
+is not JSON and a JSON array instead of an object). Its report is partial on
+purpose: the three lines are excluded and counted, the 300 events are analyzed.
+Its people and identifiers are fictional.
 
 From the repository root, regenerate the outputs with:
 
@@ -39,6 +50,7 @@ From the repository root, regenerate the outputs with:
 tabalyst report examples/input/basic.csv -o examples/output/basic/report.html --config examples/config.json --force
 tabalyst report examples/input/insurance-customers.csv -o examples/output/insurance-customers/report.html --config examples/config.json --details --force
 tabalyst report examples/input/orders.json -o examples/output/orders/report.html --config examples/config.json --force
+tabalyst report examples/input/web-events.jsonl -o examples/output/web-events/report.html --config examples/config.json --force
 ```
 
 Every output folder is independent and contains its own HTML report, canonical

{tabalyst-0.4.4 → tabalyst-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "tabalyst"
-version = "0.4.4"
+version = "0.5.0"
 description = "Open-source toolkit for understanding and working with unfamiliar data."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -69,6 +69,7 @@ testpaths = ["tests"]
 pythonpath = ["src"]
 markers = [
     "lot(name): Tabalyst Scan implementation lot that provides the tested behavior",
+    "inspect_lot(name): Tabalyst Inspect implementation lot that provides the tested behavior",
 ]
 
 [tool.ruff]

{tabalyst-0.4.4 → tabalyst-0.5.0}/src/tabalyst/__init__.py

@@ -2,6 +2,9 @@
 
 from tabalyst._version import __version__
 from tabalyst.errors import ConfigurationError, InputError, ReportError, TabalystError
+from tabalyst.inspect_service import InspectResult as InspectResult
+from tabalyst.inspect_service import generate_inspections as generate_inspections
+from tabalyst.inspect_service import inspect as inspect
 from tabalyst.report_config import ReportConfig as ReportConfig
 from tabalyst.report_service import generate_reports as generate_reports
 from tabalyst.reporting import render_report as render_report
@@ -20,15 +23,18 @@ from tabalyst.service import analyze_scan as analyze_scan
 __all__ = [
     "ConfigurationError",
     "InputError",
+    "InspectResult",
     "ReportError",
     "ScanConfig",
     "ScanResult",
     "TabalystError",
     "__version__",
     "analyze",
+    "generate_inspections",
     "generate_reports",
     "generate_samples",
     "generate_scans",
+    "inspect",
     "sample_csv",
     "scan",
 ]

{tabalyst-0.4.4 → tabalyst-0.5.0}/src/tabalyst/batch.py

@@ -13,6 +13,15 @@ from pathlib import Path
 
 from tabalyst.errors import ConfigurationError, InputError, ReportError
 
+# The visible Inspect file of a source is its full name plus this suffix
+# (design inspect 12.1). It is never a source itself.
+INSPECT_SUFFIX = "-inspect.json"
+
+
+def is_inspect_name(name: str) -> bool:
+    """Whether a file name is that of an Inspect file."""
+    return name.endswith(INSPECT_SUFFIX)
+
 
 def path_key(path: Path) -> str:
     """Identity of a path for comparisons, whatever its spelling and case."""
@@ -20,7 +29,11 @@ def path_key(path: Path) -> str:
 
 
 def resolve_input_specs(input_specs: Sequence[str | Path]) -> list[Path]:
-    """Resolve explicit files and shell-independent, non-recursive glob patterns."""
+    """Resolve explicit files and shell-independent, non-recursive glob patterns.
+
+    A pattern skips the Inspect files it matches; an explicit path to one is
+    refused (design inspect 12.1).
+    """
     if not input_specs:
         raise ConfigurationError("At least one input file is required.")
 
@@ -34,7 +47,11 @@ def resolve_input_specs(input_specs: Sequence[str | Path]) -> list[Path]:
             )
         if glob.has_magic(text):
             matches = sorted(
-                (Path(match) for match in glob.glob(text) if Path(match).is_file()),
+                (
+                    Path(match)
+                    for match in glob.glob(text)
+                    if Path(match).is_file() and not is_inspect_name(Path(match).name)
+                ),
                 key=path_key,
             )
             if not matches:
@@ -45,6 +62,11 @@ def resolve_input_specs(input_specs: Sequence[str | Path]) -> list[Path]:
                 raise InputError(f"Input file does not exist: {path}")
             if not path.is_file():
                 raise InputError(f"Input path is not a file: {path}")
+            if is_inspect_name(path.name):
+                raise ConfigurationError(
+                    f"{path} is an Inspect file, not a source. Give the source it "
+                    "describes; edit the Inspect file to change how it is read."
+                )
             matches = [path]
 
         for path in matches:

{tabalyst-0.4.4 → tabalyst-0.5.0}/src/tabalyst/cli/app.py

@@ -6,6 +6,7 @@ import typer
 
 from tabalyst import __version__
 from tabalyst.cli.cache import cache_app
+from tabalyst.cli.inspect import inspect_command
 from tabalyst.cli.report import report_command
 from tabalyst.cli.sample import sample_command
 from tabalyst.cli.scan import scan_command
@@ -39,6 +40,7 @@ def root(
     """Understand, explore, clean, and validate structured data."""
 
 
+app.command("inspect")(inspect_command)
 app.command("report")(report_command)
 app.command("sample")(sample_command)
 app.command("scan")(scan_command)
@@ -46,5 +48,10 @@ app.add_typer(cache_app, name="cache")
 
 
 def main() -> None:
-    """Run the Tabalyst command-line application."""
-    app()
+    """Run the Tabalyst command-line application.
+
+    Windows wildcard expansion is left to Tabalyst: its own pattern resolver
+    skips the Inspect files that a shell-expanded ``*.json`` would pass as
+    explicit paths.
+    """
+    app(windows_expand_args=False)