tabalyst 0.2.1__tar.gz → 0.4.0__tar.gz

{tabalyst-0.2.1/src/tabalyst.egg-info → tabalyst-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tabalyst
-Version: 0.2.1
+Version: 0.4.0
 Summary: Open-source toolkit for understanding and working with unfamiliar data.
 Author: Gregory Borelli
 License-Expression: MIT
@@ -21,9 +21,9 @@ Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: ijson>=3.3
 Requires-Dist: jinja2>=3.1
-Requires-Dist: pandas>=2.2
-Requires-Dist: pydantic>=2.10
+Requires-Dist: pydantic>=2.11
 Requires-Dist: typer>=0.12
 Provides-Extra: dev
 Requires-Dist: build>=1.2; extra == "dev"
@@ -51,9 +51,12 @@ the shared toolkit architecture is being established.
 Tabalyst supports Python 3.11, 3.12, 3.13, and 3.14.
 
 ```console
-pip install tabalyst
+pip install --upgrade tabalyst
 ```
 
+The same command installs Tabalyst or updates it. Tabalyst is in alpha and
+changes often: update it before each new test.
+
 ## Tabalyst Report
 
 | Use case | Command | Destination |
@@ -62,6 +65,8 @@ pip install tabalyst
 | One file, custom name | `tabalyst report data.csv -o report.html` | The file `report.html` |
 | Several files, automatic names | `tabalyst report *.csv` | Beside each source |
 | Several files, one directory | `tabalyst report *.csv -d reports/` | The directory `reports/` |
+| One JSON file | `tabalyst report data.json` | `data.report.html` beside the source |
+| From a scan document | `tabalyst report --scan data.scan.json` | `data.html` beside the scan |
 
 The simplest command keeps the source filename:
 
@@ -84,6 +89,18 @@ Use `-o` to choose a different HTML filename for one source:
 tabalyst report customers.csv -o customer-analysis.html
 ```
 
+### JSON files
+
+A JSON file gets a report too, named `<stem>.report.html` so its profile
+`<stem>.report.json` never replaces the source. Each collection of records,
+such as the `customers` array of `{"customers": [...]}`, is a dataset of the
+report, and nested fields are columns named by their path, such as
+`address.city`:
+
+```console
+tabalyst report orders.json
+```
+
 ### Multiple files
 
 Report several CSV files at once:
@@ -170,16 +187,81 @@ tabalyst --version
 
 `python -m tabalyst` accepts the same commands.
 
+## Tabalyst Sample
+
+Create a smaller CSV without modifying the source:
+
+```console
+tabalyst sample customers.csv --sample-method random --rows 1000 --seed 42
+```
+
+The default output is `customers.sample.csv` beside the source. Use `-o` to
+name the output for one input, or `-d` to sample several files into one
+directory:
+
+```console
+tabalyst sample customers.csv --sample-method first --rows 100 -o test.csv
+tabalyst sample *.csv --sample-method random --percent 5 -d samples/
+```
+
+Available methods are `first`, `last`, `random` and `stratified`. Stratified
+sampling approximately preserves the distribution of a selected field:
+
+```console
+tabalyst sample customers.csv --sample-method stratified --field province --rows 1000 --seed 42
+```
+
+Sampling reads CSV records as a stream. Random and stratified sampling keep
+only the requested sample, plus stratum counts, in memory. Existing outputs
+require `--force`, and an input file is never overwritten.
+
+## Tabalyst Scan
+
+Describe every field of a CSV or JSON file in one JSON document:
+
+```console
+tabalyst scan customers.csv
+tabalyst scan orders.json --collection "$.customers[]"
+```
+
+The default output is `customers.scan.json` beside the source; `-o`, `-d` and
+`--force` work as for the other commands. Tabalyst Scan reads the file once as
+a stream, with memory bounded by configurable limits, and records for each
+field its presence, native types, missing values, frequencies, exact
+statistics, normalization variants, technical type and the result of every
+detector: numbers with decimal commas, dates, booleans, enumerations, email
+addresses, URLs, phone numbers, postal codes, currency amounts, percentages,
+quantities, UUIDs and IP addresses. Values of sensitive fields, such as email
+addresses, are masked by default. Results are written atomically: an
+interrupted scan never leaves a partial file.
+
+Build the report from a scan document without reading the source again:
+
+```console
+tabalyst report --scan customers.scan.json
+```
+
+Tabalyst refuses a scan whose source changed since it was written, or whose
+settings differ from the `scan` settings of `--config`.
+
+See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
+and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
+
 ## What the report analyzes
 
 - Dataset dimensions, missing cells, duplicates, and quality observations.
 - Physical and semantic types with confidence and error rates.
 - Numeric, date, string-length, normalization, and value distributions.
-- Distinct values, representative examples, enum candidates, and date formats.
+- Distinct values, representative examples, date formats, and semantic types
+  such as enumerations, email addresses, phone numbers, and postal codes.
 - CSV record widths, quoting, encoding, and delimiter configuration.
 - A bounded raw-data preview while every record is analyzed.
 
-The HTML report is self-contained and works without a CDN or network connection.
+The report is built on Tabalyst Scan: it reads the CSV once as a stream and
+masks values of sensitive columns, such as email addresses, by default.
+Ambiguous dates stay ambiguous; the report shows the evidence of the column
+without applying it. The HTML report is self-contained and works without a CDN
+or network connection.
 The JSON profile contains the same canonical analysis result for scripts and
 future Tabalyst tools.
 
@@ -200,10 +282,24 @@ batch = tabalyst.generate_reports(
     ["*.csv"],
     output_dir="reports",
 )
+
+sample = tabalyst.sample_csv(
+    "customers.csv",
+    method="random",
+    rows=1000,
+    seed=42,
+)
+
+scan = tabalyst.scan("orders.json")
+scans = tabalyst.generate_scans(["data/*.json"], output_dir="scans")
+reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 ```
 
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
+`scan()` returns a scan result without writing anything; `generate_scans()`
+writes one `.scan.json` document per source; `from_scan=True` builds reports
+from such documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
 Existing artifacts are never replaced silently. Pass `force=True` when
@@ -215,16 +311,22 @@ Configuration files are strict JSON. A minimal file is:
 
 ```json
 {
-  "csv": {
-    "delimiter": ";",
-    "encoding": "cp1252"
+  "scan": {
+    "csv": {
+      "delimiter": ";",
+      "encoding": "cp1252"
+    },
+    "values": {"null_markers": ["N/A"]}
   }
 }
 ```
 
-Explicit CLI or Python arguments override the configuration file, which overrides
-Tabalyst defaults. See the
-[configuration reference](https://github.com/loribel-labs/tabalyst/blob/main/docs/configuration.md)
+Analysis settings, CSV reading included, go in the `scan` object, shared by
+`tabalyst report` and `tabalyst scan`. Top-level settings shape the report
+presentation, and `csv` configures `tabalyst sample`. Explicit CLI or Python
+arguments override the configuration file, which overrides Tabalyst defaults.
+No configuration file is loaded unless it is passed with `--config`. See the
+[configuration reference](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/configuration.md)
 for all analysis settings.
 
 ## Local-first behavior and current limits
@@ -233,9 +335,11 @@ Tabalyst performs analysis locally and adds no telemetry or remote processing.
 Generated JSON and HTML may contain source values and should be shared
 accordingly.
 
-The complete CSV is currently loaded into memory. File and phase progress is
-available now; row percentages, throughput estimates, recursive directory input,
-and parallel batch execution will require later ingestion work.
+Reports, samples, and scans read files as a stream, with memory bounded by
+configurable limits, including duplicate-row detection
+(`scan.limits.max_tracked_records`). Reports and scans show the share of the file read;
+throughput estimates, recursive directory input, and parallel batch execution
+will require later work.
 
 ## Examples and development
 
@@ -251,8 +355,8 @@ python -m build
 
 Additional documentation:
 
-- [Architecture](https://github.com/loribel-labs/tabalyst/blob/main/docs/architecture.md)
-- [Configuration](https://github.com/loribel-labs/tabalyst/blob/main/docs/configuration.md)
+- [Architecture](https://github.com/loribel-labs/tabalyst/blob/main/docs/dev/architecture.md)
+- [Configuration](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/configuration.md)
 - [Release procedure](https://github.com/loribel-labs/tabalyst/blob/main/RELEASING.md)
 
 Please report defects and feature requests through the

{tabalyst-0.2.1 → tabalyst-0.4.0}/README.md

@@ -17,9 +17,12 @@ the shared toolkit architecture is being established.
 Tabalyst supports Python 3.11, 3.12, 3.13, and 3.14.
 
 ```console
-pip install tabalyst
+pip install --upgrade tabalyst
 ```
 
+The same command installs Tabalyst or updates it. Tabalyst is in alpha and
+changes often: update it before each new test.
+
 ## Tabalyst Report
 
 | Use case | Command | Destination |
@@ -28,6 +31,8 @@ pip install tabalyst
 | One file, custom name | `tabalyst report data.csv -o report.html` | The file `report.html` |
 | Several files, automatic names | `tabalyst report *.csv` | Beside each source |
 | Several files, one directory | `tabalyst report *.csv -d reports/` | The directory `reports/` |
+| One JSON file | `tabalyst report data.json` | `data.report.html` beside the source |
+| From a scan document | `tabalyst report --scan data.scan.json` | `data.html` beside the scan |
 
 The simplest command keeps the source filename:
 
@@ -50,6 +55,18 @@ Use `-o` to choose a different HTML filename for one source:
 tabalyst report customers.csv -o customer-analysis.html
 ```
 
+### JSON files
+
+A JSON file gets a report too, named `<stem>.report.html` so its profile
+`<stem>.report.json` never replaces the source. Each collection of records,
+such as the `customers` array of `{"customers": [...]}`, is a dataset of the
+report, and nested fields are columns named by their path, such as
+`address.city`:
+
+```console
+tabalyst report orders.json
+```
+
 ### Multiple files
 
 Report several CSV files at once:
@@ -136,16 +153,81 @@ tabalyst --version
 
 `python -m tabalyst` accepts the same commands.
 
+## Tabalyst Sample
+
+Create a smaller CSV without modifying the source:
+
+```console
+tabalyst sample customers.csv --sample-method random --rows 1000 --seed 42
+```
+
+The default output is `customers.sample.csv` beside the source. Use `-o` to
+name the output for one input, or `-d` to sample several files into one
+directory:
+
+```console
+tabalyst sample customers.csv --sample-method first --rows 100 -o test.csv
+tabalyst sample *.csv --sample-method random --percent 5 -d samples/
+```
+
+Available methods are `first`, `last`, `random` and `stratified`. Stratified
+sampling approximately preserves the distribution of a selected field:
+
+```console
+tabalyst sample customers.csv --sample-method stratified --field province --rows 1000 --seed 42
+```
+
+Sampling reads CSV records as a stream. Random and stratified sampling keep
+only the requested sample, plus stratum counts, in memory. Existing outputs
+require `--force`, and an input file is never overwritten.
+
+## Tabalyst Scan
+
+Describe every field of a CSV or JSON file in one JSON document:
+
+```console
+tabalyst scan customers.csv
+tabalyst scan orders.json --collection "$.customers[]"
+```
+
+The default output is `customers.scan.json` beside the source; `-o`, `-d` and
+`--force` work as for the other commands. Tabalyst Scan reads the file once as
+a stream, with memory bounded by configurable limits, and records for each
+field its presence, native types, missing values, frequencies, exact
+statistics, normalization variants, technical type and the result of every
+detector: numbers with decimal commas, dates, booleans, enumerations, email
+addresses, URLs, phone numbers, postal codes, currency amounts, percentages,
+quantities, UUIDs and IP addresses. Values of sensitive fields, such as email
+addresses, are masked by default. Results are written atomically: an
+interrupted scan never leaves a partial file.
+
+Build the report from a scan document without reading the source again:
+
+```console
+tabalyst report --scan customers.scan.json
+```
+
+Tabalyst refuses a scan whose source changed since it was written, or whose
+settings differ from the `scan` settings of `--config`.
+
+See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
+and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
+
 ## What the report analyzes
 
 - Dataset dimensions, missing cells, duplicates, and quality observations.
 - Physical and semantic types with confidence and error rates.
 - Numeric, date, string-length, normalization, and value distributions.
-- Distinct values, representative examples, enum candidates, and date formats.
+- Distinct values, representative examples, date formats, and semantic types
+  such as enumerations, email addresses, phone numbers, and postal codes.
 - CSV record widths, quoting, encoding, and delimiter configuration.
 - A bounded raw-data preview while every record is analyzed.
 
-The HTML report is self-contained and works without a CDN or network connection.
+The report is built on Tabalyst Scan: it reads the CSV once as a stream and
+masks values of sensitive columns, such as email addresses, by default.
+Ambiguous dates stay ambiguous; the report shows the evidence of the column
+without applying it. The HTML report is self-contained and works without a CDN
+or network connection.
 The JSON profile contains the same canonical analysis result for scripts and
 future Tabalyst tools.
 
@@ -166,10 +248,24 @@ batch = tabalyst.generate_reports(
     ["*.csv"],
     output_dir="reports",
 )
+
+sample = tabalyst.sample_csv(
+    "customers.csv",
+    method="random",
+    rows=1000,
+    seed=42,
+)
+
+scan = tabalyst.scan("orders.json")
+scans = tabalyst.generate_scans(["data/*.json"], output_dir="scans")
+reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 ```
 
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
+`scan()` returns a scan result without writing anything; `generate_scans()`
+writes one `.scan.json` document per source; `from_scan=True` builds reports
+from such documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
 Existing artifacts are never replaced silently. Pass `force=True` when
@@ -181,16 +277,22 @@ Configuration files are strict JSON. A minimal file is:
 
 ```json
 {
-  "csv": {
-    "delimiter": ";",
-    "encoding": "cp1252"
+  "scan": {
+    "csv": {
+      "delimiter": ";",
+      "encoding": "cp1252"
+    },
+    "values": {"null_markers": ["N/A"]}
   }
 }
 ```
 
-Explicit CLI or Python arguments override the configuration file, which overrides
-Tabalyst defaults. See the
-[configuration reference](https://github.com/loribel-labs/tabalyst/blob/main/docs/configuration.md)
+Analysis settings, CSV reading included, go in the `scan` object, shared by
+`tabalyst report` and `tabalyst scan`. Top-level settings shape the report
+presentation, and `csv` configures `tabalyst sample`. Explicit CLI or Python
+arguments override the configuration file, which overrides Tabalyst defaults.
+No configuration file is loaded unless it is passed with `--config`. See the
+[configuration reference](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/configuration.md)
 for all analysis settings.
 
 ## Local-first behavior and current limits
@@ -199,9 +301,11 @@ Tabalyst performs analysis locally and adds no telemetry or remote processing.
 Generated JSON and HTML may contain source values and should be shared
 accordingly.
 
-The complete CSV is currently loaded into memory. File and phase progress is
-available now; row percentages, throughput estimates, recursive directory input,
-and parallel batch execution will require later ingestion work.
+Reports, samples, and scans read files as a stream, with memory bounded by
+configurable limits, including duplicate-row detection
+(`scan.limits.max_tracked_records`). Reports and scans show the share of the file read;
+throughput estimates, recursive directory input, and parallel batch execution
+will require later work.
 
 ## Examples and development
 
@@ -217,8 +321,8 @@ python -m build
 
 Additional documentation:
 
-- [Architecture](https://github.com/loribel-labs/tabalyst/blob/main/docs/architecture.md)
-- [Configuration](https://github.com/loribel-labs/tabalyst/blob/main/docs/configuration.md)
+- [Architecture](https://github.com/loribel-labs/tabalyst/blob/main/docs/dev/architecture.md)
+- [Configuration](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/configuration.md)
 - [Release procedure](https://github.com/loribel-labs/tabalyst/blob/main/RELEASING.md)
 
 Please report defects and feature requests through the

{tabalyst-0.2.1 → tabalyst-0.4.0}/examples/README.md

@@ -8,13 +8,18 @@ examples/
 |-- config.json
 |-- input/
 |   |-- basic.csv
-|   `-- insurance-customers.csv
+|   |-- insurance-customers.csv
+|   `-- orders.json
 `-- output/
     |-- basic/
     |   |-- report.html
     |   |-- report.json
     |   `-- executions.json
-    `-- insurance-customers/
+    |-- insurance-customers/
+    |   |-- report.html
+    |   |-- report.json
+    |   `-- executions.json
+    `-- orders/
         |-- report.html
         |-- report.json
         `-- executions.json
@@ -23,13 +28,16 @@ examples/
 `basic.csv` is a five-row smoke example. `insurance-customers.csv` contains 3,000
 synthetic customer and contract records across 34 columns. Its people, contact
 details, and contracts are fictional; email addresses use reserved `.example`
-domains.
+domains. `orders.json` is a synthetic shop export: 60 customers with nested
+addresses, tags and orders, whose report has two datasets, `$` and
+`$.customers[]`.
 
-From the repository root, regenerate both outputs with:
+From the repository root, regenerate the outputs with:
 
 ```console
 tabalyst report examples/input/basic.csv -o examples/output/basic/report.html --config examples/config.json --force
 tabalyst report examples/input/insurance-customers.csv -o examples/output/insurance-customers/report.html --config examples/config.json --force
+tabalyst report examples/input/orders.json -o examples/output/orders/report.html --config examples/config.json --force
 ```
 
 Every output folder is independent and contains its own HTML report, canonical

tabalyst-0.4.0/examples/config.json

@@ -0,0 +1,60 @@
+{
+  "string_analysis": {
+    "very_short_max_length": 5,
+    "short_max_length": 20,
+    "medium_max_length": 50,
+    "long_max_length": 255,
+    "length_distribution_max_length": 50,
+    "examples_per_length": 10
+  },
+  "value_examples": {
+    "full_distribution_max_distinct": 50,
+    "short_text_max_length": 20,
+    "short_text_percentile": 0.95,
+    "short_text_result_size": 20,
+    "long_text_result_size": 20,
+    "long_text_truncate_at": 30,
+    "truncation_suffix": "...",
+    "inline_display_size": 3
+  },
+  "scan": {
+    "csv": {
+      "encoding": "utf-8-sig",
+      "delimiter": ","
+    },
+    "values": {
+      "null_markers": ["N/A", "NULL"],
+      "missing": ["absent", "null", "empty", "blank", "marker"]
+    },
+    "normalization": {
+      "trim": true,
+      "collapse_whitespace": true
+    },
+    "types": {
+      "minimum_confidence": 0.95
+    },
+    "detectors": {
+      "date": {
+        "orders": ["YMD", "MDY", "DMY"],
+        "separators": ["-", "/", "."],
+        "ambiguous_order": null
+      },
+      "enumeration": {
+        "minimum_values": 500,
+        "maximum_distinct": 49,
+        "case_sensitive": true
+      }
+    },
+    "limits": {
+      "max_samples": 100
+    },
+    "records": {
+      "preview": 20,
+      "duplicates": true
+    },
+    "exposure": {
+      "sensitive_values": "mask"
+    },
+    "random_seed": 42
+  }
+}

{tabalyst-0.2.1 → tabalyst-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "tabalyst"
-version = "0.2.1"
+version = "0.4.0"
 description = "Open-source toolkit for understanding and working with unfamiliar data."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -34,9 +34,9 @@ classifiers = [
     "Topic :: Scientific/Engineering :: Information Analysis",
 ]
 dependencies = [
+    "ijson>=3.3",
     "jinja2>=3.1",
-    "pandas>=2.2",
-    "pydantic>=2.10",
+    "pydantic>=2.11",
     "typer>=0.12",
 ]
 
@@ -65,6 +65,9 @@ Issues = "https://github.com/loribel-labs/tabalyst/issues"
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 pythonpath = ["src"]
+markers = [
+    "lot(name): Tabalyst Scan implementation lot that provides the tested behavior",
+]
 
 [tool.ruff]
 line-length = 88

tabalyst-0.4.0/src/tabalyst/__init__.py

@@ -0,0 +1,34 @@
+"""Public Python interface for Tabalyst."""
+
+from tabalyst._version import __version__
+from tabalyst.errors import ConfigurationError, InputError, ReportError, TabalystError
+from tabalyst.report_config import ReportConfig as ReportConfig
+from tabalyst.report_service import generate_reports as generate_reports
+from tabalyst.reporting import render_report as render_report
+from tabalyst.sampling import SampleMethod as SampleMethod
+from tabalyst.sampling import SampleResult as SampleResult
+from tabalyst.sampling import sample_csv as sample_csv
+from tabalyst.sampling_service import generate_samples as generate_samples
+from tabalyst.scan_service import generate_scans as generate_scans
+from tabalyst.scanner import ScanConfig as ScanConfig
+from tabalyst.scanner import ScanResult as ScanResult
+from tabalyst.scanner import scan as scan
+from tabalyst.service import analyze
+from tabalyst.service import analyze_csv as analyze_csv
+from tabalyst.service import analyze_scan as analyze_scan
+
+__all__ = [
+    "ConfigurationError",
+    "InputError",
+    "ReportError",
+    "ScanConfig",
+    "ScanResult",
+    "TabalystError",
+    "__version__",
+    "analyze",
+    "generate_reports",
+    "generate_samples",
+    "generate_scans",
+    "sample_csv",
+    "scan",
+]