tabalyst 0.6.2__tar.gz → 0.7.1__tar.gz

tabalyst-0.7.1/MANIFEST.in

@@ -0,0 +1,14 @@
+include LICENSE
+include README.md
+include examples/README.md
+include examples/config.json
+recursive-include examples/datasets *.csv
+include examples/generate/README.md
+include examples/generate/generate_examples.py
+recursive-include src/tabalyst/generate_examples/crm *.yaml
+recursive-include src/tabalyst/generate_examples/insurance *.yaml *.csv *.md
+recursive-include src/tabalyst/generate_examples/computer *.yaml *.csv *.md
+exclude src/tabalyst/generate_examples/computer/data/csv_files.md
+recursive-include examples/generate/common-references *.yaml
+recursive-include examples/generate/cohorts-anomalies *.yaml
+recursive-include src/tabalyst/generate_resources *.csv *.md

tabalyst-0.7.1/PKG-INFO

@@ -0,0 +1,382 @@
+Metadata-Version: 2.4
+Name: tabalyst
+Version: 0.7.1
+Summary: Open-source toolkit for understanding and working with unfamiliar data.
+Author: Gregory Borelli
+License-Expression: MPL-2.0
+Project-URL: Homepage, https://github.com/loribel-labs/tabalyst
+Project-URL: Repository, https://github.com/loribel-labs/tabalyst.git
+Project-URL: Issues, https://github.com/loribel-labs/tabalyst/issues
+Keywords: csv,data-analysis,data-profiling,data-quality,eda,html-report,json
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: duckdb==1.5.5
+Requires-Dist: ijson>=3.3
+Requires-Dist: jinja2>=3.1
+Requires-Dist: platformdirs>=4
+Requires-Dist: pydantic>=2.11
+Requires-Dist: python-calamine>=0.8
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: jsonschema>=4; extra == "dev"
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-cov>=6; extra == "dev"
+Requires-Dist: ruff>=0.11; extra == "dev"
+Requires-Dist: xlsxwriter>=3; extra == "dev"
+Dynamic: license-file
+
+# Tabalyst
+
+**Open-source Python toolkit for data profiling and data quality analysis.**
+
+*Tools for unfamiliar data.*
+
+Tabalyst helps you understand CSV, Excel, JSON and JSONL datasets with minimal
+configuration. Detect missing values, duplicates, mixed types, data formats and
+quality issues. Generate interactive HTML reports and structured JSON profiles,
+entirely on your computer.
+
+**Simple. Clear. Smart.**
+
+## Quick start
+
+Tabalyst supports Python 3.11, 3.12, 3.13, and 3.14.
+
+```console
+pip install --upgrade tabalyst
+tabalyst report insurance-customers.csv
+```
+
+Save [`insurance-customers.csv`](https://github.com/loribel-labs/tabalyst/blob/main/examples/datasets/insurance-customers.csv),
+a synthetic example, in your working folder, or use your own file. Explore the
+interactive HTML report or use the generated JSON profile in your own
+workflows. The same `pip` command installs Tabalyst or updates it.
+
+Tabalyst is in beta. Commands and interfaces may change between releases.
+Update before each new test.
+
+## Tabalyst Websites
+
+- [tabalyst.com](https://tabalyst.com/): the official website, with an interactive demo.
+- [docs.tabalyst.com](https://docs.tabalyst.com/): the documentation, for people and for AI assistants.
+- [Interactive report](https://tabalyst.com/examples/report.html): a real Tabalyst report you can explore in your browser.
+
+## What can Tabalyst do?
+
+| Tool | Purpose |
+| --- | --- |
+| [**Tabalyst Report**](https://docs.tabalyst.com/report/) | Generate interactive HTML profiling reports and JSON profiles |
+| [**Tabalyst Scan**](https://docs.tabalyst.com/scan/) | Profile datasets and generate JSON and HTML scan summaries |
+| [**Tabalyst Inspect**](https://docs.tabalyst.com/inspect/) | Discover and configure JSON collections and Excel tables |
+| [**Tabalyst Sample**](https://docs.tabalyst.com/sample/) | Create random, stratified and smart CSV samples |
+| [**Tabalyst Dedup**](https://docs.tabalyst.com/dedup/) | Find the exact duplicate rows of CSV files and write the unique and repeated rows |
+| [**Tabalyst Generate**](https://docs.tabalyst.com/generate/) | Generate synthetic CSV datasets for testing and demos |
+
+## Tabalyst Report
+
+| Use case | Command | Destination |
+| --- | --- | --- |
+| One file, automatic name | `tabalyst report data.csv` | `data.html` beside the source |
+| One file, custom name | `tabalyst report data.csv -o report.html` | The file `report.html` |
+| Several files, automatic names | `tabalyst report *.csv` | Beside each source |
+| Several files, one directory | `tabalyst report *.csv -d reports/` | The directory `reports/` |
+| One JSON file | `tabalyst report data.json` | `data.report.html` beside the source |
+| One JSONL file | `tabalyst report events.jsonl` | `events.report.html` beside the source |
+| One Excel workbook | `tabalyst report sales.xlsx` | `sales.report.html` beside the source |
+| Every table of a workbook | `tabalyst report shop.xlsx --all-collections` | `shop.costs.html`, `shop.sales.html`, ... beside the source |
+| From a scan document | `tabalyst report --scan data.scan.json` | `data.html` beside the scan |
+
+The simplest command keeps the source filename:
+
+```console
+tabalyst report customers.csv
+```
+
+It creates these files beside the source:
+
+```text
+customers.csv
+customers.html
+customers.json
+executions.json
+```
+
+Add `--details` to generate one self-contained HTML page per column, and `-o`
+to choose another HTML filename for one source:
+
+```console
+tabalyst report customers.csv --details
+tabalyst report customers.csv -o customer-analysis.html
+```
+
+Existing outputs are never replaced silently: add `--force` when replacement is
+intentional. A batch is checked for name collisions before any file is
+processed, and a malformed file does not stop the others.
+
+JSON, JSONL and Excel sources work the same way. When a JSON file or a workbook
+holds several equally plausible collections, Tabalyst stops and prints one
+command per candidate to copy, such as
+`tabalyst report shop.xlsx --collection Costs`.
+
+Useful options: `--delimiter`, `--encoding`, `--config`, `--workers`,
+`--verbose`, `--quiet` and `--help`. `python -m tabalyst` accepts the same
+commands.
+
+Details: [Tabalyst Report](https://docs.tabalyst.com/report/),
+[JSON and JSONL](https://docs.tabalyst.com/report/json/),
+[Excel](https://docs.tabalyst.com/report/excel/),
+[CSV batches and options](https://docs.tabalyst.com/report/csv/) and the
+[`report` command reference](https://docs.tabalyst.com/reference/cli/report/).
+
+### What the report analyzes
+
+- Dataset dimensions, missing cells, duplicates, and quality observations.
+- A family and a type for each column (identifier, text, date, boolean...),
+  with the evidence behind them, and the technical type with its error rates.
+- A summary table with one row per column: its type, a sentence about its
+  values, the missing share, its Errors and Issues with a preview of each, and
+  its details; above it, the Column quality and Signals cards.
+- Numeric, date, string-length, normalization, and value distributions.
+- Distinct values, representative examples, date formats, and semantic types
+  such as enumerations, email addresses, phone numbers, and postal codes.
+- CSV record widths, quoting, encoding, and delimiter configuration.
+- A bounded raw-data preview while every record is analyzed.
+
+The report is built on Tabalyst Scan; values of sensitive columns can be masked
+or hidden. Ambiguous dates stay
+ambiguous; the report shows the evidence of the column without applying it. The
+HTML report is self-contained and works without a CDN or network connection.
+The JSON profile contains the same canonical analysis result for scripts and
+future Tabalyst tools. See
+[how to read the report](https://docs.tabalyst.com/report/read-report/).
+
+## Tabalyst Scan
+
+Describe every field of a CSV, JSON, JSONL or Excel file in one JSON document:
+
+```console
+tabalyst scan customers.csv
+tabalyst scan orders.json --collection "$.customers[]"
+tabalyst scan sales.xlsx --collection Orders
+```
+
+Scan reads a text file once as a stream, with bounded memory. An Excel sheet is
+loaded into memory. It records for each
+field its types, missing values, frequencies, statistics and the result of every
+detector (dates, numbers with decimal commas, emails, URLs, phone numbers,
+postal codes, currencies, UUIDs, IP addresses and more). It also writes
+`customers.scan.html`, a page that shows the scan (`--no-html` skips it).
+
+A second reading can provide exact distinct counts, frequencies, histograms,
+medians, patterns, correlations and more, within the limits of your data. `--plan` shows what it would add and
+cost without running it, and `--deep` runs it:
+
+```console
+tabalyst scan customers.csv --plan
+tabalyst scan customers.csv --deep
+```
+
+`tabalyst report` decides by itself whether that second reading is worth its
+cost, and says what it added. `tabalyst report` also reuses a stored scan
+while the source is unchanged.
+
+Details: [Scan CSV and JSON files](https://docs.tabalyst.com/scan/files/),
+[the second reading](https://docs.tabalyst.com/scan/deep/),
+[reading exact, estimated and limited results](https://docs.tabalyst.com/scan/reading-results/),
+[scan storage and cache](https://docs.tabalyst.com/scan/cache/) and the
+[scan format](https://docs.tabalyst.com/scan/format/).
+
+## Tabalyst Inspect
+
+A JSON file can hold several arrays, and an Excel workbook several sheets.
+**Tabalyst Inspect** finds which array, sheet or table holds the records and
+writes its answer in `<source>-inspect.json` beside the source:
+
+```console
+tabalyst inspect orders.json
+tabalyst inspect sales.xlsx
+```
+
+The `config` section of that file holds the rules that `tabalyst scan` and
+`tabalyst report` apply (collection, flatten settings, header row). Inspect is
+optional: both commands inspect a source themselves, and stop with exit code
+`2` and a list of candidates when the choice is ambiguous. Pass
+`--collection` to choose.
+
+Details: [Inspect JSON and JSONL files](https://docs.tabalyst.com/inspect/json/),
+[Inspect Excel workbooks](https://docs.tabalyst.com/inspect/excel/) and the
+[Inspect format](https://docs.tabalyst.com/inspect/format/).
+
+## Tabalyst Sample
+
+Create a smaller CSV without modifying the source:
+
+```console
+tabalyst sample customers.csv --sample-method random --rows 1000 --seed 42
+tabalyst sample customers.csv --smart --rows 1000
+```
+
+Methods are `first`, `last`, `random`, `stratified` and `smart`. A smart sample
+keeps the diversity of the file (rare values, extremes, formats, odd cases)
+instead of its proportions, and writes a report of what it covers, so never
+estimate the population from it. Sampling reads records as a stream.
+
+Details: [Tabalyst Sample](https://docs.tabalyst.com/sample/).
+
+## Tabalyst Dedup
+
+Find the rows that occur more than once in a CSV file, without modifying it:
+
+```console
+tabalyst dedup customers.csv
+tabalyst dedup "*.csv" -d deduped
+```
+
+Two rows are the same only when every cell is identical (case, spaces and
+leading zeros count). Dedup writes, beside the source or in `-d`,
+`customers.unique.csv` (every distinct row once, at its first place),
+`customers.duplicates.csv` (each repeated row once) and, when a row occurs three
+times or more, `customers.duplicates_all.csv` (every occurrence of the repeated
+rows), plus `customers.dedup.json`, a report whose counts always add up. A file
+without duplicate gets its report only. A batch also writes
+`dedup.summary.json`; a file that is already processed and unchanged is skipped.
+
+A result is exact or the file fails with a reason: the rows are kept in memory up
+to a budget (`--memory-budget-mb`, 512 by default) and then on disk, a file may
+hold 10,000,000 records by default (`--max-rows`), and nothing is ever
+approximated. Dedup reads CSV files only and asks no question.
+
+Details: [Tabalyst Dedup](https://docs.tabalyst.com/dedup/),
+[batches and already processed files](https://docs.tabalyst.com/dedup/batch/),
+[exactness, memory and limits](https://docs.tabalyst.com/dedup/limits/) and the
+[`dedup` command reference](https://docs.tabalyst.com/reference/cli/dedup/).
+
+## Tabalyst Generate
+
+Generate synthetic CSV data from a packaged example (`crm`, `insurance` or
+`computer`) or from a Generate package:
+
+```console
+tabalyst generate --example crm -o crm-contacts.csv
+tabalyst generate --example insurance --rows 3000 -d insurance-output/
+tabalyst generate --example computer --rows 10000 -d computer-output/
+tabalyst generate --example insurance --rows 3000 --anomaly-profile dirty_realistic -d insurance-dirty/
+```
+
+An anomaly profile writes an altered CSV, a cell change log and a summary
+beside the untouched clean CSV.
+
+Details: [Tabalyst Generate](https://docs.tabalyst.com/generate/),
+[`examples/generate/README.md`](examples/generate/README.md) and the
+[Computer package contract](docs/dev/generate/computer.md).
+
+## Python API
+
+The same operations are available without the CLI:
+
+```python
+import tabalyst
+
+result = tabalyst.analyze("customers.csv", "customers.html")
+batch = tabalyst.generate_reports(["*.csv"], output_dir="reports")
+sample = tabalyst.sample_csv("customers.csv", method="random", rows=1000, seed=42)
+dedup = tabalyst.dedup_csv("customers.csv")
+scan = tabalyst.scan("orders.json")
+inspection = tabalyst.inspect("orders.json")
+```
+
+Existing artifacts are never replaced silently: pass `force=True` when
+replacement is intentional. Expected failures derive from
+`tabalyst.TabalystError`. See the
+[Python API reference](https://docs.tabalyst.com/reference/python-api/).
+
+## Configuration
+
+Configuration files are strict JSON, loaded only with `--config`:
+
+```json
+{
+  "scan": {
+    "csv": {"delimiter": ";", "encoding": "cp1252"},
+    "values": {"null_markers": ["N/A"]}
+  }
+}
+```
+
+Explicit CLI or Python arguments override the configuration file, which
+overrides Tabalyst defaults. See the
+[configuration reference](https://docs.tabalyst.com/reference/configuration/).
+
+## Local-first behavior and current limits
+
+Tabalyst performs analysis locally and adds no telemetry or remote processing.
+Generated JSON and HTML may contain source values and should be shared
+accordingly.
+
+Reports, samples, and scans read CSV, JSON and JSONL files as a stream, with
+memory bounded by configurable limits; Dedup keeps the rows it has seen in memory
+up to a budget, then on disk. An Excel workbook is read by sheet into
+memory, and a sheet above 1 GiB of XML is refused. See
+[limitations](https://docs.tabalyst.com/about/limitations/).
+
+## Examples and development
+
+The repository includes small and synthetic public examples under `examples/`.
+See the [examples README](https://github.com/loribel-labs/tabalyst/blob/main/examples/README.md)
+for regeneration commands.
+
+```console
+python -m pytest
+python -m ruff check .
+python -m build
+```
+
+Additional documentation:
+
+- [Documentation site](https://docs.tabalyst.com/)
+- [Architecture](https://github.com/loribel-labs/tabalyst/blob/main/docs/dev/architecture.md)
+- [Release procedure](https://github.com/loribel-labs/tabalyst/blob/main/RELEASING.md)
+
+Please report defects and feature requests through the
+[GitHub issue tracker](https://github.com/loribel-labs/tabalyst/issues).
+
+## For AI assistants
+
+To get help from an AI assistant, paste this prompt into a chat that can read
+web pages:
+
+```text
+Read the complete Tabalyst documentation at
+https://docs.tabalyst.com/llms-full.txt, then help me use Tabalyst well.
+Ask me what my data looks like and what I want to learn from it, then suggest
+the right commands and options. Base your answers only on that documentation.
+```
+
+If the assistant cannot open links, paste the content of the file instead.
+
+The documentation site publishes plain-text versions of its pages:
+[llms.txt](https://docs.tabalyst.com/llms.txt) lists every page with a
+one-line description, and
+[llms-full.txt](https://docs.tabalyst.com/llms-full.txt) holds the whole
+documentation in one file. Every page is also available as Markdown by adding
+`.md` to its URL, for example
+[report.md](https://docs.tabalyst.com/report.md).
+
+## License
+
+Tabalyst is released under the [Mozilla Public License 2.0](LICENSE).
+
+Created by Gregory Borelli. Open source project by Loribel Labs.