tabalyst 0.4.2__tar.gz → 0.4.4__tar.gz

{tabalyst-0.4.2/src/tabalyst.egg-info → tabalyst-0.4.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tabalyst
-Version: 0.4.2
+Version: 0.4.4
 Summary: Open-source toolkit for understanding and working with unfamiliar data.
 Author: Gregory Borelli
 License-Expression: MIT
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://github.com/loribel-labs/tabalyst
 Project-URL: Repository, https://github.com/loribel-labs/tabalyst.git
 Project-URL: Issues, https://github.com/loribel-labs/tabalyst/issues
 Keywords: csv,data-analysis,data-profiling,data-quality,eda,html-report,json
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
@@ -21,8 +21,10 @@ Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: duckdb==1.5.5
 Requires-Dist: ijson>=3.3
 Requires-Dist: jinja2>=3.1
+Requires-Dist: platformdirs>=4
 Requires-Dist: pydantic>=2.11
 Requires-Dist: typer>=0.12
 Provides-Extra: dev
@@ -42,8 +44,10 @@ with structured data.
 Its first tool is **Tabalyst Report**. The current CSV implementation,
 **Tabalyst CSV Report**, analyzes a CSV file and produces both a structured JSON
 profile and a self-contained interactive HTML report.
+**Tabalyst Scan** describes CSV and JSON files in a scan document, and
+**Tabalyst Sample** creates smaller CSV files.
 
-Tabalyst is in active alpha development. Its interfaces may still change while
+Tabalyst is in beta. Its interfaces may still change while
 the shared toolkit architecture is being established.
 
 ## Install
@@ -54,7 +58,7 @@ Tabalyst supports Python 3.11, 3.12, 3.13, and 3.14.
 pip install --upgrade tabalyst
 ```
 
-The same command installs Tabalyst or updates it. Tabalyst is in alpha and
+The same command installs Tabalyst or updates it. Tabalyst is in beta and
 changes often: update it before each new test.
 
 ## Tabalyst Report
@@ -83,6 +87,20 @@ customers.json
 executions.json
 ```
 
+Add `--details` to generate one self-contained HTML page per column:
+
+```console
+tabalyst report customers.csv --details
+```
+
+The **Columns** table then links to those pages. Each page has its own sidebar
+and shows values, counts, detected formats and other analysis directly. The
+subfolder takes the HTML filename without `.html`; for `-o report.html`, pages
+are in `report/`. Names use `col-01-` and a shortened slug of the column name,
+in source order. Details are off by default. `--no-details` states that choice
+explicitly; with `--force`, it removes prior generated column pages while
+preserving other files in the folder.
+
 Use `-o` to choose a different HTML filename for one source:
 
 ```console
@@ -153,8 +171,10 @@ instead.
 
 ### Safe batch behavior
 
-Before processing begins, Tabalyst resolves every input and planned output. It
-stops the entire batch if output names collide or if an output already exists.
+Before processing begins, Tabalyst resolves every input and planned main HTML
+and JSON output. It stops the entire batch if those names collide or if an
+output already exists. With `--details`, each column page is checked after its
+profile has been built, before that report's files are written.
 Use `--force` only when replacing all matching report artifacts is intentional:
 
 ```console
@@ -225,8 +245,10 @@ tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
 ```
 
-The default output is `customers.scan.json` beside the source; `-o`, `-d` and
-`--force` work as for the other commands. Tabalyst Scan reads the file once as
+For a CSV, the default scan is a reusable `scan.json` under Tabalyst's local
+storage directory (or `TABALYST_HOME`). The default workflow does not build a
+DuckDB database. `-o` or `-d` writes
+a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
 a stream, with memory bounded by configurable limits, and records for each
 field its presence, native types, missing values, frequencies, exact
 statistics, normalization variants, technical type and the result of every
@@ -238,7 +260,13 @@ interrupted scan never leaves a partial file. Files of 16 MiB or more are
 analyzed by several worker processes, with the same result; `--workers`
 chooses their number, `--workers 1` keeps one process.
 
-Build the report from a scan document without reading the source again:
+`tabalyst report customers.csv` reuses the verified stored scan when the CSV
+content and requested scan settings are current. It creates a scan when none
+exists and atomically replaces a stale one. The source check uses size,
+modification time and SHA-256. Existing DuckDB projects from 0.4.3 are left
+untouched.
+
+Build the report from a standalone scan document without reading the source again:
 
 ```console
 tabalyst report --scan customers.scan.json
@@ -247,6 +275,19 @@ tabalyst report --scan customers.scan.json
 Tabalyst refuses a scan whose source changed since it was written, or whose
 settings differ from the `scan` settings of `--config`.
 
+Inspect or clean disposable query caches of existing DuckDB projects, for every
+project or for one CSV file:
+
+```console
+tabalyst cache info
+tabalyst cache info customers.csv
+tabalyst cache clean
+tabalyst cache clean customers.csv
+```
+
+`cache clean` leaves project scans and databases in place. Ordinary scans and
+reports do not create these query caches.
+
 See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
 and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
 
@@ -301,8 +342,9 @@ reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
 `scan()` returns a scan result without writing anything; `generate_scans()`
-writes one `.scan.json` document per source; `from_scan=True` builds reports
-from such documents.
+writes one standalone `.scan.json` document per source by default. Pass
+`project_storage=True` to use the command's scan-only storage for CSV files. `from_scan=True`
+builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
 Existing artifacts are never replaced silently. Pass `force=True` when

{tabalyst-0.4.2 → tabalyst-0.4.4}/README.md

@@ -8,8 +8,10 @@ with structured data.
 Its first tool is **Tabalyst Report**. The current CSV implementation,
 **Tabalyst CSV Report**, analyzes a CSV file and produces both a structured JSON
 profile and a self-contained interactive HTML report.
+**Tabalyst Scan** describes CSV and JSON files in a scan document, and
+**Tabalyst Sample** creates smaller CSV files.
 
-Tabalyst is in active alpha development. Its interfaces may still change while
+Tabalyst is in beta. Its interfaces may still change while
 the shared toolkit architecture is being established.
 
 ## Install
@@ -20,7 +22,7 @@ Tabalyst supports Python 3.11, 3.12, 3.13, and 3.14.
 pip install --upgrade tabalyst
 ```
 
-The same command installs Tabalyst or updates it. Tabalyst is in alpha and
+The same command installs Tabalyst or updates it. Tabalyst is in beta and
 changes often: update it before each new test.
 
 ## Tabalyst Report
@@ -49,6 +51,20 @@ customers.json
 executions.json
 ```
 
+Add `--details` to generate one self-contained HTML page per column:
+
+```console
+tabalyst report customers.csv --details
+```
+
+The **Columns** table then links to those pages. Each page has its own sidebar
+and shows values, counts, detected formats and other analysis directly. The
+subfolder takes the HTML filename without `.html`; for `-o report.html`, pages
+are in `report/`. Names use `col-01-` and a shortened slug of the column name,
+in source order. Details are off by default. `--no-details` states that choice
+explicitly; with `--force`, it removes prior generated column pages while
+preserving other files in the folder.
+
 Use `-o` to choose a different HTML filename for one source:
 
 ```console
@@ -119,8 +135,10 @@ instead.
 
 ### Safe batch behavior
 
-Before processing begins, Tabalyst resolves every input and planned output. It
-stops the entire batch if output names collide or if an output already exists.
+Before processing begins, Tabalyst resolves every input and planned main HTML
+and JSON output. It stops the entire batch if those names collide or if an
+output already exists. With `--details`, each column page is checked after its
+profile has been built, before that report's files are written.
 Use `--force` only when replacing all matching report artifacts is intentional:
 
 ```console
@@ -191,8 +209,10 @@ tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
 ```
 
-The default output is `customers.scan.json` beside the source; `-o`, `-d` and
-`--force` work as for the other commands. Tabalyst Scan reads the file once as
+For a CSV, the default scan is a reusable `scan.json` under Tabalyst's local
+storage directory (or `TABALYST_HOME`). The default workflow does not build a
+DuckDB database. `-o` or `-d` writes
+a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
 a stream, with memory bounded by configurable limits, and records for each
 field its presence, native types, missing values, frequencies, exact
 statistics, normalization variants, technical type and the result of every
@@ -204,7 +224,13 @@ interrupted scan never leaves a partial file. Files of 16 MiB or more are
 analyzed by several worker processes, with the same result; `--workers`
 chooses their number, `--workers 1` keeps one process.
 
-Build the report from a scan document without reading the source again:
+`tabalyst report customers.csv` reuses the verified stored scan when the CSV
+content and requested scan settings are current. It creates a scan when none
+exists and atomically replaces a stale one. The source check uses size,
+modification time and SHA-256. Existing DuckDB projects from 0.4.3 are left
+untouched.
+
+Build the report from a standalone scan document without reading the source again:
 
 ```console
 tabalyst report --scan customers.scan.json
@@ -213,6 +239,19 @@ tabalyst report --scan customers.scan.json
 Tabalyst refuses a scan whose source changed since it was written, or whose
 settings differ from the `scan` settings of `--config`.
 
+Inspect or clean disposable query caches of existing DuckDB projects, for every
+project or for one CSV file:
+
+```console
+tabalyst cache info
+tabalyst cache info customers.csv
+tabalyst cache clean
+tabalyst cache clean customers.csv
+```
+
+`cache clean` leaves project scans and databases in place. Ordinary scans and
+reports do not create these query caches.
+
 See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
 and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
 
@@ -267,8 +306,9 @@ reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
 `scan()` returns a scan result without writing anything; `generate_scans()`
-writes one `.scan.json` document per source; `from_scan=True` builds reports
-from such documents.
+writes one standalone `.scan.json` document per source by default. Pass
+`project_storage=True` to use the command's scan-only storage for CSV files. `from_scan=True`
+builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
 Existing artifacts are never replaced silently. Pass `force=True` when

{tabalyst-0.4.2 → tabalyst-0.4.4}/examples/README.md

@@ -18,6 +18,7 @@ examples/
     |-- insurance-customers/
     |   |-- report.html
     |   |-- report.json
+    |   |-- report/ (34 standalone column pages with --details)
     |   `-- executions.json
     `-- orders/
         |-- report.html
@@ -36,12 +37,13 @@ From the repository root, regenerate the outputs with:
 
 ```console
 tabalyst report examples/input/basic.csv -o examples/output/basic/report.html --config examples/config.json --force
-tabalyst report examples/input/insurance-customers.csv -o examples/output/insurance-customers/report.html --config examples/config.json --force
+tabalyst report examples/input/insurance-customers.csv -o examples/output/insurance-customers/report.html --config examples/config.json --details --force
 tabalyst report examples/input/orders.json -o examples/output/orders/report.html --config examples/config.json --force
 ```
 
 Every output folder is independent and contains its own HTML report, canonical
-JSON profile, and cumulative `executions.json` history. Delete an output folder
+JSON profile and cumulative `executions.json` history. The insurance example
+also has standalone HTML pages under `report/`. Delete an output folder
 before regenerating when a fresh one-entry execution history is required.
 
 The input datasets and generated output are public repository examples. The wheel

{tabalyst-0.4.2 → tabalyst-0.4.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "tabalyst"
-version = "0.4.2"
+version = "0.4.4"
 description = "Open-source toolkit for understanding and working with unfamiliar data."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -22,7 +22,7 @@ keywords = [
     "json",
 ]
 classifiers = [
-    "Development Status :: 3 - Alpha",
+    "Development Status :: 4 - Beta",
     "Intended Audience :: Developers",
     "Intended Audience :: Science/Research",
     "Operating System :: OS Independent",
@@ -34,8 +34,10 @@ classifiers = [
     "Topic :: Scientific/Engineering :: Information Analysis",
 ]
 dependencies = [
+    "duckdb==1.5.5",
     "ijson>=3.3",
     "jinja2>=3.1",
+    "platformdirs>=4",
     "pydantic>=2.11",
     "typer>=0.12",
 ]

tabalyst-0.4.4/src/tabalyst/cache_service.py

@@ -0,0 +1,114 @@
+"""Public cleanup of disposable project query caches."""
+
+from dataclasses import dataclass
+from pathlib import Path
+
+from tabalyst.errors import InputError
+from tabalyst.projects._cache import clean_query_caches, inventory_query_caches
+from tabalyst.projects._locks import ProjectBusyError
+from tabalyst.projects.identity import is_project_id
+from tabalyst.projects.index import (
+    document_key,
+    find_project,
+    read_index,
+    source_key,
+)
+from tabalyst.projects.location import StorageLocation
+from tabalyst.projects.store import read_project
+
+
+@dataclass(frozen=True)
+class CacheCleanResult:
+    projects: int
+    removed: int
+    removed_bytes: int
+    skipped: tuple[str, ...]
+
+
+@dataclass(frozen=True)
+class CacheInfo:
+    projects: int
+    directories: int
+    logical_bytes: int
+    skipped: tuple[str, ...]
+
+
+def _selected_projects(
+    source: str | Path | None = None,
+    *,
+    location: StorageLocation | None = None,
+) -> tuple[StorageLocation, list]:
+    location = location or StorageLocation.local()
+    if source is None:
+        try:
+            directories = sorted(location.projects_dir.iterdir())
+        except FileNotFoundError:
+            directories = []
+        projects = []
+        for directory in directories:
+            if is_project_id(directory.name):
+                try:
+                    projects.append(read_project(location, directory.name))
+                except InputError:
+                    continue
+    else:
+        key = source_key(Path(source))
+        index = read_index(location)
+        project_id = index.projects.get(key) if index else None
+        try:
+            selected = read_project(location, project_id) if project_id else None
+        except InputError:
+            selected = None
+        if selected is None or document_key(selected) != key:
+            selected = find_project(location, Path(source))
+        if selected is None:
+            raise InputError(f"No Tabalyst project for source: {source}")
+        projects = [selected]
+    return location, projects
+
+
+def cache_info(
+    source: str | Path | None = None,
+    *,
+    location: StorageLocation | None = None,
+) -> CacheInfo:
+    """Summarize owned disposable cache files without modifying them."""
+    location, projects = _selected_projects(source, location=location)
+    directories = 0
+    logical_bytes = 0
+    skipped = []
+    for project in projects:
+        inventory = inventory_query_caches(location, project.project_id)
+        for entry in inventory.entries:
+            if entry.status == "candidate":
+                directories += 1
+                logical_bytes += entry.logical_bytes
+            else:
+                skipped.append(f"{project.project_id}/{entry.path}: {entry.reason}")
+    return CacheInfo(len(projects), directories, logical_bytes, tuple(skipped))
+
+
+def clean_cache(
+    source: str | Path | None = None,
+    *,
+    location: StorageLocation | None = None,
+) -> CacheCleanResult:
+    """Clean owned disposable cache files for one source or all projects."""
+    location, projects = _selected_projects(source, location=location)
+    removed = 0
+    removed_bytes = 0
+    skipped = []
+    for project in projects:
+        plan = inventory_query_caches(location, project.project_id)
+        try:
+            result = clean_query_caches(location, project.project_id, plan=plan)
+        except ProjectBusyError as exc:
+            skipped.append(f"{project.project_id}/cache: {exc}")
+            continue
+        for entry in result.entries:
+            if entry.status == "removed":
+                removed += 1
+                removed_bytes += entry.removed_bytes
+            elif entry.status in ("skipped", "error"):
+                skipped.append(f"{project.project_id}/{entry.path}: {entry.reason}")
+    return CacheCleanResult(len(projects), removed, removed_bytes, tuple(skipped))

{tabalyst-0.4.2 → tabalyst-0.4.4}/src/tabalyst/cli/app.py

@@ -5,6 +5,7 @@ from typing import Annotated
 import typer
 
 from tabalyst import __version__
+from tabalyst.cli.cache import cache_app
 from tabalyst.cli.report import report_command
 from tabalyst.cli.sample import sample_command
 from tabalyst.cli.scan import scan_command
@@ -41,6 +42,7 @@ def root(
 app.command("report")(report_command)
 app.command("sample")(sample_command)
 app.command("scan")(scan_command)
+app.add_typer(cache_app, name="cache")
 
 
 def main() -> None:

tabalyst-0.4.4/src/tabalyst/cli/cache.py

@@ -0,0 +1,64 @@
+"""CLI adapter for disposable project cache cleanup."""
+
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+from tabalyst.cache_service import cache_info, clean_cache
+from tabalyst.cli.terminal import error_exit_code
+from tabalyst.errors import TabalystError
+
+cache_app = typer.Typer(help="Manage disposable project caches.")
+
+
+def _count(value: int, singular: str) -> str:
+    if value == 1:
+        return f"{value} {singular}"
+    plural = singular.removesuffix("y") + "ies" if singular.endswith("y") else singular + "s"
+    return f"{value} {plural}"
+
+
+@cache_app.command("info")
+def info_command(
+    source: Annotated[
+        Path | None,
+        typer.Argument(help="CSV file whose project cache should be inspected."),
+    ] = None,
+) -> None:
+    """Show the size and directory count of all caches or one source cache."""
+    try:
+        result = cache_info(source)
+    except TabalystError as exc:
+        typer.echo(f"Error: {exc}", err=True)
+        raise typer.Exit(error_exit_code(exc)) from exc
+    typer.echo(
+        f"Cache: {_count(result.directories, 'directory')} in "
+        f"{_count(result.projects, 'project')} "
+        f"({result.logical_bytes:,} bytes)."
+    )
+    for warning in result.skipped:
+        typer.echo(f"Retained: {warning}")
+
+
+@cache_app.command("clean")
+def clean_command(
+    source: Annotated[
+        Path | None,
+        typer.Argument(help="CSV file whose project cache should be cleaned."),
+    ] = None,
+) -> None:
+    """Clean all project caches, or the cache for one source file."""
+    try:
+        result = clean_cache(source)
+    except TabalystError as exc:
+        typer.echo(f"Error: {exc}", err=True)
+        raise typer.Exit(error_exit_code(exc)) from exc
+    typer.echo(
+        f"Cleaned {_count(result.removed, 'cache directory')} in "
+        f"{_count(result.projects, 'project')} "
+        f"({result.removed_bytes:,} bytes).",
+        err=True,
+    )
+    for warning in result.skipped:
+        typer.echo(f"Warning: retained {warning}", err=True)

{tabalyst-0.4.2 → tabalyst-0.4.4}/src/tabalyst/cli/report.py

@@ -58,6 +58,8 @@ def _print_batch_result(
                     typer.echo("  " + " | ".join(details), err=True)
 
     for success in batch.successes:
+        for warning in success.warnings:
+            typer.echo(f"Warning [{success.job.source}]: {warning}", err=True)
         if success.source_checked is False:
             typer.echo(
                 f"Warning [{success.job.source}]: source "
@@ -128,6 +130,13 @@ def report_command(
         bool,
         typer.Option("--force", "-f", help="Replace all existing report artifacts."),
     ] = False,
+    details: Annotated[
+        bool,
+        typer.Option(
+            "--details/--no-details",
+            help="Generate standalone HTML pages for every column (off by default).",
+        ),
+    ] = False,
     quiet: Annotated[
         bool,
         typer.Option("--quiet", "-q", help="Suppress success messages."),
@@ -171,6 +180,7 @@ def report_command(
             encoding=encoding,
             config_path=config,
             force=force,
+            details=details,
             on_progress=progress,
             from_scan=from_scan,
             workers=workers,

{tabalyst-0.4.2 → tabalyst-0.4.4}/src/tabalyst/cli/scan.py

@@ -76,6 +76,8 @@ def _print_batch_result(
     for success in batch.successes:
         if success.result.status == "partial":
             _print_partial(success)
+        for warning in success.warnings:
+            typer.echo(f"Warning [{success.job.source}]: {warning}", err=True)
     for failure in batch.failures:
         typer.echo(f"Error [{failure.job.source}]: {failure.error}", err=True)
 
@@ -180,6 +182,7 @@ def scan_command(
             force=force,
             on_progress=progress,
             workers=workers,
+            project_storage=True,
         )
     except TabalystError as exc:
         typer.echo(f"Error: {exc}", err=True)

{tabalyst-0.4.2 → tabalyst-0.4.4}/src/tabalyst/models.py

@@ -1,14 +1,28 @@
-"""Serializable report profile (revision 6), independent from presentation.
+"""Serializable report profile (revision 9), independent from presentation.
 
 Built from a Tabalyst Scan result by ``report_profile.py``.
 """
 
 from datetime import datetime
-from typing import Literal
+from typing import Any, Literal
 
 from pydantic import BaseModel, ConfigDict, FiniteFloat
 
 from tabalyst.report_config import ReportConfig
+from tabalyst.scanner.models import (
+    Adaptive,
+    BooleanCounts,
+    Coverage,
+    Evidence,
+    Missing,
+    Presence,
+    StringCategories,
+    StringCharacteristics,
+    StringLengths,
+    TemporalStats,
+    ValueAt,
+)
+from tabalyst.scanner.models import NumericStats as ScanNumericStats
 
 
 class ResultModel(BaseModel):
@@ -176,6 +190,29 @@ class DetectorProfile(ResultModel):
     ambiguous_count: int
     invalid_count: int
     formats: list[DetectorFormat]
+    coverage: Coverage | None = None
+    evidence: Evidence | None = None
+    details: dict[str, Any] = {}
+    adaptive: Adaptive | None = None
+
+
+class ScanFieldDetails(ResultModel):
+    """Useful bounded field evidence retained from the exposed Scan document."""
+
+    first_record: int | None
+    occurrences: int
+    value_count: int
+    presence: Presence
+    native_types: dict[str, int]
+    strings: StringCategories
+    missing: Missing
+    first: ValueAt | None
+    last: ValueAt | None
+    string_characteristics: StringCharacteristics
+    string_lengths: StringLengths | None
+    numeric: ScanNumericStats | None
+    booleans: BooleanCounts | None
+    temporal: TemporalStats | None
 
 
 class ColumnProfile(ResultModel):
@@ -208,6 +245,7 @@ class ColumnProfile(ResultModel):
     numeric: NumericStats | None = None
     # Detectors that recognized values, in scan order; failed ones included.
     detectors: list[DetectorProfile] = []
+    scan_details: ScanFieldDetails | None = None
 
 
 class DatasetSummary(ResultModel):
@@ -378,7 +416,7 @@ class DatasetProfile(ResultModel):
 
 class ReportProfile(ResultModel):
     format_version: Literal["0.1.0a"] = "0.1.0a"
-    format_revision: Literal[8] = 8
+    format_revision: Literal[9] = 9
     generated_at: datetime
     processing_seconds: FiniteFloat
     source: SourceInfo