tabalyst 0.4.1__tar.gz → 0.4.3__tar.gz

{tabalyst-0.4.1/src/tabalyst.egg-info → tabalyst-0.4.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tabalyst
-Version: 0.4.1
+Version: 0.4.3
 Summary: Open-source toolkit for understanding and working with unfamiliar data.
 Author: Gregory Borelli
 License-Expression: MIT
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://github.com/loribel-labs/tabalyst
 Project-URL: Repository, https://github.com/loribel-labs/tabalyst.git
 Project-URL: Issues, https://github.com/loribel-labs/tabalyst/issues
 Keywords: csv,data-analysis,data-profiling,data-quality,eda,html-report,json
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
@@ -21,8 +21,10 @@ Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: duckdb==1.5.5
 Requires-Dist: ijson>=3.3
 Requires-Dist: jinja2>=3.1
+Requires-Dist: platformdirs>=4
 Requires-Dist: pydantic>=2.11
 Requires-Dist: typer>=0.12
 Provides-Extra: dev
@@ -42,8 +44,10 @@ with structured data.
 Its first tool is **Tabalyst Report**. The current CSV implementation,
 **Tabalyst CSV Report**, analyzes a CSV file and produces both a structured JSON
 profile and a self-contained interactive HTML report.
+**Tabalyst Scan** describes CSV and JSON files in a scan document, and
+**Tabalyst Sample** creates smaller CSV files.
 
-Tabalyst is in active alpha development. Its interfaces may still change while
+Tabalyst is in beta. Its interfaces may still change while
 the shared toolkit architecture is being established.
 
 ## Install
@@ -54,7 +58,7 @@ Tabalyst supports Python 3.11, 3.12, 3.13, and 3.14.
 pip install --upgrade tabalyst
 ```
 
-The same command installs Tabalyst or updates it. Tabalyst is in alpha and
+The same command installs Tabalyst or updates it. Tabalyst is in beta and
 changes often: update it before each new test.
 
 ## Tabalyst Report
@@ -181,6 +185,7 @@ tabalyst report data.csv --delimiter ";"
 tabalyst report data.csv --encoding cp1252
 tabalyst report data.csv --config tabalyst.json
 tabalyst report data.csv --verbose
+tabalyst report data.csv --workers 4
 tabalyst report --help
 tabalyst --version
 ```
@@ -224,8 +229,10 @@ tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
 ```
 
-The default output is `customers.scan.json` beside the source; `-o`, `-d` and
-`--force` work as for the other commands. Tabalyst Scan reads the file once as
+For a CSV, the default scan is the project's generation-bound `scan.json` under
+Tabalyst's local storage directory (or `TABALYST_HOME`). The project also holds
+the database and disposable `cache/` used by future tools. `-o` or `-d` writes
+a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
 a stream, with memory bounded by configurable limits, and records for each
 field its presence, native types, missing values, frequencies, exact
 statistics, normalization variants, technical type and the result of every
@@ -233,9 +240,18 @@ detector: numbers with decimal commas, dates, booleans, enumerations, email
 addresses, URLs, phone numbers, postal codes, currency amounts, percentages,
 quantities, UUIDs and IP addresses. Values of sensitive fields, such as email
 addresses, are masked by default. Results are written atomically: an
-interrupted scan never leaves a partial file.
+interrupted scan never leaves a partial file. Files of 16 MiB or more are
+analyzed by several worker processes, with the same result; `--workers`
+chooses their number, `--workers 1` keeps one process.
 
-Build the report from a scan document without reading the source again:
+`tabalyst report customers.csv` reuses the verified project scan when the CSV
+content and requested scan settings are current. It creates a project scan when
+none exists. If the CSV changed, it publishes a new generation, removes the old
+generation and its owned cache when no reader holds it, then builds the report.
+The source check uses size, modification time and SHA-256. An old generation
+held by a reader is retained with a warning.
+
+Build the report from a standalone scan document without reading the source again:
 
 ```console
 tabalyst report --scan customers.scan.json
@@ -244,6 +260,18 @@ tabalyst report --scan customers.scan.json
 Tabalyst refuses a scan whose source changed since it was written, or whose
 settings differ from the `scan` settings of `--config`.
 
+Inspect or clean disposable project query caches for every project, or for one
+CSV file:
+
+```console
+tabalyst cache info
+tabalyst cache info customers.csv
+tabalyst cache clean
+tabalyst cache clean customers.csv
+```
+
+`cache clean` leaves project scans and databases in place.
+
 See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
 and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
 
@@ -298,8 +326,9 @@ reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
 `scan()` returns a scan result without writing anything; `generate_scans()`
-writes one `.scan.json` document per source; `from_scan=True` builds reports
-from such documents.
+writes one standalone `.scan.json` document per source by default. Pass
+`project_storage=True` to use project storage for CSV files. `from_scan=True`
+builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
 Existing artifacts are never replaced silently. Pass `force=True` when

{tabalyst-0.4.1 → tabalyst-0.4.3}/README.md

@@ -8,8 +8,10 @@ with structured data.
 Its first tool is **Tabalyst Report**. The current CSV implementation,
 **Tabalyst CSV Report**, analyzes a CSV file and produces both a structured JSON
 profile and a self-contained interactive HTML report.
+**Tabalyst Scan** describes CSV and JSON files in a scan document, and
+**Tabalyst Sample** creates smaller CSV files.
 
-Tabalyst is in active alpha development. Its interfaces may still change while
+Tabalyst is in beta. Its interfaces may still change while
 the shared toolkit architecture is being established.
 
 ## Install
@@ -20,7 +22,7 @@ Tabalyst supports Python 3.11, 3.12, 3.13, and 3.14.
 pip install --upgrade tabalyst
 ```
 
-The same command installs Tabalyst or updates it. Tabalyst is in alpha and
+The same command installs Tabalyst or updates it. Tabalyst is in beta and
 changes often: update it before each new test.
 
 ## Tabalyst Report
@@ -147,6 +149,7 @@ tabalyst report data.csv --delimiter ";"
 tabalyst report data.csv --encoding cp1252
 tabalyst report data.csv --config tabalyst.json
 tabalyst report data.csv --verbose
+tabalyst report data.csv --workers 4
 tabalyst report --help
 tabalyst --version
 ```
@@ -190,8 +193,10 @@ tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
 ```
 
-The default output is `customers.scan.json` beside the source; `-o`, `-d` and
-`--force` work as for the other commands. Tabalyst Scan reads the file once as
+For a CSV, the default scan is the project's generation-bound `scan.json` under
+Tabalyst's local storage directory (or `TABALYST_HOME`). The project also holds
+the database and disposable `cache/` used by future tools. `-o` or `-d` writes
+a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
 a stream, with memory bounded by configurable limits, and records for each
 field its presence, native types, missing values, frequencies, exact
 statistics, normalization variants, technical type and the result of every
@@ -199,9 +204,18 @@ detector: numbers with decimal commas, dates, booleans, enumerations, email
 addresses, URLs, phone numbers, postal codes, currency amounts, percentages,
 quantities, UUIDs and IP addresses. Values of sensitive fields, such as email
 addresses, are masked by default. Results are written atomically: an
-interrupted scan never leaves a partial file.
+interrupted scan never leaves a partial file. Files of 16 MiB or more are
+analyzed by several worker processes, with the same result; `--workers`
+chooses their number, `--workers 1` keeps one process.
 
-Build the report from a scan document without reading the source again:
+`tabalyst report customers.csv` reuses the verified project scan when the CSV
+content and requested scan settings are current. It creates a project scan when
+none exists. If the CSV changed, it publishes a new generation, removes the old
+generation and its owned cache when no reader holds it, then builds the report.
+The source check uses size, modification time and SHA-256. An old generation
+held by a reader is retained with a warning.
+
+Build the report from a standalone scan document without reading the source again:
 
 ```console
 tabalyst report --scan customers.scan.json
@@ -210,6 +224,18 @@ tabalyst report --scan customers.scan.json
 Tabalyst refuses a scan whose source changed since it was written, or whose
 settings differ from the `scan` settings of `--config`.
 
+Inspect or clean disposable project query caches for every project, or for one
+CSV file:
+
+```console
+tabalyst cache info
+tabalyst cache info customers.csv
+tabalyst cache clean
+tabalyst cache clean customers.csv
+```
+
+`cache clean` leaves project scans and databases in place.
+
 See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
 and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
 
@@ -264,8 +290,9 @@ reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
 `scan()` returns a scan result without writing anything; `generate_scans()`
-writes one `.scan.json` document per source; `from_scan=True` builds reports
-from such documents.
+writes one standalone `.scan.json` document per source by default. Pass
+`project_storage=True` to use project storage for CSV files. `from_scan=True`
+builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
 Existing artifacts are never replaced silently. Pass `force=True` when

{tabalyst-0.4.1 → tabalyst-0.4.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "tabalyst"
-version = "0.4.1"
+version = "0.4.3"
 description = "Open-source toolkit for understanding and working with unfamiliar data."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -22,7 +22,7 @@ keywords = [
     "json",
 ]
 classifiers = [
-    "Development Status :: 3 - Alpha",
+    "Development Status :: 4 - Beta",
     "Intended Audience :: Developers",
     "Intended Audience :: Science/Research",
     "Operating System :: OS Independent",
@@ -34,8 +34,10 @@ classifiers = [
     "Topic :: Scientific/Engineering :: Information Analysis",
 ]
 dependencies = [
+    "duckdb==1.5.5",
     "ijson>=3.3",
     "jinja2>=3.1",
+    "platformdirs>=4",
     "pydantic>=2.11",
     "typer>=0.12",
 ]

tabalyst-0.4.3/src/tabalyst/cache_service.py

@@ -0,0 +1,114 @@
+"""Public cleanup of disposable project query caches."""
+
+from dataclasses import dataclass
+from pathlib import Path
+
+from tabalyst.errors import InputError
+from tabalyst.projects._cache import clean_query_caches, inventory_query_caches
+from tabalyst.projects._locks import ProjectBusyError
+from tabalyst.projects.identity import is_project_id
+from tabalyst.projects.index import (
+    document_key,
+    find_project,
+    read_index,
+    source_key,
+)
+from tabalyst.projects.location import StorageLocation
+from tabalyst.projects.store import read_project
+
+
+@dataclass(frozen=True)
+class CacheCleanResult:
+    projects: int
+    removed: int
+    removed_bytes: int
+    skipped: tuple[str, ...]
+
+
+@dataclass(frozen=True)
+class CacheInfo:
+    projects: int
+    directories: int
+    logical_bytes: int
+    skipped: tuple[str, ...]
+
+
+def _selected_projects(
+    source: str | Path | None = None,
+    *,
+    location: StorageLocation | None = None,
+) -> tuple[StorageLocation, list]:
+    location = location or StorageLocation.local()
+    if source is None:
+        try:
+            directories = sorted(location.projects_dir.iterdir())
+        except FileNotFoundError:
+            directories = []
+        projects = []
+        for directory in directories:
+            if is_project_id(directory.name):
+                try:
+                    projects.append(read_project(location, directory.name))
+                except InputError:
+                    continue
+    else:
+        key = source_key(Path(source))
+        index = read_index(location)
+        project_id = index.projects.get(key) if index else None
+        try:
+            selected = read_project(location, project_id) if project_id else None
+        except InputError:
+            selected = None
+        if selected is None or document_key(selected) != key:
+            selected = find_project(location, Path(source))
+        if selected is None:
+            raise InputError(f"No Tabalyst project for source: {source}")
+        projects = [selected]
+    return location, projects
+
+
+def cache_info(
+    source: str | Path | None = None,
+    *,
+    location: StorageLocation | None = None,
+) -> CacheInfo:
+    """Summarize owned disposable cache files without modifying them."""
+    location, projects = _selected_projects(source, location=location)
+    directories = 0
+    logical_bytes = 0
+    skipped = []
+    for project in projects:
+        inventory = inventory_query_caches(location, project.project_id)
+        for entry in inventory.entries:
+            if entry.status == "candidate":
+                directories += 1
+                logical_bytes += entry.logical_bytes
+            else:
+                skipped.append(f"{project.project_id}/{entry.path}: {entry.reason}")
+    return CacheInfo(len(projects), directories, logical_bytes, tuple(skipped))
+
+
+def clean_cache(
+    source: str | Path | None = None,
+    *,
+    location: StorageLocation | None = None,
+) -> CacheCleanResult:
+    """Clean owned disposable cache files for one source or all projects."""
+    location, projects = _selected_projects(source, location=location)
+    removed = 0
+    removed_bytes = 0
+    skipped = []
+    for project in projects:
+        plan = inventory_query_caches(location, project.project_id)
+        try:
+            result = clean_query_caches(location, project.project_id, plan=plan)
+        except ProjectBusyError as exc:
+            skipped.append(f"{project.project_id}/cache: {exc}")
+            continue
+        for entry in result.entries:
+            if entry.status == "removed":
+                removed += 1
+                removed_bytes += entry.removed_bytes
+            elif entry.status in ("skipped", "error"):
+                skipped.append(f"{project.project_id}/{entry.path}: {entry.reason}")
+    return CacheCleanResult(len(projects), removed, removed_bytes, tuple(skipped))

{tabalyst-0.4.1 → tabalyst-0.4.3}/src/tabalyst/cli/app.py

@@ -5,6 +5,7 @@ from typing import Annotated
 import typer
 
 from tabalyst import __version__
+from tabalyst.cli.cache import cache_app
 from tabalyst.cli.report import report_command
 from tabalyst.cli.sample import sample_command
 from tabalyst.cli.scan import scan_command
@@ -41,6 +42,7 @@ def root(
 app.command("report")(report_command)
 app.command("sample")(sample_command)
 app.command("scan")(scan_command)
+app.add_typer(cache_app, name="cache")
 
 
 def main() -> None:

tabalyst-0.4.3/src/tabalyst/cli/cache.py

@@ -0,0 +1,64 @@
+"""CLI adapter for disposable project cache cleanup."""
+
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+from tabalyst.cache_service import cache_info, clean_cache
+from tabalyst.cli.terminal import error_exit_code
+from tabalyst.errors import TabalystError
+
+cache_app = typer.Typer(help="Manage disposable project caches.")
+
+
+def _count(value: int, singular: str) -> str:
+    if value == 1:
+        return f"{value} {singular}"
+    plural = singular.removesuffix("y") + "ies" if singular.endswith("y") else singular + "s"
+    return f"{value} {plural}"
+
+
+@cache_app.command("info")
+def info_command(
+    source: Annotated[
+        Path | None,
+        typer.Argument(help="CSV file whose project cache should be inspected."),
+    ] = None,
+) -> None:
+    """Show the size and directory count of all caches or one source cache."""
+    try:
+        result = cache_info(source)
+    except TabalystError as exc:
+        typer.echo(f"Error: {exc}", err=True)
+        raise typer.Exit(error_exit_code(exc)) from exc
+    typer.echo(
+        f"Cache: {_count(result.directories, 'directory')} in "
+        f"{_count(result.projects, 'project')} "
+        f"({result.logical_bytes:,} bytes)."
+    )
+    for warning in result.skipped:
+        typer.echo(f"Retained: {warning}")
+
+
+@cache_app.command("clean")
+def clean_command(
+    source: Annotated[
+        Path | None,
+        typer.Argument(help="CSV file whose project cache should be cleaned."),
+    ] = None,
+) -> None:
+    """Clean all project caches, or the cache for one source file."""
+    try:
+        result = clean_cache(source)
+    except TabalystError as exc:
+        typer.echo(f"Error: {exc}", err=True)
+        raise typer.Exit(error_exit_code(exc)) from exc
+    typer.echo(
+        f"Cleaned {_count(result.removed, 'cache directory')} in "
+        f"{_count(result.projects, 'project')} "
+        f"({result.removed_bytes:,} bytes).",
+        err=True,
+    )
+    for warning in result.skipped:
+        typer.echo(f"Warning: retained {warning}", err=True)

{tabalyst-0.4.1 → tabalyst-0.4.3}/src/tabalyst/cli/report.py

@@ -58,6 +58,8 @@ def _print_batch_result(
                     typer.echo("  " + " | ".join(details), err=True)
 
     for success in batch.successes:
+        for warning in success.warnings:
+            typer.echo(f"Warning [{success.job.source}]: {warning}", err=True)
         if success.source_checked is False:
             typer.echo(
                 f"Warning [{success.job.source}]: source "
@@ -143,6 +145,17 @@ def report_command(
             help="Disable interactive file and phase progress.",
         ),
     ] = False,
+    workers: Annotated[
+        int | None,
+        typer.Option(
+            "--workers",
+            min=1,
+            help=(
+                "Processes that analyze values: 1 for one process. By default, "
+                "files of 16 MiB or more use one per spare processor."
+            ),
+        ),
+    ] = None,
 ) -> None:
     """Analyze and profile datasets with Tabalyst Report."""
     if quiet and verbose:
@@ -162,6 +175,7 @@ def report_command(
             force=force,
             on_progress=progress,
             from_scan=from_scan,
+            workers=workers,
         )
     except TabalystError as exc:
         typer.echo(f"Error: {exc}", err=True)

{tabalyst-0.4.1 → tabalyst-0.4.3}/src/tabalyst/cli/scan.py

@@ -76,6 +76,8 @@ def _print_batch_result(
     for success in batch.successes:
         if success.result.status == "partial":
             _print_partial(success)
+        for warning in success.warnings:
+            typer.echo(f"Warning [{success.job.source}]: {warning}", err=True)
     for failure in batch.failures:
         typer.echo(f"Error [{failure.job.source}]: {failure.error}", err=True)
 
@@ -149,6 +151,17 @@ def scan_command(
         bool,
         typer.Option("--no-progress", help="Disable interactive progress."),
     ] = False,
+    workers: Annotated[
+        int | None,
+        typer.Option(
+            "--workers",
+            min=1,
+            help=(
+                "Processes that analyze values: 1 for one process. By default, "
+                "files of 16 MiB or more use one per spare processor."
+            ),
+        ),
+    ] = None,
 ) -> None:
     """Scan CSV and JSON files into a complete JSON description of their data."""
     if quiet and verbose:
@@ -168,6 +181,8 @@ def scan_command(
             collections=collection,
             force=force,
             on_progress=progress,
+            workers=workers,
+            project_storage=True,
         )
     except TabalystError as exc:
         typer.echo(f"Error: {exc}", err=True)

{tabalyst-0.4.1 → tabalyst-0.4.3}/src/tabalyst/models.py

@@ -378,7 +378,7 @@ class DatasetProfile(ResultModel):
 
 class ReportProfile(ResultModel):
     format_version: Literal["0.1.0a"] = "0.1.0a"
-    format_revision: Literal[7] = 7
+    format_revision: Literal[8] = 8
     generated_at: datetime
     processing_seconds: FiniteFloat
     source: SourceInfo