tabalyst 0.4.2__tar.gz → 0.4.3__tar.gz

{tabalyst-0.4.2/src/tabalyst.egg-info → tabalyst-0.4.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tabalyst
-Version: 0.4.2
+Version: 0.4.3
 Summary: Open-source toolkit for understanding and working with unfamiliar data.
 Author: Gregory Borelli
 License-Expression: MIT
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://github.com/loribel-labs/tabalyst
 Project-URL: Repository, https://github.com/loribel-labs/tabalyst.git
 Project-URL: Issues, https://github.com/loribel-labs/tabalyst/issues
 Keywords: csv,data-analysis,data-profiling,data-quality,eda,html-report,json
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
@@ -21,8 +21,10 @@ Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: duckdb==1.5.5
 Requires-Dist: ijson>=3.3
 Requires-Dist: jinja2>=3.1
+Requires-Dist: platformdirs>=4
 Requires-Dist: pydantic>=2.11
 Requires-Dist: typer>=0.12
 Provides-Extra: dev
@@ -42,8 +44,10 @@ with structured data.
 Its first tool is **Tabalyst Report**. The current CSV implementation,
 **Tabalyst CSV Report**, analyzes a CSV file and produces both a structured JSON
 profile and a self-contained interactive HTML report.
+**Tabalyst Scan** describes CSV and JSON files in a scan document, and
+**Tabalyst Sample** creates smaller CSV files.
 
-Tabalyst is in active alpha development. Its interfaces may still change while
+Tabalyst is in beta. Its interfaces may still change while
 the shared toolkit architecture is being established.
 
 ## Install
@@ -54,7 +58,7 @@ Tabalyst supports Python 3.11, 3.12, 3.13, and 3.14.
 pip install --upgrade tabalyst
 ```
 
-The same command installs Tabalyst or updates it. Tabalyst is in alpha and
+The same command installs Tabalyst or updates it. Tabalyst is in beta and
 changes often: update it before each new test.
 
 ## Tabalyst Report
@@ -225,8 +229,10 @@ tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
 ```
 
-The default output is `customers.scan.json` beside the source; `-o`, `-d` and
-`--force` work as for the other commands. Tabalyst Scan reads the file once as
+For a CSV, the default scan is the project's generation-bound `scan.json` under
+Tabalyst's local storage directory (or `TABALYST_HOME`). The project also holds
+the database and disposable `cache/` used by future tools. `-o` or `-d` writes
+a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
 a stream, with memory bounded by configurable limits, and records for each
 field its presence, native types, missing values, frequencies, exact
 statistics, normalization variants, technical type and the result of every
@@ -238,7 +244,14 @@ interrupted scan never leaves a partial file. Files of 16 MiB or more are
 analyzed by several worker processes, with the same result; `--workers`
 chooses their number, `--workers 1` keeps one process.
 
-Build the report from a scan document without reading the source again:
+`tabalyst report customers.csv` reuses the verified project scan when the CSV
+content and requested scan settings are current. It creates a project scan when
+none exists. If the CSV changed, it publishes a new generation, removes the old
+generation and its owned cache when no reader holds it, then builds the report.
+The source check uses size, modification time and SHA-256. An old generation
+held by a reader is retained with a warning.
+
+Build the report from a standalone scan document without reading the source again:
 
 ```console
 tabalyst report --scan customers.scan.json
@@ -247,6 +260,18 @@ tabalyst report --scan customers.scan.json
 Tabalyst refuses a scan whose source changed since it was written, or whose
 settings differ from the `scan` settings of `--config`.
 
+Inspect or clean disposable project query caches for every project, or for one
+CSV file:
+
+```console
+tabalyst cache info
+tabalyst cache info customers.csv
+tabalyst cache clean
+tabalyst cache clean customers.csv
+```
+
+`cache clean` leaves project scans and databases in place.
+
 See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
 and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
 
@@ -301,8 +326,9 @@ reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
 `scan()` returns a scan result without writing anything; `generate_scans()`
-writes one `.scan.json` document per source; `from_scan=True` builds reports
-from such documents.
+writes one standalone `.scan.json` document per source by default. Pass
+`project_storage=True` to use project storage for CSV files. `from_scan=True`
+builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
 Existing artifacts are never replaced silently. Pass `force=True` when

{tabalyst-0.4.2 → tabalyst-0.4.3}/README.md

@@ -8,8 +8,10 @@ with structured data.
 Its first tool is **Tabalyst Report**. The current CSV implementation,
 **Tabalyst CSV Report**, analyzes a CSV file and produces both a structured JSON
 profile and a self-contained interactive HTML report.
+**Tabalyst Scan** describes CSV and JSON files in a scan document, and
+**Tabalyst Sample** creates smaller CSV files.
 
-Tabalyst is in active alpha development. Its interfaces may still change while
+Tabalyst is in beta. Its interfaces may still change while
 the shared toolkit architecture is being established.
 
 ## Install
@@ -20,7 +22,7 @@ Tabalyst supports Python 3.11, 3.12, 3.13, and 3.14.
 pip install --upgrade tabalyst
 ```
 
-The same command installs Tabalyst or updates it. Tabalyst is in alpha and
+The same command installs Tabalyst or updates it. Tabalyst is in beta and
 changes often: update it before each new test.
 
 ## Tabalyst Report
@@ -191,8 +193,10 @@ tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
 ```
 
-The default output is `customers.scan.json` beside the source; `-o`, `-d` and
-`--force` work as for the other commands. Tabalyst Scan reads the file once as
+For a CSV, the default scan is the project's generation-bound `scan.json` under
+Tabalyst's local storage directory (or `TABALYST_HOME`). The project also holds
+the database and disposable `cache/` used by future tools. `-o` or `-d` writes
+a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
 a stream, with memory bounded by configurable limits, and records for each
 field its presence, native types, missing values, frequencies, exact
 statistics, normalization variants, technical type and the result of every
@@ -204,7 +208,14 @@ interrupted scan never leaves a partial file. Files of 16 MiB or more are
 analyzed by several worker processes, with the same result; `--workers`
 chooses their number, `--workers 1` keeps one process.
 
-Build the report from a scan document without reading the source again:
+`tabalyst report customers.csv` reuses the verified project scan when the CSV
+content and requested scan settings are current. It creates a project scan when
+none exists. If the CSV changed, it publishes a new generation, removes the old
+generation and its owned cache when no reader holds it, then builds the report.
+The source check uses size, modification time and SHA-256. An old generation
+held by a reader is retained with a warning.
+
+Build the report from a standalone scan document without reading the source again:
 
 ```console
 tabalyst report --scan customers.scan.json
@@ -213,6 +224,18 @@ tabalyst report --scan customers.scan.json
 Tabalyst refuses a scan whose source changed since it was written, or whose
 settings differ from the `scan` settings of `--config`.
 
+Inspect or clean disposable project query caches for every project, or for one
+CSV file:
+
+```console
+tabalyst cache info
+tabalyst cache info customers.csv
+tabalyst cache clean
+tabalyst cache clean customers.csv
+```
+
+`cache clean` leaves project scans and databases in place.
+
 See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
 and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
 
@@ -267,8 +290,9 @@ reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 `analyze()` returns the JSON-serializable profile for one report.
 `generate_reports()` returns the complete batch plan, successes, and failures.
 `scan()` returns a scan result without writing anything; `generate_scans()`
-writes one `.scan.json` document per source; `from_scan=True` builds reports
-from such documents.
+writes one standalone `.scan.json` document per source by default. Pass
+`project_storage=True` to use project storage for CSV files. `from_scan=True`
+builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 
 Existing artifacts are never replaced silently. Pass `force=True` when

{tabalyst-0.4.2 → tabalyst-0.4.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "tabalyst"
-version = "0.4.2"
+version = "0.4.3"
 description = "Open-source toolkit for understanding and working with unfamiliar data."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -22,7 +22,7 @@ keywords = [
     "json",
 ]
 classifiers = [
-    "Development Status :: 3 - Alpha",
+    "Development Status :: 4 - Beta",
     "Intended Audience :: Developers",
     "Intended Audience :: Science/Research",
     "Operating System :: OS Independent",
@@ -34,8 +34,10 @@ classifiers = [
     "Topic :: Scientific/Engineering :: Information Analysis",
 ]
 dependencies = [
+    "duckdb==1.5.5",
     "ijson>=3.3",
     "jinja2>=3.1",
+    "platformdirs>=4",
     "pydantic>=2.11",
     "typer>=0.12",
 ]

tabalyst-0.4.3/src/tabalyst/cache_service.py

@@ -0,0 +1,114 @@
+"""Public cleanup of disposable project query caches."""
+
+from dataclasses import dataclass
+from pathlib import Path
+
+from tabalyst.errors import InputError
+from tabalyst.projects._cache import clean_query_caches, inventory_query_caches
+from tabalyst.projects._locks import ProjectBusyError
+from tabalyst.projects.identity import is_project_id
+from tabalyst.projects.index import (
+    document_key,
+    find_project,
+    read_index,
+    source_key,
+)
+from tabalyst.projects.location import StorageLocation
+from tabalyst.projects.store import read_project
+
+
+@dataclass(frozen=True)
+class CacheCleanResult:
+    projects: int
+    removed: int
+    removed_bytes: int
+    skipped: tuple[str, ...]
+
+
+@dataclass(frozen=True)
+class CacheInfo:
+    projects: int
+    directories: int
+    logical_bytes: int
+    skipped: tuple[str, ...]
+
+
+def _selected_projects(
+    source: str | Path | None = None,
+    *,
+    location: StorageLocation | None = None,
+) -> tuple[StorageLocation, list]:
+    location = location or StorageLocation.local()
+    if source is None:
+        try:
+            directories = sorted(location.projects_dir.iterdir())
+        except FileNotFoundError:
+            directories = []
+        projects = []
+        for directory in directories:
+            if is_project_id(directory.name):
+                try:
+                    projects.append(read_project(location, directory.name))
+                except InputError:
+                    continue
+    else:
+        key = source_key(Path(source))
+        index = read_index(location)
+        project_id = index.projects.get(key) if index else None
+        try:
+            selected = read_project(location, project_id) if project_id else None
+        except InputError:
+            selected = None
+        if selected is None or document_key(selected) != key:
+            selected = find_project(location, Path(source))
+        if selected is None:
+            raise InputError(f"No Tabalyst project for source: {source}")
+        projects = [selected]
+    return location, projects
+
+
+def cache_info(
+    source: str | Path | None = None,
+    *,
+    location: StorageLocation | None = None,
+) -> CacheInfo:
+    """Summarize owned disposable cache files without modifying them."""
+    location, projects = _selected_projects(source, location=location)
+    directories = 0
+    logical_bytes = 0
+    skipped = []
+    for project in projects:
+        inventory = inventory_query_caches(location, project.project_id)
+        for entry in inventory.entries:
+            if entry.status == "candidate":
+                directories += 1
+                logical_bytes += entry.logical_bytes
+            else:
+                skipped.append(f"{project.project_id}/{entry.path}: {entry.reason}")
+    return CacheInfo(len(projects), directories, logical_bytes, tuple(skipped))
+
+
+def clean_cache(
+    source: str | Path | None = None,
+    *,
+    location: StorageLocation | None = None,
+) -> CacheCleanResult:
+    """Clean owned disposable cache files for one source or all projects."""
+    location, projects = _selected_projects(source, location=location)
+    removed = 0
+    removed_bytes = 0
+    skipped = []
+    for project in projects:
+        plan = inventory_query_caches(location, project.project_id)
+        try:
+            result = clean_query_caches(location, project.project_id, plan=plan)
+        except ProjectBusyError as exc:
+            skipped.append(f"{project.project_id}/cache: {exc}")
+            continue
+        for entry in result.entries:
+            if entry.status == "removed":
+                removed += 1
+                removed_bytes += entry.removed_bytes
+            elif entry.status in ("skipped", "error"):
+                skipped.append(f"{project.project_id}/{entry.path}: {entry.reason}")
+    return CacheCleanResult(len(projects), removed, removed_bytes, tuple(skipped))

{tabalyst-0.4.2 → tabalyst-0.4.3}/src/tabalyst/cli/app.py

@@ -5,6 +5,7 @@ from typing import Annotated
 import typer
 
 from tabalyst import __version__
+from tabalyst.cli.cache import cache_app
 from tabalyst.cli.report import report_command
 from tabalyst.cli.sample import sample_command
 from tabalyst.cli.scan import scan_command
@@ -41,6 +42,7 @@ def root(
 app.command("report")(report_command)
 app.command("sample")(sample_command)
 app.command("scan")(scan_command)
+app.add_typer(cache_app, name="cache")
 
 
 def main() -> None:

tabalyst-0.4.3/src/tabalyst/cli/cache.py

@@ -0,0 +1,64 @@
+"""CLI adapter for disposable project cache cleanup."""
+
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+from tabalyst.cache_service import cache_info, clean_cache
+from tabalyst.cli.terminal import error_exit_code
+from tabalyst.errors import TabalystError
+
+cache_app = typer.Typer(help="Manage disposable project caches.")
+
+
+def _count(value: int, singular: str) -> str:
+    if value == 1:
+        return f"{value} {singular}"
+    plural = singular.removesuffix("y") + "ies" if singular.endswith("y") else singular + "s"
+    return f"{value} {plural}"
+
+
+@cache_app.command("info")
+def info_command(
+    source: Annotated[
+        Path | None,
+        typer.Argument(help="CSV file whose project cache should be inspected."),
+    ] = None,
+) -> None:
+    """Show the size and directory count of all caches or one source cache."""
+    try:
+        result = cache_info(source)
+    except TabalystError as exc:
+        typer.echo(f"Error: {exc}", err=True)
+        raise typer.Exit(error_exit_code(exc)) from exc
+    typer.echo(
+        f"Cache: {_count(result.directories, 'directory')} in "
+        f"{_count(result.projects, 'project')} "
+        f"({result.logical_bytes:,} bytes)."
+    )
+    for warning in result.skipped:
+        typer.echo(f"Retained: {warning}")
+
+
+@cache_app.command("clean")
+def clean_command(
+    source: Annotated[
+        Path | None,
+        typer.Argument(help="CSV file whose project cache should be cleaned."),
+    ] = None,
+) -> None:
+    """Clean all project caches, or the cache for one source file."""
+    try:
+        result = clean_cache(source)
+    except TabalystError as exc:
+        typer.echo(f"Error: {exc}", err=True)
+        raise typer.Exit(error_exit_code(exc)) from exc
+    typer.echo(
+        f"Cleaned {_count(result.removed, 'cache directory')} in "
+        f"{_count(result.projects, 'project')} "
+        f"({result.removed_bytes:,} bytes).",
+        err=True,
+    )
+    for warning in result.skipped:
+        typer.echo(f"Warning: retained {warning}", err=True)

{tabalyst-0.4.2 → tabalyst-0.4.3}/src/tabalyst/cli/report.py

@@ -58,6 +58,8 @@ def _print_batch_result(
                     typer.echo("  " + " | ".join(details), err=True)
 
     for success in batch.successes:
+        for warning in success.warnings:
+            typer.echo(f"Warning [{success.job.source}]: {warning}", err=True)
         if success.source_checked is False:
             typer.echo(
                 f"Warning [{success.job.source}]: source "

{tabalyst-0.4.2 → tabalyst-0.4.3}/src/tabalyst/cli/scan.py

@@ -76,6 +76,8 @@ def _print_batch_result(
     for success in batch.successes:
         if success.result.status == "partial":
             _print_partial(success)
+        for warning in success.warnings:
+            typer.echo(f"Warning [{success.job.source}]: {warning}", err=True)
     for failure in batch.failures:
         typer.echo(f"Error [{failure.job.source}]: {failure.error}", err=True)
 
@@ -180,6 +182,7 @@ def scan_command(
             force=force,
             on_progress=progress,
             workers=workers,
+            project_storage=True,
         )
     except TabalystError as exc:
         typer.echo(f"Error: {exc}", err=True)

tabalyst-0.4.3/src/tabalyst/project_open_service.py

@@ -0,0 +1,179 @@
+"""Private first-open policy for an eventual Explore adapter (S18).
+
+The decision is transient and advisory. A caller must retain it for an explicit
+snapshot or rescan choice; every action revalidates the selected generation.
+No CLI, public API, automatic scan or presentation is provided here.
+"""
+
+from contextlib import ExitStack, contextmanager
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Literal
+
+from tabalyst.errors import InputError
+from tabalyst.project_scan_service import rescan_project
+from tabalyst.projects._generation import GenerationConflictError
+from tabalyst.projects._session import (
+    ProjectSession,
+    SessionAssessment,
+    SessionRefusedError,
+    inspect_project,
+    open_project,
+)
+from tabalyst.projects.location import StorageLocation
+from tabalyst.scan_reuse import SourceState
+from tabalyst.scanner import ScanConfig
+from tabalyst.scanner.config import config_sha256
+
+OpeningAction = Literal["open_current", "choose", "resolve_config"]
+
+
+@dataclass(frozen=True)
+class ProjectOpenDecision:
+    storage_root: Path
+    assessment: SessionAssessment
+    action: OpeningAction
+    choices: tuple[str, ...]
+
+
+@dataclass(frozen=True)
+class ProjectEntry:
+    decision: ProjectOpenDecision
+    session: ProjectSession | None
+
+
+def _decision(
+    location: StorageLocation, assessment: SessionAssessment
+) -> ProjectOpenDecision:
+    root = location.root.resolve()
+    if assessment.errors:
+        choices = ("use_recorded_config",)
+        if (
+            assessment.source_check is not None
+            and assessment.source_check.state is not SourceState.MISSING
+        ):
+            choices += ("rescan",)
+        return ProjectOpenDecision(root, assessment, "resolve_config", choices)
+    if assessment.current_ready:
+        return ProjectOpenDecision(
+            root, assessment, "open_current", ("open_current", "rescan")
+        )
+    if assessment.source_check is None:
+        return ProjectOpenDecision(
+            root, assessment, "choose", ("snapshot", "reinspect")
+        )
+    if assessment.source_check.state is SourceState.MISSING:
+        return ProjectOpenDecision(root, assessment, "choose", ("snapshot",))
+    return ProjectOpenDecision(root, assessment, "choose", ("snapshot", "rescan"))
+
+
+def _check_decision(
+    location: StorageLocation, decision: ProjectOpenDecision
+) -> SessionAssessment:
+    if not isinstance(decision, ProjectOpenDecision):
+        raise TypeError("A ProjectOpenDecision from inspection is required")
+    assessment = decision.assessment
+    if assessment.workspace_id != location.workspace_id:
+        raise GenerationConflictError("Opening decision belongs to another workspace")
+    if decision.storage_root != location.root.resolve():
+        raise GenerationConflictError(
+            "Opening decision belongs to another storage root"
+        )
+    if decision != _decision(location, assessment):
+        raise ValueError("Inconsistent project opening decision")
+    return assessment
+
+
+def inspect_opening(
+    location: StorageLocation,
+    project_id: str,
+    *,
+    requested_config: ScanConfig | None = None,
+) -> ProjectOpenDecision:
+    """Inspect one verified project, without opening a query session or scanning."""
+    return _decision(
+        location,
+        inspect_project(location, project_id, requested_config=requested_config),
+    )
+
+
+@contextmanager
+def enter_project(
+    location: StorageLocation,
+    project_id: str,
+    *,
+    requested_config: ScanConfig | None = None,
+):
+    """Open a fresh source as current; return a decision for every other state."""
+    decision = inspect_opening(location, project_id, requested_config=requested_config)
+    if decision.action != "open_current":
+        yield ProjectEntry(decision, None)
+        return
+    with ExitStack() as resources:
+        try:
+            session = resources.enter_context(
+                open_project(
+                    location,
+                    project_id,
+                    requested_config=requested_config,
+                    expected_generation_id=decision.assessment.generation_id,
+                )
+            )
+        except SessionRefusedError as exc:
+            # The source changed between inspection and opening. Report the
+            # newer assessment instead of yielding a stale current session.
+            yield ProjectEntry(_decision(location, exc.assessment), None)
+            return
+        if session.assessment.source != decision.assessment.source:
+            raise GenerationConflictError("Selected project source binding changed")
+        yield ProjectEntry(_decision(location, session.assessment), session)
+
+
+@contextmanager
+def open_snapshot(location: StorageLocation, decision: ProjectOpenDecision):
+    """Explicitly query the stored generation, carrying its warning and id."""
+    assessment = _check_decision(location, decision)
+    if "snapshot" not in decision.choices:
+        raise InputError("Snapshot is not an available choice for this assessment")
+    with open_project(
+        location,
+        assessment.project_id,
+        intent="snapshot",
+        expected_generation_id=assessment.generation_id,
+    ) as session:
+        if session.assessment.source != assessment.source:
+            raise GenerationConflictError("Selected project source binding changed")
+        yield session
+
+
+@contextmanager
+def rescan_from_decision(
+    location: StorageLocation,
+    decision: ProjectOpenDecision,
+    *,
+    config: ScanConfig | None = None,
+    value_limit: int = 10000,
+    intent: Literal["require_current", "snapshot"] = "require_current",
+):
+    """Explicitly rescan only the assessed project/source/generation."""
+    assessment = _check_decision(location, decision)
+    if "rescan" not in decision.choices:
+        raise InputError("Rescan is not an available choice for this assessment")
+    if decision.action == "resolve_config":
+        if config is None or not isinstance(config, ScanConfig):
+            raise InputError("A complete replacement ScanConfig is required")
+        validated = ScanConfig.model_validate(
+            config.model_dump(mode="json", by_alias=True)
+        )
+        if config_sha256(validated) != assessment.requested_config_sha256:
+            raise InputError("Replacement config differs from the inspected request")
+    with rescan_project(
+        location,
+        assessment.project_id,
+        expected_generation_id=assessment.generation_id,
+        expected_source=Path(assessment.source),
+        config=config,
+        value_limit=value_limit,
+        intent=intent,
+    ) as result:
+        yield result