tabalyst 0.4.3__tar.gz → 0.4.4__tar.gz

{tabalyst-0.4.3/src/tabalyst.egg-info → tabalyst-0.4.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tabalyst
-Version: 0.4.3
+Version: 0.4.4
 Summary: Open-source toolkit for understanding and working with unfamiliar data.
 Author: Gregory Borelli
 License-Expression: MIT
@@ -87,6 +87,20 @@ customers.json
 executions.json
 ```
 
+Add `--details` to generate one self-contained HTML page per column:
+
+```console
+tabalyst report customers.csv --details
+```
+
+The **Columns** table then links to those pages. Each page has its own sidebar
+and shows values, counts, detected formats and other analysis directly. The
+subfolder takes the HTML filename without `.html`; for `-o report.html`, pages
+are in `report/`. Names use `col-01-` and a shortened slug of the column name,
+in source order. Details are off by default. `--no-details` states that choice
+explicitly; with `--force`, it removes prior generated column pages while
+preserving other files in the folder.
+
 Use `-o` to choose a different HTML filename for one source:
 
 ```console
@@ -157,8 +171,10 @@ instead.
 
 ### Safe batch behavior
 
-Before processing begins, Tabalyst resolves every input and planned output. It
-stops the entire batch if output names collide or if an output already exists.
+Before processing begins, Tabalyst resolves every input and planned main HTML
+and JSON output. It stops the entire batch if those names collide or if an
+output already exists. With `--details`, each column page is checked after its
+profile has been built, before that report's files are written.
 Use `--force` only when replacing all matching report artifacts is intentional:
 
 ```console
@@ -229,9 +245,9 @@ tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
 ```
 
-For a CSV, the default scan is the project's generation-bound `scan.json` under
-Tabalyst's local storage directory (or `TABALYST_HOME`). The project also holds
-the database and disposable `cache/` used by future tools. `-o` or `-d` writes
+For a CSV, the default scan is a reusable `scan.json` under Tabalyst's local
+storage directory (or `TABALYST_HOME`). The default workflow does not build a
+DuckDB database. `-o` or `-d` writes
 a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
 a stream, with memory bounded by configurable limits, and records for each
 field its presence, native types, missing values, frequencies, exact
@@ -244,12 +260,11 @@ interrupted scan never leaves a partial file. Files of 16 MiB or more are
 analyzed by several worker processes, with the same result; `--workers`
 chooses their number, `--workers 1` keeps one process.
 
-`tabalyst report customers.csv` reuses the verified project scan when the CSV
-content and requested scan settings are current. It creates a project scan when
-none exists. If the CSV changed, it publishes a new generation, removes the old
-generation and its owned cache when no reader holds it, then builds the report.
-The source check uses size, modification time and SHA-256. An old generation
-held by a reader is retained with a warning.
+`tabalyst report customers.csv` reuses the verified stored scan when the CSV
+content and requested scan settings are current. It creates a scan when none
+exists and atomically replaces a stale one. The source check uses size,
+modification time and SHA-256. Existing DuckDB projects from 0.4.3 are left
+untouched.
 
 Build the report from a standalone scan document without reading the source again:
 
@@ -260,8 +275,8 @@ tabalyst report --scan customers.scan.json
 Tabalyst refuses a scan whose source changed since it was written, or whose
 settings differ from the `scan` settings of `--config`.
 
-Inspect or clean disposable project query caches for every project, or for one
-CSV file:
+Inspect or clean disposable query caches of existing DuckDB projects, for every
+project or for one CSV file:
 
 ```console
 tabalyst cache info
@@ -270,7 +285,8 @@ tabalyst cache clean
 tabalyst cache clean customers.csv
 ```
 
-`cache clean` leaves project scans and databases in place.
+`cache clean` leaves project scans and databases in place. Ordinary scans and
+reports do not create these query caches.
 
 See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
 and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
@@ -327,7 +343,7 @@ reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 `generate_reports()` returns the complete batch plan, successes, and failures.
 `scan()` returns a scan result without writing anything; `generate_scans()`
 writes one standalone `.scan.json` document per source by default. Pass
-`project_storage=True` to use project storage for CSV files. `from_scan=True`
+`project_storage=True` to use the command's scan-only storage for CSV files. `from_scan=True`
 builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 

{tabalyst-0.4.3 → tabalyst-0.4.4}/README.md

@@ -51,6 +51,20 @@ customers.json
 executions.json
 ```
 
+Add `--details` to generate one self-contained HTML page per column:
+
+```console
+tabalyst report customers.csv --details
+```
+
+The **Columns** table then links to those pages. Each page has its own sidebar
+and shows values, counts, detected formats and other analysis directly. The
+subfolder takes the HTML filename without `.html`; for `-o report.html`, pages
+are in `report/`. Names use `col-01-` and a shortened slug of the column name,
+in source order. Details are off by default. `--no-details` states that choice
+explicitly; with `--force`, it removes prior generated column pages while
+preserving other files in the folder.
+
 Use `-o` to choose a different HTML filename for one source:
 
 ```console
@@ -121,8 +135,10 @@ instead.
 
 ### Safe batch behavior
 
-Before processing begins, Tabalyst resolves every input and planned output. It
-stops the entire batch if output names collide or if an output already exists.
+Before processing begins, Tabalyst resolves every input and planned main HTML
+and JSON output. It stops the entire batch if those names collide or if an
+output already exists. With `--details`, each column page is checked after its
+profile has been built, before that report's files are written.
 Use `--force` only when replacing all matching report artifacts is intentional:
 
 ```console
@@ -193,9 +209,9 @@ tabalyst scan customers.csv
 tabalyst scan orders.json --collection "$.customers[]"
 ```
 
-For a CSV, the default scan is the project's generation-bound `scan.json` under
-Tabalyst's local storage directory (or `TABALYST_HOME`). The project also holds
-the database and disposable `cache/` used by future tools. `-o` or `-d` writes
+For a CSV, the default scan is a reusable `scan.json` under Tabalyst's local
+storage directory (or `TABALYST_HOME`). The default workflow does not build a
+DuckDB database. `-o` or `-d` writes
 a standalone `<stem>.scan.json` export instead. Tabalyst Scan reads the file once as
 a stream, with memory bounded by configurable limits, and records for each
 field its presence, native types, missing values, frequencies, exact
@@ -208,12 +224,11 @@ interrupted scan never leaves a partial file. Files of 16 MiB or more are
 analyzed by several worker processes, with the same result; `--workers`
 chooses their number, `--workers 1` keeps one process.
 
-`tabalyst report customers.csv` reuses the verified project scan when the CSV
-content and requested scan settings are current. It creates a project scan when
-none exists. If the CSV changed, it publishes a new generation, removes the old
-generation and its owned cache when no reader holds it, then builds the report.
-The source check uses size, modification time and SHA-256. An old generation
-held by a reader is retained with a warning.
+`tabalyst report customers.csv` reuses the verified stored scan when the CSV
+content and requested scan settings are current. It creates a scan when none
+exists and atomically replaces a stale one. The source check uses size,
+modification time and SHA-256. Existing DuckDB projects from 0.4.3 are left
+untouched.
 
 Build the report from a standalone scan document without reading the source again:
 
@@ -224,8 +239,8 @@ tabalyst report --scan customers.scan.json
 Tabalyst refuses a scan whose source changed since it was written, or whose
 settings differ from the `scan` settings of `--config`.
 
-Inspect or clean disposable project query caches for every project, or for one
-CSV file:
+Inspect or clean disposable query caches of existing DuckDB projects, for every
+project or for one CSV file:
 
 ```console
 tabalyst cache info
@@ -234,7 +249,8 @@ tabalyst cache clean
 tabalyst cache clean customers.csv
 ```
 
-`cache clean` leaves project scans and databases in place.
+`cache clean` leaves project scans and databases in place. Ordinary scans and
+reports do not create these query caches.
 
 See [Scan CSV and JSON files](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/how-to/scan-files.md)
 and the [scan format](https://github.com/loribel-labs/tabalyst/blob/main/docs/en/reference/scan-format.md).
@@ -291,7 +307,7 @@ reports = tabalyst.generate_reports(["scans/*.scan.json"], from_scan=True)
 `generate_reports()` returns the complete batch plan, successes, and failures.
 `scan()` returns a scan result without writing anything; `generate_scans()`
 writes one standalone `.scan.json` document per source by default. Pass
-`project_storage=True` to use project storage for CSV files. `from_scan=True`
+`project_storage=True` to use the command's scan-only storage for CSV files. `from_scan=True`
 builds reports from standalone scan documents.
 Expected failures derive from `tabalyst.TabalystError`.
 

{tabalyst-0.4.3 → tabalyst-0.4.4}/examples/README.md

@@ -18,6 +18,7 @@ examples/
     |-- insurance-customers/
     |   |-- report.html
     |   |-- report.json
+    |   |-- report/ (34 standalone column pages with --details)
     |   `-- executions.json
     `-- orders/
         |-- report.html
@@ -36,12 +37,13 @@ From the repository root, regenerate the outputs with:
 
 ```console
 tabalyst report examples/input/basic.csv -o examples/output/basic/report.html --config examples/config.json --force
-tabalyst report examples/input/insurance-customers.csv -o examples/output/insurance-customers/report.html --config examples/config.json --force
+tabalyst report examples/input/insurance-customers.csv -o examples/output/insurance-customers/report.html --config examples/config.json --details --force
 tabalyst report examples/input/orders.json -o examples/output/orders/report.html --config examples/config.json --force
 ```
 
 Every output folder is independent and contains its own HTML report, canonical
-JSON profile, and cumulative `executions.json` history. Delete an output folder
+JSON profile and cumulative `executions.json` history. The insurance example
+also has standalone HTML pages under `report/`. Delete an output folder
 before regenerating when a fresh one-entry execution history is required.
 
 The input datasets and generated output are public repository examples. The wheel

{tabalyst-0.4.3 → tabalyst-0.4.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "tabalyst"
-version = "0.4.3"
+version = "0.4.4"
 description = "Open-source toolkit for understanding and working with unfamiliar data."
 readme = "README.md"
 requires-python = ">=3.11"

{tabalyst-0.4.3 → tabalyst-0.4.4}/src/tabalyst/cli/report.py

@@ -130,6 +130,13 @@ def report_command(
         bool,
         typer.Option("--force", "-f", help="Replace all existing report artifacts."),
     ] = False,
+    details: Annotated[
+        bool,
+        typer.Option(
+            "--details/--no-details",
+            help="Generate standalone HTML pages for every column (off by default).",
+        ),
+    ] = False,
     quiet: Annotated[
         bool,
         typer.Option("--quiet", "-q", help="Suppress success messages."),
@@ -173,6 +180,7 @@ def report_command(
             encoding=encoding,
             config_path=config,
             force=force,
+            details=details,
             on_progress=progress,
             from_scan=from_scan,
             workers=workers,

{tabalyst-0.4.3 → tabalyst-0.4.4}/src/tabalyst/models.py

@@ -1,14 +1,28 @@
-"""Serializable report profile (revision 6), independent from presentation.
+"""Serializable report profile (revision 9), independent from presentation.
 
 Built from a Tabalyst Scan result by ``report_profile.py``.
 """
 
 from datetime import datetime
-from typing import Literal
+from typing import Any, Literal
 
 from pydantic import BaseModel, ConfigDict, FiniteFloat
 
 from tabalyst.report_config import ReportConfig
+from tabalyst.scanner.models import (
+    Adaptive,
+    BooleanCounts,
+    Coverage,
+    Evidence,
+    Missing,
+    Presence,
+    StringCategories,
+    StringCharacteristics,
+    StringLengths,
+    TemporalStats,
+    ValueAt,
+)
+from tabalyst.scanner.models import NumericStats as ScanNumericStats
 
 
 class ResultModel(BaseModel):
@@ -176,6 +190,29 @@ class DetectorProfile(ResultModel):
     ambiguous_count: int
     invalid_count: int
     formats: list[DetectorFormat]
+    coverage: Coverage | None = None
+    evidence: Evidence | None = None
+    details: dict[str, Any] = {}
+    adaptive: Adaptive | None = None
+
+
+class ScanFieldDetails(ResultModel):
+    """Useful bounded field evidence retained from the exposed Scan document."""
+
+    first_record: int | None
+    occurrences: int
+    value_count: int
+    presence: Presence
+    native_types: dict[str, int]
+    strings: StringCategories
+    missing: Missing
+    first: ValueAt | None
+    last: ValueAt | None
+    string_characteristics: StringCharacteristics
+    string_lengths: StringLengths | None
+    numeric: ScanNumericStats | None
+    booleans: BooleanCounts | None
+    temporal: TemporalStats | None
 
 
 class ColumnProfile(ResultModel):
@@ -208,6 +245,7 @@ class ColumnProfile(ResultModel):
     numeric: NumericStats | None = None
     # Detectors that recognized values, in scan order; failed ones included.
     detectors: list[DetectorProfile] = []
+    scan_details: ScanFieldDetails | None = None
 
 
 class DatasetSummary(ResultModel):
@@ -378,7 +416,7 @@ class DatasetProfile(ResultModel):
 
 class ReportProfile(ResultModel):
     format_version: Literal["0.1.0a"] = "0.1.0a"
-    format_revision: Literal[8] = 8
+    format_revision: Literal[9] = 9
     generated_at: datetime
     processing_seconds: FiniteFloat
     source: SourceInfo

{tabalyst-0.4.3 → tabalyst-0.4.4}/src/tabalyst/projects/location.py

@@ -9,6 +9,7 @@ or the directory named by ``TABALYST_HOME``, which tests and a server can use
 to relocate storage.
 """
 
+import hashlib
 import os
 import sys
 from dataclasses import dataclass
@@ -16,6 +17,7 @@ from pathlib import Path
 
 import platformdirs
 
+from tabalyst.batch import path_key
 from tabalyst.errors import ConfigurationError, InputError
 from tabalyst.projects.identity import is_project_id
 
@@ -71,6 +73,11 @@ class StorageLocation:
     def projects_dir(self) -> Path:
         return self.workspace_dir / "projects"
 
+    def shared_scan_path(self, source: Path) -> Path:
+        """The scan-only document for a resolved source, independent of DuckDB projects."""
+        key = path_key(source.resolve()).encode("utf-8", "surrogatepass")
+        return self.workspace_dir / "scans" / hashlib.sha256(key).hexdigest() / SCAN_FILE_NAME
+
     @property
     def workspace_dir(self) -> Path:
         return self.root / "workspaces" / self.workspace_id

{tabalyst-0.4.3 → tabalyst-0.4.4}/src/tabalyst/report_profile.py

@@ -1,4 +1,4 @@
-"""Report profile built from a Tabalyst Scan result (profile revision 6).
+"""Report profile built from a Tabalyst Scan result (profile revision 9).
 
 ``build_profile`` turns a scan result, fresh or read back from a scan
 document, into the profile that ``reporting.py`` renders: the columns come
@@ -45,6 +45,7 @@ from tabalyst.models import (
     NumericStats,
     PreviewRow,
     ReportProfile,
+    ScanFieldDetails,
     SourceInfo,
     StringLengthDistribution,
     StringLengthExample,
@@ -293,7 +294,10 @@ def build_value_profile(
     """
     settings = config.value_examples
     listed, complete = _listed_values(field)
-    if complete and len(listed) <= settings.full_distribution_max_distinct:
+    if complete and (
+        len(listed) <= settings.full_distribution_max_distinct
+        or field.interpretations.primary == "enumeration"
+    ):
         return ValueProfile(
             selection="complete",
             sampled_distinct_count=len(listed),
@@ -558,6 +562,10 @@ def build_detectors(field: FieldResult) -> list[DetectorProfile]:
                     )
                     for item in detector.formats
                 ],
+                coverage=coverage,
+                evidence=detector.evidence,
+                details=detector.details,
+                adaptive=detector.adaptive,
             )
         )
     return detectors
@@ -613,6 +621,30 @@ def build_column(
         string_profile=build_string_profile(field, inferred, config),
         numeric=build_numeric(field, inferred),
         detectors=build_detectors(field),
+        scan_details=ScanFieldDetails(
+            first_record=field.first_record,
+            occurrences=field.occurrences,
+            value_count=field.values.count,
+            presence=field.presence,
+            native_types=dict(field.native_types),
+            strings=field.strings,
+            missing=field.missing,
+            first=field.values.first,
+            last=field.values.last,
+            string_characteristics=field.string_characteristics,
+            string_lengths=(
+                field.string_lengths.value
+                if isinstance(field.string_lengths, Complete)
+                else None
+            ),
+            numeric=field.numeric.value if isinstance(field.numeric, Complete) else None,
+            booleans=(
+                field.booleans.value if isinstance(field.booleans, Complete) else None
+            ),
+            temporal=(
+                field.temporal.value if isinstance(field.temporal, Complete) else None
+            ),
+        ),
     )
 
 

{tabalyst-0.4.3 → tabalyst-0.4.4}/src/tabalyst/report_service.py

@@ -174,6 +174,7 @@ def generate_reports(
     encoding: str | None = None,
     config_path: ConfigPath | None = None,
     force: bool = False,
+    details: bool = False,
     on_progress: ProgressCallback | None = None,
     from_scan: bool = False,
     workers: int | None = None,
@@ -184,6 +185,7 @@ def generate_reports(
     ``tabalyst scan``: each report is built from its document without reading
     the source again, after the staleness checks of ``tabalyst.scan_reuse``.
     Otherwise ``workers`` is passed to ``tabalyst.scan()``.
+    Set ``details=True`` to write a standalone HTML page for each column.
     """
     if from_scan and (separator is not None or encoding is not None):
         raise ConfigurationError(
@@ -245,6 +247,7 @@ def generate_reports(
                     scan_layer,
                     config_paths,
                     force=force,
+                    details=details,
                     on_progress=forward,
                 )
             else:
@@ -254,6 +257,7 @@ def generate_reports(
                     config,
                     config_paths,
                     force=force,
+                    details=details,
                     on_progress=forward,
                     workers=workers,
                     scan_layer=report_scan_layer,

{tabalyst-0.4.3 → tabalyst-0.4.4}/src/tabalyst/reporting.py

@@ -1,6 +1,7 @@
 """Render self-contained analysis results as an interactive HTML report."""
 
 import re
+import unicodedata
 from collections import defaultdict
 from importlib.resources import files
 from pathlib import Path
@@ -9,7 +10,7 @@ from jinja2 import Environment, StrictUndefined
 from markupsafe import Markup, escape
 
 from tabalyst._version import get_version
-from tabalyst.models import ReportProfile
+from tabalyst.models import ColumnProfile, DatasetProfile, ReportProfile
 
 # Plain names of the scan measures a limit can stop (design 8 and 11).
 MEASURE_LABELS = {
@@ -73,7 +74,24 @@ def load_profile(path: str | Path) -> ReportProfile:
     return ReportProfile.model_validate_json(Path(path).read_text(encoding="utf-8"))
 
 
-def render_report(profile: ReportProfile) -> str:
+def column_pages(
+    profile: ReportProfile, report_path: Path
+) -> list[tuple[Path, DatasetProfile, ColumnProfile]]:
+    """Name standalone column pages in source order, across all datasets."""
+    pages = []
+    number = 0
+    for dataset in profile.datasets:
+        for column in dataset.columns:
+            number += 1
+            plain = unicodedata.normalize("NFKD", column.name).encode("ascii", "ignore").decode()
+            slug = re.sub(r"[^a-z0-9]+", "-", plain.lower()).strip("-")[:48].rstrip("-")
+            slug = slug or "unnamed"
+            path = report_path.with_suffix("") / f"col-{number:02d}-{slug}.html"
+            pages.append((path, dataset, column))
+    return pages
+
+
+def render_report(profile: ReportProfile, *, column_links: dict[tuple[str, str], str] | None = None) -> str:
     """Render exclusively from serialized analysis results; CSV access is unnecessary."""
     resources = files("tabalyst")
     environment = Environment(autoescape=True, undefined=StrictUndefined)
@@ -141,4 +159,36 @@ def render_report(profile: ReportProfile) -> str:
         reason_settings=REASON_SETTINGS,
         theme_css=resources.joinpath("static/theme.css").read_text(encoding="utf-8"),
         report_js=resources.joinpath("static/report.js").read_text(encoding="utf-8"),
+        column_links=column_links or {},
     )
+
+
+def render_column_report(
+    profile: ReportProfile,
+    dataset: DatasetProfile,
+    column: ColumnProfile,
+    report_href: str,
+) -> str:
+    """Render a self-contained page from one serialized column profile."""
+    resources = files("tabalyst")
+    environment = Environment(autoescape=True, undefined=StrictUndefined)
+    environment.filters["count"] = lambda number: f"{number:,}"
+    environment.filters["number"] = format_number
+    template = environment.from_string(
+        resources.joinpath("templates/column.html").read_text(encoding="utf-8")
+    )
+    rendered = template.render(
+        report=profile,
+        ds=dataset,
+        column=column,
+        report_href=report_href,
+        tabalyst_version=get_version(),
+        limits=[item for item in dataset.limits.measures if item.column_id == column.id],
+        issues=[item for item in dataset.issues if column.id in item.column_ids],
+        diagnostics=[
+            item for item in dataset.limits.diagnostics if item.path == column.path
+        ],
+        theme_css=resources.joinpath("static/theme.css").read_text(encoding="utf-8"),
+        report_js=resources.joinpath("static/report.js").read_text(encoding="utf-8"),
+    )
+    return re.sub(r"(?m)^[ \t]+$", "", rendered)