agentsor-file 0.2.0__py3-none-any.whl

agentsor_file-0.2.0.dist-info/METADATA

@@ -0,0 +1,247 @@
+Metadata-Version: 2.4
+Name: agentsor-file
+Version: 0.2.0
+Summary: Local CSV and Parquet contracts with redacted hosted receipts
+Author: Agentsor
+License-Expression: MIT
+Project-URL: Homepage, https://agentsor.ai/file-contracts
+Project-URL: Repository, https://github.com/linkoinsight/agentsor-file
+Project-URL: Issues, https://github.com/linkoinsight/agentsor-file/issues
+Keywords: csv,parquet,data-contracts,schema-validation
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyarrow<24,>=16
+Provides-Extra: dev
+Requires-Dist: build<2,>=1.2; extra == "dev"
+Requires-Dist: jsonschema<5,>=4.23; extra == "dev"
+Requires-Dist: pytest<10,>=8; extra == "dev"
+Requires-Dist: ruff<0.14,>=0.12; extra == "dev"
+Dynamic: license-file
+
+# Agentsor File Contracts
+
+Agentsor File Contracts is a free, MIT-licensed command-line tool for checking
+one local CSV or Parquet file against an explicit TOML contract. It checks:
+
+- file readability;
+- required columns and Arrow types;
+- unexpected columns;
+- row and byte bounds;
+- optional event-time freshness; and
+- optional duplicate history.
+
+`init`, `check`, and `schema` run offline. The separate `report` command checks
+the file locally and sends only the fixed redacted result envelope to the
+single Agentsor collector. The package has no file-upload or telemetry path.
+
+## Local quickstart
+
+Python 3.11 or newer is required. From this checkout:
+
+```bash
+python -m venv .venv
+. .venv/bin/activate
+python -m pip install .
+agentsor-file init \
+  --format parquet \
+  --contract file-contract.toml \
+  --fingerprint-key-file .agentsor-file.key
+```
+
+`init` refuses to replace either target or follow a target symlink. It creates
+a random 32-byte fingerprint key with mode `0600` and never prints the key.
+Edit the generated `[schema]` table to describe the file, then run:
+
+```bash
+agentsor-file check YOUR_FILE.parquet \
+  --contract file-contract.toml \
+  --fingerprint-key-file .agentsor-file.key \
+  --state .agentsor-file-state.json
+```
+
+The `init`, `check`, and `schema` commands make no network requests. The CLI
+requires the fingerprint key to be an owner-only regular file and refuses
+links or group/world-readable modes. Keep it private and stable within one
+project. Back it up if fingerprint continuity matters.
+
+The CLI exits `0` for a passed contract, `1` for a failed or inconclusive
+result, and `2` for a usage, contract, credential, input, transport, or hosted
+receipt error.
+
+## Hosted deadline reporting
+
+After creating a free monitor at
+[agentsor.ai/file-contracts](https://agentsor.ai/file-contracts), put the
+one-time ingest token in an owner-only credential file without placing it in a
+shell argument:
+
+```bash
+mkdir -m 700 -p ~/.config/agentsor
+umask 077
+${EDITOR:-vi} ~/.config/agentsor/file-token
+chmod 600 ~/.config/agentsor/file-token
+```
+
+Then check and submit one redacted result:
+
+```bash
+agentsor-file report YOUR_FILE.parquet \
+  --contract file-contract.toml \
+  --fingerprint-key-file .agentsor-file.key \
+  --token-file ~/.config/agentsor/file-token
+```
+
+`report` has a fixed HTTPS destination, disables inherited proxies and
+redirects, verifies TLS, bounds request/response sizes, validates the complete
+result before network I/O, and never prints the token. It prints a fixed
+acknowledgement containing the run ID, local overall result, duplicate-receipt
+flag, and next hosted deadline.
+
+Hosted reporting does not currently accept `--state`; configure
+`reject_duplicates = false` for that command. Use offline `check --state` when
+local duplicate-output detection is required. This avoids mutating duplicate
+state before a network failure can be retried safely.
+
+## Contract format
+
+```toml
+[contract]
+format = "parquet"
+allow_extra_columns = false
+min_rows = 1
+max_rows = 1000000
+min_bytes = 1
+max_bytes = 104857600
+csv_delimiter = ","
+reject_duplicates = false
+
+# Optional; configure both together.
+# event_time_column = "event_time"
+# max_age_seconds = 86400
+
+[schema]
+id = "string"
+amount = "double"
+# event_time = "timestamp[ms]"
+```
+
+`format` may be `csv` or `parquet`, or omitted when one contract intentionally
+supports both. Schema values use PyArrow type aliases. Parquet physical types
+must match exactly. CSV has no physical type metadata, so required CSV columns
+must safely cast to their configured Arrow types.
+
+Freshness uses the newest non-null event time. Timestamp columns without a
+timezone are interpreted as UTC. `reject_duplicates = true` requires
+`--state`; without usable state, the result is inconclusive. State contains
+only project-keyed output fingerprints and retains the latest 4,096 unique
+values.
+
+## Redacted result envelope
+
+`check` writes exactly one JSON object to stdout. The v1 envelope contains a
+canonical run UUID and UTC timestamps, format, bounded row/byte counts, six
+fixed check statuses, fixed reason codes, and three fingerprints. It does not
+contain file paths, file names, column names, raw values, exception messages,
+or arbitrary text.
+
+The contract, schema, and output fingerprints are domain-separated
+HMAC-SHA256 values keyed by the supplied project key. They support equality
+and drift checks inside that project without exposing a global raw SHA-256
+that could correlate low-entropy files or schemas across projects.
+
+Redaction is not zero knowledge: row/byte counts, timing, outcomes, and
+within-project equality can still reveal metadata. Review that boundary
+before sharing a result.
+
+Print the installed formal JSON Schema with:
+
+```bash
+agentsor-file schema
+```
+
+The fixed checks are `readability`, `schema`, `rowBounds`, `byteBounds`,
+`eventFreshness`, and `duplicate`. Required checks use
+`passed|failed|inconclusive`; the two optional checks may also use
+`not_configured`. Overall status is:
+
+- `passed` when every check is passed or not configured and reasons are empty;
+- `failed` when at least one check failed; or
+- `inconclusive` when none failed and at least one could not complete.
+
+## Python API
+
+```python
+from pathlib import Path
+
+from parquet_guard import check_file, load_file_contract
+
+contract = load_file_contract("file-contract.toml")
+key = Path(".agentsor-file.key").read_bytes()
+result = check_file("output.parquet", contract, fingerprint_key=key)
+print(result.as_dict())
+```
+
+Applications should protect the key at least as carefully as the local CLI
+does and should not log it.
+
+## Deliberate limits
+
+- Files larger than 1 TiB and row counts above 1 trillion are outside the v1
+  result envelope.
+- Files are read locally by PyArrow; set a realistic `max_bytes` to establish
+  a resource boundary before parsing.
+- Duplicate state is atomic and fail-closed but assumes one writer at a time.
+- This release checks completed files; it does not watch producer directories
+  or establish that a producer has finished writing.
+- A same-size file that changes during a run is detected by a second keyed
+  fingerprint and produces an inconclusive result.
+- Hosted reporting sends only aggregate result metadata; it does not send the
+  file, its name or path, column names, values, storage location, or arbitrary
+  notes.
+
+## Historical Parquet demonstration
+
+This repository began as an owned engineering demonstration of a fail-closed,
+idempotent Parquet batch pipeline. That truthful history remains available
+through the compatibility command:
+
+```bash
+python examples/generate_sample.py
+parquet-guard --config examples/config.toml
+```
+
+It demonstrates validation, deterministic decimal transformation, quarantine,
+atomic output/state, and rerun integrity. It is not client work and is not
+evidence that customer data has been processed. Agentsor File Contracts does
+not silently invoke or quarantine files through that historical command.
+
+## Development
+
+```bash
+python -m pip install -e '.[dev]'
+pytest
+ruff check src tests
+python -m build
+```
+
+The tests cover both formats, config validation, redaction, formal envelope
+validation, project-key isolation, schema stability, bounds, freshness,
+duplicate state, fail-closed state handling, CLI exit behavior, secure init,
+and the historical batch demonstration.
+
+For bounded implementation work, see
+[Agentsor Automation Reliability](https://agentsor.ai/automation-reliability).
+Do not send
+credentials, source code, personal data, production records, or confidential
+material by email.
+
+License: MIT.

agentsor_file-0.2.0.dist-info/RECORD

@@ -0,0 +1,18 @@
+agentsor_file-0.2.0.dist-info/licenses/LICENSE,sha256=6IPSBYHE6H43vEEeQf7EuUy6ZteosiENo2f1a5QJZX4,1065
+parquet_guard/__init__.py,sha256=sagb0CpHaB_BYG8NVk0v02ROE9SzAz8XIeovFbukU8c,766
+parquet_guard/cli.py,sha256=9F0JPsBENiU_-BC-4caxq7nR5VruttvWZyGzg6VmS6M,2145
+parquet_guard/config.py,sha256=mr4n8w3BsZ9dQc6i3RMzWXfB6LKZn5g304mpdD9Eboo,7070
+parquet_guard/contracts.py,sha256=xOtPuMRr9do8uV3JvRl84cEUZIME7vMMRje2tMOIUrs,9792
+parquet_guard/file_cli.py,sha256=_o3-jCNJzZZi0cZH-0XVyYLDgFGElFZRH2tpaRkS_00,8837
+parquet_guard/file_contracts.py,sha256=5Ixk5kjDFDdhleHqImmsTvfr-rxCIdiEvAyt67qI9IE,22440
+parquet_guard/fingerprints.py,sha256=687YT3XCMBdb4mIk2evmyhw5q0xc3Mn5SPbY52EzHyQ,1552
+parquet_guard/hosted.py,sha256=tqwPFnhKhNZW6JrhZdqKl24JLDRt5WN95Myamo4SZzM,16752
+parquet_guard/pipeline.py,sha256=X_XTOpUwYCf-HnKmlFfiU05kX2ADbnsQ5CdRuXfznPM,13887
+parquet_guard/py.typed,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
+parquet_guard/schemas/error-envelope-v1.schema.json,sha256=1UjfvJtJ4-_UhQGq0_tZS_ELA-_wnRXxGbfxTM11uoY,1001
+parquet_guard/schemas/result-envelope-v1.schema.json,sha256=2gcvGHJpEGRTS0-lFbiY-AThaHbEfAnpP2alcg-mZXs,7073
+agentsor_file-0.2.0.dist-info/METADATA,sha256=2QzxRTGasrgw1Rv8zQxEMFeKnjZkW8vfp4iwh5qCgbo,8748
+agentsor_file-0.2.0.dist-info/WHEEL,sha256=K260EYznzXsJYBQGqmI8VTxEdiZYNvDZwW9cBh9-_MA,91
+agentsor_file-0.2.0.dist-info/entry_points.txt,sha256=ufwXlFdGw7SL_oWPXTil8gEEb44AVln8D5atMMKbXgE,101
+agentsor_file-0.2.0.dist-info/top_level.txt,sha256=_KMYeKeboLcrR6Lu73fwwg46QX3AC7Mo1s7NEUaTPAw,14
+agentsor_file-0.2.0.dist-info/RECORD,,

agentsor_file-0.2.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (83.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

agentsor_file-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+agentsor-file = parquet_guard.file_cli:main
+parquet-guard = parquet_guard.cli:main

agentsor_file-0.2.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Agentsor
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agentsor_file-0.2.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+parquet_guard

parquet_guard/__init__.py

@@ -0,0 +1,28 @@
+"""Offline file contracts and the historical Parquet batch demonstration."""
+
+from .config import PipelineConfig, load_config
+from .contracts import ContractConfigError, FileContract, load_file_contract
+from .file_contracts import (
+    ContractResult,
+    DuplicateStateError,
+    FileContractInputError,
+    check_file,
+)
+from .fingerprints import FingerprintKeyError, project_fingerprint
+from .pipeline import PipelineSummary, run_pipeline
+
+__all__ = [
+    "ContractConfigError",
+    "ContractResult",
+    "DuplicateStateError",
+    "FileContract",
+    "FileContractInputError",
+    "FingerprintKeyError",
+    "PipelineConfig",
+    "PipelineSummary",
+    "check_file",
+    "load_config",
+    "load_file_contract",
+    "project_fingerprint",
+    "run_pipeline",
+]

parquet_guard/cli.py

@@ -0,0 +1,75 @@
+"""Command-line entry point."""
+
+from __future__ import annotations
+
+import argparse
+from datetime import datetime, timezone
+import json
+import logging
+import sys
+from typing import Any
+
+from .config import ConfigError, load_config
+from .pipeline import run_pipeline, summary_dict
+
+
+class JsonFormatter(logging.Formatter):
+    """Emit one small structured event per log line."""
+
+    def format(self, record: logging.LogRecord) -> str:
+        value: dict[str, Any] = {
+            "timestamp": datetime.fromtimestamp(
+                record.created, tz=timezone.utc
+            ).isoformat(),
+            "level": record.levelname.lower(),
+            "event": getattr(record, "event", record.getMessage()),
+        }
+        for key in (
+            "run_id",
+            "file",
+            "reason",
+            "rows",
+            "discovered",
+            "succeeded",
+            "quarantined",
+            "skipped",
+            "failed",
+        ):
+            if hasattr(record, key):
+                value[key] = getattr(record, key)
+        return json.dumps(value, sort_keys=True, separators=(",", ":"))
+
+
+def _parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Process, validate, transform, and quarantine Parquet inputs."
+    )
+    parser.add_argument(
+        "--config", required=True, help="Path to the TOML configuration"
+    )
+    return parser
+
+
+def main() -> int:
+    handler = logging.StreamHandler(sys.stderr)
+    handler.setFormatter(JsonFormatter())
+    logger = logging.getLogger("parquet_guard")
+    logger.handlers[:] = [handler]
+    logger.setLevel(logging.INFO)
+    logger.propagate = False
+
+    arguments = _parser().parse_args()
+    try:
+        summary = run_pipeline(load_config(arguments.config))
+    except ConfigError as exc:
+        print(
+            json.dumps({"error": "invalid_config", "message": str(exc)}),
+            file=sys.stderr,
+        )
+        return 2
+    print(json.dumps(summary_dict(summary), sort_keys=True))
+    return 0 if summary.quarantined == 0 and summary.failed == 0 else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

parquet_guard/config.py

@@ -0,0 +1,176 @@
+"""External configuration for the demonstration pipeline."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from decimal import Decimal, InvalidOperation
+from pathlib import Path
+import tomllib
+
+import pyarrow as pa
+
+
+class ConfigError(ValueError):
+    """Raised when a configuration cannot be used safely."""
+
+
+@dataclass(frozen=True)
+class PipelineConfig:
+    """Validated runtime configuration."""
+
+    input_dir: Path
+    output_dir: Path
+    quarantine_dir: Path
+    state_dir: Path
+    required_schema: dict[str, str]
+    output_compression: str = "zstd"
+    allow_extra_columns: bool = False
+    amount_column: str | None = None
+    amount_multiplier: str = "100"
+    amount_output_column: str = "amount_cents"
+    amount_rounding: str = "half_even"
+
+    def validate(self) -> "PipelineConfig":
+        directories = (
+            self.input_dir.resolve(),
+            self.output_dir.resolve(),
+            self.quarantine_dir.resolve(),
+            self.state_dir.resolve(),
+        )
+        if len(set(directories)) != len(directories):
+            raise ConfigError("pipeline directories must be distinct")
+        if not self.required_schema:
+            raise ConfigError("schema must contain at least one required column")
+        for column, type_name in self.required_schema.items():
+            if not column or not isinstance(column, str):
+                raise ConfigError("schema column names must be non-empty strings")
+            try:
+                pa.type_for_alias(type_name)
+            except (ValueError, TypeError) as exc:
+                raise ConfigError(
+                    f"unsupported Arrow type alias for {column!r}: {type_name!r}"
+                ) from exc
+        if self.amount_column is not None:
+            configured_type = self.required_schema.get(self.amount_column)
+            if configured_type not in {"double", "float64", "float", "float32"}:
+                raise ConfigError(
+                    "amount_column must name a required floating-point column"
+                )
+            if (
+                not isinstance(self.amount_output_column, str)
+                or not self.amount_output_column.strip()
+            ):
+                raise ConfigError("amount_output_column must be a non-empty string")
+            if self.amount_output_column in self.required_schema:
+                raise ConfigError(
+                    "amount_output_column must not overwrite a required input column"
+                )
+            if isinstance(self.amount_multiplier, bool):
+                raise ConfigError("amount_multiplier must be a decimal number")
+            try:
+                multiplier = Decimal(self.amount_multiplier)
+            except (InvalidOperation, TypeError, ValueError) as exc:
+                raise ConfigError("amount_multiplier must be a decimal number") from exc
+            if not multiplier.is_finite():
+                raise ConfigError("amount_multiplier must be finite")
+            if self.amount_rounding not in {
+                "half_even",
+                "half_up",
+                "half_down",
+                "down",
+                "up",
+            }:
+                raise ConfigError("unsupported amount_rounding rule")
+        if self.output_compression not in {
+            "none",
+            "snappy",
+            "gzip",
+            "brotli",
+            "zstd",
+            "lz4",
+        }:
+            raise ConfigError("unsupported Parquet output compression")
+        if self.output_compression != "none" and not pa.Codec.is_available(
+            self.output_compression
+        ):
+            raise ConfigError(
+                f"Parquet compression codec is unavailable: {self.output_compression}"
+            )
+        return self
+
+
+def _required_string(table: dict[str, object], key: str) -> str:
+    value = table.get(key)
+    if not isinstance(value, str) or not value.strip():
+        raise ConfigError(f"pipeline.{key} must be a non-empty string")
+    return value.strip()
+
+
+def load_config(path: Path | str) -> PipelineConfig:
+    """Load a TOML configuration, resolving paths relative to that file."""
+
+    config_path = Path(path).resolve()
+    try:
+        raw = tomllib.loads(config_path.read_text(encoding="utf-8"))
+    except (OSError, tomllib.TOMLDecodeError) as exc:
+        raise ConfigError(f"could not load configuration: {exc}") from exc
+
+    pipeline = raw.get("pipeline")
+    schema = raw.get("schema")
+    if not isinstance(pipeline, dict) or not isinstance(schema, dict):
+        raise ConfigError("configuration requires [pipeline] and [schema] tables")
+
+    base = config_path.parent
+
+    def configured_path(key: str) -> Path:
+        value = Path(_required_string(pipeline, key))
+        return value.resolve() if value.is_absolute() else (base / value).resolve()
+
+    if not all(
+        isinstance(key, str) and isinstance(value, str) for key, value in schema.items()
+    ):
+        raise ConfigError("schema entries must map column names to Arrow type aliases")
+
+    amount_column = pipeline.get("amount_column")
+    if amount_column is not None and (
+        not isinstance(amount_column, str) or not amount_column.strip()
+    ):
+        raise ConfigError("pipeline.amount_column must be a non-empty string")
+
+    allow_extra = pipeline.get("allow_extra_columns", False)
+    if not isinstance(allow_extra, bool):
+        raise ConfigError("pipeline.allow_extra_columns must be true or false")
+
+    compression = pipeline.get("output_compression", "zstd")
+    if not isinstance(compression, str):
+        raise ConfigError("pipeline.output_compression must be a string")
+
+    amount_multiplier = pipeline.get("amount_multiplier", "100")
+    if isinstance(amount_multiplier, (int, float)):
+        amount_multiplier = str(amount_multiplier)
+    if not isinstance(amount_multiplier, str) or not amount_multiplier.strip():
+        raise ConfigError(
+            "pipeline.amount_multiplier must be a decimal string or number"
+        )
+
+    amount_output_column = pipeline.get("amount_output_column", "amount_cents")
+    if not isinstance(amount_output_column, str) or not amount_output_column.strip():
+        raise ConfigError("pipeline.amount_output_column must be a non-empty string")
+
+    amount_rounding = pipeline.get("amount_rounding", "half_even")
+    if not isinstance(amount_rounding, str):
+        raise ConfigError("pipeline.amount_rounding must be a string")
+
+    return PipelineConfig(
+        input_dir=configured_path("input_dir"),
+        output_dir=configured_path("output_dir"),
+        quarantine_dir=configured_path("quarantine_dir"),
+        state_dir=configured_path("state_dir"),
+        required_schema=dict(schema),
+        output_compression=compression.lower(),
+        allow_extra_columns=allow_extra,
+        amount_column=amount_column.strip() if isinstance(amount_column, str) else None,
+        amount_multiplier=amount_multiplier.strip(),
+        amount_output_column=amount_output_column.strip(),
+        amount_rounding=amount_rounding.strip().lower(),
+    ).validate()