sheet-unmess 1.0.0__tar.gz

sheet_unmess-1.0.0/.gitignore

@@ -0,0 +1,51 @@
+
+# Python
+__pycache__/
+*.py[cod]
+*.pyd
+*.so
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Build artifacts
+build/
+dist/
+*.egg-info/
+.eggs/
+
+# IDE and local AI configuration
+.vscode/
+.cursor/
+.cursorrules
+.windsurfrules
+
+# OS files
+.DS_Store
+Thumbs.db
+desktop.ini
+
+# Local test outputs
+sample.csv
+cleaned.csv
+audit.md
+merged_test.xlsx
+merged_cleaned.csv
+merged_audit.md
+nested_test.xlsx
+nested_cleaned.csv
+nested_audit.md
+multi_sheet_test.xlsx
+sales_cleaned.csv
+sales_audit.md
+expenses_cleaned.csv
+expenses_audit.md
+employees_cleaned.csv
+employees_audit.md

sheet_unmess-1.0.0/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.5
+Name: sheet-unmess
+Version: 1.0.0
+Summary: Forensic Messy-Excel & Dirty-CSV Ingestor with Tamper-Evident Audit Receipts and Native MCP Server
+Project-URL: Homepage, https://github.com/sheet-unmess/sheet-unmess
+Project-URL: Documentation, https://github.com/sheet-unmess/sheet-unmess#readme
+Project-URL: Repository, https://github.com/sheet-unmess/sheet-unmess
+Author: Sheet Unmess Contributors
+License: MIT
+Keywords: audit,claude,csv,cursor,data-cleaning,excel,llm-agent,mcp,mcp-server,pandas,windsurf
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.10
+Requires-Dist: openpyxl>=3.1.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: python-dateutil>=2.8.2
+Provides-Extra: all
+Requires-Dist: polars>=0.20.0; extra == 'all'
+Provides-Extra: polars
+Requires-Dist: polars>=0.20.0; extra == 'polars'
+Description-Content-Type: text/markdown
+
+# sheet-unmess
+
+Forensic-style ingestion for messy Excel (`.xlsx`) and CSV files. It attempts to
+identify headers, remove common non-data rows, normalize common number formats,
+and produce a human-readable audit receipt.
+
+> **Important:** automatic header/row classification is heuristic. Always inspect
+> the output and reconciliation receipt before using cleaned data for financial,
+> regulatory, or other high-stakes decisions. A receipt is an audit report, not a
+> cryptographic signature or proof that no data was lost.
+
+## Features
+
+- Detects likely header rows and combines adjacent stacked headers.
+- Propagates values from the top-left cell across merged Excel ranges.
+- Classifies common title, blank, footnote, subtotal, and grand-total rows.
+- Parses common US/EU number formats, accounting negatives, and percentages.
+- Optionally returns a cleaning report with dropped-row details and total checks.
+- Provides a CLI and a stdio MCP server for compatible AI clients.
+
+## Install from source
+
+Python 3.10 or newer is required.
+
+```bash
+python -m venv .venv
+# Windows PowerShell:
+.venv\Scripts\Activate.ps1
+# macOS/Linux:
+source .venv/bin/activate
+
+python -m pip install --upgrade pip
+pip install -e .
+```
+
+## Quick start
+
+```python
+from sheet_unmess import read_messy
+
+df = read_messy("quarterly_report.xlsx")
+print(df.head())
+
+report = read_messy("quarterly_report.xlsx", return_report=True)
+print(report.data.head())
+print(report.receipt.to_markdown())
+```
+
+For European day-first dates, pass `date_dayfirst=True`.
+
+## CLI
+
+```bash
+sheet-unmess clean input.xlsx --out cleaned.csv --receipt audit.md
+sheet-unmess clean input.csv --out cleaned.csv --dayfirst
+sheet-unmess clean input.xlsx --sheet Financials --out cleaned.csv
+sheet-unmess init-ai
+```
+
+`--out` writes CSV by default; use a `.parquet` extension for Parquet output
+(and install a suitable Parquet engine such as `pyarrow`).
+
+## Tests
+
+```bash
+python -m pip install -e ".[dev]"
+python -m pytest
+```
+
+Or run the included test file directly:
+
+```bash
+python tests/test_sheet_unmess.py
+```
+
+## MCP server
+
+The package exposes `sheet-unmess-mcp` and `sheet-unmess mcp`. Example generic
+stdio MCP configuration:
+
+```json
+{
+  "mcpServers": {
+    "sheet-unmess": {
+      "command": "python",
+      "args": ["-m", "sheet_unmess.mcp_server"]
+    }
+  }
+}
+```
+
+Ensure the `python` executable points to the environment where `sheet-unmess`
+is installed. The `sheet-unmess init-ai` command writes project rules, a Cursor
+MCP configuration, and attempts to update the Claude Desktop configuration.
+Review generated configuration before using it.
+
+## Current limitations
+
+- `.xls` legacy Excel files are not supported by the current openpyxl loader.
+- Header detection and row classification are heuristic and may need manual review.
+- Reconciliation currently compares embedded totals against aggregated clean-row
+  sums; nested subtotals and multiple independent sections can require more
+  sophisticated grouping logic.
+- Mixed ambiguous dates cannot be inferred reliably; use `date_dayfirst` when
+  appropriate.
+- The MCP server reads files by local path and therefore needs filesystem access
+  to the file being processed.
+
+## License
+
+MIT

sheet_unmess-1.0.0/README.md

@@ -0,0 +1,111 @@
+# sheet-unmess
+
+Forensic-style ingestion for messy Excel (`.xlsx`) and CSV files. It attempts to
+identify headers, remove common non-data rows, normalize common number formats,
+and produce a human-readable audit receipt.
+
+> **Important:** automatic header/row classification is heuristic. Always inspect
+> the output and reconciliation receipt before using cleaned data for financial,
+> regulatory, or other high-stakes decisions. A receipt is an audit report, not a
+> cryptographic signature or proof that no data was lost.
+
+## Features
+
+- Detects likely header rows and combines adjacent stacked headers.
+- Propagates values from the top-left cell across merged Excel ranges.
+- Classifies common title, blank, footnote, subtotal, and grand-total rows.
+- Parses common US/EU number formats, accounting negatives, and percentages.
+- Optionally returns a cleaning report with dropped-row details and total checks.
+- Provides a CLI and a stdio MCP server for compatible AI clients.
+
+## Install from source
+
+Python 3.10 or newer is required.
+
+```bash
+python -m venv .venv
+# Windows PowerShell:
+.venv\Scripts\Activate.ps1
+# macOS/Linux:
+source .venv/bin/activate
+
+python -m pip install --upgrade pip
+pip install -e .
+```
+
+## Quick start
+
+```python
+from sheet_unmess import read_messy
+
+df = read_messy("quarterly_report.xlsx")
+print(df.head())
+
+report = read_messy("quarterly_report.xlsx", return_report=True)
+print(report.data.head())
+print(report.receipt.to_markdown())
+```
+
+For European day-first dates, pass `date_dayfirst=True`.
+
+## CLI
+
+```bash
+sheet-unmess clean input.xlsx --out cleaned.csv --receipt audit.md
+sheet-unmess clean input.csv --out cleaned.csv --dayfirst
+sheet-unmess clean input.xlsx --sheet Financials --out cleaned.csv
+sheet-unmess init-ai
+```
+
+`--out` writes CSV by default; use a `.parquet` extension for Parquet output
+(and install a suitable Parquet engine such as `pyarrow`).
+
+## Tests
+
+```bash
+python -m pip install -e ".[dev]"
+python -m pytest
+```
+
+Or run the included test file directly:
+
+```bash
+python tests/test_sheet_unmess.py
+```
+
+## MCP server
+
+The package exposes `sheet-unmess-mcp` and `sheet-unmess mcp`. Example generic
+stdio MCP configuration:
+
+```json
+{
+  "mcpServers": {
+    "sheet-unmess": {
+      "command": "python",
+      "args": ["-m", "sheet_unmess.mcp_server"]
+    }
+  }
+}
+```
+
+Ensure the `python` executable points to the environment where `sheet-unmess`
+is installed. The `sheet-unmess init-ai` command writes project rules, a Cursor
+MCP configuration, and attempts to update the Claude Desktop configuration.
+Review generated configuration before using it.
+
+## Current limitations
+
+- `.xls` legacy Excel files are not supported by the current openpyxl loader.
+- Header detection and row classification are heuristic and may need manual review.
+- Reconciliation currently compares embedded totals against aggregated clean-row
+  sums; nested subtotals and multiple independent sections can require more
+  sophisticated grouping logic.
+- Mixed ambiguous dates cannot be inferred reliably; use `date_dayfirst` when
+  appropriate.
+- The MCP server reads files by local path and therefore needs filesystem access
+  to the file being processed.
+
+## License
+
+MIT

sheet_unmess-1.0.0/pyproject.toml

@@ -0,0 +1,46 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "sheet-unmess"
+version = "1.0.0"
+description = "Forensic Messy-Excel & Dirty-CSV Ingestor with Tamper-Evident Audit Receipts and Native MCP Server"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+    { name = "Sheet Unmess Contributors" }
+]
+keywords = [
+    "excel", "csv", "data-cleaning", "pandas", "mcp", "mcp-server",
+    "cursor", "windsurf", "claude", "llm-agent", "audit"
+]
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Financial and Insurance Industry",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+    "pandas>=2.0.0",
+    "openpyxl>=3.1.0",
+    "python-dateutil>=2.8.2",
+]
+
+[project.optional-dependencies]
+polars = ["polars>=0.20.0"]
+all = ["polars>=0.20.0"]
+
+[project.scripts]
+sheet-unmess = "sheet_unmess.cli:main"
+sheet-unmess-mcp = "sheet_unmess.mcp_server:main"
+
+[project.urls]
+Homepage = "https://github.com/sheet-unmess/sheet-unmess"
+Documentation = "https://github.com/sheet-unmess/sheet-unmess#readme"
+Repository = "https://github.com/sheet-unmess/sheet-unmess"

sheet_unmess-1.0.0/sheet_unmess/__init__.py

@@ -0,0 +1,16 @@
+"""
+sheet-unmess: Forensic Messy-Excel & Dirty-CSV Ingestor + Audit Receipt.
+"""
+
+from sheet_unmess.core import read_messy, unmess
+from sheet_unmess.receipt import AuditReceipt, CleaningReport
+from sheet_unmess.ai_rules import install_ai_rules
+
+__version__ = "1.0.0"
+__all__ = [
+    "read_messy",
+    "unmess",
+    "AuditReceipt",
+    "CleaningReport",
+    "install_ai_rules",
+]

sheet_unmess-1.0.0/sheet_unmess/ai_rules.py

@@ -0,0 +1,79 @@
+"""AI editor rules and MCP configuration installer."""
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+
+CURSOR_RULE_CONTENT = """# Spreadsheet ingestion rules
+
+When working with Excel (.xlsx) or CSV files, prefer `sheet_unmess.read_messy`
+over raw pandas readers when the file may contain merged/multi-row headers,
+embedded subtotals, mixed locale-formatted numbers, or title/footnote rows.
+
+```python
+from sheet_unmess import read_messy
+
+report = read_messy("report.xlsx", return_report=True)
+df = report.data
+print(report.receipt.to_markdown())
+```
+"""
+
+def _merge_json(path: Path, key: str, value: dict) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    current = {}
+    if path.exists():
+        try:
+            current = json.loads(path.read_text(encoding="utf-8"))
+        except (json.JSONDecodeError, OSError):
+            current = {}
+    current.setdefault(key, {}).update(value)
+    path.write_text(json.dumps(current, indent=2), encoding="utf-8")
+
+def install_ai_rules(project_root: str = ".") -> dict[str, bool]:
+    """Install project-local rules and Cursor MCP config; preserve other MCP entries."""
+    root = Path(project_root).resolve()
+    results: dict[str, bool] = {}
+
+    try:
+        (root / ".cursorrules").write_text(CURSOR_RULE_CONTENT, encoding="utf-8")
+        results[".cursorrules"] = True
+    except OSError:
+        results[".cursorrules"] = False
+
+    try:
+        (root / ".windsurfrules").write_text(CURSOR_RULE_CONTENT, encoding="utf-8")
+        results[".windsurfrules"] = True
+    except OSError:
+        results[".windsurfrules"] = False
+
+    try:
+        _merge_json(
+            root / ".cursor" / "mcp.json",
+            "mcpServers",
+            {"sheet-unmess": {"command": "python", "args": ["-m", "sheet_unmess.mcp_server"]}},
+        )
+        results[".cursor/mcp.json"] = True
+    except (OSError, TypeError):
+        results[".cursor/mcp.json"] = False
+
+    # Claude Desktop locations differ by operating system.
+    home = Path.home()
+    if os.name == "nt":
+        claude_path = home / "AppData" / "Roaming" / "Claude" / "claude_desktop_config.json"
+    elif os.uname().sysname == "Darwin":
+        claude_path = home / "Library" / "Application Support" / "Claude" / "claude_desktop_config.json"
+    else:
+        claude_path = home / ".config" / "Claude" / "claude_desktop_config.json"
+
+    try:
+        _merge_json(
+            claude_path,
+            "mcpServers",
+            {"sheet-unmess": {"command": "python", "args": ["-m", "sheet_unmess.mcp_server"]}},
+        )
+        results["claude_desktop"] = True
+    except (OSError, TypeError):
+        results["claude_desktop"] = False
+    return results

sheet_unmess-1.0.0/sheet_unmess/classifier.py

@@ -0,0 +1,94 @@
+"""
+Row Classification Engine.
+Forensically categorizes every row into:
+- TITLE_BANNER
+- DATA
+- SUBTOTAL
+- GRAND_TOTAL
+- FOOTNOTE
+- BLANK_PADDING
+"""
+
+from __future__ import annotations
+import re
+from typing import Any, List, Optional
+from sheet_unmess.normalizer import parse_forensic_number
+
+RE_GRAND_TOTAL = re.compile(
+    r"\b(grand\s*total|overall\s*total|gesamtsumme|total\s*general|solde\s*final)\b",
+    re.IGNORECASE,
+)
+RE_SUBTOTAL = re.compile(
+    r"\b(sub\s*total|subtotal|summe|zwischensumme|sous\s*total|totales?|total)\b",
+    re.IGNORECASE,
+)
+RE_FOOTNOTE = re.compile(
+    r"^\s*([*†‡§#^]|note:|notes:|source:|disclaimer:|confidential|prepared\s+by|copyright)",
+    re.IGNORECASE,
+)
+
+
+class RowClassifier:
+    @staticmethod
+    def is_blank(row: List[Any]) -> bool:
+        for cell in row:
+            if cell is not None and str(cell).strip() not in ("", "None", "nan"):
+                return False
+        return True
+
+    @staticmethod
+    def is_footnote(row: List[Any], total_cols: int) -> bool:
+        # A footnote typically has content in the first 1-2 columns and is empty everywhere else
+        non_empty = [c for c in row if c is not None and str(c).strip() not in ("", "None", "nan")]
+        if not non_empty:
+            return False
+        first_str = str(non_empty[0]).strip()
+        if RE_FOOTNOTE.match(first_str):
+            return True
+        # If there is only 1 non-empty cell in a wide table, and it is a long text string (>30 chars)
+        if len(non_empty) == 1 and len(first_str) > 30 and total_cols >= 3:
+            return True
+        return False
+
+    @classmethod
+    def classify_row(
+        cls,
+        row: List[Any],
+        row_idx: int,
+        header_end_idx: int,
+        total_rows: int,
+        col_names: Optional[List[str]] = None,
+    ) -> str:
+        if cls.is_blank(row):
+            return "BLANK_PADDING"
+
+        # Rows prior to data block
+        if row_idx < header_end_idx:
+            return "TITLE_BANNER"
+
+        total_cols = len(row)
+        if cls.is_footnote(row, total_cols):
+            return "FOOTNOTE"
+
+        # Check for subtotal or grand total markers
+        first_cells_str = " ".join([str(c) for c in row[:3] if c is not None])
+
+        if RE_GRAND_TOTAL.search(first_cells_str):
+            return "GRAND_TOTAL"
+
+        if RE_SUBTOTAL.search(first_cells_str):
+            return "SUBTOTAL"
+
+        return "DATA"
+
+    @classmethod
+    def extract_row_numerics(cls, row: List[Any], col_names: List[str]) -> dict[str, float]:
+        """Extracts all parsable numbers from a row keyed by column name."""
+        numerics = {}
+        for c_idx, cell in enumerate(row):
+            if c_idx < len(col_names):
+                col = col_names[c_idx]
+                parsed, is_num = parse_forensic_number(cell)
+                if is_num and parsed is not None:
+                    numerics[col] = float(parsed)
+        return numerics

sheet_unmess-1.0.0/sheet_unmess/cli.py

@@ -0,0 +1,98 @@
+"""
+Rich Command-Line Interface for sheet-unmess.
+"""
+
+from __future__ import annotations
+import argparse
+import sys
+from pathlib import Path
+from sheet_unmess.core import read_messy
+from sheet_unmess.ai_rules import install_ai_rules
+from sheet_unmess.mcp_server import main as mcp_main
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="sheet-unmess",
+        description="Forensic Messy-Excel & Dirty-CSV Ingestor + Tamper-Evident Audit Receipt."
+    )
+    subparsers = parser.add_subparsers(dest="command")
+
+    # Command: clean
+    clean_p = subparsers.add_parser("clean", help="Clean a messy spreadsheet")
+    clean_p.add_argument("file", help="Path to .xlsx or .csv")
+    clean_p.add_argument("--sheet", default=0, help="Sheet name or index (for Excel)")
+    clean_p.add_argument("--out", "-o", help="Output path (.csv or .parquet)")
+    clean_p.add_argument("--receipt", "-r", help="Save Audit Receipt markdown file path")
+    clean_p.add_argument("--dayfirst", action="store_true", help="Parse dates EU style (DD/MM/YYYY)")
+
+    # Command: init-ai
+    subparsers.add_parser("init-ai", help="Inject MCP server and rules into Cursor, Windsurf, and Claude")
+
+    # Command: mcp
+    subparsers.add_parser("mcp", help="Launch native Stdio Model Context Protocol (MCP) server")
+
+    args = parser.parse_args()
+
+    if args.command == "mcp":
+        mcp_main()
+        return
+
+    if args.command == "init-ai":
+        res = install_ai_rules()
+        print("🚀 Successfully provisioned AI agent configurations:")
+        for target, status in res.items():
+            icon = "✅" if status else "⚠️"
+            print(f"  {icon} {target}")
+        print("\nYour AI assistant (Cursor, Windsurf, Claude) is now primed to automatically clean spreadsheets with sheet-unmess!")
+        return
+
+    if args.command == "clean":
+        file_path = Path(args.file)
+        if not file_path.exists():
+            print(f"❌ File not found: {file_path}", file=sys.stderr)
+            sys.exit(1)
+
+        print(f"🔍 Forensically ingesting `{file_path}`...")
+        report = read_messy(
+            file_path,
+            sheet_name=args.sheet,
+            date_dayfirst=args.dayfirst,
+            return_report=True
+        )
+
+        df = report.data
+        receipt = report.receipt
+
+        print(f"✨ Ingestion complete: {receipt.raw_shape} ➔ {receipt.clean_shape} in {receipt.execution_time_ms:.1f}ms")
+
+        if receipt.reconciliations:
+            print("\n📊 Invariant Verification:")
+            for rec in receipt.reconciliations:
+                icon = "✅" if rec.status in ("VERIFIED_EXACT", "ROUNDING_TOLERANCE") else "🚨"
+                print(f"  {icon} {rec.column}: reported {rec.reported_total:,.2f} vs sum {rec.computed_sum:,.2f} ({rec.status})")
+
+        # Save output data
+        if args.out:
+            out_p = Path(args.out)
+            if out_p.suffix == ".parquet":
+                df.to_parquet(out_p, index=False)
+            else:
+                df.to_csv(out_p, index=False)
+            print(f"\n💾 Clean data written to: {out_p}")
+        else:
+            print("\nPreview of cleaned data:")
+            print(df.head(10).to_string())
+
+        # Save receipt
+        if args.receipt:
+            rec_p = Path(args.receipt)
+            rec_p.write_text(receipt.to_markdown(), encoding="utf-8")
+            print(f"🛡️ Audit Receipt written to: {rec_p}")
+
+    else:
+        parser.print_help()
+
+
+if __name__ == "__main__":
+    main()