csvsmith 0.2.0__tar.gz

csvsmith-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Eiichi YAMAMOTO 
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights  
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell  
+copies of the Software, and to permit persons to whom the Software is  
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in  
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR  
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,  
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE  
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER  
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING  
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS  
+IN THE SOFTWARE.

csvsmith-0.2.0/PKG-INFO

@@ -0,0 +1,221 @@
+Metadata-Version: 2.4
+Name: csvsmith
+Version: 0.2.0
+Summary: Small CSV utilities: classification, duplicates, row digests, and CLI helpers.
+Author-email: Eiichi YAMAMOTO <info@yeiichi.com>
+License: MIT License
+        
+        Copyright (c) 2025 Eiichi YAMAMOTO 
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights  
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell  
+        copies of the Software, and to permit persons to whom the Software is  
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in  
+        all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR  
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,  
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE  
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER  
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING  
+        FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS  
+        IN THE SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/yeiichi/csvsmith
+Project-URL: Repository, https://github.com/yeiichi/csvsmith
+Keywords: csv,pandas,duplicates,data-cleaning,file-organization
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=2.0
+Dynamic: license-file
+
+# csvsmith
+
+[![PyPI version](https://img.shields.io/pypi/v/csvsmith.svg)](https://pypi.org/project/csvsmith/)
+![Python versions](https://img.shields.io/pypi/pyversions/csvsmith.svg)
+[![License](https://img.shields.io/pypi/l/csvsmith.svg)](https://pypi.org/project/csvsmith/)
+
+## Introduction
+
+`csvsmith` is a lightweight collection of CSV utilities designed for
+data integrity, deduplication, and organization. It provides a robust
+Python API for programmatic data cleaning and a convenient CLI for quick
+operations. Whether you need to organize thousands of files based on
+their structural signatures or pinpoint duplicate rows in a complex
+dataset, `csvsmith` ensures the process is predictable, transparent, and
+reversible.
+
+## Table of Contents
+
+-   [Installation](#installation)
+
+-   
+
+    [Python API Usage](#python-api-usage)
+
+    :   -   [Count duplicate values](#count-duplicate-values)
+        -   [Find duplicate rows in a
+            DataFrame](#find-duplicate-rows-in-a-dataframe)
+        -   [Deduplicate with report](#deduplicate-with-report)
+        -   [CSV File Classification](#csv-file-classification)
+
+-   
+
+    [CLI Usage](#cli-usage)
+
+    :   -   [Show duplicate rows](#show-duplicate-rows)
+        -   [Deduplicate and generate a duplicate
+            report](#deduplicate-and-generate-a-duplicate-report)
+        -   [Classify CSVs](#classify-csvs)
+
+-   [Philosophy](#philosophy)
+
+-   [License](#license)
+
+## Installation
+
+From PyPI:
+
+``` bash
+pip install csvsmith
+```
+
+For local development:
+
+``` bash
+git clone https://github.com/yeiichi/csvsmith.git
+cd csvsmith
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .[dev]
+```
+
+## Python API Usage
+
+### Count duplicate values
+
+Works on any iterable of hashable items.
+
+``` python
+from csvsmith import count_duplicates_sorted
+
+items = ["a", "b", "a", "c", "a", "b"]
+print(count_duplicates_sorted(items))
+# [('a', 3), ('b', 2)]
+```
+
+### Find duplicate rows in a DataFrame
+
+``` python
+import pandas as pd
+from csvsmith import find_duplicate_rows
+
+df = pd.read_csv("input.csv")
+dup_rows = find_duplicate_rows(df)
+print(dup_rows)
+```
+
+### Deduplicate with report
+
+``` python
+import pandas as pd
+from csvsmith import dedupe_with_report
+
+df = pd.read_csv("input.csv")
+
+# Use all columns
+deduped, report = dedupe_with_report(df)
+deduped.to_csv("deduped.csv", index=False)
+report.to_csv("duplicate_report.csv", index=False)
+
+# Use all columns except an ID column
+deduped_no_id, report_no_id = dedupe_with_report(df, exclude=["id"])
+```
+
+### CSV File Classification
+
+Organize files into directories based on their headers.
+
+``` python
+from csvsmith.classify import CSVClassifier
+
+classifier = CSVClassifier(
+    source_dir="./raw_data",
+    dest_dir="./organized",
+    auto=True  # Automatically group files with identical headers
+)
+
+# Execute the classification
+classifier.run()
+
+# Or rollback a previous run using its manifest
+classifier.rollback("./organized/manifest_20260121_120000.json")
+```
+
+## CLI Usage
+
+`csvsmith` includes a command-line interface for duplicate detection and
+file organization.
+
+### Show duplicate rows
+
+``` bash
+csvsmith row-duplicates input.csv
+```
+
+Save only duplicate rows to a file:
+
+``` bash
+csvsmith row-duplicates input.csv -o duplicates_only.csv
+```
+
+### Deduplicate and generate a duplicate report
+
+``` bash
+csvsmith dedupe input.csv --deduped deduped.csv --report duplicate_report.csv
+```
+
+### Classify CSVs
+
+Organize a mess of CSV files into structured folders based on their
+column headers.
+
+``` bash
+# Preview what would happen (Dry Run)
+csvsmith classify --src ./raw_data --dest ./organized --auto --dry-run
+
+# Run classification with a signature config
+csvsmith classify --src ./raw_data --dest ./organized --config signatures.json
+
+# Undo a classification run
+csvsmith classify --rollback ./organized/manifest_20260121_120000.json
+```
+
+## Philosophy
+
+1.  CSVs deserve tools that are simple, predictable, and transparent.
+2.  A row has meaning only when its identity is stable and hashable.
+3.  Collisions are sin; determinism is virtue.
+4.  Let no delimiter sow ambiguity among fields.
+5.  **Love thy \\x1f.** The unseen separator, the quiet guardian of
+    clean hashes.
+6.  The pipeline should be silent unless something is wrong.
+7.  Your data deserves respect --- and your tools should help you give
+    it.
+
+For more, see `MANIFESTO.md`.
+
+## License
+
+MIT License.

csvsmith-0.2.0/README.md

@@ -0,0 +1,179 @@
+# csvsmith
+
+[![PyPI version](https://img.shields.io/pypi/v/csvsmith.svg)](https://pypi.org/project/csvsmith/)
+![Python versions](https://img.shields.io/pypi/pyversions/csvsmith.svg)
+[![License](https://img.shields.io/pypi/l/csvsmith.svg)](https://pypi.org/project/csvsmith/)
+
+## Introduction
+
+`csvsmith` is a lightweight collection of CSV utilities designed for
+data integrity, deduplication, and organization. It provides a robust
+Python API for programmatic data cleaning and a convenient CLI for quick
+operations. Whether you need to organize thousands of files based on
+their structural signatures or pinpoint duplicate rows in a complex
+dataset, `csvsmith` ensures the process is predictable, transparent, and
+reversible.
+
+## Table of Contents
+
+-   [Installation](#installation)
+
+-   
+
+    [Python API Usage](#python-api-usage)
+
+    :   -   [Count duplicate values](#count-duplicate-values)
+        -   [Find duplicate rows in a
+            DataFrame](#find-duplicate-rows-in-a-dataframe)
+        -   [Deduplicate with report](#deduplicate-with-report)
+        -   [CSV File Classification](#csv-file-classification)
+
+-   
+
+    [CLI Usage](#cli-usage)
+
+    :   -   [Show duplicate rows](#show-duplicate-rows)
+        -   [Deduplicate and generate a duplicate
+            report](#deduplicate-and-generate-a-duplicate-report)
+        -   [Classify CSVs](#classify-csvs)
+
+-   [Philosophy](#philosophy)
+
+-   [License](#license)
+
+## Installation
+
+From PyPI:
+
+``` bash
+pip install csvsmith
+```
+
+For local development:
+
+``` bash
+git clone https://github.com/yeiichi/csvsmith.git
+cd csvsmith
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .[dev]
+```
+
+## Python API Usage
+
+### Count duplicate values
+
+Works on any iterable of hashable items.
+
+``` python
+from csvsmith import count_duplicates_sorted
+
+items = ["a", "b", "a", "c", "a", "b"]
+print(count_duplicates_sorted(items))
+# [('a', 3), ('b', 2)]
+```
+
+### Find duplicate rows in a DataFrame
+
+``` python
+import pandas as pd
+from csvsmith import find_duplicate_rows
+
+df = pd.read_csv("input.csv")
+dup_rows = find_duplicate_rows(df)
+print(dup_rows)
+```
+
+### Deduplicate with report
+
+``` python
+import pandas as pd
+from csvsmith import dedupe_with_report
+
+df = pd.read_csv("input.csv")
+
+# Use all columns
+deduped, report = dedupe_with_report(df)
+deduped.to_csv("deduped.csv", index=False)
+report.to_csv("duplicate_report.csv", index=False)
+
+# Use all columns except an ID column
+deduped_no_id, report_no_id = dedupe_with_report(df, exclude=["id"])
+```
+
+### CSV File Classification
+
+Organize files into directories based on their headers.
+
+``` python
+from csvsmith.classify import CSVClassifier
+
+classifier = CSVClassifier(
+    source_dir="./raw_data",
+    dest_dir="./organized",
+    auto=True  # Automatically group files with identical headers
+)
+
+# Execute the classification
+classifier.run()
+
+# Or rollback a previous run using its manifest
+classifier.rollback("./organized/manifest_20260121_120000.json")
+```
+
+## CLI Usage
+
+`csvsmith` includes a command-line interface for duplicate detection and
+file organization.
+
+### Show duplicate rows
+
+``` bash
+csvsmith row-duplicates input.csv
+```
+
+Save only duplicate rows to a file:
+
+``` bash
+csvsmith row-duplicates input.csv -o duplicates_only.csv
+```
+
+### Deduplicate and generate a duplicate report
+
+``` bash
+csvsmith dedupe input.csv --deduped deduped.csv --report duplicate_report.csv
+```
+
+### Classify CSVs
+
+Organize a mess of CSV files into structured folders based on their
+column headers.
+
+``` bash
+# Preview what would happen (Dry Run)
+csvsmith classify --src ./raw_data --dest ./organized --auto --dry-run
+
+# Run classification with a signature config
+csvsmith classify --src ./raw_data --dest ./organized --config signatures.json
+
+# Undo a classification run
+csvsmith classify --rollback ./organized/manifest_20260121_120000.json
+```
+
+## Philosophy
+
+1.  CSVs deserve tools that are simple, predictable, and transparent.
+2.  A row has meaning only when its identity is stable and hashable.
+3.  Collisions are sin; determinism is virtue.
+4.  Let no delimiter sow ambiguity among fields.
+5.  **Love thy \\x1f.** The unseen separator, the quiet guardian of
+    clean hashes.
+6.  The pipeline should be silent unless something is wrong.
+7.  Your data deserves respect --- and your tools should help you give
+    it.
+
+For more, see `MANIFESTO.md`.
+
+## License
+
+MIT License.

csvsmith-0.2.0/pyproject.toml

@@ -0,0 +1,38 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "csvsmith"
+version = "0.2.0"
+description = "Small CSV utilities: classification, duplicates, row digests, and CLI helpers."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { file = "LICENSE" }
+
+authors = [
+  { name = "Eiichi YAMAMOTO", email = "info@yeiichi.com" }
+]
+keywords = ["csv", "pandas", "duplicates", "data-cleaning", "file-organization"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "License :: OSI Approved :: MIT License",
+  "Intended Audience :: Developers",
+  "Topic :: Software Development :: Libraries",
+  "Topic :: Utilities",
+]
+
+dependencies = [
+  "pandas>=2.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/yeiichi/csvsmith"
+Repository = "https://github.com/yeiichi/csvsmith"
+
+[project.scripts]
+csvsmith = "csvsmith.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

csvsmith-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

csvsmith-0.2.0/src/csvsmith/__init__.py

@@ -0,0 +1,26 @@
+"""
+csvsmith: small, focused CSV utilities.
+
+Current submodules:
+- csvsmith.duplicates
+- csvsmith.classify
+- csvsmith.cli (CLI entrypoint)
+"""
+
+__version__ = "0.2.0"
+
+from .duplicates import (
+    count_duplicates_sorted,
+    add_row_digest,
+    find_duplicate_rows,
+    dedupe_with_report,
+)
+from .classify import CSVClassifier
+
+__all__ = [
+    "count_duplicates_sorted",
+    "add_row_digest",
+    "find_duplicate_rows",
+    "dedupe_with_report",
+    "CSVClassifier",
+]

csvsmith-0.2.0/src/csvsmith/classify.py

@@ -0,0 +1,143 @@
+import csv
+import json
+import shutil
+from datetime import datetime
+from pathlib import Path
+
+
+class CSVClassifier:
+    """
+    Classifies CSV files into folders based on header signatures.
+    Supports predefined mapping, auto-discovery, dry-runs, and rollbacks.
+    """
+
+    def __init__(self, source_dir, dest_dir, signatures=None, auto=False, dry_run=False):
+        self.source = Path(source_dir)
+        self.dest = Path(dest_dir)
+        self.signatures = signatures or {}
+        self.auto = auto
+        self.dry_run = dry_run
+        self.manifest = {
+            "source_path": str(self.source.absolute()),
+            "timestamp": datetime.now().isoformat(),
+            "operations": []
+        }
+
+    def _get_headers(self, file_path):
+        """Validates if file is a CSV and extracts the first row as a header."""
+        if not file_path.suffix.lower() == '.csv':
+            return None
+        try:
+            with open(file_path, 'r', encoding='utf-8-sig', newline='') as f:
+                reader = csv.reader(f)
+                header = next(reader, None)
+
+                if not header:
+                    return None
+
+                # Rule: If the first row is purely numeric, it is data, not a header.
+                if all(str(c).strip().replace('.', '', 1).isdigit() for c in header if c.strip()):
+                    return None
+
+                return [h.strip() for h in header if h.strip()]
+        except (UnicodeDecodeError, csv.Error):
+            return None
+
+    def _move_file(self, file_path, category, headers):
+        """Executes move with duplicate protection and records in manifest."""
+        target_dir = self.dest / category
+        dest_file = target_dir / file_path.name
+
+        # Handle duplicate filenames in destination
+        if dest_file.exists() and not self.dry_run:
+            timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
+            dest_file = target_dir / f"{file_path.stem}_{timestamp}{file_path.suffix}"
+
+        operation_log = {
+            "original_path": str(file_path.absolute()),
+            "moved_to": str(dest_file.absolute()) if not self.dry_run else "simulated",
+            "category": category,
+            "headers": headers,
+            "status": "pending"
+        }
+
+        if self.dry_run:
+            print(f"[DRY RUN] Would move: {file_path.name} -> {category}/")
+            operation_log["status"] = "simulated"
+        else:
+            try:
+                target_dir.mkdir(parents=True, exist_ok=True)
+                shutil.move(str(file_path), str(dest_file))
+                print(f"Moved: {file_path.name} -> {category}/")
+                operation_log["status"] = "success"
+            except Exception as e:
+                print(f"Failed to move {file_path.name}: {e}")
+                operation_log["status"] = "failed"
+
+        self.manifest["operations"].append(operation_log)
+
+    def _save_manifest(self):
+        """Saves the session manifest to the destination directory."""
+        if not self.manifest["operations"] or self.dry_run:
+            return
+
+        ts = datetime.now().strftime('%Y%m%d_%H%M%S')
+        manifest_path = self.dest / f"manifest_{ts}.json"
+
+        with open(manifest_path, 'w', encoding='utf-8') as f:
+            json.dump(self.manifest, f, indent=4)
+        print(f"\nManifest saved: {manifest_path}")
+
+    def rollback(self, manifest_path):
+        """Reverses operations defined in a manifest file."""
+        m_path = Path(manifest_path)
+        if not m_path.exists():
+            print(f"Error: Manifest {manifest_path} not found.")
+            return
+
+        with open(m_path, 'r') as f:
+            data = json.load(f)
+
+        print(f"Starting rollback for session: {data.get('timestamp')}")
+        for op in data.get("operations", []):
+            if op["status"] != "success":
+                continue
+
+            current_loc = Path(op["moved_to"])
+            original_loc = Path(op["original_path"])
+
+            if current_loc.exists():
+                if self.dry_run:
+                    print(f"[DRY RUN] Would restore: {current_loc.name} -> {original_loc}")
+                else:
+                    original_loc.parent.mkdir(parents=True, exist_ok=True)
+                    shutil.move(str(current_loc), str(original_loc))
+                    print(f"Restored: {current_loc.name}")
+            else:
+                print(f"Warning: Could not find file to restore: {current_loc}")
+
+    def run(self):
+        """Standard classification run."""
+        if not self.source.is_dir():
+            print(f"Error: Source directory {self.source} does not exist.")
+            return
+
+        for file in self.source.glob("*.csv"):
+            headers = self._get_headers(file)
+            target_sub = "unclassified"
+
+            if headers:
+                match_found = False
+                for cat, reqs in self.signatures.items():
+                    if all(r in headers for r in reqs):
+                        target_sub = cat
+                        match_found = True
+                        break
+
+                if not match_found and self.auto:
+                    slug = "_".join(sorted(headers))[:50]
+                    target_sub = f"cluster_{slug}"
+
+            self._move_file(file, target_sub, headers)
+
+        self._save_manifest()