csvnorm 0.3.0__tar.gz

csvnorm-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 aborruso@gmail.com
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

csvnorm-0.3.0/PKG-INFO

@@ -0,0 +1,217 @@
+Metadata-Version: 2.4
+Name: csvnorm
+Version: 0.3.0
+Summary: A command-line utility to validate and normalize CSV files
+Author-email: aborruso <aborruso@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 aborruso@gmail.com
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/aborruso/prepare_data
+Project-URL: Issues, https://github.com/aborruso/prepare_data/issues
+Keywords: csv,data,normalization,validation,etl
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: charset-normalizer>=3.0.0
+Requires-Dist: duckdb>=0.9.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: rich-argparse>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Provides-Extra: banner
+Requires-Dist: pyfiglet>=1.0.0; extra == "banner"
+Dynamic: license-file
+
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/aborruso/prepare_data)
+[![PyPI version](https://badge.fury.io/py/csvnorm.svg)](https://pypi.org/project/csvnorm/)
+
+# csvnorm
+
+A command-line utility to validate and normalize CSV files for initial exploration.
+
+## Installation
+
+Recommended (uv):
+
+```bash
+uv tool install csvnorm
+```
+
+Or with pip:
+
+```bash
+pip install csvnorm
+```
+
+For ASCII banner in verbose mode:
+
+```bash
+uv tool install csvnorm[banner]
+# or
+pip install csvnorm[banner]
+```
+
+## Purpose
+
+This tool prepares CSV files for **basic exploratory data analysis (EDA)**, not for complex transformations. It focuses on achieving a clean, standardized baseline format that allows you to quickly assess data quality and structure before designing more sophisticated ETL pipelines.
+
+**What it does:**
+- Validates CSV structure and reports errors
+- Normalizes encoding to UTF-8
+- Normalizes delimiters and field names
+- Creates a consistent starting point for data exploration
+
+**What it doesn't do:**
+- Complex data transformations or business logic
+- Type inference or data validation beyond structure
+- Heavy processing or aggregations
+
+## Features
+
+- **CSV Validation**: Checks for common CSV errors and inconsistencies using DuckDB
+- **Delimiter Normalization**: Converts all field separators to standard commas (`,`)
+- **Field Name Normalization**: Converts column headers to snake_case format
+- **Encoding Normalization**: Auto-detects encoding and converts to UTF-8
+- **Error Reporting**: Exports detailed error file for invalid rows
+
+## Usage
+
+```bash
+csvnorm input.csv [options]
+```
+
+### Options
+
+| Option | Description |
+|--------|-------------|
+| `-f, --force` | Force overwrite of existing output files |
+| `-n, --keep-names` | Keep original column names (disable snake_case) |
+| `-d, --delimiter CHAR` | Set custom output delimiter (default: `,`) |
+| `-o, --output-dir DIR` | Set output directory (default: current dir) |
+| `-v, --verbose` | Enable verbose output for debugging |
+| `--version` | Show version number |
+| `-h, --help` | Show help message |
+
+### Examples
+
+```bash
+# Basic usage
+csvnorm data.csv
+
+# With semicolon delimiter
+csvnorm data.csv -d ';'
+
+# Custom output directory
+csvnorm data.csv -o ./output
+
+# Keep original headers
+csvnorm data.csv --keep-names
+
+# Force overwrite with verbose output
+csvnorm data.csv -f -v
+```
+
+### Output
+
+Creates a normalized CSV file in the specified output directory with:
+- UTF-8 encoding
+- Consistent field delimiters
+- Normalized column names (unless `--keep-names` is specified)
+- Error report if any invalid rows are found (saved as `{input_name}_reject_errors.csv`)
+
+The tool provides modern terminal output with:
+- Progress indicators for multi-step processing
+- Color-coded error messages with panels
+- Success summary table showing encoding, paths, and settings
+- Optional ASCII art banner in verbose mode (requires `pyfiglet`)
+
+### Exit Codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | Success |
+| 1 | Error (validation failed, file not found, etc.) |
+
+## Requirements
+
+- Python 3.8+
+- Dependencies (automatically installed):
+  - `charset-normalizer>=3.0.0` - Encoding detection
+  - `duckdb>=0.9.0` - CSV validation and normalization
+  - `rich>=13.0.0` - Modern terminal output formatting
+  - `rich-argparse>=1.0.0` - Enhanced CLI help formatting
+
+Optional:
+- `pyfiglet>=1.0.0` - ASCII art banner in verbose mode (install with `pip install csvnorm[banner]`)
+
+## Development
+
+### Setup
+
+```bash
+git clone https://github.com/aborruso/prepare_data
+cd prepare_data
+pip install -e ".[dev]"
+```
+
+### Testing
+
+```bash
+pytest tests/ -v
+```
+
+### Project Structure
+
+```
+prepare_data/
+├── src/csvnorm/
+│   ├── __init__.py      # Package version
+│   ├── __main__.py      # python -m support
+│   ├── cli.py           # CLI argument parsing
+│   ├── core.py          # Main processing pipeline
+│   ├── encoding.py      # Encoding detection/conversion
+│   ├── validation.py    # DuckDB validation
+│   └── utils.py         # Helper functions
+├── tests/               # Test suite
+├── test/                # CSV fixtures
+└── pyproject.toml       # Package configuration
+```
+
+## License
+
+MIT License (c) 2026 aborruso@gmail.com - See LICENSE file for details

csvnorm-0.3.0/README.md

@@ -0,0 +1,158 @@
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/aborruso/prepare_data)
+[![PyPI version](https://badge.fury.io/py/csvnorm.svg)](https://pypi.org/project/csvnorm/)
+
+# csvnorm
+
+A command-line utility to validate and normalize CSV files for initial exploration.
+
+## Installation
+
+Recommended (uv):
+
+```bash
+uv tool install csvnorm
+```
+
+Or with pip:
+
+```bash
+pip install csvnorm
+```
+
+For ASCII banner in verbose mode:
+
+```bash
+uv tool install csvnorm[banner]
+# or
+pip install csvnorm[banner]
+```
+
+## Purpose
+
+This tool prepares CSV files for **basic exploratory data analysis (EDA)**, not for complex transformations. It focuses on achieving a clean, standardized baseline format that allows you to quickly assess data quality and structure before designing more sophisticated ETL pipelines.
+
+**What it does:**
+- Validates CSV structure and reports errors
+- Normalizes encoding to UTF-8
+- Normalizes delimiters and field names
+- Creates a consistent starting point for data exploration
+
+**What it doesn't do:**
+- Complex data transformations or business logic
+- Type inference or data validation beyond structure
+- Heavy processing or aggregations
+
+## Features
+
+- **CSV Validation**: Checks for common CSV errors and inconsistencies using DuckDB
+- **Delimiter Normalization**: Converts all field separators to standard commas (`,`)
+- **Field Name Normalization**: Converts column headers to snake_case format
+- **Encoding Normalization**: Auto-detects encoding and converts to UTF-8
+- **Error Reporting**: Exports detailed error file for invalid rows
+
+## Usage
+
+```bash
+csvnorm input.csv [options]
+```
+
+### Options
+
+| Option | Description |
+|--------|-------------|
+| `-f, --force` | Force overwrite of existing output files |
+| `-n, --keep-names` | Keep original column names (disable snake_case) |
+| `-d, --delimiter CHAR` | Set custom output delimiter (default: `,`) |
+| `-o, --output-dir DIR` | Set output directory (default: current dir) |
+| `-v, --verbose` | Enable verbose output for debugging |
+| `--version` | Show version number |
+| `-h, --help` | Show help message |
+
+### Examples
+
+```bash
+# Basic usage
+csvnorm data.csv
+
+# With semicolon delimiter
+csvnorm data.csv -d ';'
+
+# Custom output directory
+csvnorm data.csv -o ./output
+
+# Keep original headers
+csvnorm data.csv --keep-names
+
+# Force overwrite with verbose output
+csvnorm data.csv -f -v
+```
+
+### Output
+
+Creates a normalized CSV file in the specified output directory with:
+- UTF-8 encoding
+- Consistent field delimiters
+- Normalized column names (unless `--keep-names` is specified)
+- Error report if any invalid rows are found (saved as `{input_name}_reject_errors.csv`)
+
+The tool provides modern terminal output with:
+- Progress indicators for multi-step processing
+- Color-coded error messages with panels
+- Success summary table showing encoding, paths, and settings
+- Optional ASCII art banner in verbose mode (requires `pyfiglet`)
+
+### Exit Codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | Success |
+| 1 | Error (validation failed, file not found, etc.) |
+
+## Requirements
+
+- Python 3.8+
+- Dependencies (automatically installed):
+  - `charset-normalizer>=3.0.0` - Encoding detection
+  - `duckdb>=0.9.0` - CSV validation and normalization
+  - `rich>=13.0.0` - Modern terminal output formatting
+  - `rich-argparse>=1.0.0` - Enhanced CLI help formatting
+
+Optional:
+- `pyfiglet>=1.0.0` - ASCII art banner in verbose mode (install with `pip install csvnorm[banner]`)
+
+## Development
+
+### Setup
+
+```bash
+git clone https://github.com/aborruso/prepare_data
+cd prepare_data
+pip install -e ".[dev]"
+```
+
+### Testing
+
+```bash
+pytest tests/ -v
+```
+
+### Project Structure
+
+```
+prepare_data/
+├── src/csvnorm/
+│   ├── __init__.py      # Package version
+│   ├── __main__.py      # python -m support
+│   ├── cli.py           # CLI argument parsing
+│   ├── core.py          # Main processing pipeline
+│   ├── encoding.py      # Encoding detection/conversion
+│   ├── validation.py    # DuckDB validation
+│   └── utils.py         # Helper functions
+├── tests/               # Test suite
+├── test/                # CSV fixtures
+└── pyproject.toml       # Package configuration
+```
+
+## License
+
+MIT License (c) 2026 aborruso@gmail.com - See LICENSE file for details

csvnorm-0.3.0/pyproject.toml

@@ -0,0 +1,62 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "csvnorm"
+version = "0.3.0"
+description = "A command-line utility to validate and normalize CSV files"
+readme = "README.md"
+requires-python = ">=3.9"
+license = {file = "LICENSE"}
+authors = [
+    {name = "aborruso", email = "aborruso@gmail.com"}
+]
+keywords = ["csv", "data", "normalization", "validation", "etl"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+    "Topic :: Software Development :: Libraries",
+    "Topic :: Utilities",
+]
+
+dependencies = [
+    "charset-normalizer>=3.0.0",
+    "duckdb>=0.9.0",
+    "rich>=13.0.0",
+    "rich-argparse>=1.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0.0",
+    "pytest-cov>=4.0.0",
+    "ruff>=0.1.0",
+]
+banner = [
+    "pyfiglet>=1.0.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/aborruso/prepare_data"
+Issues = "https://github.com/aborruso/prepare_data/issues"
+
+[project.scripts]
+csvnorm = "csvnorm.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 88
+target-version = "py38"

csvnorm-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

csvnorm-0.3.0/src/csvnorm/__init__.py

@@ -0,0 +1,8 @@
+"""csvnorm - Validate and normalize CSV files."""
+
+__version__ = "0.3.0"
+__all__ = ["normalize_csv", "detect_encoding", "process_csv"]
+
+from csvnorm.core import process_csv
+from csvnorm.encoding import detect_encoding
+from csvnorm.validation import normalize_csv

csvnorm-0.3.0/src/csvnorm/__main__.py

@@ -0,0 +1,6 @@
+"""Entry point for python -m csvnorm."""
+
+from csvnorm.cli import main
+
+if __name__ == "__main__":
+    main()

csvnorm-0.3.0/src/csvnorm/cli.py

@@ -0,0 +1,137 @@
+"""Command-line interface for csvnorm."""
+
+import argparse
+import sys
+from pathlib import Path
+
+from rich.console import Console
+from rich_argparse import RichHelpFormatter
+
+from csvnorm import __version__
+from csvnorm.core import process_csv
+from csvnorm.utils import setup_logger
+
+console = Console()
+
+
+def show_banner() -> None:
+    """Show ASCII art banner if pyfiglet is available."""
+    try:
+        from pyfiglet import figlet_format
+        banner = figlet_format("csvnorm", font="slant")
+        console.print(banner, style="bold cyan")
+    except ImportError:
+        # pyfiglet not installed, skip banner
+        pass
+
+
+def create_parser() -> argparse.ArgumentParser:
+    """Create and return the argument parser."""
+    parser = argparse.ArgumentParser(
+        prog="csvnorm",
+        description="Validate and normalize CSV files for exploratory data analysis",
+        formatter_class=RichHelpFormatter,
+        epilog="""\
+Examples:
+  csvnorm data.csv -d ';' -o output_folder --force
+  csvnorm data.csv --keep-names --delimiter '\\t'
+  csvnorm data.csv -v
+""",
+    )
+
+    parser.add_argument(
+        "input_file",
+        type=Path,
+        help="Input CSV file path",
+    )
+
+    parser.add_argument(
+        "-f",
+        "--force",
+        action="store_true",
+        help="Force overwrite of existing output files",
+    )
+
+    parser.add_argument(
+        "-n",
+        "--keep-names",
+        action="store_true",
+        help=(
+            "Keep original column names (disable snake_case normalization). "
+            "By default, column names are converted to snake_case format "
+            "(e.g., 'Column Name' becomes 'column_name')."
+        ),
+    )
+
+    parser.add_argument(
+        "-d",
+        "--delimiter",
+        default=",",
+        help="Set custom field delimiter (default: comma). Example: -d ';'",
+    )
+
+    parser.add_argument(
+        "-o",
+        "--output-dir",
+        type=Path,
+        default=Path.cwd(),
+        help="Set custom output directory (default: current working directory)",
+    )
+
+    parser.add_argument(
+        "-v",
+        "--verbose",
+        action="store_true",
+        help="Enable verbose output for debugging",
+    )
+
+    parser.add_argument(
+        "--version",
+        action="version",
+        version=f"%(prog)s {__version__}",
+    )
+
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    """Main entry point for the CLI.
+
+    Args:
+        argv: Command line arguments (defaults to sys.argv[1:]).
+
+    Returns:
+        Exit code: 0 for success, 1 for error.
+    """
+    parser = create_parser()
+
+    # Handle missing arguments gracefully
+    if argv is None:
+        argv = sys.argv[1:]
+
+    if not argv or (len(argv) == 1 and argv[0] in ['-h', '--help']):
+        parser.print_help()
+        return 0 if argv else 2
+
+    args = parser.parse_args(argv)
+
+    # Show banner in verbose mode
+    if args.verbose:
+        show_banner()
+
+    # Setup logging
+    setup_logger(args.verbose)
+
+    # Run processing
+    return process_csv(
+        input_file=args.input_file,
+        output_dir=args.output_dir,
+        force=args.force,
+        keep_names=args.keep_names,
+        delimiter=args.delimiter,
+        verbose=args.verbose,
+    )
+
+
+if __name__ == "__main__":
+    sys.exit(main())