datacompose 0.2.4__py3-none-any.whl

datacompose-0.2.4.dist-info/METADATA

@@ -0,0 +1,431 @@
+Metadata-Version: 2.4
+Name: datacompose
+Version: 0.2.4
+Summary: Copy-pasteable data transformation primitives for PySpark. Inspired by shadcn-svelte.
+Author: Datacompose Contributors
+Maintainer: Datacompose Contributors
+License: MIT
+Project-URL: Homepage, https://github.com/datacompose/datacompose
+Project-URL: Documentation, https://github.com/datacompose/datacompose/tree/main/docs
+Project-URL: Repository, https://github.com/datacompose/datacompose.git
+Project-URL: Issues, https://github.com/datacompose/datacompose/issues
+Project-URL: Changelog, https://github.com/datacompose/datacompose/blob/main/CHANGELOG.md
+Keywords: data-cleaning,data-quality,udf,spark,postgres,code-generation,data-pipeline,etl
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Code Generators
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: jinja2>=3.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: click>=8.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=7.2.0; extra == "docs"
+Requires-Dist: furo>=2024.1.0; extra == "docs"
+Requires-Dist: myst-parser>=2.0.0; extra == "docs"
+Requires-Dist: sphinx-autodoc-typehints>=1.25.0; extra == "docs"
+Requires-Dist: sphinx-copybutton>=0.5.2; extra == "docs"
+Requires-Dist: sphinx-tabs>=3.4.0; extra == "docs"
+Requires-Dist: sphinx-click>=5.1.0; extra == "docs"
+Dynamic: license-file
+
+# Datacompose
+
+A powerful data transformation framework for building reusable, composable data cleaning pipelines in PySpark.
+
+## Overview
+
+Datacompose provides a declarative way to build data transformation pipelines using composable primitives. It generates optimized, standalone PySpark code that can be deployed without runtime dependencies.
+
+## Key Features
+
+- **Composable Primitives**: Build complex transformations from simple, reusable functions
+- **Smart Partial Application**: Configure transformations with parameters for reuse
+- **Pipeline Compilation**: Convert declarative pipeline definitions into optimized Spark operations
+- **Code Generation**: Generate standalone PySpark code with embedded dependencies
+- **Comprehensive Libraries**: Pre-built primitives for emails, addresses, and phone numbers
+- **Conditional Logic**: Support for if/else branching in pipelines
+- **Type-Safe Operations**: All transformations maintain Spark column type safety
+
+## Installation
+
+```bash
+pip install datacompose
+```
+
+## Quick Start
+
+### 1. Initialize a Project
+
+```bash
+datacompose init
+```
+
+This creates a `datacompose.json` configuration file with default settings.
+
+### 2. Generate Transformation Code
+
+```bash
+# Generate email cleaning primitives
+datacompose add clean_emails --target pyspark
+
+# Generate address standardization primitives  
+datacompose add clean_addresses --target pyspark
+
+# Generate phone number validation primitives
+datacompose add clean_phone_numbers --target pyspark
+```
+
+### 3. Use the Generated Code
+
+```python
+from pyspark.sql import SparkSession
+from pyspark.sql import functions as F
+
+# Import the generated primitives
+from build.pyspark.clean_emails.email_primitives import emails
+
+# Create Spark session
+spark = SparkSession.builder.appName("DataCleaning").getOrCreate()
+
+# Load your data
+df = spark.read.csv("data.csv", header=True)
+
+# Apply email transformations
+cleaned_df = df.withColumn(
+    "email_clean",
+    emails.standardize_email(F.col("email"))
+).withColumn(
+    "email_domain",
+    emails.extract_domain(F.col("email_clean"))
+).withColumn(
+    "is_valid",
+    emails.is_valid_email(F.col("email_clean"))
+)
+
+# Filter to valid emails only
+valid_emails = cleaned_df.filter(F.col("is_valid"))
+```
+
+## Core Concepts
+
+### PrimitiveRegistry
+
+A container for organizing related transformation functions:
+
+```python
+from datacompose.operators.primitives import PrimitiveRegistry
+
+# Create a registry for text operations
+text = PrimitiveRegistry("text")
+
+# Register transformation functions
+@text.register()
+def lowercase(col):
+    return F.lower(col)
+
+@text.register()
+def remove_spaces(col):
+    return F.regexp_replace(col, r'\s+', '')
+
+# Use the transformations
+df = df.withColumn("clean_text", text.lowercase(F.col("input")))
+```
+
+### SmartPrimitive
+
+Enables partial application of transformations:
+
+```python
+@text.register()
+def trim(col, chars=' '):
+    return F.trim(col, chars)
+
+# Direct usage
+df = df.withColumn("trimmed", text.trim(F.col("input")))
+
+# Pre-configured usage
+trim_tabs = text.trim(chars='\t')
+df = df.withColumn("no_tabs", trim_tabs(F.col("input")))
+```
+
+### Pipeline Composition
+
+Build complex pipelines from simple primitives:
+
+```python
+@text.compose(text=text)
+def clean_pipeline():
+    text.trim()
+    text.lowercase()
+    text.remove_spaces()
+
+# Apply the entire pipeline
+df = df.withColumn("cleaned", clean_pipeline(F.col("input")))
+```
+
+### Conditional Pipelines
+
+Add conditional logic to your transformations:
+
+```python
+@text.register(is_conditional=True)
+def is_valid_length(col):
+    return F.length(col) > 5
+
+@text.register()
+def truncate(col):
+    return F.substring(col, 1, 5)
+
+@text.compose(text=text)
+def smart_truncate():
+    if text.is_valid_length():
+        text.truncate()
+```
+
+## Available Primitives
+
+### Email Primitives
+
+```python
+from build.pyspark.clean_emails.email_primitives import emails
+
+# Validation
+emails.is_valid_email(col)
+emails.is_business_email(col)
+emails.is_disposable_email(col)
+
+# Extraction
+emails.extract_domain(col)
+emails.extract_username(col)
+emails.extract_tld(col)
+
+# Standardization
+emails.standardize_email(col)
+emails.normalize_gmail(col)
+emails.fix_common_typos(col)
+
+# Filtering
+emails.filter_valid_emails(col)
+emails.filter_business_emails(col)
+```
+
+### Address Primitives
+
+```python
+from build.pyspark.clean_addresses.address_primitives import addresses
+
+# Extraction
+addresses.extract_street_number(col)
+addresses.extract_street_name(col)
+addresses.extract_city(col)
+addresses.extract_state(col)
+addresses.extract_zip_code(col)
+
+# Standardization
+addresses.standardize_state(col)
+addresses.standardize_street_suffix(col)
+addresses.standardize_direction(col)
+
+# Validation
+addresses.is_valid_zip_code(col)
+addresses.is_valid_state(col)
+addresses.is_po_box(col)
+```
+
+### Phone Number Primitives
+
+```python
+from build.pyspark.clean_phone_numbers.phone_primitives import phones
+
+# Validation
+phones.is_valid_nanp(col)
+phones.is_valid_international(col)
+phones.is_toll_free(col)
+
+# Extraction
+phones.extract_country_code(col)
+phones.extract_area_code(col)
+phones.extract_exchange(col)
+phones.extract_subscriber(col)
+
+# Formatting
+phones.format_nanp(col)
+phones.format_e164(col)
+phones.format_international(col)
+
+# Standardization
+phones.standardize_phone(col)
+phones.clean_phone(col)
+```
+
+## Advanced Usage
+
+### Creating Custom Primitives
+
+```python
+from datacompose.operators.primitives import PrimitiveRegistry
+
+# Create your own registry
+custom = PrimitiveRegistry("custom")
+
+@custom.register()
+def remove_special_chars(col):
+    return F.regexp_replace(col, r'[^a-zA-Z0-9\s]', '')
+
+@custom.register()
+def capitalize_words(col):
+    return F.initcap(col)
+
+@custom.register(is_conditional=True)
+def contains_numbers(col):
+    return col.rlike(r'\d+')
+
+# Create a pipeline with your custom primitives
+@custom.compose(custom=custom)
+def clean_text():
+    custom.remove_special_chars()
+    if custom.contains_numbers():
+        custom.capitalize_words()
+```
+
+### Working with Parameters
+
+```python
+@custom.register()
+def pad_string(col, length=10, fill_char='0'):
+    return F.lpad(col, length, fill_char)
+
+# Use with different parameters
+df = df.withColumn("padded_10", custom.pad_string(F.col("id")))
+df = df.withColumn("padded_5", custom.pad_string(length=5)(F.col("id")))
+df = df.withColumn("padded_x", custom.pad_string(length=8, fill_char='X')(F.col("id")))
+```
+
+### Combining Multiple Registries
+
+```python
+from build.pyspark.clean_emails.email_primitives import emails
+from build.pyspark.clean_phones.phone_primitives import phones
+
+# Create a combined validation pipeline
+validation = PrimitiveRegistry("validation")
+
+@validation.compose(emails=emails, phones=phones)
+def validate_contact_info():
+    # Check email
+    if emails.is_valid_email():
+        emails.standardize_email()
+    
+    # Check phone
+    if phones.is_valid_phone():
+        phones.standardize_phone()
+```
+
+## CLI Commands
+
+### Initialize a Project
+```bash
+datacompose init [--yes]
+```
+
+### Add Transformers
+```bash
+datacompose add <transformer> [--target TARGET] [--output OUTPUT] [--verbose]
+
+# Examples
+datacompose add clean_emails --target pyspark
+datacompose add clean_addresses --target pyspark --output ./custom/path
+datacompose add clean_phone_numbers --target pyspark --verbose
+```
+
+### List Available Transformers
+```bash
+datacompose list transformers
+datacompose list generators
+```
+
+## Project Structure
+
+After running `datacompose add`, your project will have the following structure:
+
+```
+project/
+├── datacompose.json                    # Configuration file
+├── build/
+│   └── pyspark/
+│       ├── clean_emails/
+│       │   ├── email_primitives.py     # Generated email primitives
+│       │   └── utils/
+│       │       └── primitives.py       # Core framework (embedded)
+│       ├── clean_addresses/
+│       │   ├── address_primitives.py
+│       │   └── utils/
+│       │       └── primitives.py
+│       └── clean_phone_numbers/
+│           ├── phone_primitives.py
+│           └── utils/
+│               └── primitives.py
+```
+
+## Configuration
+
+The `datacompose.json` file configures default settings:
+
+```json
+{
+  "version": "1.0.0",
+  "targets": {
+    "pyspark": {
+      "output": "./build/pyspark",
+      "generator": "SparkPandasUDFGenerator"
+    }
+  },
+  "templates": {
+    "directory": "src/transformers/templates"
+  }
+}
+```
+
+
+
+## Performance Considerations
+
+- Primitives are designed to be efficient Spark operations
+- Pipelines are compiled to minimize intermediate columns
+- Conditional logic uses Spark's `when/otherwise` for vectorized operations
+- Generated code has no runtime dependencies beyond PySpark
+
+## Philosophy & Inspiration
+
+Datacompose is inspired by [shadcn-svelte](https://www.shadcn-svelte.com/) and [huntabyte](https://github.com/huntabyte)'s approach to component libraries. Just as shadcn-svelte provides "copy and paste" components rather than npm packages, Datacompose generates data transformation code that becomes part of YOUR codebase.
+
+**Why we believe in this approach:**
+
+- **You Own Your Code**: No external dependencies to manage or worry about breaking changes
+- **Full Transparency**: Every transformation is readable, debuggable PySpark code you can understand
+- **Customization First**: Need to adjust transformation? Just edit the code
+- **Learn by Reading**: The generated code serves as documentation and learning material
+
+This is NOT a traditional library - it's a code generator that gives you production-ready data transformation primitives that you can modify to fit your exact needs.
+
+
+## License
+
+MIT License - see LICENSE file for details

datacompose-0.2.4.dist-info/RECORD

@@ -0,0 +1,31 @@
+datacompose/__init__.py,sha256=kbQEzEvheAMsm3-MT4nWSuX42fjrfgDWoR8WZmC_WX4,34
+datacompose/cli/__init__.py,sha256=a5UAbPnAm87gUq4YaoTPN5umknN5s2xgnztatI03So0,107
+datacompose/cli/colors.py,sha256=Ax7jHhdAIuq5x3663gJ7_MzFCBOJv38DqNXts5t4SLs,1756
+datacompose/cli/main.py,sha256=NjA6Uy1_A-xGaAEKKdXOrtMbAxOZ9Cn1aNDNYgHW9rg,1273
+datacompose/cli/validation.py,sha256=8WMZ9wtPgFk9eBgMS_wtkncFz_-BmH4E8V57tjp3YoI,2526
+datacompose/cli/commands/__init__.py,sha256=Bu58UsnkGRbVFS92U2Px_KxlUPrdlbSY6wlvP6tet2o,38
+datacompose/cli/commands/add.py,sha256=dWussDjA_7hlkM6J0mv4GGXg5iKwIqjScbstPDZcPYo,7023
+datacompose/cli/commands/init.py,sha256=MTO48fM18s4qnUCPpEC5RdmdKW6OsJc_nyN_wrHbi3g,16740
+datacompose/cli/commands/list.py,sha256=MmRxMnghBLagg6IEh4lqCK0WR-0Ku-jxH8AT6WlajuU,3867
+datacompose/cli/commands/upgrade.py,sha256=F0ra-HLVCP5MEdYOkKbvZ_cnhXFmKKw6IRBhmuWBGVI,163
+datacompose/generators/__init__.py,sha256=dFJWJScu8mkP0ZKIQtVlJ36PQW-LwCYBijuNwLSevZw,48
+datacompose/generators/base.py,sha256=YodC9Ai58eWCPNFx1IIOAcQvd5CGu3ySf6Z6m_-UVTI,7002
+datacompose/generators/pyspark/__init__.py,sha256=ayoKDGtbt2KwFcNt2QxHKt8z83Kzy4ySw9Gg7j9ZMTY,33
+datacompose/generators/pyspark/generator.py,sha256=U0OCJVRI9Jc9aWsubRNBQ6dEq8IFNf_j5IVGDZuglvs,1987
+datacompose/operators/__init__.py,sha256=6g7Hp5261TkPghRgTfxKrizx0OH3Zga3OKHZ37I9_4E,586
+datacompose/operators/primitives.py,sha256=rIERyKfPIULngHs9fRewXo6VjmbjyiOXvTCqiHGIur8,22022
+datacompose/transformers/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+datacompose/transformers/discovery.py,sha256=JgMtmd8PkhmwLqS17NeKSv9MRneY9tdOsOK32A6luOQ,7048
+datacompose/transformers/text/__init__.py,sha256=Mq0UmgYlBV8T18IkvHAS1TImEzWyGciCqxaCv324hFQ,36
+datacompose/transformers/text/clean_addresses/__init__.py,sha256=l5TItGrGBn69Mlq0CaRGJa-SwpyuUEYWvG5N26s3Pco,39
+datacompose/transformers/text/clean_addresses/pyspark/pyspark_primitives.py,sha256=wiENqpdcVAlfNhsyIM-JjX7lye4xcW0h1INHYMRrYlE,60249
+datacompose/transformers/text/clean_emails/__init__.py,sha256=snZLOJsxrPDOi8gIISlRxc6YlskKxUyu0NnOZCE5cIU,34
+datacompose/transformers/text/clean_emails/pyspark/pyspark_primitives.py,sha256=vIgPAcc6t8UCYSzvi79UplkTFqY9jIR9brZyhAhtLwY,21802
+datacompose/transformers/text/clean_phone_numbers/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+datacompose/transformers/text/clean_phone_numbers/pyspark/pyspark_primitives.py,sha256=BsCfgncFxM77M3k0hEZyARPJk4kq1PQZB40YRc9RR8M,26279
+datacompose-0.2.4.dist-info/licenses/LICENSE,sha256=SCPOqmPhMikiyYDlKZ877fGHaE2O45cDBoJIomrlpDU,1067
+datacompose-0.2.4.dist-info/METADATA,sha256=eEXoMoQtlhoJXojWoX8weZ3uH5dGyBT7MJG8f76B_To,11876
+datacompose-0.2.4.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+datacompose-0.2.4.dist-info/entry_points.txt,sha256=oeG9oGgDwajk4v0C1awdUTBx2GmhLpuNHCTAV-jurUc,58
+datacompose-0.2.4.dist-info/top_level.txt,sha256=AX1qGkuJMD2YJLZKo40h-w4MeFxDZL6W1vbKKuTpW8I,12
+datacompose-0.2.4.dist-info/RECORD,,

datacompose-0.2.4.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (80.9.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

datacompose-0.2.4.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+datacompose = datacompose.cli.main:main

datacompose-0.2.4.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 datacompose
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

datacompose-0.2.4.dist-info/top_level.txt

@@ -0,0 +1 @@
+datacompose