sf-synth 0.1.0__tar.gz

sf_synth-0.1.0/.gitignore

@@ -0,0 +1,218 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#   Usually these files are written by a python script from a template
+#   before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+# Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+# uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+# poetry.lock
+# poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+# pdm.lock
+# pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+# pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# Redis
+*.rdb
+*.aof
+*.pid
+
+# RabbitMQ
+mnesia/
+rabbitmq/
+rabbitmq-data/
+
+# ActiveMQ
+activemq-data/
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#   and can be added to the global gitignore or merged into this file.  For a more nuclear
+#   option (not recommended) you can uncomment the following to ignore the entire idea folder.
+# .idea/
+
+# Abstra
+#   Abstra is an AI-powered process automation framework.
+#   Ignore directories containing user credentials, local state, and settings.
+#   Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#   Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#   that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#   and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#   you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+# Temporary file for partial code execution
+tempCodeRunnerFile.py
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Streamlit
+.streamlit/secrets.toml

sf_synth-0.1.0/CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+All notable changes to sf-synth are documented here.
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+
+## [0.1.0] - 2026-05-08
+
+### Added
+- Snowpark-first execution engine — data is generated entirely inside Snowflake.
+- Auto-discovery of tables, columns, types, PK/FK/UNIQUE/NOT NULL from `INFORMATION_SCHEMA`.
+- DAG-based generation order using `networkx` topological sort.
+- Self-referential FK support via two-pass generation (insert NULL → UPDATE).
+- Distribution-preserving generators using `APPROX_TOP_K`, `APPROX_PERCENTILE`, and `HLL`.
+- SQL-native generators: `seq`, `uniform`, `choice`, `range`, `regex`.
+- Faker UDF generators for rich fake data (email, name, address, phone, etc.).
+- Zipf-weighted FK sampling for realistic skewed distributions.
+- Pydantic v2 configuration with YAML loader and strict validation.
+- Column-name semantic type inference (80+ patterns).
+- Typer CLI with `discover`, `plan`, `generate`, and `clean` commands.
+- `plan` command shows generation order, row counts, and byte estimates without writing.
+- `--seed` flag for fully deterministic output.
+- Multi-schema support: FK references across schemas within the same database.
+- Backend abstraction (`Backend` ABC + `SnowparkBackend` implementation).
+- Unit tests and opt-in Snowflake integration tests.
+- Example configs: e-commerce, self-referential employees, multi-schema enterprise.

sf_synth-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ayush Pareek
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sf_synth-0.1.0/PKG-INFO

@@ -0,0 +1,288 @@
+Metadata-Version: 2.4
+Name: sf-synth
+Version: 0.1.0
+Summary: High-fidelity synthetic data generation for Snowflake
+Project-URL: Homepage, https://github.com/apareek/snowflake-synthesizer
+Project-URL: Repository, https://github.com/apareek/snowflake-synthesizer
+Project-URL: Bug Tracker, https://github.com/apareek/snowflake-synthesizer/issues
+Project-URL: Changelog, https://github.com/apareek/snowflake-synthesizer/blob/main/CHANGELOG.md
+Author-email: Apareek <ayushpareek1608@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: data-generation,faker,snowflake,snowpark,synthetic-data
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Requires-Dist: faker>=18.0.0
+Requires-Dist: networkx>=3.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: snowflake-connector-python>=3.0.0
+Requires-Dist: snowflake-snowpark-python>=1.11.0
+Requires-Dist: typer[all]>=0.9.0
+Provides-Extra: dev
+Requires-Dist: duckdb>=0.9.0; extra == 'dev'
+Requires-Dist: mypy>=1.0.0; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
+Requires-Dist: pytest>=7.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Requires-Dist: types-pyyaml>=6.0.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# sf-synth
+
+High-fidelity synthetic data generation for Snowflake.
+
+A Snowpark-first Python library and CLI that generates realistic synthetic data inside Snowflake using auto-discovered schema, distribution statistics, Faker-based rules, and a DAG-driven referential-integrity engine. All generation runs server-side, so PII never leaves the account.
+
+## Features
+
+- **Snowpark-first execution**: Data is generated entirely within Snowflake using Snowpark. No data egress required.
+- **Auto-discovery**: Automatically detects tables, columns, types, constraints (PK, FK, UNIQUE, NOT NULL) from `INFORMATION_SCHEMA`.
+- **Referential integrity**: DAG-based generation ensures parent tables are populated before children. FK values are sampled from actual parent keys.
+- **Self-referential tables**: Handles self-referential FKs (e.g., `employees.manager_id → employees.id`) via two-pass generation.
+- **Multi-schema support**: Reference tables across different schemas within the same database (e.g., `SALES.CUSTOMERS` → `CORE.COUNTRIES`).
+- **Distribution-preserving**: Sample from real column statistics (`APPROX_TOP_K`, `APPROX_PERCENTILE`, `HLL`) to preserve data distributions without exposing PII.
+- **Skewed FK distributions**: Support for Zipf-weighted FK sampling (e.g., 80% of orders belong to 20% of customers).
+- **Semantic inference**: Automatically infers generators based on column names (e.g., `email`, `phone`, `created_at`).
+- **Deterministic output**: Seed-based generation for reproducible results.
+- **YAML configuration**: Simple, validated config with Pydantic.
+
+## Installation
+
+```bash
+pip install sf-synth
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/apareek/snowflake-synthesizer.git
+cd snowflake-synthesizer
+pip install -e ".[dev]"
+```
+
+## Quick Start
+
+### 1. Discover your schema
+
+Generate a starter config by discovering your existing Snowflake schema:
+
+```bash
+sf-synth discover MY_DATABASE --output config.yaml
+```
+
+### 2. Edit the config
+
+Customize row counts, add generators, and define relationships:
+
+```yaml
+defaults:
+  seed: 42
+  database: MY_DATABASE
+  schema: PUBLIC
+
+tables:
+  - name: CUSTOMERS
+    rows: 10000
+    columns:
+      EMAIL:
+        generator: faker
+        provider: email
+        unique: true
+      MEMBERSHIP:
+        generator: choice
+        values: [Gold, Silver, Bronze]
+        weights: [0.1, 0.3, 0.6]
+
+  - name: ORDERS
+    rows: 50000
+    relationships:
+      - column: CUSTOMER_ID
+        references: CUSTOMERS.ID
+        skew: zipf
+```
+
+### 3. Preview the plan
+
+See the generation order and dependencies without executing:
+
+```bash
+sf-synth plan config.yaml
+```
+
+### 4. Generate data
+
+```bash
+sf-synth generate config.yaml
+```
+
+### 5. Clean up
+
+Remove temporary tables created during generation:
+
+```bash
+sf-synth clean config.yaml
+```
+
+## Configuration Reference
+
+### Defaults
+
+```yaml
+defaults:
+  seed: 42                    # Random seed for reproducibility
+  locale: en_US               # Faker locale
+  database: MY_DB             # Default database
+  schema: PUBLIC              # Default schema
+  null_ratio: 0.0             # Default null ratio for all columns
+```
+
+### Generator Types
+
+| Generator | Description | Required Parameters |
+|-----------|-------------|---------------------|
+| `seq` | Sequential integers | `start`, `step` |
+| `uniform` | Uniform random numbers | `min_value`, `max_value` |
+| `choice` | Random selection from list | `values`, `weights` (optional) |
+| `range` | Values in numeric/date range | `min_value`, `max_value` |
+| `faker` | Faker provider | `provider`, `locale` (optional) |
+| `distribution` | Sample from source column stats | `source` (FQN: DB.SCHEMA.TABLE.COL) |
+| `regex` | Pattern-based strings | `pattern` |
+
+### Faker Providers
+
+Common providers: `email`, `name`, `first_name`, `last_name`, `phone_number`, `address`, `city`, `state`, `zipcode`, `country`, `company`, `job`, `date`, `date_time`, `uuid4`, `url`, `ipv4`, `ssn`, `credit_card_number`.
+
+### Relationships
+
+```yaml
+relationships:
+  - column: CUSTOMER_ID           # FK column in this table
+    references: CUSTOMERS.ID       # Parent table.column
+    null_ratio: 0.05              # 5% null FKs
+    skew: zipf                    # Distribution: uniform or zipf
+    skew_param: 1.5               # Zipf exponent (higher = more skewed)
+```
+
+## Python API
+
+```python
+from sf_synth import SynthConfig, SynthEngine, discover_schema
+from sf_synth.backend import SnowparkBackend
+
+# Connect to Snowflake
+backend = SnowparkBackend(connection_name="my_connection")
+backend.connect()
+
+# Discover schema
+schema = backend.discover_schema("MY_DATABASE")
+
+# Load config
+from sf_synth.config import load_config
+config = load_config("config.yaml")
+
+# Generate
+engine = SynthEngine(backend.session, config, schema_model=schema)
+result = engine.generate()
+
+print(f"Generated {result.total_rows} rows in {result.total_elapsed_seconds:.2f}s")
+
+# Cleanup
+engine.cleanup()
+backend.disconnect()
+```
+
+## Examples
+
+The `examples/` directory contains ready-to-use configurations:
+
+| Example | Description |
+|---------|-------------|
+| [`ecommerce.yaml`](examples/ecommerce.yaml) | E-commerce schema with customers, products, orders, and reviews. Demonstrates FK relationships, Zipf-skewed distributions, and various generators. |
+| [`selfref_employees.yaml`](examples/selfref_employees.yaml) | HR schema with self-referential `manager_id` FK. Shows how sf-synth handles circular references via two-pass generation. |
+| [`multi_schema.yaml`](examples/multi_schema.yaml) | Enterprise schema spanning CORE, HR, SALES, and FINANCE schemas. Demonstrates cross-schema FK relationships within a single database. |
+
+## Architecture
+
+```
+┌─────────────┐     ┌──────────────┐     ┌─────────────┐
+│   CLI       │────▶│   Config     │────▶│  Discovery  │
+│   (Typer)   │     │  (Pydantic)  │     │  (INFO_SCH) │
+└─────────────┘     └──────────────┘     └─────────────┘
+                           │                    │
+                           ▼                    ▼
+                    ┌─────────────┐     ┌─────────────┐
+                    │  DAG Builder│────▶│   Schema    │
+                    │  (networkx) │     │   Model     │
+                    └─────────────┘     └─────────────┘
+                           │
+                           ▼
+┌─────────────┐     ┌─────────────┐     ┌─────────────┐
+│   Stats     │────▶│   Engine    │────▶│ RI Manager  │
+│   Sampler   │     │  (Snowpark) │     │ (Parent Keys│
+│ (APPROX_*)  │     └─────────────┘     └─────────────┘
+└─────────────┘            │
+                           ▼
+                    ┌─────────────┐
+                    │  Snowflake  │
+                    │   Tables    │
+                    └─────────────┘
+```
+
+## Connection Configuration
+
+sf-synth uses standard Snowflake connection methods:
+
+1. **Named connection** (recommended): `~/.snowflake/connections.toml`
+2. **Environment variables**: `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_USER`, etc.
+3. **CLI parameters**: `--connection`, `--account`, etc.
+
+Example `~/.snowflake/connections.toml`:
+
+```toml
+[my_connection]
+account = "myaccount"
+user = "myuser"
+authenticator = "externalbrowser"
+database = "MY_DB"
+schema = "PUBLIC"
+warehouse = "COMPUTE_WH"
+```
+
+## Performance Notes
+
+- **SQL-first generators** (seq, uniform, choice, range) are fast and scale to billions of rows.
+- **Faker UDFs** are slower due to Python UDF overhead. Use them only when SQL alternatives don't exist.
+- **Distribution sampling** requires one-time stats queries per column but generates data efficiently.
+- For very large tables (>100M rows), consider chunked generation or Snowflake-native `GENERATOR()` patterns.
+
+## Development
+
+```bash
+# Install dev dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest tests/unit/
+
+# Run integration tests (requires Snowflake credentials)
+SF_SYNTH_INTEGRATION_TESTS=1 pytest tests/integration/
+
+# Lint
+ruff check src/ tests/
+
+# Type check
+mypy src/
+```
+
+## License
+
+MIT License. See [LICENSE](LICENSE) for details.