snowflake-data-exchange-agent 0.0.19__tar.gz

snowflake_data_exchange_agent-0.0.19/.gitignore

@@ -0,0 +1,126 @@
+__pycache__/
+*.py[codz]
+*$py.class
+*.so
+
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+*.manifest
+*.spec
+
+pip-log.txt
+pip-delete-this-directory.txt
+
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+*.mo
+*.pot
+
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+instance/
+.webassets-cache
+
+.scrapy
+
+docs/_build/
+
+.pybuilder/
+target/
+
+.ipynb_checkpoints
+
+profile_default/
+ipython_config.py
+
+.pdm-python
+.pdm-build/
+.pixi
+__pypackages__/
+
+celerybeat-schedule
+celerybeat.pid
+
+*.rdb
+*.aof
+*.pid
+
+mnesia/
+rabbitmq/
+rabbitmq-data/
+activemq-data/
+
+*.sage.py
+
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+.spyderproject
+.spyproject
+.ropeproject
+
+/site
+
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pyre/
+.pytype/
+
+cython_debug/
+.abstra/
+.ruff_cache/
+.pypirc
+
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+.streamlit/secrets.toml
+
+.DS_Store
+.AppleDouble
+.LSOverride
+._*
+
+configuration.toml
+src/mocked_api/*
+*.code-workspace

snowflake_data_exchange_agent-0.0.19/PKG-INFO

@@ -0,0 +1,248 @@
+Metadata-Version: 2.4
+Name: snowflake-data-exchange-agent
+Version: 0.0.19
+Summary: Data exchange agent for migrations and validation
+Project-URL: Bug Tracker, https://github.com/snowflakedb/migrations-data-validation/issues
+Project-URL: Source code, https://github.com/snowflakedb/migrations-data-validation/
+Project-URL: homepage, https://www.snowflake.com/
+Author-email: "Snowflake, Inc." <snowflake-python-libraries-dl@snowflake.com>
+License: Apache License, Version 2.0
+Keywords: Snowflake,analytics,cloud,data,data-analysis,data-analytics,data-engineering,data-management,data-processing,data-science,data-visualization,data-warehouse,database
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Environment :: Other Environment
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: SQL
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: <3.13,>=3.10
+Requires-Dist: azure-identity
+Requires-Dist: azure-storage-blob
+Requires-Dist: boto3
+Requires-Dist: dependency-injector
+Requires-Dist: flask
+Requires-Dist: psutil
+Requires-Dist: psycopg2-binary
+Requires-Dist: pyspark
+Requires-Dist: requests>=2.28.0
+Requires-Dist: snowflake-connector-python>=3.0.0
+Requires-Dist: sqlparse
+Requires-Dist: toml
+Requires-Dist: urllib3>=2.5.0
+Requires-Dist: waitress
+Provides-Extra: all
+Requires-Dist: pytest-cov>=4.0.0; extra == 'all'
+Requires-Dist: pytest-mock>=3.10.0; extra == 'all'
+Requires-Dist: pytest>=7.0.0; extra == 'all'
+Requires-Dist: ruff>=0.1.0; extra == 'all'
+Provides-Extra: development
+Requires-Dist: pytest-cov>=4.0.0; extra == 'development'
+Requires-Dist: pytest-mock>=3.10.0; extra == 'development'
+Requires-Dist: pytest>=7.0.0; extra == 'development'
+Requires-Dist: ruff>=0.1.0; extra == 'development'
+Description-Content-Type: text/markdown
+
+# Snowflake Data Exchange Agent
+
+[![License Apache-2.0](https://img.shields.io/:license-Apache%202-brightgreen.svg)](http://www.apache.org/licenses/LICENSE-2.0.txt)
+[![Python](https://img.shields.io/badge/python-3.10--3.12-blue)](https://www.python.org/downloads/)
+
+A REST API service for database migrations and data validation. Supports multiple databases including Snowflake, PostgreSQL, and SQL Server with queue-based task processing.
+
+## Quick Start
+
+```bash
+# Install
+pip install snowflake-data-exchange-agent
+
+# Run
+data-exchange-agent --port 8080
+
+# Test
+curl http://localhost:8080/health
+```
+
+## Installation
+
+### From PyPI (Production)
+```bash
+pip install snowflake-data-exchange-agent
+```
+
+### Requirements & Dependencies
+
+**Python Version**: 3.10, 3.11, or 3.12 (3.13 not yet supported)
+
+**Available dependency groups**:
+- `development`: Testing and development tools (pytest, ruff, etc.)
+- `all`: Includes all development dependencies
+
+**Core dependencies include**:
+- Snowflake Connector for Python
+- PySpark for data processing
+- Flask + Waitress for REST API
+- PostgreSQL support (psycopg2-binary)
+- AWS SDK (boto3)
+
+## Configuration
+
+Create `src/data_exchange_agent/configuration.toml`:
+
+```toml
+# API settings
+[api_configuration]
+key = "your-api-key"
+host = "0.0.0.0"
+port = 5001
+workers = 4
+
+# Database connections
+[connection.postgresql]
+driver_name = "postgresql"
+username = "user"
+password = "password"
+host = "localhost"
+port = 5432
+database = "mydb"
+
+# Task queue
+[task_queue]
+type = "sqlite"
+database_path = "~/.data_exchange_agent/tasks.db"
+```
+
+For Snowflake, create `~/.snowflake/config.toml`:
+
+```toml
+[connections.default]
+account = "your_account.region"
+user = "your_username"
+password = "your_password"
+warehouse = "COMPUTE_WH"
+database = "PRODUCTION_DB"
+```
+
+## API Usage
+
+### Command Line
+```bash
+# Basic usage
+data-exchange-agent
+
+# Production settings
+data-exchange-agent --workers 8 --port 8080
+
+# Debug mode
+data-exchange-agent --debug --port 5001
+```
+
+### Health Check
+```http
+GET /health
+```
+```json
+{
+  "status": "healthy",
+  "version": "0.0.18",
+  "database_connections": {
+    "snowflake": "connected"
+  }
+}
+```
+
+### Task Management
+```http
+# Start processing
+GET /handle_tasks
+
+# Stop processing
+GET /stop
+
+# Get status
+GET /get_handling_tasks_status
+
+# Task count
+GET /get_tasks_count
+```
+
+### Add Task
+```http
+POST /tasks
+Content-Type: application/json
+```
+```json
+{
+  "task_type": "data_extraction",
+  "source_config": {
+    "database": "postgresql",
+    "query": "SELECT * FROM users"
+  },
+  "destination_config": {
+    "type": "snowflake_stage",
+    "stage": "@data_stage/users/"
+  }
+}
+```
+
+## Development
+
+### Setup
+```bash
+git clone https://github.com/snowflakedb/migrations-data-validation.git
+cd migrations-data-validation/data-exchange-agent
+pip install -e .[development]
+```
+
+### Testing
+```bash
+# Run all tests
+pytest
+
+# With coverage
+pytest --cov=src/data_exchange_agent
+
+# Run specific test types
+pytest tests/unit/           # Unit tests only
+pytest -m "not integration" # Non-integration tests
+```
+
+### Code Quality
+```bash
+# Format code
+ruff format .
+
+# Lint code
+ruff check .
+
+# Auto-fix linting issues
+ruff check --fix .
+```
+
+## 🤝 Contributing
+
+We welcome contributions! See our [Contributing Guide](../CONTRIBUTING.md) for details on how to collaborate, set up your development environment, and submit PRs.
+
+---
+
+## 📄 License
+
+This project is licensed under the Apache License 2.0. See the [LICENSE](../LICENSE) file for details.
+
+## 🆘 Support
+
+- **Documentation**: [Full documentation](https://github.com/snowflakedb/migrations-data-validation)
+- **Issues**: [GitHub Issues](https://github.com/snowflakedb/migrations-data-validation/issues)
+
+---
+
+**Developed with ❄️ by Snowflake**

snowflake_data_exchange_agent-0.0.19/README.md

@@ -0,0 +1,194 @@
+# Snowflake Data Exchange Agent
+
+[![License Apache-2.0](https://img.shields.io/:license-Apache%202-brightgreen.svg)](http://www.apache.org/licenses/LICENSE-2.0.txt)
+[![Python](https://img.shields.io/badge/python-3.10--3.12-blue)](https://www.python.org/downloads/)
+
+A REST API service for database migrations and data validation. Supports multiple databases including Snowflake, PostgreSQL, and SQL Server with queue-based task processing.
+
+## Quick Start
+
+```bash
+# Install
+pip install snowflake-data-exchange-agent
+
+# Run
+data-exchange-agent --port 8080
+
+# Test
+curl http://localhost:8080/health
+```
+
+## Installation
+
+### From PyPI (Production)
+```bash
+pip install snowflake-data-exchange-agent
+```
+
+### Requirements & Dependencies
+
+**Python Version**: 3.10, 3.11, or 3.12 (3.13 not yet supported)
+
+**Available dependency groups**:
+- `development`: Testing and development tools (pytest, ruff, etc.)
+- `all`: Includes all development dependencies
+
+**Core dependencies include**:
+- Snowflake Connector for Python
+- PySpark for data processing
+- Flask + Waitress for REST API
+- PostgreSQL support (psycopg2-binary)
+- AWS SDK (boto3)
+
+## Configuration
+
+Create `src/data_exchange_agent/configuration.toml`:
+
+```toml
+# API settings
+[api_configuration]
+key = "your-api-key"
+host = "0.0.0.0"
+port = 5001
+workers = 4
+
+# Database connections
+[connection.postgresql]
+driver_name = "postgresql"
+username = "user"
+password = "password"
+host = "localhost"
+port = 5432
+database = "mydb"
+
+# Task queue
+[task_queue]
+type = "sqlite"
+database_path = "~/.data_exchange_agent/tasks.db"
+```
+
+For Snowflake, create `~/.snowflake/config.toml`:
+
+```toml
+[connections.default]
+account = "your_account.region"
+user = "your_username"
+password = "your_password"
+warehouse = "COMPUTE_WH"
+database = "PRODUCTION_DB"
+```
+
+## API Usage
+
+### Command Line
+```bash
+# Basic usage
+data-exchange-agent
+
+# Production settings
+data-exchange-agent --workers 8 --port 8080
+
+# Debug mode
+data-exchange-agent --debug --port 5001
+```
+
+### Health Check
+```http
+GET /health
+```
+```json
+{
+  "status": "healthy",
+  "version": "0.0.18",
+  "database_connections": {
+    "snowflake": "connected"
+  }
+}
+```
+
+### Task Management
+```http
+# Start processing
+GET /handle_tasks
+
+# Stop processing
+GET /stop
+
+# Get status
+GET /get_handling_tasks_status
+
+# Task count
+GET /get_tasks_count
+```
+
+### Add Task
+```http
+POST /tasks
+Content-Type: application/json
+```
+```json
+{
+  "task_type": "data_extraction",
+  "source_config": {
+    "database": "postgresql",
+    "query": "SELECT * FROM users"
+  },
+  "destination_config": {
+    "type": "snowflake_stage",
+    "stage": "@data_stage/users/"
+  }
+}
+```
+
+## Development
+
+### Setup
+```bash
+git clone https://github.com/snowflakedb/migrations-data-validation.git
+cd migrations-data-validation/data-exchange-agent
+pip install -e .[development]
+```
+
+### Testing
+```bash
+# Run all tests
+pytest
+
+# With coverage
+pytest --cov=src/data_exchange_agent
+
+# Run specific test types
+pytest tests/unit/           # Unit tests only
+pytest -m "not integration" # Non-integration tests
+```
+
+### Code Quality
+```bash
+# Format code
+ruff format .
+
+# Lint code
+ruff check .
+
+# Auto-fix linting issues
+ruff check --fix .
+```
+
+## 🤝 Contributing
+
+We welcome contributions! See our [Contributing Guide](../CONTRIBUTING.md) for details on how to collaborate, set up your development environment, and submit PRs.
+
+---
+
+## 📄 License
+
+This project is licensed under the Apache License 2.0. See the [LICENSE](../LICENSE) file for details.
+
+## 🆘 Support
+
+- **Documentation**: [Full documentation](https://github.com/snowflakedb/migrations-data-validation)
+- **Issues**: [GitHub Issues](https://github.com/snowflakedb/migrations-data-validation/issues)
+
+---
+
+**Developed with ❄️ by Snowflake**