data-collection-framework 0.1.0__tar.gz

data_collection_framework-0.1.0/PKG-INFO

@@ -0,0 +1,19 @@
+Metadata-Version: 2.4
+Name: data-collection-framework
+Version: 0.1.0
+Requires-Python: >=3.12
+Requires-Dist: pyspark==4.0.2
+Requires-Dist: pandas
+Requires-Dist: requests
+Requires-Dist: pyarrow
+Requires-Dist: pytz
+Requires-Dist: pydantic>=2
+Requires-Dist: typer[all]>=0.9
+Requires-Dist: google-auth>=2.29
+Requires-Dist: google-cloud-storage>=2.16
+Requires-Dist: google-cloud-secret-manager>=2.20
+Requires-Dist: kafka-python>=2.0
+Requires-Dist: google-api-python-client>=2.126
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: mcp>=1.0
+Requires-Dist: duckdb>=1.0

data_collection_framework-0.1.0/README.md

@@ -0,0 +1,209 @@
+# dcf
+
+D.ata C.ollection F.ramework
+
+It works like this
+1. User defines collectors with basic configs in a YAML (like a dbt model)
+2. dcf builds and runs the collector
+3. Data lake has data
+
+## Quickstart
+
+This guide walks you from zero to a working data collector. The example ingests your GitHub repositories.
+
+### 1. Create a project
+
+dcf is a tool you depend on, not a repo you clone. Create a fresh directory:
+
+```bash
+mkdir dcf-demo && cd dcf-demo
+```
+
+**`pyproject.toml`:**
+
+```toml
+[project]
+name = "dcf-demo"
+version = "0.1.0"
+requires-python = ">=3.12"
+dependencies = [
+    "dcf",
+]
+
+[tool.uv]
+package = false
+
+[tool.uv.sources]
+dcf = { git = "https://github.com/zephschafer/dcf.git" }    
+```
+
+**`project.yml`**
+
+```yaml
+catalog: local
+```
+
+**`.gitignore`:**
+
+```
+warehouse/
+project.yml
+.venv/
+__pycache__/
+```
+
+```bash
+mkdir collectors
+uv sync
+```
+
+---
+
+### 2. Write a collector
+
+Create `collectors/dcf_commits.yml`:
+
+```yaml
+name: dcf_commits
+namespace: github
+description: Commits to the dcf repository.
+
+source:
+  type: http
+  url: https://api.github.com/repos/zephschafer/dcf/commits
+  method: GET
+  params:
+    - name: sha
+      type: string
+      value: main
+    - name: per_page
+      type: integer
+      value: 100
+  schema:
+    columns:
+      - {name: sha,          path: sha,                type: string}
+      - {name: author,       path: commit.author.name, type: string}
+      - {name: message,      path: commit.message,     type: string}
+      - {name: committed_at, path: commit.author.date, type: timestamp}
+
+cadence:
+  strategy: incremental
+  primary_key: sha
+
+deployment:
+  schedule: "0 8 * * *"
+```
+
+---
+
+### 3. Validate
+
+```bash
+uv run dcf validate dcf_commits
+```
+
+---
+
+### 4. Run
+
+```bash
+uv run dcf run dcf_commits
+```
+
+---
+
+### 5. Query the warehouse
+
+```bash
+uv run dcf query 'SELECT * FROM github.dcf_commits'
+```
+
+You can also save your SQL to a file and run it with `--file`:
+
+```bash
+uv run dcf query --file my_query.sql
+```
+
+---
+
+### 6. Deploy
+
+```bash
+uv run dcf deploy dcf_commits
+```
+
+This schedules the collector to run daily at 8 AM UTC, as configured in `deployment.schedule`.
+
+---
+
+## Developing dcf
+
+Clone this repo, then create or point to a project for testing:
+
+```bash
+git clone https://github.com/Data-Dispatch/dcf
+cd dcf
+uv sync
+
+# Test against the demo project
+git clone https://github.com/Data-Dispatch/quipu-data-generator ../quipu-data-generator
+cd ../quipu-data-generator
+uv sync   # picks up dcf from ../dcf via editable path dep
+uv run dcf validate all
+```
+
+Or create a minimal test project:
+
+```bash
+mkdir my-test-project && cd my-test-project
+cat > pyproject.toml << 'EOF'
+[project]
+name = "my-test-project"
+version = "0.1.0"
+requires-python = ">=3.12"
+dependencies = ["dcf"]
+
+[tool.uv]
+package = false
+
+[tool.uv.sources]
+dcf = { path = "../dcf", editable = true }
+EOF
+
+cat > project.yml << 'EOF'
+catalog: local
+EOF
+
+mkdir collectors
+uv sync
+uv run dcf validate all   # "OK — 0 collector(s)"
+```
+
+---
+
+## dcf package structure
+
+```
+dcf/
+├── cli.py              Entry point (Typer app)
+├── project.py          Project root discovery (CWD walk / DCF_PROJECT_DIR)
+├── spark_session.py    PySpark + Iceberg session factory
+├── mcp_server.py       MCP server (FastMCP)
+├── warehouse_reader.py DuckDB-based warehouse query layer
+├── config/
+│   ├── models.py       Pydantic models for collector YAML
+│   └── loader.py       YAML loading + env var resolution
+├── engine/
+│   ├── runner.py       Outer loop (expand cadence → fetch → project → write)
+│   ├── fetcher.py      HTTP and Python source fetchers
+│   ├── iterator.py     Cartesian iteration over date ranges and categoricals
+│   ├── projector.py    Schema projection (path extraction, transforms)
+│   └── transforms.py   Column transforms (crs_reproject, etc.)
+├── writer/
+│   └── iceberg.py      Iceberg write strategies (incremental / append / full_refresh)
+└── gcp/
+    ├── bootstrap.py    GCS bucket + service account provisioning
+    ├── terraform.py    Terraform wrapper for lake infrastructure
+    ├── auth.py         GCP credential helpers
+    └── gcloud.py       gcloud CLI wrappers
+```

data_collection_framework-0.1.0/data_collection_framework.egg-info/PKG-INFO

@@ -0,0 +1,19 @@
+Metadata-Version: 2.4
+Name: data-collection-framework
+Version: 0.1.0
+Requires-Python: >=3.12
+Requires-Dist: pyspark==4.0.2
+Requires-Dist: pandas
+Requires-Dist: requests
+Requires-Dist: pyarrow
+Requires-Dist: pytz
+Requires-Dist: pydantic>=2
+Requires-Dist: typer[all]>=0.9
+Requires-Dist: google-auth>=2.29
+Requires-Dist: google-cloud-storage>=2.16
+Requires-Dist: google-cloud-secret-manager>=2.20
+Requires-Dist: kafka-python>=2.0
+Requires-Dist: google-api-python-client>=2.126
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: mcp>=1.0
+Requires-Dist: duckdb>=1.0

data_collection_framework-0.1.0/data_collection_framework.egg-info/SOURCES.txt

@@ -0,0 +1,53 @@
+README.md
+pyproject.toml
+data_collection_framework.egg-info/PKG-INFO
+data_collection_framework.egg-info/SOURCES.txt
+data_collection_framework.egg-info/dependency_links.txt
+data_collection_framework.egg-info/entry_points.txt
+data_collection_framework.egg-info/requires.txt
+data_collection_framework.egg-info/top_level.txt
+dcf/__init__.py
+dcf/cli.py
+dcf/local_deploy.py
+dcf/project.py
+dcf/spark_session.py
+dcf/warehouse_reader.py
+dcf/config/__init__.py
+dcf/config/loader.py
+dcf/config/models.py
+dcf/engine/__init__.py
+dcf/engine/fetcher.py
+dcf/engine/iterator.py
+dcf/engine/projector.py
+dcf/engine/runner.py
+dcf/engine/transforms.py
+dcf/gcp/__init__.py
+dcf/gcp/_collector_utils.py
+dcf/gcp/auth.py
+dcf/gcp/batch_deploy.py
+dcf/gcp/bootstrap.py
+dcf/gcp/gcloud.py
+dcf/gcp/terraform.py
+dcf/infra/modules/batch_collector/gcp/main.tf
+dcf/infra/modules/batch_collector/gcp/outputs.tf
+dcf/infra/modules/batch_collector/gcp/variables.tf
+dcf/infra/modules/batch_collector/gcp/airflow/main.tf
+dcf/infra/modules/batch_collector/gcp/airflow/outputs.tf
+dcf/infra/modules/batch_collector/gcp/airflow/variables.tf
+dcf/infra/modules/batch_collector/local/main.tf
+dcf/infra/modules/batch_collector/local/outputs.tf
+dcf/infra/modules/batch_collector/local/variables.tf
+dcf/infra/modules/batch_collector/local/airflow/main.tf
+dcf/infra/modules/batch_collector/local/airflow/outputs.tf
+dcf/infra/modules/batch_collector/local/airflow/variables.tf
+dcf/infra/templates/airflow.Dockerfile.tftpl
+dcf/infra/templates/batch_collector.Dockerfile.tftpl
+dcf/infra/templates/docker-compose.yml.tftpl
+dcf/writer/__init__.py
+dcf/writer/iceberg.py
+tests/test_deploy_cli.py
+tests/test_deploy_model.py
+tests/test_fetcher.py
+tests/test_runner_errors.py
+tests/test_transforms.py
+tests/test_warehouse_reader.py

data_collection_framework-0.1.0/data_collection_framework.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

data_collection_framework-0.1.0/data_collection_framework.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+dcf = dcf.cli:app

data_collection_framework-0.1.0/data_collection_framework.egg-info/requires.txt

@@ -0,0 +1,15 @@
+pyspark==4.0.2
+pandas
+requests
+pyarrow
+pytz
+pydantic>=2
+typer[all]>=0.9
+google-auth>=2.29
+google-cloud-storage>=2.16
+google-cloud-secret-manager>=2.20
+kafka-python>=2.0
+google-api-python-client>=2.126
+pyyaml>=6.0
+mcp>=1.0
+duckdb>=1.0

data_collection_framework-0.1.0/data_collection_framework.egg-info/top_level.txt

@@ -0,0 +1 @@
+dcf

data_collection_framework-0.1.0/dcf/__init__.py

@@ -0,0 +1,4 @@
+from .engine.runner import run_collector
+from .config import load_collector, load_all_collectors
+
+__all__ = ["run_collector", "load_collector", "load_all_collectors"]