floe-python 0.7.0__tar.gz → 0.8.1__tar.gz

floe_python-0.8.1/.dockerignore

@@ -0,0 +1,11 @@
+.git
+.github
+aws
+gcp
+bench
+orchestrators
+**/.venv
+**/__pycache__
+**/*.pyc
+**/.DS_Store
+

floe_python-0.8.1/.gitattributes

@@ -0,0 +1,20 @@
+# Ensure consistent line endings across platforms.
+# Shell scripts must always be checked out with LF so bash on Windows
+# does not include \r in variable assignments.
+*.sh        text eol=lf
+
+# YAML and TOML are text files — normalise to LF in the repo.
+*.yml       text eol=lf
+*.yaml      text eol=lf
+*.toml      text eol=lf
+*.md        text eol=lf
+*.py        text eol=lf
+*.rb        text eol=lf
+*.json      text eol=lf
+
+# Binaries — never touch line endings.
+*.png       binary
+*.jpg       binary
+*.jpeg      binary
+*.gz        binary
+*.zip       binary

floe_python-0.8.1/.github/DISCUSSION_TEMPLATE/onboarding.md

@@ -0,0 +1,33 @@
+---
+name: Onboarding / Codebase tour
+about: Start here if you're new to Floe and want a guided overview
+title: "[Onboarding] Introduce yourself + questions"
+labels: []
+---
+
+Welcome! This thread is for newcomers to share context and ask questions.
+
+## Quick intro
+- Background:
+- What you want to contribute to:
+- Any constraints (time/stack):
+
+## Codebase map (short)
+- `src/floe`: the ingestion engine and the `floe` CLI (`src/floe/_cli.py`)
+- `docs/`: user docs and support matrix
+- `example/`: sample configs + inputs
+
+## Architecture highlights
+- Storage abstraction: `src/floe/storage`
+- Input formats: `src/floe/read`
+- Output sinks: `src/floe/write`
+- Run pipeline: `src/floe/run`
+- Reports: `src/floe/report`
+
+## How to get started
+- Find a `good first issue`
+- Ask questions here if the scope or design is unclear
+- Check `CONTRIBUTING.md` for testing expectations
+
+## Questions
+Ask anything—happy to help.

floe_python-0.8.1/.github/ISSUE_TEMPLATE/bugfix.md

@@ -0,0 +1,34 @@
+---
+name: Bug report
+about: Report a defect or regression
+labels: bug
+---
+
+## Summary
+What’s broken and how does it affect usage?
+
+## Steps to reproduce
+1.
+2.
+3.
+
+## Expected behavior
+
+## Actual behavior
+
+## Environment
+- OS:
+- Floe version:
+- Config snippet (redact secrets):
+
+## Logs / error output
+```
+paste relevant logs here
+```
+
+## Suggested fix (optional)
+
+## Success criteria
+- Repro test added (unit or local integration) that fails without the fix
+- Checks from AGENTS.md §5 pass
+- Docs updated if behavior/config changed

floe_python-0.8.1/.github/ISSUE_TEMPLATE/feature.md

@@ -0,0 +1,29 @@
+---
+name: Feature request
+about: Propose a new feature or capability
+labels: enhancement
+---
+
+## Summary
+What do we want to build and why?
+
+## Scope
+- In scope:
+- Out of scope:
+
+## Success criteria
+Minimum acceptance conditions for the feature to be considered done.
+Too big for one PR? Split it into Task sub-issues (AGENTS.md §8).
+
+## Design notes
+Constraints, decisions, API shape, data model, etc.
+
+## Tests to add
+- Unit:
+- Integration (local only):
+
+## Docs to update
+- Files:
+
+## Risks / open questions
+- 

floe_python-0.8.1/.github/ISSUE_TEMPLATE/task.md

@@ -0,0 +1,24 @@
+---
+name: Task (sub-issue)
+about: One PR-sized change under a parent issue. Rules in AGENTS.md §8.
+---
+
+Parent: #
+
+## Context
+Why this change is needed. Verified `file:line` references to the code involved.
+
+## Scope
+- 
+
+## Out of scope
+- 
+
+## Acceptance criteria
+- [ ] 
+
+## Test that must fail first
+Module and test name, or the integration scenario (see `context/testing-strategy.md`).
+
+## Size estimate
+Files touched and approximate non-test src lines. Must fit ≤300 lines and ≤8 files; split if not.

floe_python-0.8.1/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,31 @@
+Closes #
+
+<!-- Exactly one issue. Rules: AGENTS.md -->
+
+## Summary
+What does this change do?
+
+## Why this is the minimum change
+Where the root cause is, and why the fix belongs there.
+
+## Deliberately not done
+Related issues you saw and left out (link the issues you opened for them).
+
+## Testing
+Commands run (see AGENTS.md §5):
+- 
+
+## Docs
+Updated files (only if user-visible):
+- 
+
+## Risks / notes
+- 
+
+## Checklist
+- [ ] One issue only; no unrelated edits, renames or reformatting
+- [ ] Within the size budget (≤300 non-test src lines, ≤8 files), or split into stacked PRs
+- [ ] Bug fix includes a test that fails without the fix
+- [ ] `/ponytail-review` run and findings resolved or justified above (non-Claude agents: ladder applied)
+- [ ] fmt / clippy / tests from AGENTS.md §5 pass
+- [ ] CHANGELOG / docs updated if user-visible

floe_python-0.8.1/.github/dependabot.yml

@@ -0,0 +1,16 @@
+version: 2
+updates:
+  - package-ecosystem: github-actions
+    directory: "/"
+    schedule:
+      interval: weekly
+
+  - package-ecosystem: pip
+    directory: "/orchestrators/dagster-floe"
+    schedule:
+      interval: weekly
+
+  - package-ecosystem: pip
+    directory: "/orchestrators/airflow-floe"
+    schedule:
+      interval: weekly

floe_python-0.8.1/.github/release.md

@@ -0,0 +1,35 @@
+# Release & CI/CD
+
+This document describes the GitHub Actions workflows that publish Floe.
+
+## Workflows
+
+- `CI` (`.github/workflows/ci.yml`)
+  - Runs on PRs, the merge queue and pushes to `main`.
+  - Runs ruff, pyright, pytest, the conformance cases and the config schema tests.
+
+- `Release` (`.github/workflows/release.yml`)
+  - Runs on version tags `vX.Y.Z` that are on `main`.
+  - Checks that the tag equals the `pyproject.toml` version.
+  - Builds `floe-python` with `uv build` (one `py3-none-any` wheel and an sdist),
+    smoke-tests the wheel and publishes both to PyPI.
+  - Creates the GitHub Release with the wheel and the sdist attached.
+  - Builds `Dockerfile.python` for amd64 and arm64 and pushes
+    `ghcr.io/openlakeforge/floe:<version>` and `:latest`.
+
+## Publishing credentials
+
+- PyPI uses a trusted publisher bound to `release.yml` and the `pypi`
+  environment; there is no token. Renaming the workflow file breaks publishing.
+- GHCR uses the built-in `GITHUB_TOKEN`.
+
+## Release flow
+
+1. Bump `version` in `pyproject.toml` in a chore PR and merge it.
+2. Tag the release on `main`:
+   - `git tag vX.Y.Z`
+   - `git push origin vX.Y.Z`
+
+For dry runs, dispatch the workflow with `dry_run: true`
+(`gh workflow run release.yml --ref <branch> -f dry_run=true`). It builds the
+wheel and the image and publishes nothing.

floe_python-0.8.1/.github/workflows/airflow-floe.yml

@@ -0,0 +1,37 @@
+name: Airflow Floe (Python)
+
+on:
+  pull_request:
+    paths:
+      - "orchestrators/airflow-floe/**"
+      - ".github/workflows/airflow-floe.yml"
+  push:
+    branches: ["main"]
+    paths:
+      - "orchestrators/airflow-floe/**"
+      - ".github/workflows/airflow-floe.yml"
+  workflow_dispatch: {}
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+      - name: Syntax check
+        run: python -m compileall orchestrators/airflow-floe/src orchestrators/airflow-floe/dags orchestrators/airflow-floe/tests
+      - name: Package install check
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install orchestrators/airflow-floe --no-deps
+      - name: Tests
+        run: python -m unittest discover -s orchestrators/airflow-floe/tests -p "test_*.py"

floe_python-0.8.1/.github/workflows/ci.yml

@@ -0,0 +1,124 @@
+name: CI
+
+on:
+  pull_request:
+  merge_group:
+  push:
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  # Single source of truth for path-based change detection. Runs `paths-filter`
+  # once and exposes every flag as a job output so the jobs below can gate at the
+  # job level (via `needs` + `if`) instead of each re-running checkout +
+  # paths-filter and booting a runner only to skip its steps.
+  changes:
+    runs-on: ubuntu-latest
+    outputs:
+      config_schema: ${{ steps.filter.outputs.config_schema }}
+      docs: ${{ steps.filter.outputs.docs }}
+      python_core: ${{ steps.filter.outputs.python_core }}
+      conformance: ${{ steps.filter.outputs.conformance }}
+    steps:
+      - uses: actions/checkout@v7
+      - uses: dorny/paths-filter@v4
+        id: filter
+        with:
+          filters: |
+            config_schema:
+              - "schemas/**"
+              - "scripts/test_config_schema.py"
+              - "src/floe/config/**"
+              - "src/floe/pii.py"
+              - "pyproject.toml"
+              - "uv.lock"
+              - "example/project/**"
+              - "example/standalone/**"
+              - ".github/workflows/ci.yml"
+            docs:
+              - "docs/**"
+              - "README.md"
+            python_core:
+              - "src/floe/**"
+              - "tests/**"
+              - "conformance/**"
+              - "pyproject.toml"
+              - "uv.lock"
+              - ".github/workflows/ci.yml"
+            conformance:
+              - "conformance/**"
+              - "src/floe/**"
+              - "pyproject.toml"
+              - "uv.lock"
+              - ".github/workflows/ci.yml"
+
+  config-schema:
+    needs: changes
+    if: needs.changes.outputs.config_schema == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v7
+      - run: uv sync --all-groups
+      - run: uv run python scripts/test_config_schema.py
+
+  python-lint:
+    needs: changes
+    if: needs.changes.outputs.python_core == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v7
+      - run: uv sync --all-groups
+      - run: uv run ruff check
+      - run: uv run ruff format --check
+
+  python-types:
+    needs: changes
+    if: needs.changes.outputs.python_core == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v7
+      - run: uv sync --all-groups
+      - run: uv run pyright
+
+  python-test:
+    needs: changes
+    # Gated per step, not per job: a skipped matrix job reports one `python-test`
+    # check, so the required `python-test (<version>)` checks would never report.
+    runs-on: ubuntu-latest
+    timeout-minutes: 20  # a stalled download or source build fails fast instead of hanging for GitHub's 6 h default
+    strategy:
+      fail-fast: false
+      matrix:
+        # Oldest supported (`requires-python`) and the newest with a wheel for every locked
+        # package. 3.15 builds duckdb, numpy, pyiceberg, pyyaml and zstandard from source;
+        # move up once they ship cp315 wheels.
+        python-version: ["3.11", "3.14"]
+    steps:
+      - uses: actions/checkout@v7
+        if: needs.changes.outputs.python_core == 'true'
+      - uses: astral-sh/setup-uv@v7
+        if: needs.changes.outputs.python_core == 'true'
+      - run: uv sync --all-groups --python ${{ matrix.python-version }}
+        if: needs.changes.outputs.python_core == 'true'
+      - run: uv run --python ${{ matrix.python-version }} pytest
+        if: needs.changes.outputs.python_core == 'true'
+
+  golden:
+    # The branch ruleset requires the check named `conformance`.
+    name: conformance
+    needs: changes
+    if: needs.changes.outputs.conformance == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v7
+      - run: uv sync --all-groups
+      - name: Python matches every golden case
+        run: uv run python conformance/check.py
+

floe_python-0.8.1/.github/workflows/coverage-python.yml

@@ -0,0 +1,46 @@
+name: Coverage (Python)
+
+# Informational only — never make this a required check. Runs on main after
+# Python core changes, on demand, and on PRs that edit this file.
+on:
+  push:
+    branches: [main]
+    paths:
+      - "src/floe/**"
+      - "tests/**"
+      - "pyproject.toml"
+      - "uv.lock"
+      - ".github/workflows/coverage-python.yml"
+  pull_request:
+    paths:
+      - ".github/workflows/coverage-python.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  coverage-python:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v7
+      - run: uv sync --all-groups
+      - name: Measure floe coverage
+        # Explicit bash turns on pipefail so a test failure isn't masked by tee.
+        shell: bash
+        run: uv run pytest --cov=floe --cov-report=term | tee coverage.txt
+      - name: Publish summary
+        run: |
+          {
+            echo '## floe (Python) coverage'
+            echo '```'
+            grep '^TOTAL' coverage.txt
+            echo '```'
+            echo '<details><summary>Per file</summary>'
+            echo
+            echo '```'
+            cat coverage.txt
+            echo '```'
+            echo '</details>'
+          } >> "$GITHUB_STEP_SUMMARY"

floe_python-0.8.1/.github/workflows/dagster-floe.yml

@@ -0,0 +1,45 @@
+name: Dagster Floe (Python)
+
+on:
+  pull_request:
+    paths:
+      - "orchestrators/dagster-floe/**"
+      - "src/floe/**"
+      - "pyproject.toml"
+      - ".github/workflows/dagster-floe.yml"
+  push:
+    branches: ["main"]
+    paths:
+      - "orchestrators/dagster-floe/**"
+      - "src/floe/**"
+      - "pyproject.toml"
+      - ".github/workflows/dagster-floe.yml"
+  workflow_dispatch: {}
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+          cache: pip
+          cache-dependency-path: |
+            orchestrators/dagster-floe/pyproject.toml
+            pyproject.toml
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -e .  # the floe core from this checkout: config mode needs floe.build_manifest
+          python -m pip install -e "orchestrators/dagster-floe[dev]"
+      - name: Tests
+        run: |
+          python -m pytest -q orchestrators/dagster-floe/tests

floe_python-0.8.1/.github/workflows/orchestrator-drift.yml

@@ -0,0 +1,39 @@
+name: Orchestrator Drift Guard
+
+on:
+  pull_request:
+    paths:
+      - "orchestrators/dagster-floe/src/floe_dagster/databricks_client.py"
+      - "orchestrators/airflow-floe/src/airflow_floe/databricks_client.py"
+      - "orchestrators/dagster-floe/src/floe_dagster/k8s_status.py"
+      - "orchestrators/airflow-floe/src/airflow_floe/k8s_status.py"
+      - "orchestrators/schemas/floe.manifest.schema.json"
+      - "orchestrators/dagster-floe/src/floe_dagster/schemas/floe.manifest.schema.json"
+      - "scripts/check_orchestrator_drift.py"
+      - ".github/workflows/orchestrator-drift.yml"
+  push:
+    branches: ["main"]
+    paths:
+      - "orchestrators/dagster-floe/src/floe_dagster/databricks_client.py"
+      - "orchestrators/airflow-floe/src/airflow_floe/databricks_client.py"
+      - "orchestrators/dagster-floe/src/floe_dagster/k8s_status.py"
+      - "orchestrators/airflow-floe/src/airflow_floe/k8s_status.py"
+      - "orchestrators/schemas/floe.manifest.schema.json"
+      - "orchestrators/dagster-floe/src/floe_dagster/schemas/floe.manifest.schema.json"
+      - "scripts/check_orchestrator_drift.py"
+      - ".github/workflows/orchestrator-drift.yml"
+  workflow_dispatch: {}
+
+permissions:
+  contents: read
+
+jobs:
+  drift:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+      - name: Check shared orchestrator modules are in sync
+        run: python scripts/check_orchestrator_drift.py

floe_python-0.8.1/.github/workflows/pr-size.yml

@@ -0,0 +1,37 @@
+name: PR size
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, edited, labeled, unlabeled]  # edited: base retarget
+  merge_group:
+
+jobs:
+  # Enforces the AGENTS.md size budget on non-test code. A maintainer can override
+  # it with the `large-pr` label; agents must never apply it.
+  size:
+    name: pr-size
+    # Skipped (reported as passing) in the merge queue: the PR was already checked.
+    if: github.event_name == 'pull_request'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          fetch-depth: 0
+      - name: Check non-test src diff against budget
+        if: >-
+          !contains(github.event.pull_request.labels.*.name, 'large-pr')
+          && github.event.pull_request.user.login != 'dependabot[bot]'
+        env:
+          BASE_REF: ${{ github.base_ref }}
+        run: |
+          git diff --numstat --no-renames "origin/$BASE_REF...HEAD" | awk -v max_lines=300 -v max_files=8 '
+            $3 ~ /^(orchestrators|src\/floe)\// && $3 !~ /\/tests?\/|\/fixtures\/|\.(md|json|lock)$/ {
+              if ($1 != "-") lines += $1 + $2; files++; print "  " $3
+            }
+            END {
+              printf "Non-test src: %d lines, %d files (budget: %d lines, %d files)\n", lines, files, max_lines, max_files
+              if (lines > max_lines || files > max_files) {
+                print "::error::Over the AGENTS.md size budget. Split into stacked PRs, or ask a maintainer for the large-pr label."
+                exit 1
+              }
+            }'

floe_python-0.8.1/.github/workflows/python-image.yml

@@ -0,0 +1,52 @@
+name: Python image
+
+# Builds Dockerfile.python and smoke-tests it (#926). Publishing is in release.yml.
+on:
+  pull_request:
+    paths:
+      - "Dockerfile.python"
+      - "src/floe/**"
+      - "pyproject.toml"
+      - "uv.lock"
+      - ".github/workflows/python-image.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  image:
+    runs-on: ubuntu-latest
+    env:
+      IMAGE: floe:ci
+    steps:
+      - uses: actions/checkout@v7
+      - uses: docker/setup-buildx-action@v4
+      - name: Build image
+        uses: docker/build-push-action@v7
+        with:
+          context: .
+          file: ./Dockerfile.python
+          load: true
+          tags: ${{ env.IMAGE }}
+          cache-from: type=gha,scope=python-image
+          cache-to: type=gha,mode=max,scope=python-image
+      - name: floe --version
+        run: docker run --rm "$IMAGE" --version
+      - name: floe validate
+        run: docker run --rm -v "$PWD/example:/work/example:ro" "$IMAGE" validate -c /work/example/standalone/config.yml
+      - name: floe run
+        run: |
+          set -euo pipefail
+          cp -r example/standalone "$RUNNER_TEMP/standalone"
+          chmod -R a+rwX "$RUNNER_TEMP/standalone"
+          docker run --rm -v "$RUNNER_TEMP/standalone:/work" "$IMAGE" run -c /work/config.yml
+          test -n "$(ls "$RUNNER_TEMP"/standalone/out/accepted/orders)"
+      - name: DuckDB extensions load offline
+        run: |
+          docker run --rm --network none --entrypoint python "$IMAGE" -c \
+            "import duckdb; c=duckdb.connect(); [c.execute(f'LOAD {e}') for e in ('avro','iceberg','httpfs','aws')]"