continuo-python-runtime 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- continuo_python_runtime-0.1.0/.dockerignore +14 -0
- continuo_python_runtime-0.1.0/.github/workflows/ci.yml +25 -0
- continuo_python_runtime-0.1.0/.github/workflows/images.yml +164 -0
- continuo_python_runtime-0.1.0/.github/workflows/publish-pypi.yml +129 -0
- continuo_python_runtime-0.1.0/.gitignore +9 -0
- continuo_python_runtime-0.1.0/Dockerfile.postgres +13 -0
- continuo_python_runtime-0.1.0/Dockerfile.trino +13 -0
- continuo_python_runtime-0.1.0/PKG-INFO +193 -0
- continuo_python_runtime-0.1.0/README.md +178 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/__init__.py +1 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/cli.py +142 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/conform.py +220 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/context.py +62 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/contract/__init__.py +1 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/contract/loader.py +363 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/contract/merge.py +80 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/contract/model.py +39 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/contract/paths.py +38 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/errors.py +36 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/harness.py +208 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/hashing.py +34 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/lint.py +295 -0
- continuo_python_runtime-0.1.0/continuo_python_runtime/types.py +208 -0
- continuo_python_runtime-0.1.0/docs/boundary-contract.md +152 -0
- continuo_python_runtime-0.1.0/docs/superpowers/plans/2026-07-31-python-runtime.md +1307 -0
- continuo_python_runtime-0.1.0/docs/superpowers/specs/2026-07-31-python-runtime-design.md +273 -0
- continuo_python_runtime-0.1.0/pyproject.toml +58 -0
- continuo_python_runtime-0.1.0/python-runtime-postgres/README.md +18 -0
- continuo_python_runtime-0.1.0/python-runtime-postgres/continuo_python_runtime_postgres/__init__.py +0 -0
- continuo_python_runtime-0.1.0/python-runtime-postgres/continuo_python_runtime_postgres/adapter.py +229 -0
- continuo_python_runtime-0.1.0/python-runtime-postgres/pyproject.toml +45 -0
- continuo_python_runtime-0.1.0/python-runtime-postgres/tests/__init__.py +0 -0
- continuo_python_runtime-0.1.0/python-runtime-postgres/tests/test_adapter_runtime_postgres.py +115 -0
- continuo_python_runtime-0.1.0/python-runtime-postgres/tests/test_integration_runtime_postgres.py +277 -0
- continuo_python_runtime-0.1.0/python-runtime-trino/README.md +77 -0
- continuo_python_runtime-0.1.0/python-runtime-trino/continuo_python_runtime_trino/__init__.py +0 -0
- continuo_python_runtime-0.1.0/python-runtime-trino/continuo_python_runtime_trino/adapter.py +388 -0
- continuo_python_runtime-0.1.0/python-runtime-trino/pyproject.toml +45 -0
- continuo_python_runtime-0.1.0/python-runtime-trino/tests/__init__.py +0 -0
- continuo_python_runtime-0.1.0/python-runtime-trino/tests/test_adapter_runtime_trino.py +210 -0
- continuo_python_runtime-0.1.0/python-runtime-trino/tests/test_integration_runtime_trino.py +366 -0
- continuo_python_runtime-0.1.0/template/.github/workflows/release.yml +58 -0
- continuo_python_runtime-0.1.0/template/Dockerfile +8 -0
- continuo_python_runtime-0.1.0/template/README.md +36 -0
- continuo_python_runtime-0.1.0/template/contracts/example.yml +12 -0
- continuo_python_runtime-0.1.0/template/scripts/example.py +3 -0
- continuo_python_runtime-0.1.0/tests/__init__.py +0 -0
- continuo_python_runtime-0.1.0/tests/conftest.py +81 -0
- continuo_python_runtime-0.1.0/tests/contract/__init__.py +0 -0
- continuo_python_runtime-0.1.0/tests/contract/test_loader.py +344 -0
- continuo_python_runtime-0.1.0/tests/contract/test_merge.py +118 -0
- continuo_python_runtime-0.1.0/tests/contract/test_model.py +25 -0
- continuo_python_runtime-0.1.0/tests/smoke/node_smoke/contracts/smoke.yml +13 -0
- continuo_python_runtime-0.1.0/tests/smoke/node_smoke/scripts/smoke.py +2 -0
- continuo_python_runtime-0.1.0/tests/smoke/postgres-stack/docker-compose.yml +16 -0
- continuo_python_runtime-0.1.0/tests/smoke/trino-stack/catalog/iceberg.properties +10 -0
- continuo_python_runtime-0.1.0/tests/smoke/trino-stack/docker-compose.yml +60 -0
- continuo_python_runtime-0.1.0/tests/test_cli.py +187 -0
- continuo_python_runtime-0.1.0/tests/test_conform.py +234 -0
- continuo_python_runtime-0.1.0/tests/test_context.py +51 -0
- continuo_python_runtime-0.1.0/tests/test_errors.py +27 -0
- continuo_python_runtime-0.1.0/tests/test_harness.py +261 -0
- continuo_python_runtime-0.1.0/tests/test_hashing.py +53 -0
- continuo_python_runtime-0.1.0/tests/test_lint.py +210 -0
- continuo_python_runtime-0.1.0/tests/test_package.py +5 -0
- continuo_python_runtime-0.1.0/tests/test_template.py +50 -0
- continuo_python_runtime-0.1.0/tests/test_types.py +83 -0
- continuo_python_runtime-0.1.0/uv.lock +600 -0
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
name: ci
|
|
2
|
+
on:
|
|
3
|
+
push: { branches: [main] }
|
|
4
|
+
pull_request:
|
|
5
|
+
jobs:
|
|
6
|
+
test:
|
|
7
|
+
runs-on: ubuntu-latest
|
|
8
|
+
steps:
|
|
9
|
+
- uses: actions/checkout@v4
|
|
10
|
+
- uses: astral-sh/setup-uv@v5
|
|
11
|
+
# continuo-validation-contract resolves from PyPI (==0.3.0).
|
|
12
|
+
- name: Sync
|
|
13
|
+
run: uv sync --all-packages --all-groups
|
|
14
|
+
- name: Lint
|
|
15
|
+
run: uv run ruff check .
|
|
16
|
+
- name: Types (runtime)
|
|
17
|
+
run: uv run mypy continuo_python_runtime
|
|
18
|
+
- name: Types (postgres adapter)
|
|
19
|
+
run: uv run --package continuo-python-runtime-postgres mypy python-runtime-postgres/continuo_python_runtime_postgres
|
|
20
|
+
- name: Types (trino adapter)
|
|
21
|
+
run: uv run --package continuo-python-runtime-trino mypy python-runtime-trino/continuo_python_runtime_trino
|
|
22
|
+
- name: Tests (runtime)
|
|
23
|
+
run: uv run pytest --cov=continuo_python_runtime -v
|
|
24
|
+
- name: Tests (adapter units)
|
|
25
|
+
run: uv run pytest python-runtime-postgres/tests python-runtime-trino/tests -m "not integration" -v
|
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
name: images
|
|
2
|
+
on:
|
|
3
|
+
pull_request:
|
|
4
|
+
paths:
|
|
5
|
+
- "Dockerfile.*"
|
|
6
|
+
- "tests/smoke/**"
|
|
7
|
+
- "continuo_python_runtime/**"
|
|
8
|
+
- "python-runtime-postgres/**"
|
|
9
|
+
- "python-runtime-trino/**"
|
|
10
|
+
- "pyproject.toml"
|
|
11
|
+
- ".github/workflows/images.yml"
|
|
12
|
+
push:
|
|
13
|
+
tags:
|
|
14
|
+
- "v*"
|
|
15
|
+
|
|
16
|
+
jobs:
|
|
17
|
+
build:
|
|
18
|
+
strategy:
|
|
19
|
+
matrix:
|
|
20
|
+
engine: [postgres, trino]
|
|
21
|
+
runs-on: ubuntu-latest
|
|
22
|
+
steps:
|
|
23
|
+
- uses: actions/checkout@v4
|
|
24
|
+
- name: Build ${{ matrix.engine }} image
|
|
25
|
+
run: docker build -f Dockerfile.${{ matrix.engine }} -t cpr-smoke-${{ matrix.engine }} .
|
|
26
|
+
- name: Save image
|
|
27
|
+
run: docker save cpr-smoke-${{ matrix.engine }} -o /tmp/cpr-smoke-${{ matrix.engine }}.tar
|
|
28
|
+
- uses: actions/upload-artifact@v4
|
|
29
|
+
with:
|
|
30
|
+
name: cpr-smoke-${{ matrix.engine }}-image
|
|
31
|
+
path: /tmp/cpr-smoke-${{ matrix.engine }}.tar
|
|
32
|
+
retention-days: 1
|
|
33
|
+
|
|
34
|
+
smoke-postgres:
|
|
35
|
+
needs: build
|
|
36
|
+
runs-on: ubuntu-latest
|
|
37
|
+
services:
|
|
38
|
+
postgres:
|
|
39
|
+
image: postgres:16
|
|
40
|
+
env:
|
|
41
|
+
POSTGRES_USER: continuo
|
|
42
|
+
POSTGRES_PASSWORD: continuo
|
|
43
|
+
POSTGRES_DB: warehouse
|
|
44
|
+
ports:
|
|
45
|
+
- 5432:5432
|
|
46
|
+
options: >-
|
|
47
|
+
--health-cmd "pg_isready -U continuo -d warehouse"
|
|
48
|
+
--health-interval 5s
|
|
49
|
+
--health-timeout 5s
|
|
50
|
+
--health-retries 10
|
|
51
|
+
steps:
|
|
52
|
+
- uses: actions/checkout@v4
|
|
53
|
+
- uses: actions/download-artifact@v4
|
|
54
|
+
with:
|
|
55
|
+
name: cpr-smoke-postgres-image
|
|
56
|
+
path: /tmp
|
|
57
|
+
- name: Load image
|
|
58
|
+
run: docker load -i /tmp/cpr-smoke-postgres.tar
|
|
59
|
+
- name: Install psql client
|
|
60
|
+
run: |
|
|
61
|
+
sudo apt-get update
|
|
62
|
+
sudo apt-get install -y --no-install-recommends postgresql-client
|
|
63
|
+
- name: Seed source table
|
|
64
|
+
env:
|
|
65
|
+
PGPASSWORD: continuo
|
|
66
|
+
run: |
|
|
67
|
+
psql -h localhost -U continuo -d warehouse -c \
|
|
68
|
+
"CREATE SCHEMA analytics; CREATE TABLE analytics.src(id int); INSERT INTO analytics.src VALUES (1),(2),(3);"
|
|
69
|
+
- name: Run smoke node
|
|
70
|
+
run: |
|
|
71
|
+
docker run --rm --network host \
|
|
72
|
+
-v "$PWD/tests/smoke/node_smoke:/app" \
|
|
73
|
+
-e NODE_ID=python-model.smoke.analytics.smoke \
|
|
74
|
+
-e TABLE_NAME=smoke \
|
|
75
|
+
-e TARGET_SCHEMA=analytics \
|
|
76
|
+
-e POSTGRES_HOST=localhost \
|
|
77
|
+
-e POSTGRES_PORT=5432 \
|
|
78
|
+
-e POSTGRES_DB=warehouse \
|
|
79
|
+
-e POSTGRES_USER=continuo \
|
|
80
|
+
-e POSTGRES_PASSWORD=continuo \
|
|
81
|
+
cpr-smoke-postgres | tee /tmp/smoke.out
|
|
82
|
+
test "${PIPESTATUS[0]}" -eq 0
|
|
83
|
+
shell: bash
|
|
84
|
+
- name: Assert success sentinel
|
|
85
|
+
run: grep -q '"status":"success"' /tmp/smoke.out
|
|
86
|
+
- name: Assert row count
|
|
87
|
+
env:
|
|
88
|
+
PGPASSWORD: continuo
|
|
89
|
+
run: |
|
|
90
|
+
count=$(psql -h localhost -U continuo -d warehouse -t -c "select count(*) from analytics.smoke" | tr -d '[:space:]')
|
|
91
|
+
test "$count" = "3"
|
|
92
|
+
|
|
93
|
+
smoke-trino:
|
|
94
|
+
needs: build
|
|
95
|
+
runs-on: ubuntu-latest
|
|
96
|
+
steps:
|
|
97
|
+
- uses: actions/checkout@v4
|
|
98
|
+
- uses: actions/download-artifact@v4
|
|
99
|
+
with:
|
|
100
|
+
name: cpr-smoke-trino-image
|
|
101
|
+
path: /tmp
|
|
102
|
+
- name: Load image
|
|
103
|
+
run: docker load -i /tmp/cpr-smoke-trino.tar
|
|
104
|
+
- name: Start trino stack
|
|
105
|
+
run: docker compose -f tests/smoke/trino-stack/docker-compose.yml up -d --wait
|
|
106
|
+
- name: Seed source table
|
|
107
|
+
run: |
|
|
108
|
+
docker exec "$(docker compose -f tests/smoke/trino-stack/docker-compose.yml ps -q trino)" \
|
|
109
|
+
trino --execute "CREATE SCHEMA IF NOT EXISTS iceberg.analytics; CREATE TABLE iceberg.analytics.src(id int); INSERT INTO iceberg.analytics.src VALUES 1,2,3"
|
|
110
|
+
- name: Run smoke node
|
|
111
|
+
run: |
|
|
112
|
+
docker run --rm \
|
|
113
|
+
-v "$PWD/tests/smoke/node_smoke:/app" \
|
|
114
|
+
-e NODE_ID=python-model.smoke.analytics.smoke \
|
|
115
|
+
-e TABLE_NAME=smoke \
|
|
116
|
+
-e TARGET_SCHEMA=analytics \
|
|
117
|
+
-e TRINO_HOST=host.docker.internal \
|
|
118
|
+
-e TRINO_PORT=18080 \
|
|
119
|
+
-e TRINO_CATALOG=iceberg \
|
|
120
|
+
-e TRINO_USER=continuo \
|
|
121
|
+
--add-host=host.docker.internal:host-gateway \
|
|
122
|
+
cpr-smoke-trino | tee /tmp/smoke-trino.out
|
|
123
|
+
test "${PIPESTATUS[0]}" -eq 0
|
|
124
|
+
shell: bash
|
|
125
|
+
- name: Assert success sentinel
|
|
126
|
+
run: grep -q '"status":"success"' /tmp/smoke-trino.out
|
|
127
|
+
- name: Assert row count
|
|
128
|
+
run: |
|
|
129
|
+
count=$(docker exec "$(docker compose -f tests/smoke/trino-stack/docker-compose.yml ps -q trino)" \
|
|
130
|
+
trino --execute "select count(*) from iceberg.analytics.smoke" | tr -d '[:space:]"')
|
|
131
|
+
test "$count" = "3"
|
|
132
|
+
- name: Tear down trino stack
|
|
133
|
+
if: always()
|
|
134
|
+
run: docker compose -f tests/smoke/trino-stack/docker-compose.yml down -v
|
|
135
|
+
|
|
136
|
+
# On tag push: log in to ghcr.io and push ghcr.io/${{ github.repository }}:<tag>-postgres
|
|
137
|
+
# and :<tag>-trino. continuo-validation-contract, continuo-python-runtime-postgres,
|
|
138
|
+
# and continuo-python-runtime-trino are all installed from the build context /
|
|
139
|
+
# PyPI already (see Dockerfile.postgres / Dockerfile.trino and
|
|
140
|
+
# publish-pypi.yml) — nothing pending here.
|
|
141
|
+
publish:
|
|
142
|
+
if: startsWith(github.ref, 'refs/tags/v')
|
|
143
|
+
needs: [smoke-postgres, smoke-trino]
|
|
144
|
+
strategy:
|
|
145
|
+
matrix:
|
|
146
|
+
engine: [postgres, trino]
|
|
147
|
+
runs-on: ubuntu-latest
|
|
148
|
+
permissions:
|
|
149
|
+
contents: read
|
|
150
|
+
packages: write
|
|
151
|
+
steps:
|
|
152
|
+
- uses: actions/checkout@v4
|
|
153
|
+
- name: Log in to ghcr.io
|
|
154
|
+
uses: docker/login-action@v3
|
|
155
|
+
with:
|
|
156
|
+
registry: ghcr.io
|
|
157
|
+
username: ${{ github.actor }}
|
|
158
|
+
password: ${{ secrets.GITHUB_TOKEN }}
|
|
159
|
+
- name: Build and push ${{ matrix.engine }} image
|
|
160
|
+
run: |
|
|
161
|
+
tag="${GITHUB_REF_NAME}"
|
|
162
|
+
image="ghcr.io/${{ github.repository }}:${tag}-${{ matrix.engine }}"
|
|
163
|
+
docker build -f Dockerfile.${{ matrix.engine }} -t "$image" .
|
|
164
|
+
docker push "$image"
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
name: publish-pypi
|
|
2
|
+
|
|
3
|
+
# Publishes the continuo-python-runtime package (and, in the two adapter jobs
|
|
4
|
+
# below, the continuo-python-runtime-postgres / continuo-python-runtime-trino
|
|
5
|
+
# packages) via PyPI Trusted Publishing (OIDC, no stored token).
|
|
6
|
+
#
|
|
7
|
+
# Tag glob note: "python-runtime-v*" requires a literal "v" immediately after
|
|
8
|
+
# the dash, so it matches only the harness tag (python-runtime-v1.2.3[-testN])
|
|
9
|
+
# and never "python-runtime-postgres-v*" / "python-runtime-trino-v*" (whose
|
|
10
|
+
# char after the shared "python-runtime-" prefix is "p"/"t", not "v") — the
|
|
11
|
+
# three globs are disjoint, so none of the three jobs below shadows another.
|
|
12
|
+
# Tag `<pkg>-v<ver>-test<n>` publishes to TestPyPI; `<pkg>-v<ver>` publishes to
|
|
13
|
+
# real PyPI. The GitHub environment name is what each PyPI "pending publisher"
|
|
14
|
+
# is registered against.
|
|
15
|
+
#
|
|
16
|
+
# Ordering: continuo-validation-contract==0.3.0 must exist on the target index
|
|
17
|
+
# before publishing here — consumers of the wheel resolve it from that index
|
|
18
|
+
# ([tool.uv.sources] is dev-only and not embedded in the wheel).
|
|
19
|
+
on:
|
|
20
|
+
push:
|
|
21
|
+
tags:
|
|
22
|
+
- "python-runtime-v*"
|
|
23
|
+
- "python-runtime-postgres-v*"
|
|
24
|
+
- "python-runtime-trino-v*"
|
|
25
|
+
|
|
26
|
+
permissions: {}
|
|
27
|
+
|
|
28
|
+
jobs:
|
|
29
|
+
publish-runtime:
|
|
30
|
+
if: startsWith(github.ref_name, 'python-runtime-v')
|
|
31
|
+
runs-on: ubuntu-latest
|
|
32
|
+
environment: ${{ contains(github.ref_name, '-test') && 'testpypi' || 'pypi' }}
|
|
33
|
+
permissions:
|
|
34
|
+
id-token: write # OIDC token for Trusted Publishing
|
|
35
|
+
contents: read # actions/checkout needs read access to the repo
|
|
36
|
+
steps:
|
|
37
|
+
- uses: actions/checkout@v4
|
|
38
|
+
- uses: astral-sh/setup-uv@v5
|
|
39
|
+
- name: Test before publishing
|
|
40
|
+
run: |
|
|
41
|
+
uv sync --all-groups
|
|
42
|
+
uv run pytest -q
|
|
43
|
+
- name: Build the sdist + wheel
|
|
44
|
+
run: uv build -o dist
|
|
45
|
+
- name: Publish to TestPyPI
|
|
46
|
+
if: contains(github.ref_name, '-test')
|
|
47
|
+
# Pinned to a commit SHA. release/v1 is a moving branch, and this job
|
|
48
|
+
# holds id-token: write for PyPI Trusted Publishing — a compromised
|
|
49
|
+
# revision could mint a token and publish under this project's name.
|
|
50
|
+
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
|
|
51
|
+
with:
|
|
52
|
+
repository-url: https://test.pypi.org/legacy/
|
|
53
|
+
packages-dir: dist
|
|
54
|
+
- name: Publish to PyPI
|
|
55
|
+
if: ${{ !contains(github.ref_name, '-test') }}
|
|
56
|
+
# Pinned to a commit SHA. release/v1 is a moving branch, and this job
|
|
57
|
+
# holds id-token: write for PyPI Trusted Publishing — a compromised
|
|
58
|
+
# revision could mint a token and publish under this project's name.
|
|
59
|
+
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
|
|
60
|
+
with:
|
|
61
|
+
packages-dir: dist
|
|
62
|
+
|
|
63
|
+
publish-runtime-postgres:
|
|
64
|
+
if: startsWith(github.ref_name, 'python-runtime-postgres-v')
|
|
65
|
+
runs-on: ubuntu-latest
|
|
66
|
+
environment: ${{ contains(github.ref_name, '-test') && 'testpypi-postgres' || 'pypi-postgres' }}
|
|
67
|
+
permissions:
|
|
68
|
+
id-token: write # OIDC token for Trusted Publishing
|
|
69
|
+
contents: read # actions/checkout needs read access to the repo
|
|
70
|
+
steps:
|
|
71
|
+
- uses: actions/checkout@v4
|
|
72
|
+
- uses: astral-sh/setup-uv@v5
|
|
73
|
+
- name: Test before publishing
|
|
74
|
+
run: |
|
|
75
|
+
uv sync --all-packages --all-groups
|
|
76
|
+
uv run pytest python-runtime-postgres/tests -m "not integration" -q
|
|
77
|
+
- name: Build the sdist + wheel
|
|
78
|
+
run: uv build --package continuo-python-runtime-postgres python-runtime-postgres -o dist
|
|
79
|
+
- name: Publish to TestPyPI
|
|
80
|
+
if: contains(github.ref_name, '-test')
|
|
81
|
+
# Pinned to a commit SHA. release/v1 is a moving branch, and this job
|
|
82
|
+
# holds id-token: write for PyPI Trusted Publishing — a compromised
|
|
83
|
+
# revision could mint a token and publish under this project's name.
|
|
84
|
+
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
|
|
85
|
+
with:
|
|
86
|
+
repository-url: https://test.pypi.org/legacy/
|
|
87
|
+
packages-dir: dist
|
|
88
|
+
- name: Publish to PyPI
|
|
89
|
+
if: ${{ !contains(github.ref_name, '-test') }}
|
|
90
|
+
# Pinned to a commit SHA. release/v1 is a moving branch, and this job
|
|
91
|
+
# holds id-token: write for PyPI Trusted Publishing — a compromised
|
|
92
|
+
# revision could mint a token and publish under this project's name.
|
|
93
|
+
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
|
|
94
|
+
with:
|
|
95
|
+
packages-dir: dist
|
|
96
|
+
|
|
97
|
+
publish-runtime-trino:
|
|
98
|
+
if: startsWith(github.ref_name, 'python-runtime-trino-v')
|
|
99
|
+
runs-on: ubuntu-latest
|
|
100
|
+
environment: ${{ contains(github.ref_name, '-test') && 'testpypi-trino' || 'pypi-trino' }}
|
|
101
|
+
permissions:
|
|
102
|
+
id-token: write # OIDC token for Trusted Publishing
|
|
103
|
+
contents: read # actions/checkout needs read access to the repo
|
|
104
|
+
steps:
|
|
105
|
+
- uses: actions/checkout@v4
|
|
106
|
+
- uses: astral-sh/setup-uv@v5
|
|
107
|
+
- name: Test before publishing
|
|
108
|
+
run: |
|
|
109
|
+
uv sync --all-packages --all-groups
|
|
110
|
+
uv run pytest python-runtime-trino/tests -m "not integration" -q
|
|
111
|
+
- name: Build the sdist + wheel
|
|
112
|
+
run: uv build --package continuo-python-runtime-trino python-runtime-trino -o dist
|
|
113
|
+
- name: Publish to TestPyPI
|
|
114
|
+
if: contains(github.ref_name, '-test')
|
|
115
|
+
# Pinned to a commit SHA. release/v1 is a moving branch, and this job
|
|
116
|
+
# holds id-token: write for PyPI Trusted Publishing — a compromised
|
|
117
|
+
# revision could mint a token and publish under this project's name.
|
|
118
|
+
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
|
|
119
|
+
with:
|
|
120
|
+
repository-url: https://test.pypi.org/legacy/
|
|
121
|
+
packages-dir: dist
|
|
122
|
+
- name: Publish to PyPI
|
|
123
|
+
if: ${{ !contains(github.ref_name, '-test') }}
|
|
124
|
+
# Pinned to a commit SHA. release/v1 is a moving branch, and this job
|
|
125
|
+
# holds id-token: write for PyPI Trusted Publishing — a compromised
|
|
126
|
+
# revision could mint a token and publish under this project's name.
|
|
127
|
+
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
|
|
128
|
+
with:
|
|
129
|
+
packages-dir: dist
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
FROM python:3.14-slim
|
|
2
|
+
# Until the runtime and adapter packages publish to (Test)PyPI, install both
|
|
3
|
+
# from the build context; the contract from PyPI.
|
|
4
|
+
COPY pyproject.toml README.md /src/
|
|
5
|
+
COPY continuo_python_runtime /src/continuo_python_runtime
|
|
6
|
+
COPY python-runtime-postgres /src/python-runtime-postgres
|
|
7
|
+
RUN pip install --no-cache-dir \
|
|
8
|
+
"continuo-validation-contract==0.3.0" \
|
|
9
|
+
/src/python-runtime-postgres \
|
|
10
|
+
/src
|
|
11
|
+
ENV CONTRACT_DIR=/app/contracts APP_ROOT=/app
|
|
12
|
+
WORKDIR /app
|
|
13
|
+
ENTRYPOINT ["continuo-runtime", "run"]
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
FROM python:3.14-slim
|
|
2
|
+
# Until the runtime and adapter packages publish to (Test)PyPI, install both
|
|
3
|
+
# from the build context; the contract from PyPI.
|
|
4
|
+
COPY pyproject.toml README.md /src/
|
|
5
|
+
COPY continuo_python_runtime /src/continuo_python_runtime
|
|
6
|
+
COPY python-runtime-trino /src/python-runtime-trino
|
|
7
|
+
RUN pip install --no-cache-dir \
|
|
8
|
+
"continuo-validation-contract==0.3.0" \
|
|
9
|
+
/src/python-runtime-trino \
|
|
10
|
+
/src
|
|
11
|
+
ENV CONTRACT_DIR=/app/contracts APP_ROOT=/app
|
|
12
|
+
WORKDIR /app
|
|
13
|
+
ENTRYPOINT ["continuo-runtime", "run"]
|
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: continuo-python-runtime
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Runtime harness, contract tooling, and CI lint for Continuo python nodes.
|
|
5
|
+
Author: Simone Carolini
|
|
6
|
+
Maintainer: Simone Carolini
|
|
7
|
+
Classifier: Development Status :: 4 - Beta
|
|
8
|
+
Classifier: Intended Audience :: Developers
|
|
9
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
10
|
+
Requires-Python: >=3.14
|
|
11
|
+
Requires-Dist: continuo-validation-contract==0.3.0
|
|
12
|
+
Requires-Dist: pyarrow==25.0.0
|
|
13
|
+
Requires-Dist: pyyaml==6.0.3
|
|
14
|
+
Description-Content-Type: text/markdown
|
|
15
|
+
|
|
16
|
+
# Continuo Python Runtime
|
|
17
|
+
|
|
18
|
+
Runtime harness, contract tooling, and CI lint for Continuo python nodes.
|
|
19
|
+
This repo is what domain data teams (marketing, finance, …) template from to
|
|
20
|
+
ship a python node into Continuo: write a contract + a `run(ctx)` script,
|
|
21
|
+
push to `main`, and CI does the rest — lint, validate, merge, build, publish,
|
|
22
|
+
and register the release with Continuo.
|
|
23
|
+
|
|
24
|
+
## What this repo is
|
|
25
|
+
|
|
26
|
+
Three artifacts come out of this repository:
|
|
27
|
+
|
|
28
|
+
- **The `continuo-python-runtime` PyPI package** — the `continuo-runtime` CLI
|
|
29
|
+
(`validate` / `merge` / `hash` / `lint` / `run`) and the harness library
|
|
30
|
+
(`conform()`, `RunContext`, the error taxonomy) that domain repos install.
|
|
31
|
+
- **Per-engine base images**, one per warehouse engine (`...-postgres`,
|
|
32
|
+
`...-trino`), that domain repos build `FROM`. Each image bakes in the
|
|
33
|
+
runtime, a single `RuntimeAdapter` for that engine, and the
|
|
34
|
+
`continuo-runtime run` entrypoint.
|
|
35
|
+
- **`template/`** — a copy-ready domain repo: `Dockerfile`, `contracts/`,
|
|
36
|
+
`scripts/`, and the `release.yml` CI/CD workflow.
|
|
37
|
+
|
|
38
|
+
Base images and the PyPI publication of this package land with this repo's
|
|
39
|
+
PR 9 (the image pipeline); until then, install the runtime from git as noted
|
|
40
|
+
in `template/.github/workflows/release.yml`.
|
|
41
|
+
|
|
42
|
+
### Package map
|
|
43
|
+
|
|
44
|
+
Each per-engine base image bakes together three PyPI packages: the harness,
|
|
45
|
+
the engine adapter, and the published contract. The harness and both engine
|
|
46
|
+
adapters live in *this* repo (`continuo-python-runtime`) as a uv workspace;
|
|
47
|
+
`continuo-validation-runners` owns the *validation-side* (offline lint/merge)
|
|
48
|
+
counterparts of the same engines, which is a separate concern from the
|
|
49
|
+
data-plane adapters here.
|
|
50
|
+
|
|
51
|
+
| Package (distribution name) | Module | Lives in | Role |
|
|
52
|
+
| --- | --- | --- | --- |
|
|
53
|
+
| `continuo-python-runtime` | `continuo_python_runtime` | this repo (root) | Harness: CLI, `conform()`, `RunContext`, error taxonomy. |
|
|
54
|
+
| `continuo-python-runtime-postgres` | `continuo_python_runtime_postgres` | this repo, `python-runtime-postgres/` | Data-plane `RuntimeAdapter` for Postgres (`fetch`/`ensure_table`/`load`). |
|
|
55
|
+
| `continuo-python-runtime-trino` | `continuo_python_runtime_trino` | this repo, `python-runtime-trino/` | Data-plane `RuntimeAdapter` for Trino/Iceberg. |
|
|
56
|
+
| `continuo-validation-contract` | `continuo_validation_contract` | `continuo-validation-runners` | The published contract (schema, `RuntimeAdapter` port, result-block format) both sides depend on. |
|
|
57
|
+
| `continuo-validation-postgres` / `continuo-validation-trino` | `continuo_validation_postgres` / `continuo_validation_trino` | `continuo-validation-runners` | Validation-side (lint/merge, no live warehouse I/O) adapters — not to be confused with the data-plane adapters above. |
|
|
58
|
+
|
|
59
|
+
All three packages built in this repo resolve `continuo-validation-contract`
|
|
60
|
+
from PyPI (`==0.3.0`); the two adapter packages are uv workspace members
|
|
61
|
+
(`[tool.uv.workspace]` in the root `pyproject.toml`), so `uv sync
|
|
62
|
+
--all-packages --all-groups` at the repo root installs everything for local
|
|
63
|
+
development.
|
|
64
|
+
|
|
65
|
+
## Quickstart for domain teams
|
|
66
|
+
|
|
67
|
+
1. Copy `template/` into a new repository.
|
|
68
|
+
2. Edit `template/.github/workflows/release.yml` and set `SERVICE` to your
|
|
69
|
+
service name (one service name per domain repo).
|
|
70
|
+
3. Configure repository variables in GitHub (Settings → Secrets and
|
|
71
|
+
variables → Actions): `REGISTRY` (your Docker registry), `BUCKET` (your
|
|
72
|
+
S3 bucket for contract artifacts), `RELEASE_ENDPOINT` (the release
|
|
73
|
+
webhook endpoint). `RELEASE_ENDPOINT` is the **base URL** of the Continuo
|
|
74
|
+
API (no `/releases` suffix) — the workflow appends `/releases` itself.
|
|
75
|
+
4. Configure repository secrets: `AWS_ACCESS_KEY_ID` and
|
|
76
|
+
`AWS_SECRET_ACCESS_KEY` for the S3 upload. The template workflow pushes
|
|
77
|
+
the built image to GHCR using the workflow's own `GITHUB_TOKEN` (granted
|
|
78
|
+
`packages: write`) — no registry secret is needed for that. If you point
|
|
79
|
+
`REGISTRY` at a different or private registry, add your own `docker
|
|
80
|
+
login` step to `release.yml`.
|
|
81
|
+
5. Write a contract file under `contracts/` (see
|
|
82
|
+
`template/contracts/example.yml`) and a script under `scripts/` that
|
|
83
|
+
implements `run(ctx)` (see `template/scripts/example.py`).
|
|
84
|
+
6. Push to `main`. The `release.yml` workflow lints the scripts, validates
|
|
85
|
+
and merges the contracts, builds and pushes the image, uploads the merged
|
|
86
|
+
contract to S3, and POSTs the release.
|
|
87
|
+
|
|
88
|
+
## The script API
|
|
89
|
+
|
|
90
|
+
A node script is a Python file with exactly one required entry point:
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
def run(ctx):
|
|
94
|
+
...
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
- `ctx` is a `RunContext` (`continuo_python_runtime.context.RunContext`).
|
|
98
|
+
Its only method is `ctx.read(name)`, where `name` is one of the read
|
|
99
|
+
names declared under the node's `reads:` map in the contract — reading
|
|
100
|
+
anything else raises `ReadError`. Each declared read is fetched once and
|
|
101
|
+
memoized; `ctx.read(name)` returns a `pyarrow.Table`.
|
|
102
|
+
- `run(ctx)` can return anything Arrow-convertible: a `pyarrow.Table`
|
|
103
|
+
as-is, a pandas `DataFrame` (converted via
|
|
104
|
+
`pa.Table.from_pandas(..., preserve_index=False)`), or any object
|
|
105
|
+
implementing the Arrow C stream protocol (`__arrow_c_stream__`) — for
|
|
106
|
+
example a polars DataFrame. Returning anything else raises `ScriptError`.
|
|
107
|
+
- The dataframe library is your choice. No dataframe library is baked into
|
|
108
|
+
the base image — the package's own runtime dependencies are `pyarrow`,
|
|
109
|
+
`PyYAML`, and `continuo-validation-contract`. Add whatever you script
|
|
110
|
+
against (pandas, polars, …) as a `RUN pip install` line in your own
|
|
111
|
+
`Dockerfile`, on top of the base image.
|
|
112
|
+
- Scripts do not import warehouse drivers, write raw SQL literals, or call
|
|
113
|
+
data-access methods directly — `continuo-runtime lint` rejects those:
|
|
114
|
+
- forbidden driver imports (`psycopg2`/`sqlalchemy`/`trino`/etc.),
|
|
115
|
+
- SQL string literals (in plain strings, f-strings, and `+` concatenation),
|
|
116
|
+
- forbidden data-access calls (`execute`/`read_sql`/etc.), including ones
|
|
117
|
+
reached via a `from ... import` alias (e.g. `from pandas import
|
|
118
|
+
read_sql as rs` then calling `rs(...)`),
|
|
119
|
+
- private/protected attribute access (`obj._x`) — except on `self`/`cls`,
|
|
120
|
+
so a script's own class-private helpers (`self._helper()`) aren't
|
|
121
|
+
flagged.
|
|
122
|
+
|
|
123
|
+
The SQL-literal rule is best-effort: docstrings are exempt, but other
|
|
124
|
+
prose may still occasionally match. The hard guarantees enforced by lint
|
|
125
|
+
are the driver-import and data-access-call rules, together with the fact
|
|
126
|
+
that `RunContext` only exposes `ctx.read()` — all warehouse access goes
|
|
127
|
+
through it.
|
|
128
|
+
- The harness — not the script — performs the write. It calls `conform()`
|
|
129
|
+
on whatever `run()` returned and issues the only INSERT; the script never
|
|
130
|
+
writes directly.
|
|
131
|
+
|
|
132
|
+
## Conform rules
|
|
133
|
+
|
|
134
|
+
`conform()` (`continuo_python_runtime/conform.py`) enforces the node's
|
|
135
|
+
declared `output_columns` on the table `run()` returned, in this order:
|
|
136
|
+
|
|
137
|
+
| Check | Behavior |
|
|
138
|
+
| --- | --- |
|
|
139
|
+
| Duplicate columns | Any duplicate column name in the returned table always raises `ConformError`. |
|
|
140
|
+
| Extra columns | Governed by the node's `extra_columns` policy: `raise` (default) fails the run; `warn` drops the undeclared column(s) and logs a warning. |
|
|
141
|
+
| Missing columns | Any declared column absent from the returned table always raises `ConformError`. |
|
|
142
|
+
| Column order | The table is reselected into the declared column order. |
|
|
143
|
+
| Strict cast | Each column is cast to its declared Arrow type with `safe=True`. Casts pyarrow's `safe=True` would silently accept but that are not value-lossless are rejected before the cast even runs: floating → decimal (rounds to scale), non-boolean → boolean (coerces truthiness), timestamp → date (drops time-of-day). Any other cast failure also raises `ConformError`. |
|
|
144
|
+
| Not-null | A column declared `nullable: false` that contains any null raises `ConformError`. |
|
|
145
|
+
| VARCHAR/CHAR length | A column declared `VARCHAR(n)`/`CHAR(n)` whose longest value exceeds `n` raises `ConformError`. |
|
|
146
|
+
|
|
147
|
+
## Error taxonomy
|
|
148
|
+
|
|
149
|
+
Every runtime failure is one of five `HarnessError` subclasses
|
|
150
|
+
(`continuo_python_runtime/errors.py`). The sentinel result block's message
|
|
151
|
+
is prefixed `<ErrorClass>: ` so failures can be triaged without parsing free
|
|
152
|
+
text.
|
|
153
|
+
|
|
154
|
+
| Class | Meaning | Typical fix target |
|
|
155
|
+
| --- | --- | --- |
|
|
156
|
+
| `ContractError` | Contract missing or invalid, node not found for `NODE_ID`, or the declared script is missing/unreachable. | The contract yaml or the `script:` path. |
|
|
157
|
+
| `ReadError` | `ctx.read()` was called with an undeclared name, or a declared read failed at the warehouse. | The `reads:` map, or the upstream query/warehouse access. |
|
|
158
|
+
| `ScriptError` | `run()` raised, has no callable `run`, or returned a value that isn't Arrow-convertible. | The node script. |
|
|
159
|
+
| `ConformError` | Structural mismatch (extra/missing/duplicate columns), a strict-cast failure, a not-null violation, or a VARCHAR/CHAR overflow. | The script's output shape, or the `output_columns` declaration. |
|
|
160
|
+
| `LoadError` | Adapter construction failed, or the DDL/INSERT failed at the warehouse during the write. | Warehouse connectivity/permissions, or the target table. |
|
|
161
|
+
|
|
162
|
+
## Engine selection
|
|
163
|
+
|
|
164
|
+
A domain repo picks its warehouse engine by which base image it builds
|
|
165
|
+
`FROM`:
|
|
166
|
+
|
|
167
|
+
```dockerfile
|
|
168
|
+
FROM ghcr.io/carolsimone/continuo-python-runtime:v0.1.0-postgres
|
|
169
|
+
# or
|
|
170
|
+
FROM ghcr.io/carolsimone/continuo-python-runtime:v0.1.0-trino
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
Each image bakes in exactly one `RuntimeAdapter` for that engine — installed
|
|
174
|
+
from this repo's `python-runtime-postgres/` or `python-runtime-trino/`
|
|
175
|
+
package (see the package map above; both adapters live in this repo, not
|
|
176
|
+
`continuo-validation-runners`) — registered under the
|
|
177
|
+
`continuo_runtime.adapters` entry-point group (entry names `postgres` /
|
|
178
|
+
`trino`). The harness discovers it via `discover_runtime_adapter()` at run
|
|
179
|
+
time, so a single image serves every node in the service. The executor
|
|
180
|
+
injects the
|
|
181
|
+
warehouse connection as environment variables (engine-native, e.g.
|
|
182
|
+
`POSTGRES_HOST`/`POSTGRES_DB`/`POSTGRES_USER`) plus the node-selection
|
|
183
|
+
environment (`NODE_ID`, `TABLE_NAME`, `TARGET_SCHEMA`, and optionally
|
|
184
|
+
`CONTRACT_DIR`/`APP_ROOT`) that `continuo-runtime run` reads to dispatch the
|
|
185
|
+
right node's script.
|
|
186
|
+
|
|
187
|
+
## Further reading
|
|
188
|
+
|
|
189
|
+
- `docs/superpowers/specs/2026-07-31-python-runtime-design.md` — this
|
|
190
|
+
repo's design.
|
|
191
|
+
- `docs/boundary-contract.md` — the parent design's boundary contract (§13):
|
|
192
|
+
the five surfaces (S3 artifact, `content_hash`, the release call, the
|
|
193
|
+
runtime image, and the domain repo's CI/CD) that this repo implements.
|