continuo-trino-adapter 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- continuo_trino_adapter-0.2.0/.gitignore +11 -0
- continuo_trino_adapter-0.2.0/PKG-INFO +101 -0
- continuo_trino_adapter-0.2.0/README.md +86 -0
- continuo_trino_adapter-0.2.0/continuo_trino_adapter/__init__.py +0 -0
- continuo_trino_adapter-0.2.0/continuo_trino_adapter/adapter.py +610 -0
- continuo_trino_adapter-0.2.0/pyproject.toml +57 -0
- continuo_trino_adapter-0.2.0/tests/__init__.py +0 -0
- continuo_trino_adapter-0.2.0/tests/test_adapter_runtime_trino.py +501 -0
- continuo_trino_adapter-0.2.0/tests/test_adapter_trino_validation.py +120 -0
- continuo_trino_adapter-0.2.0/tests/test_integration_runtime_trino.py +370 -0
- continuo_trino_adapter-0.2.0/tests/test_integration_trino_validation.py +469 -0
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: continuo-trino-adapter
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Trino warehouse adapter for Continuo: validation and python-node runtime.
|
|
5
|
+
Author: Simone Carolini
|
|
6
|
+
Maintainer: Simone Carolini
|
|
7
|
+
Classifier: Development Status :: 4 - Beta
|
|
8
|
+
Classifier: Intended Audience :: Developers
|
|
9
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
10
|
+
Requires-Python: >=3.14
|
|
11
|
+
Requires-Dist: continuo-engine-contract==0.7.3
|
|
12
|
+
Requires-Dist: pyarrow==25.0.1
|
|
13
|
+
Requires-Dist: trino==0.338.0
|
|
14
|
+
Description-Content-Type: text/markdown
|
|
15
|
+
|
|
16
|
+
# continuo-trino-adapter
|
|
17
|
+
|
|
18
|
+
Trino (Iceberg connector) engine-adapter library for Continuo python nodes.
|
|
19
|
+
`TrinoAdapter` implements `continuo_engine_contract.port.WarehouseAdapter`
|
|
20
|
+
(from the `continuo-engine-contract` package) — one class covering both the
|
|
21
|
+
data plane (`fetch` / `ensure_table` / `load`) and release-time validation
|
|
22
|
+
(`ensure_schema` / `drop_schema` / `build_empty_from_sql` /
|
|
23
|
+
`build_empty_from_columns` / `clone_empty_from_prod` / `check_binds`) — and
|
|
24
|
+
registers itself under entry-point group `continuo_engine.adapters` as
|
|
25
|
+
`trino`.
|
|
26
|
+
|
|
27
|
+
Connection env: `TRINO_HOST`, `TRINO_CATALOG` (required); `TRINO_PORT` (default
|
|
28
|
+
8080), `TRINO_USER` (default `continuo`), `TRINO_HTTP_SCHEME` (default `http`),
|
|
29
|
+
`TRINO_PASSWORD` (optional; requires `TRINO_HTTP_SCHEME=https`).
|
|
30
|
+
|
|
31
|
+
The `config:` vocabulary is the union of the two vocabularies this adapter
|
|
32
|
+
inherited from the merge: `partitioning`, `sorted_by`, `format`, and
|
|
33
|
+
`format_version`. `format` takes the `PARQUET` / `ORC` / `AVRO` allowlist on
|
|
34
|
+
every path. `location` and `extra_properties` are deliberately absent — see
|
|
35
|
+
the adapter module docstring.
|
|
36
|
+
|
|
37
|
+
## No multi-statement transactions
|
|
38
|
+
|
|
39
|
+
Unlike the postgres runtime adapter, Trino/Iceberg has no multi-statement
|
|
40
|
+
transactions, so `load()` cannot be a single atomic TRUNCATE+INSERT. Two
|
|
41
|
+
atomic-replace primitives were verified live (Trino 483 + Iceberg REST catalog)
|
|
42
|
+
before choosing one — see the full writeup in `continuo_trino_adapter/adapter.py`'s
|
|
43
|
+
module docstring:
|
|
44
|
+
|
|
45
|
+
- `CREATE OR REPLACE TABLE t AS SELECT * FROM stage` is a single Iceberg
|
|
46
|
+
metadata commit but **silently drops NOT NULL constraints** on the replaced
|
|
47
|
+
table (verified live: a NULL insert into a "NOT NULL" column succeeds
|
|
48
|
+
afterwards, and this connector/version does not support re-adding NOT NULL via
|
|
49
|
+
`ALTER TABLE ... ALTER COLUMN ... SET NOT NULL`). Rejected for this reason.
|
|
50
|
+
- `CREATE TABLE stage (LIKE target INCLUDING PROPERTIES)` at a fresh sibling
|
|
51
|
+
Iceberg location + populate + a two-statement `ALTER TABLE ... RENAME TO` swap
|
|
52
|
+
preserves the target's exact columns, NOT NULL constraints, partitioning,
|
|
53
|
+
format, and other connector properties (verified live). **This is what
|
|
54
|
+
`load()` uses.**
|
|
55
|
+
|
|
56
|
+
The swap's exact atomicity guarantee: nothing under the target name is touched
|
|
57
|
+
until every row has been inserted into staging, so a failure before the swap
|
|
58
|
+
begins (e.g. a NOT-NULL-violating row) leaves the prior target contents
|
|
59
|
+
untouched. The two-statement rename swap itself is *not* single-statement
|
|
60
|
+
atomic — there is a brief window between the two renames during which the
|
|
61
|
+
target name refers to neither table. `load()` makes a best-effort recovery
|
|
62
|
+
attempt (rename the private old relation back) if the second rename raises, and
|
|
63
|
+
best-effort drops only a private staging relation created by that load. If
|
|
64
|
+
recovery itself fails, the logged private old relation remains available for
|
|
65
|
+
manual recovery because it contains the original target data.
|
|
66
|
+
|
|
67
|
+
`load()` assumes a single writer per table: Continuo's scheduler runs at most
|
|
68
|
+
one Job per node at a time. Concurrent `load()` calls against the same table are
|
|
69
|
+
unsupported because their target renames would race. Each load uses UUID-named
|
|
70
|
+
private swap relations, so distinct targets cannot collide with each other or
|
|
71
|
+
with legitimate user tables.
|
|
72
|
+
|
|
73
|
+
## NOT NULL is supported here
|
|
74
|
+
|
|
75
|
+
Verified live against Trino 483 + the Iceberg REST catalog: NOT NULL column
|
|
76
|
+
constraints in `CREATE TABLE` **are** supported and enforced by INSERT
|
|
77
|
+
(CONSTRAINT_VIOLATION on a NULL value). `ensure_table()` emits NOT NULL for
|
|
78
|
+
`nullable=False` columns, same as the postgres adapter.
|
|
79
|
+
|
|
80
|
+
## Type-name mapping
|
|
81
|
+
|
|
82
|
+
The contract's SQL-type grammar (copied verbatim from the postgres adapter, same
|
|
83
|
+
regex) admits spellings Trino does not recognize as type names: `TEXT`,
|
|
84
|
+
`DOUBLE PRECISION`, `NUMERIC(p,s)`. These are mapped to Trino's own spellings
|
|
85
|
+
(`VARCHAR`, `DOUBLE`, `DECIMAL(p,s)`) after the grammar guard has already
|
|
86
|
+
rejected anything injection-shaped. Every other grammar token is valid Trino DDL
|
|
87
|
+
unchanged.
|
|
88
|
+
|
|
89
|
+
## Identifier quoting
|
|
90
|
+
|
|
91
|
+
The trino DBAPI has no psycopg2-style `sql.Identifier`. Catalog, schema, table,
|
|
92
|
+
and column names are double-quoted by hand, with every embedded double quote
|
|
93
|
+
doubled. This preserves legal delimited identifiers such as `sales-prod`,
|
|
94
|
+
`order id`, and `has"quote` without allowing names to escape their quotes.
|
|
95
|
+
|
|
96
|
+
Verification tier: unit tests are mock-free pure-logic tests (type-grammar
|
|
97
|
+
validation, type-name mapping, identifier quoting, location derivation, Arrow
|
|
98
|
+
table construction from rows); DDL/swap behavior (schema/table creation,
|
|
99
|
+
load-owned swap relations, property and NOT NULL preservation, failure cleanup)
|
|
100
|
+
is verified against a live Trino 483 + Iceberg REST stack in
|
|
101
|
+
`tests/test_integration_runtime_trino.py`.
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
# continuo-trino-adapter
|
|
2
|
+
|
|
3
|
+
Trino (Iceberg connector) engine-adapter library for Continuo python nodes.
|
|
4
|
+
`TrinoAdapter` implements `continuo_engine_contract.port.WarehouseAdapter`
|
|
5
|
+
(from the `continuo-engine-contract` package) — one class covering both the
|
|
6
|
+
data plane (`fetch` / `ensure_table` / `load`) and release-time validation
|
|
7
|
+
(`ensure_schema` / `drop_schema` / `build_empty_from_sql` /
|
|
8
|
+
`build_empty_from_columns` / `clone_empty_from_prod` / `check_binds`) — and
|
|
9
|
+
registers itself under entry-point group `continuo_engine.adapters` as
|
|
10
|
+
`trino`.
|
|
11
|
+
|
|
12
|
+
Connection env: `TRINO_HOST`, `TRINO_CATALOG` (required); `TRINO_PORT` (default
|
|
13
|
+
8080), `TRINO_USER` (default `continuo`), `TRINO_HTTP_SCHEME` (default `http`),
|
|
14
|
+
`TRINO_PASSWORD` (optional; requires `TRINO_HTTP_SCHEME=https`).
|
|
15
|
+
|
|
16
|
+
The `config:` vocabulary is the union of the two vocabularies this adapter
|
|
17
|
+
inherited from the merge: `partitioning`, `sorted_by`, `format`, and
|
|
18
|
+
`format_version`. `format` takes the `PARQUET` / `ORC` / `AVRO` allowlist on
|
|
19
|
+
every path. `location` and `extra_properties` are deliberately absent — see
|
|
20
|
+
the adapter module docstring.
|
|
21
|
+
|
|
22
|
+
## No multi-statement transactions
|
|
23
|
+
|
|
24
|
+
Unlike the postgres runtime adapter, Trino/Iceberg has no multi-statement
|
|
25
|
+
transactions, so `load()` cannot be a single atomic TRUNCATE+INSERT. Two
|
|
26
|
+
atomic-replace primitives were verified live (Trino 483 + Iceberg REST catalog)
|
|
27
|
+
before choosing one — see the full writeup in `continuo_trino_adapter/adapter.py`'s
|
|
28
|
+
module docstring:
|
|
29
|
+
|
|
30
|
+
- `CREATE OR REPLACE TABLE t AS SELECT * FROM stage` is a single Iceberg
|
|
31
|
+
metadata commit but **silently drops NOT NULL constraints** on the replaced
|
|
32
|
+
table (verified live: a NULL insert into a "NOT NULL" column succeeds
|
|
33
|
+
afterwards, and this connector/version does not support re-adding NOT NULL via
|
|
34
|
+
`ALTER TABLE ... ALTER COLUMN ... SET NOT NULL`). Rejected for this reason.
|
|
35
|
+
- `CREATE TABLE stage (LIKE target INCLUDING PROPERTIES)` at a fresh sibling
|
|
36
|
+
Iceberg location + populate + a two-statement `ALTER TABLE ... RENAME TO` swap
|
|
37
|
+
preserves the target's exact columns, NOT NULL constraints, partitioning,
|
|
38
|
+
format, and other connector properties (verified live). **This is what
|
|
39
|
+
`load()` uses.**
|
|
40
|
+
|
|
41
|
+
The swap's exact atomicity guarantee: nothing under the target name is touched
|
|
42
|
+
until every row has been inserted into staging, so a failure before the swap
|
|
43
|
+
begins (e.g. a NOT-NULL-violating row) leaves the prior target contents
|
|
44
|
+
untouched. The two-statement rename swap itself is *not* single-statement
|
|
45
|
+
atomic — there is a brief window between the two renames during which the
|
|
46
|
+
target name refers to neither table. `load()` makes a best-effort recovery
|
|
47
|
+
attempt (rename the private old relation back) if the second rename raises, and
|
|
48
|
+
best-effort drops only a private staging relation created by that load. If
|
|
49
|
+
recovery itself fails, the logged private old relation remains available for
|
|
50
|
+
manual recovery because it contains the original target data.
|
|
51
|
+
|
|
52
|
+
`load()` assumes a single writer per table: Continuo's scheduler runs at most
|
|
53
|
+
one Job per node at a time. Concurrent `load()` calls against the same table are
|
|
54
|
+
unsupported because their target renames would race. Each load uses UUID-named
|
|
55
|
+
private swap relations, so distinct targets cannot collide with each other or
|
|
56
|
+
with legitimate user tables.
|
|
57
|
+
|
|
58
|
+
## NOT NULL is supported here
|
|
59
|
+
|
|
60
|
+
Verified live against Trino 483 + the Iceberg REST catalog: NOT NULL column
|
|
61
|
+
constraints in `CREATE TABLE` **are** supported and enforced by INSERT
|
|
62
|
+
(CONSTRAINT_VIOLATION on a NULL value). `ensure_table()` emits NOT NULL for
|
|
63
|
+
`nullable=False` columns, same as the postgres adapter.
|
|
64
|
+
|
|
65
|
+
## Type-name mapping
|
|
66
|
+
|
|
67
|
+
The contract's SQL-type grammar (copied verbatim from the postgres adapter, same
|
|
68
|
+
regex) admits spellings Trino does not recognize as type names: `TEXT`,
|
|
69
|
+
`DOUBLE PRECISION`, `NUMERIC(p,s)`. These are mapped to Trino's own spellings
|
|
70
|
+
(`VARCHAR`, `DOUBLE`, `DECIMAL(p,s)`) after the grammar guard has already
|
|
71
|
+
rejected anything injection-shaped. Every other grammar token is valid Trino DDL
|
|
72
|
+
unchanged.
|
|
73
|
+
|
|
74
|
+
## Identifier quoting
|
|
75
|
+
|
|
76
|
+
The trino DBAPI has no psycopg2-style `sql.Identifier`. Catalog, schema, table,
|
|
77
|
+
and column names are double-quoted by hand, with every embedded double quote
|
|
78
|
+
doubled. This preserves legal delimited identifiers such as `sales-prod`,
|
|
79
|
+
`order id`, and `has"quote` without allowing names to escape their quotes.
|
|
80
|
+
|
|
81
|
+
Verification tier: unit tests are mock-free pure-logic tests (type-grammar
|
|
82
|
+
validation, type-name mapping, identifier quoting, location derivation, Arrow
|
|
83
|
+
table construction from rows); DDL/swap behavior (schema/table creation,
|
|
84
|
+
load-owned swap relations, property and NOT NULL preservation, failure cleanup)
|
|
85
|
+
is verified against a live Trino 483 + Iceberg REST stack in
|
|
86
|
+
`tests/test_integration_runtime_trino.py`.
|
|
File without changes
|