spark-kindling-ext-databricks 0.14.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- spark_kindling_ext_databricks-0.14.0/PKG-INFO +489 -0
- spark_kindling_ext_databricks-0.14.0/README.md +466 -0
- spark_kindling_ext_databricks-0.14.0/kindling_ext_databricks/__init__.py +52 -0
- spark_kindling_ext_databricks-0.14.0/kindling_ext_databricks/auto_cdc.py +175 -0
- spark_kindling_ext_databricks-0.14.0/kindling_ext_databricks/engine.py +425 -0
- spark_kindling_ext_databricks-0.14.0/kindling_ext_databricks/engine_extension.py +37 -0
- spark_kindling_ext_databricks-0.14.0/kindling_ext_databricks/lakeflow_app_selector.py +441 -0
- spark_kindling_ext_databricks-0.14.0/kindling_ext_databricks/temporal_lowering.py +697 -0
- spark_kindling_ext_databricks-0.14.0/pyproject.toml +44 -0
- spark_kindling_ext_databricks-0.14.0/pyproject.toml.orig +42 -0
|
@@ -0,0 +1,489 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: spark-kindling-ext-databricks
|
|
3
|
+
Version: 0.14.0
|
|
4
|
+
Summary: Databricks Lakeflow adapter for Kindling's SDP declaration engine
|
|
5
|
+
Author: SEP Engineering
|
|
6
|
+
Author-email: SEP Engineering <engineering@sep.com>
|
|
7
|
+
License-Expression: MIT
|
|
8
|
+
Classifier: Development Status :: 4 - Beta
|
|
9
|
+
Classifier: Intended Audience :: Developers
|
|
10
|
+
Classifier: Operating System :: OS Independent
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
13
|
+
Classifier: Topic :: Database
|
|
14
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
15
|
+
Requires-Dist: spark-kindling>=0.14.0,<0.15
|
|
16
|
+
Requires-Dist: spark-kindling-ext-sdp>=0.14.0,<0.15
|
|
17
|
+
Requires-Python: >=3.10
|
|
18
|
+
Project-URL: Homepage, https://github.com/sep/spark-kindling-framework
|
|
19
|
+
Project-URL: Documentation, https://github.com/sep/spark-kindling-framework/tree/main/docs
|
|
20
|
+
Project-URL: Changelog, https://github.com/sep/spark-kindling-framework/blob/main/CHANGELOG.md
|
|
21
|
+
Project-URL: Issues, https://github.com/sep/spark-kindling-framework/issues
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
|
|
24
|
+
# kindling_ext_databricks
|
|
25
|
+
|
|
26
|
+
Databricks Lakeflow adapter for Kindling's SDP declaration engine.
|
|
27
|
+
|
|
28
|
+
Lakeflow "extends and is interoperable with" the OSS `pyspark.pipelines`
|
|
29
|
+
API, so this adapter **augments** the `kindling_ext_sdp` core rather than
|
|
30
|
+
translating it: the OSS emission (materialized views with comments, table
|
|
31
|
+
properties, partitioning, clustering, schema) runs unchanged, and
|
|
32
|
+
Databricks-only capabilities are layered on top.
|
|
33
|
+
|
|
34
|
+
Install it from PyPI with `pip install spark-kindling-ext-databricks`; in a uv project,
|
|
35
|
+
`kindling env add spark-kindling-ext-databricks` pins the version that matches the
|
|
36
|
+
project's Kindling release.
|
|
37
|
+
|
|
38
|
+
Selected with:
|
|
39
|
+
|
|
40
|
+
```python
|
|
41
|
+
kindling.initialize(engine="databricks_sdp")
|
|
42
|
+
...
|
|
43
|
+
kindling.declare_pipeline()
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
— resolved through kindling core's engine-extension mapping
|
|
47
|
+
(`kindling_<name>.engine_extension()`); adding this engine required zero
|
|
48
|
+
core changes.
|
|
49
|
+
|
|
50
|
+
## Phase 3 (this package today): expectations
|
|
51
|
+
|
|
52
|
+
```yaml
|
|
53
|
+
datapipes:
|
|
54
|
+
silver.orders:
|
|
55
|
+
engine:
|
|
56
|
+
databricks_sdp:
|
|
57
|
+
expectations: # violations counted, rows kept (warn)
|
|
58
|
+
valid_order_id: "order_id IS NOT NULL"
|
|
59
|
+
expectations_drop: # violating rows dropped
|
|
60
|
+
positive_qty: "quantity > 0"
|
|
61
|
+
expectations_fail: # violation fails the update
|
|
62
|
+
no_future_dates: "order_date <= current_date()"
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Blocks map to `expect_all` / `expect_all_or_drop` / `expect_all_or_fail`
|
|
66
|
+
decorators. Declaring them while targeting OSS fails fast at validation
|
|
67
|
+
(capability gating in the core); declaring them here against a runtime
|
|
68
|
+
without the Lakeflow decorators fails at declaration with an actionable
|
|
69
|
+
error.
|
|
70
|
+
|
|
71
|
+
`refresh_policy: incremental` is accepted (adapter-tier gated) but emits
|
|
72
|
+
nothing yet — incremental MV refresh on Databricks (Enzyme) is engine
|
|
73
|
+
behavior rather than a declaration keyword; treated as a documented hint
|
|
74
|
+
pending verification against a live workspace.
|
|
75
|
+
|
|
76
|
+
## Phase 5: SCD declared flows → AUTO CDC
|
|
77
|
+
|
|
78
|
+
An SCD-tagged output entity (`scd.type` plus the declared-flow tags from
|
|
79
|
+
kindling core) is declared as the Lakeflow CDC pattern — a pipeline-scoped
|
|
80
|
+
source view (the pipe's DataFrame), a streaming table target, and an AUTO
|
|
81
|
+
CDC flow — instead of a materialized view:
|
|
82
|
+
|
|
83
|
+
| Kindling declaration | AUTO CDC |
|
|
84
|
+
|---|---|
|
|
85
|
+
| `scd.type: "2"` (the only type core registration accepts) | `stored_as_scd_type` |
|
|
86
|
+
| entity `merge_columns` | `keys` |
|
|
87
|
+
| `scd.sequence_by` | `sequence_by` (change feed) |
|
|
88
|
+
| `scd.source_kind: change_feed` (default) | `create_auto_cdc_flow` |
|
|
89
|
+
| `scd.source_kind: snapshot` / `scd.close_on_missing: true` | `create_auto_cdc_from_snapshot_flow` |
|
|
90
|
+
| `scd.delete_when` | `apply_as_deletes=expr(...)` |
|
|
91
|
+
| `scd.tracked` | `track_history_column_list` |
|
|
92
|
+
| `scd.untracked` | `track_history_except_column_list=[*untracked, sequence_by]` |
|
|
93
|
+
| (default) sequence column ≠ content | `track_history_except_column_list=[sequence_by]` |
|
|
94
|
+
|
|
95
|
+
The tracking list goes to both the change-feed and the snapshot flow, and
|
|
96
|
+
only one list is ever passed. `sequence_by` is added to the exclude list
|
|
97
|
+
only when declared. Columns outside the tracked set are updated in place on
|
|
98
|
+
the current version, which is also what the runner does.
|
|
99
|
+
|
|
100
|
+
Fail-fast mapping requirements (validated with everything else, all
|
|
101
|
+
errors at once): change-feed targets need `scd.sequence_by`
|
|
102
|
+
(`scd_sequence_by_required`); `scd.type` must be `2`, matching core registration
|
|
103
|
+
(`scd_type_unsupported` — bitemporal is deliberately excluded while in
|
|
104
|
+
Beta); keys are required (`scd_keys_required`); `scd.tracked` and
|
|
105
|
+
`scd.untracked` can't both be set (`scd_tracked_untracked_exclusive`).
|
|
106
|
+
|
|
107
|
+
Documented divergences from the runner engine (dual-engine parity
|
|
108
|
+
criterion): history columns are `__START_AT`/`__END_AT` (the entity's
|
|
109
|
+
runner-shape schema is not passed to the streaming table); snapshot
|
|
110
|
+
ordering is ingestion order, not `scd.sequence_by`; out-of-order
|
|
111
|
+
change-feed rows are reconciled into history rather than ignored under
|
|
112
|
+
the runner's strictly-later rule; the `scd.current_entity_id`
|
|
113
|
+
current-view companion stays runner-only (an ordinary declared view is
|
|
114
|
+
the SDP-native replacement, tracked separately); `scd.untracked` on a
|
|
115
|
+
keyed SCD1 merge (no `scd.type`) is runner-only, because keyed SCD1 is not
|
|
116
|
+
lowered to AUTO CDC. Expectations, when
|
|
117
|
+
configured, attach to the source view — quality is checked on the
|
|
118
|
+
incoming feed.
|
|
119
|
+
|
|
120
|
+
Change-feed sources are consumed as a **stream**: every input selected by
|
|
121
|
+
`driving_entity_ids` is read with `spark.readStream.table()` so the AUTO CDC
|
|
122
|
+
flow processes it incrementally — the declarative replacement for
|
|
123
|
+
hand-rolled foreachBatch+MERGE. Remaining inputs stay batch reads
|
|
124
|
+
(stream-static joins). Snapshot sources keep batch reads: the snapshot API
|
|
125
|
+
diffs whole snapshots per update.
|
|
126
|
+
|
|
127
|
+
## Provider-owned streaming sources
|
|
128
|
+
|
|
129
|
+
An external provider stream, initially Event Hubs over Kafka, can drive a
|
|
130
|
+
normal Kindling ingestion pipe when targeting `engine="databricks_sdp"`:
|
|
131
|
+
|
|
132
|
+
```python
|
|
133
|
+
DataEntities.entity(
|
|
134
|
+
entityid="stream.telemetry",
|
|
135
|
+
name="Telemetry Event Hub",
|
|
136
|
+
tags={
|
|
137
|
+
"provider_type": "eventhub",
|
|
138
|
+
"provider.eventhub.connectionString": "@secret:lakeflow:eh-conn",
|
|
139
|
+
"provider.eventhub.name": "telemetry",
|
|
140
|
+
"provider.transport": "kafka",
|
|
141
|
+
"provider.kafka.includeHeaders": "true",
|
|
142
|
+
"provider.preprocess": "kafka",
|
|
143
|
+
},
|
|
144
|
+
schema=None,
|
|
145
|
+
)
|
|
146
|
+
|
|
147
|
+
@DataPipes.pipe(
|
|
148
|
+
pipeid="bronze.telemetry.ingest",
|
|
149
|
+
input_entity_ids=["stream.telemetry", "ref.devices"],
|
|
150
|
+
output_entity_id="bronze.telemetry",
|
|
151
|
+
driving_entity_ids=["stream.telemetry"],
|
|
152
|
+
)
|
|
153
|
+
def ingest_telemetry(stream_telemetry, ref_devices):
|
|
154
|
+
...
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
The shared declaration planner asks the provider for a secret-safe
|
|
158
|
+
`StreamingSourceSpec`. Invalid source configuration is reported before
|
|
159
|
+
emission without printing connection strings, SAS keys, JAAS values, or
|
|
160
|
+
resolved `@secret` values. The Databricks adapter then emits exactly one
|
|
161
|
+
`dp.create_streaming_table(...)` target and one `dp.append_flow(...)`. When
|
|
162
|
+
Lakeflow evaluates the append-flow function, Kindling resolves the provider,
|
|
163
|
+
calls `read_entity_as_stream()` with the declarative-source option, passes
|
|
164
|
+
later inputs as static DataFrames, invokes the registered pipe transform, and
|
|
165
|
+
returns the streaming DataFrame to Lakeflow.
|
|
166
|
+
|
|
167
|
+
The output entity's declared `schema` is not forwarded to
|
|
168
|
+
`create_streaming_table` pending Lakeflow platform evidence; the target schema
|
|
169
|
+
is inferred from the append-flow DataFrame on this path.
|
|
170
|
+
|
|
171
|
+
Lakeflow owns query startup, checkpoint placement, retries, update scheduling,
|
|
172
|
+
and target persistence for this path. Kindling does not call `writeStream`,
|
|
173
|
+
does not pass `checkpointLocation`, and does not start a streaming query.
|
|
174
|
+
|
|
175
|
+
## Incremental reads of external Delta inputs
|
|
176
|
+
|
|
177
|
+
A normal pipe whose driving input is an ordinary Delta table — one produced
|
|
178
|
+
by a separate ingestion pipeline, not by a streaming provider — can be read
|
|
179
|
+
incrementally by opting in per pipe:
|
|
180
|
+
|
|
181
|
+
```yaml
|
|
182
|
+
datapipes:
|
|
183
|
+
silver.device_telemetry:
|
|
184
|
+
engine:
|
|
185
|
+
databricks_sdp:
|
|
186
|
+
streaming_inputs: ["bronze.device_telemetry"]
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
The output is then declared as a streaming table plus one append flow, the
|
|
190
|
+
named input is read with `spark.readStream.table(...)`, and every other
|
|
191
|
+
input stays a batch read — so reference-table joins remain stream-static.
|
|
192
|
+
Everything else on this path (expectations, table properties, partitioning,
|
|
193
|
+
clustering, the deferred `schema` forwarding) behaves exactly as it does for
|
|
194
|
+
a provider-owned streaming source; both shapes reach the same emission.
|
|
195
|
+
|
|
196
|
+
**The opt-in is never inferred.** An incremental read changes what a pipe
|
|
197
|
+
means — append-only, with no reprocessing of revised rows — so eligibility
|
|
198
|
+
is the app's decision, not the adapter's. Nothing changes for a Delta input
|
|
199
|
+
that is not named here.
|
|
200
|
+
|
|
201
|
+
It is adapter-tier: `engine="sdp"` reports `capability_not_supported`.
|
|
202
|
+
Rejected shapes, all at declaration time:
|
|
203
|
+
|
|
204
|
+
| Code | Shape |
|
|
205
|
+
| --- | --- |
|
|
206
|
+
| `streaming_input_not_an_input` | names an entity the pipe does not read |
|
|
207
|
+
| `streaming_input_not_driving` | names a non-driving input |
|
|
208
|
+
| `streaming_inputs_partial` | does not name every driving input |
|
|
209
|
+
| `multiple_streaming_inputs` | names more than one input |
|
|
210
|
+
| `streaming_input_not_delta` | names a non-Delta entity |
|
|
211
|
+
| `streaming_input_internal` | names a dataset produced in this pipeline |
|
|
212
|
+
| `streaming_input_provider_owned` | names a provider-owned stream (already streamed) |
|
|
213
|
+
| `streaming_dataset_type_conflict` | output explicitly requests a materialized view |
|
|
214
|
+
|
|
215
|
+
`streaming_inputs_partial` exists because the append flow streams whatever
|
|
216
|
+
`driving_entity_ids` selects: a partial opt-in would silently stream an
|
|
217
|
+
input the app never opted in, so the sets must match.
|
|
218
|
+
|
|
219
|
+
## Temporal chain execution mode
|
|
220
|
+
|
|
221
|
+
A temporal chain lowers its event strata `<events>__g0..gK` as streaming
|
|
222
|
+
tables fed by append flows, so each base-event transform runs over a
|
|
223
|
+
Structured Streaming DataFrame. That rejects ordered analytic windows —
|
|
224
|
+
`row_number`, `lag`, unbounded forward fill — even though the same transform
|
|
225
|
+
runs correctly under the runner engine. Set the mode to `batch` to lower
|
|
226
|
+
those strata as materialized views with batch reads instead:
|
|
227
|
+
|
|
228
|
+
```yaml
|
|
229
|
+
kindling:
|
|
230
|
+
lakeflow:
|
|
231
|
+
temporal_mode: batch
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
From a Databricks Asset Bundle, use the canonical SparkConf spelling in the
|
|
235
|
+
pipeline's `configuration:` mapping — it is point-looked-up by the selector,
|
|
236
|
+
so it also works on serverless and shared-access runtimes that cannot
|
|
237
|
+
enumerate Spark configuration:
|
|
238
|
+
|
|
239
|
+
```yaml
|
|
240
|
+
configuration:
|
|
241
|
+
spark.kindling.lakeflow.temporal_mode: "batch"
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
| Component | `streaming` (default, omitted) | `batch` |
|
|
245
|
+
| --- | --- | --- |
|
|
246
|
+
| `<events>__g0` | Streaming table, one append flow per base declaration | One materialized view, union of the transformed base inputs |
|
|
247
|
+
| Base reads | `spark.readStream.table(...)` | `spark.table(...)` |
|
|
248
|
+
| `<events>__g1..gK` | Streaming tables and append flows | Materialized view per generation, batch reads of lower strata |
|
|
249
|
+
| Empty generation | Empty streaming projection of `__g0` | Empty batch projection of `__g0`, same schema and dependency |
|
|
250
|
+
| Episode snapshot, episodes, determinations, higher-order strata, public events | unchanged | unchanged |
|
|
251
|
+
|
|
252
|
+
### Unpersisted generations
|
|
253
|
+
|
|
254
|
+
In `batch` mode the numbered strata can be declared as pipeline-scoped
|
|
255
|
+
temporary views instead of materialized views, so no `__g*` tables are
|
|
256
|
+
created and nothing intermediate is written:
|
|
257
|
+
|
|
258
|
+
```yaml
|
|
259
|
+
kindling:
|
|
260
|
+
lakeflow:
|
|
261
|
+
temporal_mode: batch
|
|
262
|
+
temporal_strata_materialization: view # default: table
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
Only the numbered strata change. The determinations view, higher-order
|
|
266
|
+
boundaries, the episodes Auto CDC target and the public `events` union are
|
|
267
|
+
declared identically, so query results are unchanged.
|
|
268
|
+
|
|
269
|
+
The cost is recomputation. Generation `k` reads every stratum below it, so a
|
|
270
|
+
temporary view is re-expanded once per reference rather than read back from
|
|
271
|
+
storage: the plan behind `events` grows exponentially in
|
|
272
|
+
`kindling.temporal.max_generations`, on the order of `2^K` rescans of the
|
|
273
|
+
base stratum at ceiling `K`. That is fine at a ceiling of 1–3 and a serious
|
|
274
|
+
problem at the default of 10. You also give up per-generation observability —
|
|
275
|
+
there is no `__g*` table to query when working out which generation produced
|
|
276
|
+
an event, and no per-stratum metrics in the event log.
|
|
277
|
+
|
|
278
|
+
`view` requires `temporal_mode: batch`. A streaming stratum is an append-flow
|
|
279
|
+
target and a temporary view cannot be one, so the combination fails the
|
|
280
|
+
declaration instead of quietly downgrading to batch reads.
|
|
281
|
+
|
|
282
|
+
### How many strata get declared
|
|
283
|
+
|
|
284
|
+
`kindling.temporal.max_generations` is a ceiling, not a target. A chain whose
|
|
285
|
+
conditions are all registry-declared (`DataConditions.register`) declares
|
|
286
|
+
exactly as many numbered strata as those rules reach — a single-generation
|
|
287
|
+
app gets `__g0` and `__g1`, nothing else — because both emission and
|
|
288
|
+
`validate()`'s name reservation can compute that depth in-process from the
|
|
289
|
+
condition registry. The ceiling still applies: a rule set reaching past it
|
|
290
|
+
fails the declaration rather than being truncated.
|
|
291
|
+
|
|
292
|
+
A chain with any table-sourced condition engine keeps the ceiling as its
|
|
293
|
+
topology, declaring `__g0..__gK` whether or not each generation holds rules.
|
|
294
|
+
Rule rows are data that changes between updates, and the reservation path
|
|
295
|
+
runs with no Spark session to read their depth with, so a fixed shape is the
|
|
296
|
+
only way emission and reservation can agree. Empty strata there are real
|
|
297
|
+
datasets with a flow that filters everything out, so set the ceiling to the
|
|
298
|
+
depth you actually use.
|
|
299
|
+
|
|
300
|
+
Shrinking the depth of a registry chain leaves the highest `__g*` dataset
|
|
301
|
+
behind, no longer updated. That is a deploy-time event; drop the stale table
|
|
302
|
+
once.
|
|
303
|
+
|
|
304
|
+
Multi-source fan-in is unchanged: every base declaration lands in the one
|
|
305
|
+
stratum-0 dataset, each input keeping its own transform. Base sources still
|
|
306
|
+
resolve to external physical names through `EntityNameMapper` in both modes,
|
|
307
|
+
so a producer selected in the same pipeline establishes no local dependency
|
|
308
|
+
edge — keep those sources in an upstream resource. A batch chain declares no
|
|
309
|
+
`readStream` and no append flow at all; a chain *with* episodes still calls
|
|
310
|
+
`create_streaming_table` for the snapshot-CDC target only, because that is
|
|
311
|
+
what the AUTO CDC FROM SNAPSHOT API requires.
|
|
312
|
+
|
|
313
|
+
Semantics to plan for:
|
|
314
|
+
|
|
315
|
+
- Batch strata carry batch-query semantics over the rows available at each
|
|
316
|
+
refresh. A refresh can revise or remove previously produced events, and
|
|
317
|
+
late arrivals can change window results across a whole subject partition.
|
|
318
|
+
They are not append-only event archives and they do not use runner
|
|
319
|
+
watermarks.
|
|
320
|
+
- Retain the source history the computation needs. Removing input rows can
|
|
321
|
+
remove derived events and change the episode snapshot; snapshot CDC keeps
|
|
322
|
+
its SCD2 version history but cannot reconstruct source history that was
|
|
323
|
+
never retained.
|
|
324
|
+
- Use stable event identity and deterministic window ordering, including a
|
|
325
|
+
tie breaker for equal timestamps. Never use a mutable row rank as event
|
|
326
|
+
identity.
|
|
327
|
+
- Changing the mode changes `__g0..gK` dataset types. It is not a hot toggle
|
|
328
|
+
with portable checkpoint state: select the mode for newly provisioned
|
|
329
|
+
pipeline outputs, and treat conversion of an existing pipeline as a
|
|
330
|
+
deployment operation. Nothing is dropped or reset automatically.
|
|
331
|
+
- The value ignores surrounding whitespace and case. Anything other than
|
|
332
|
+
`streaming` or `batch` fails the declaration before any Lakeflow object is
|
|
333
|
+
created. The setting is inert when no temporal chain-events pipe is
|
|
334
|
+
selected, and never changes general SDP pipe execution, Lakeflow triggered
|
|
335
|
+
versus continuous scheduling, or the runner engine's watermark behavior.
|
|
336
|
+
|
|
337
|
+
Batch mode enables batch analysis; it does not guarantee that arbitrary user
|
|
338
|
+
code is declarable, or that a query refreshes incrementally.
|
|
339
|
+
|
|
340
|
+
|
|
341
|
+
## Deferred
|
|
342
|
+
|
|
343
|
+
- Multiple provider-owned streaming inputs and stream-stream joins.
|
|
344
|
+
- Bitemporal AUTO CDC (`stored_as_scd_type="bitemporal"`) — Beta;
|
|
345
|
+
tracked, not adopted (proposal decision).
|
|
346
|
+
- Current-view companion as a declared view.
|
|
347
|
+
|
|
348
|
+
|
|
349
|
+
## Output dataset naming
|
|
350
|
+
|
|
351
|
+
Lakeflow uses the shared [SDP dataset naming configuration](https://github.com/sep/spark-kindling-framework/blob/main/packages/extensions/kindling_ext_sdp/README.md#output-dataset-naming):
|
|
352
|
+
|
|
353
|
+
```yaml
|
|
354
|
+
kindling:
|
|
355
|
+
sdp:
|
|
356
|
+
dataset_naming: leaf
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
With a pipeline destination of catalog `dev_silver`, schema `cwmdp`,
|
|
360
|
+
`silver.device_telemetry` is declared as `device_telemetry`, producing
|
|
361
|
+
`dev_silver.cwmdp.device_telemetry`. Set separate destinations and select
|
|
362
|
+
the appropriate pipes for each medallion resource. Same-leaf outputs in
|
|
363
|
+
one plan are rejected.
|
|
364
|
+
|
|
365
|
+
This also controls AUTO CDC targets and temporary sources, temporal
|
|
366
|
+
events/episodes, and their generated strata, snapshots, and internal
|
|
367
|
+
references. Cross-pipeline external table resolution remains independent,
|
|
368
|
+
including temporal reads through `EntityNameMapper` and entity
|
|
369
|
+
`provider.table_catalog` overrides. Omitting the setting preserves the
|
|
370
|
+
existing `silver_device_telemetry` convention.
|
|
371
|
+
|
|
372
|
+
|
|
373
|
+
When consuming these outputs, explicitly align external entity metadata with
|
|
374
|
+
the leaf table, for example
|
|
375
|
+
`provider.table_name: dev_silver.cwmdp.device_telemetry`. Otherwise a consumer
|
|
376
|
+
configured with catalog and schema still resolves the historical flattened
|
|
377
|
+
name. See [external reads and generated-name reservations](https://github.com/sep/spark-kindling-framework/blob/main/packages/extensions/kindling_ext_sdp/README.md#reading-leaf-named-outputs-from-elsewhere)
|
|
378
|
+
for the complete example and the temporal source limitation.
|
|
379
|
+
|
|
380
|
+
## Canonical configuration from a Bundle
|
|
381
|
+
|
|
382
|
+
Lakeflow uses the same Kindling configuration hierarchy as jobs and notebooks.
|
|
383
|
+
Publish ordinary Kindling settings files and pass them through the canonical
|
|
384
|
+
bootstrap key `spark.kindling.bootstrap.config_files`, or publish them through
|
|
385
|
+
`spark.kindling.bootstrap.artifacts_storage_path` and let bootstrap discover
|
|
386
|
+
base, platform, workspace, environment, and app overlays.
|
|
387
|
+
|
|
388
|
+
The selector does not parse YAML, inspect files, or validate structured
|
|
389
|
+
sections. It bridges Lakeflow pipeline configuration into
|
|
390
|
+
`kindling.initialize(..., app_name=<selected>, engine="databricks_sdp")` and
|
|
391
|
+
sets `declaration_only`; the shared bootstrap/Dynaconf path owns transport,
|
|
392
|
+
parsing, validation, precedence, and overlays. Flat bridged pipeline keys still
|
|
393
|
+
act as bootstrap overrides and win over settings files.
|
|
394
|
+
|
|
395
|
+
Use Bundle sync to deploy config to a stable workspace or volume path. This
|
|
396
|
+
matches the transport pattern in the
|
|
397
|
+
[DAB config promotion guide](https://github.com/sep/spark-kindling-framework/blob/main/docs/guide/dab_config_promotion.md):
|
|
398
|
+
|
|
399
|
+
```yaml
|
|
400
|
+
# databricks.yml
|
|
401
|
+
bundle:
|
|
402
|
+
name: telemetry-lakeflow
|
|
403
|
+
|
|
404
|
+
targets:
|
|
405
|
+
dev:
|
|
406
|
+
workspace:
|
|
407
|
+
host: https://adb-<dev>.azuredatabricks.net
|
|
408
|
+
file_path: /Workspace/Shared/kindling/dev
|
|
409
|
+
|
|
410
|
+
sync:
|
|
411
|
+
include:
|
|
412
|
+
- config/**
|
|
413
|
+
- data-apps/**
|
|
414
|
+
|
|
415
|
+
resources:
|
|
416
|
+
pipelines:
|
|
417
|
+
telemetry_bronze:
|
|
418
|
+
name: telemetry-bronze
|
|
419
|
+
catalog: dev_bronze
|
|
420
|
+
target: cwmdp
|
|
421
|
+
configuration:
|
|
422
|
+
"kindling.data_app": telemetry
|
|
423
|
+
"kindling.lakeflow.pipes": bronze.ingest_telemetry
|
|
424
|
+
"spark.kindling.bootstrap.environment": dev
|
|
425
|
+
"spark.kindling.bootstrap.workspace_id": adb-dev
|
|
426
|
+
"spark.kindling.bootstrap.config_files": '["/Workspace/Shared/kindling/dev/config/settings.yaml", "/Workspace/Shared/kindling/dev/config/settings.databricks.yaml", "/Workspace/Shared/kindling/dev/data-apps/telemetry/settings.yaml"]'
|
|
427
|
+
telemetry_silver:
|
|
428
|
+
name: telemetry-silver
|
|
429
|
+
catalog: dev_silver
|
|
430
|
+
target: cwmdp
|
|
431
|
+
configuration:
|
|
432
|
+
"kindling.data_app": telemetry
|
|
433
|
+
"kindling.lakeflow.pipes": silver.build_telemetry,silver.derive_events,silver.derive_episodes
|
|
434
|
+
"spark.kindling.bootstrap.environment": dev
|
|
435
|
+
"spark.kindling.bootstrap.workspace_id": adb-dev
|
|
436
|
+
"spark.kindling.bootstrap.config_files": '["/Workspace/Shared/kindling/dev/config/settings.yaml", "/Workspace/Shared/kindling/dev/config/settings.databricks.yaml", "/Workspace/Shared/kindling/dev/data-apps/telemetry/settings.yaml"]'
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
```yaml
|
|
440
|
+
# data-apps/telemetry/settings.yaml
|
|
441
|
+
kindling:
|
|
442
|
+
sdp:
|
|
443
|
+
dataset_naming: leaf
|
|
444
|
+
|
|
445
|
+
dataentities:
|
|
446
|
+
bronze.device_telemetry:
|
|
447
|
+
tags:
|
|
448
|
+
provider.table_name: dev_bronze.cwmdp.device_telemetry
|
|
449
|
+
silver.device_telemetry:
|
|
450
|
+
tags:
|
|
451
|
+
provider.table_name: dev_silver.cwmdp.device_telemetry
|
|
452
|
+
silver.events:
|
|
453
|
+
tags:
|
|
454
|
+
provider.table_name: dev_silver.cwmdp.events
|
|
455
|
+
silver.episodes:
|
|
456
|
+
tags:
|
|
457
|
+
provider.table_name: dev_silver.cwmdp.episodes
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
With `dataset_naming: leaf`, dotted logical IDs stay unchanged in the Kindling
|
|
461
|
+
registry while Lakeflow emits single-part output names inside each pipeline
|
|
462
|
+
destination:
|
|
463
|
+
|
|
464
|
+
| Logical ID | Pipeline destination | Physical table |
|
|
465
|
+
|---|---|---|
|
|
466
|
+
| `bronze.device_telemetry` | `dev_bronze.cwmdp` | `dev_bronze.cwmdp.device_telemetry` |
|
|
467
|
+
| `silver.device_telemetry` | `dev_silver.cwmdp` | `dev_silver.cwmdp.device_telemetry` |
|
|
468
|
+
| `silver.events` | `dev_silver.cwmdp` | `dev_silver.cwmdp.events` |
|
|
469
|
+
| `silver.episodes` | `dev_silver.cwmdp` | `dev_silver.cwmdp.episodes` |
|
|
470
|
+
|
|
471
|
+
Pipeline-local outputs get those names from `dataset_naming: leaf` plus the
|
|
472
|
+
pipeline catalog/schema. External reads of the same tables still use the
|
|
473
|
+
ordinary entity resolver, so they need the fully qualified
|
|
474
|
+
`provider.table_name` overrides shown in the structured YAML. A catalog/schema
|
|
475
|
+
override alone is not enough: it would still resolve the default flattened
|
|
476
|
+
table name. Temporal chain base-event sources always resolve externally, so
|
|
477
|
+
keep their external table overrides in the same structured YAML.
|
|
478
|
+
|
|
479
|
+
Workspace files under `/Workspace/...` are documented as driver-readable for
|
|
480
|
+
clusters and jobs. Readability from a Lakeflow serverless pipeline has not yet
|
|
481
|
+
been confirmed; if that is required, deploy the same YAML to a Unity Catalog
|
|
482
|
+
volume path and reference that path in
|
|
483
|
+
`spark.kindling.bootstrap.config_files`.
|
|
484
|
+
|
|
485
|
+
`kindling.lakeflow.config_files`, the comma-separated alias that 0.12.35
|
|
486
|
+
deprecated, was removed in 0.12.47 and is now ignored. Bundle configuration
|
|
487
|
+
that still sets it loads no files from it: move the paths to
|
|
488
|
+
`spark.kindling.bootstrap.config_files` as a JSON array string (or a bare
|
|
489
|
+
string for one file).
|