spark-kindling-ext-databricks 0.14.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,489 @@
1
+ Metadata-Version: 2.4
2
+ Name: spark-kindling-ext-databricks
3
+ Version: 0.14.0
4
+ Summary: Databricks Lakeflow adapter for Kindling's SDP declaration engine
5
+ Author: SEP Engineering
6
+ Author-email: SEP Engineering <engineering@sep.com>
7
+ License-Expression: MIT
8
+ Classifier: Development Status :: 4 - Beta
9
+ Classifier: Intended Audience :: Developers
10
+ Classifier: Operating System :: OS Independent
11
+ Classifier: Programming Language :: Python :: 3
12
+ Classifier: Programming Language :: Python :: 3 :: Only
13
+ Classifier: Topic :: Database
14
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
15
+ Requires-Dist: spark-kindling>=0.14.0,<0.15
16
+ Requires-Dist: spark-kindling-ext-sdp>=0.14.0,<0.15
17
+ Requires-Python: >=3.10
18
+ Project-URL: Homepage, https://github.com/sep/spark-kindling-framework
19
+ Project-URL: Documentation, https://github.com/sep/spark-kindling-framework/tree/main/docs
20
+ Project-URL: Changelog, https://github.com/sep/spark-kindling-framework/blob/main/CHANGELOG.md
21
+ Project-URL: Issues, https://github.com/sep/spark-kindling-framework/issues
22
+ Description-Content-Type: text/markdown
23
+
24
+ # kindling_ext_databricks
25
+
26
+ Databricks Lakeflow adapter for Kindling's SDP declaration engine.
27
+
28
+ Lakeflow "extends and is interoperable with" the OSS `pyspark.pipelines`
29
+ API, so this adapter **augments** the `kindling_ext_sdp` core rather than
30
+ translating it: the OSS emission (materialized views with comments, table
31
+ properties, partitioning, clustering, schema) runs unchanged, and
32
+ Databricks-only capabilities are layered on top.
33
+
34
+ Install it from PyPI with `pip install spark-kindling-ext-databricks`; in a uv project,
35
+ `kindling env add spark-kindling-ext-databricks` pins the version that matches the
36
+ project's Kindling release.
37
+
38
+ Selected with:
39
+
40
+ ```python
41
+ kindling.initialize(engine="databricks_sdp")
42
+ ...
43
+ kindling.declare_pipeline()
44
+ ```
45
+
46
+ — resolved through kindling core's engine-extension mapping
47
+ (`kindling_<name>.engine_extension()`); adding this engine required zero
48
+ core changes.
49
+
50
+ ## Phase 3 (this package today): expectations
51
+
52
+ ```yaml
53
+ datapipes:
54
+ silver.orders:
55
+ engine:
56
+ databricks_sdp:
57
+ expectations: # violations counted, rows kept (warn)
58
+ valid_order_id: "order_id IS NOT NULL"
59
+ expectations_drop: # violating rows dropped
60
+ positive_qty: "quantity > 0"
61
+ expectations_fail: # violation fails the update
62
+ no_future_dates: "order_date <= current_date()"
63
+ ```
64
+
65
+ Blocks map to `expect_all` / `expect_all_or_drop` / `expect_all_or_fail`
66
+ decorators. Declaring them while targeting OSS fails fast at validation
67
+ (capability gating in the core); declaring them here against a runtime
68
+ without the Lakeflow decorators fails at declaration with an actionable
69
+ error.
70
+
71
+ `refresh_policy: incremental` is accepted (adapter-tier gated) but emits
72
+ nothing yet — incremental MV refresh on Databricks (Enzyme) is engine
73
+ behavior rather than a declaration keyword; treated as a documented hint
74
+ pending verification against a live workspace.
75
+
76
+ ## Phase 5: SCD declared flows → AUTO CDC
77
+
78
+ An SCD-tagged output entity (`scd.type` plus the declared-flow tags from
79
+ kindling core) is declared as the Lakeflow CDC pattern — a pipeline-scoped
80
+ source view (the pipe's DataFrame), a streaming table target, and an AUTO
81
+ CDC flow — instead of a materialized view:
82
+
83
+ | Kindling declaration | AUTO CDC |
84
+ |---|---|
85
+ | `scd.type: "2"` (the only type core registration accepts) | `stored_as_scd_type` |
86
+ | entity `merge_columns` | `keys` |
87
+ | `scd.sequence_by` | `sequence_by` (change feed) |
88
+ | `scd.source_kind: change_feed` (default) | `create_auto_cdc_flow` |
89
+ | `scd.source_kind: snapshot` / `scd.close_on_missing: true` | `create_auto_cdc_from_snapshot_flow` |
90
+ | `scd.delete_when` | `apply_as_deletes=expr(...)` |
91
+ | `scd.tracked` | `track_history_column_list` |
92
+ | `scd.untracked` | `track_history_except_column_list=[*untracked, sequence_by]` |
93
+ | (default) sequence column ≠ content | `track_history_except_column_list=[sequence_by]` |
94
+
95
+ The tracking list goes to both the change-feed and the snapshot flow, and
96
+ only one list is ever passed. `sequence_by` is added to the exclude list
97
+ only when declared. Columns outside the tracked set are updated in place on
98
+ the current version, which is also what the runner does.
99
+
100
+ Fail-fast mapping requirements (validated with everything else, all
101
+ errors at once): change-feed targets need `scd.sequence_by`
102
+ (`scd_sequence_by_required`); `scd.type` must be `2`, matching core registration
103
+ (`scd_type_unsupported` — bitemporal is deliberately excluded while in
104
+ Beta); keys are required (`scd_keys_required`); `scd.tracked` and
105
+ `scd.untracked` can't both be set (`scd_tracked_untracked_exclusive`).
106
+
107
+ Documented divergences from the runner engine (dual-engine parity
108
+ criterion): history columns are `__START_AT`/`__END_AT` (the entity's
109
+ runner-shape schema is not passed to the streaming table); snapshot
110
+ ordering is ingestion order, not `scd.sequence_by`; out-of-order
111
+ change-feed rows are reconciled into history rather than ignored under
112
+ the runner's strictly-later rule; the `scd.current_entity_id`
113
+ current-view companion stays runner-only (an ordinary declared view is
114
+ the SDP-native replacement, tracked separately); `scd.untracked` on a
115
+ keyed SCD1 merge (no `scd.type`) is runner-only, because keyed SCD1 is not
116
+ lowered to AUTO CDC. Expectations, when
117
+ configured, attach to the source view — quality is checked on the
118
+ incoming feed.
119
+
120
+ Change-feed sources are consumed as a **stream**: every input selected by
121
+ `driving_entity_ids` is read with `spark.readStream.table()` so the AUTO CDC
122
+ flow processes it incrementally — the declarative replacement for
123
+ hand-rolled foreachBatch+MERGE. Remaining inputs stay batch reads
124
+ (stream-static joins). Snapshot sources keep batch reads: the snapshot API
125
+ diffs whole snapshots per update.
126
+
127
+ ## Provider-owned streaming sources
128
+
129
+ An external provider stream, initially Event Hubs over Kafka, can drive a
130
+ normal Kindling ingestion pipe when targeting `engine="databricks_sdp"`:
131
+
132
+ ```python
133
+ DataEntities.entity(
134
+ entityid="stream.telemetry",
135
+ name="Telemetry Event Hub",
136
+ tags={
137
+ "provider_type": "eventhub",
138
+ "provider.eventhub.connectionString": "@secret:lakeflow:eh-conn",
139
+ "provider.eventhub.name": "telemetry",
140
+ "provider.transport": "kafka",
141
+ "provider.kafka.includeHeaders": "true",
142
+ "provider.preprocess": "kafka",
143
+ },
144
+ schema=None,
145
+ )
146
+
147
+ @DataPipes.pipe(
148
+ pipeid="bronze.telemetry.ingest",
149
+ input_entity_ids=["stream.telemetry", "ref.devices"],
150
+ output_entity_id="bronze.telemetry",
151
+ driving_entity_ids=["stream.telemetry"],
152
+ )
153
+ def ingest_telemetry(stream_telemetry, ref_devices):
154
+ ...
155
+ ```
156
+
157
+ The shared declaration planner asks the provider for a secret-safe
158
+ `StreamingSourceSpec`. Invalid source configuration is reported before
159
+ emission without printing connection strings, SAS keys, JAAS values, or
160
+ resolved `@secret` values. The Databricks adapter then emits exactly one
161
+ `dp.create_streaming_table(...)` target and one `dp.append_flow(...)`. When
162
+ Lakeflow evaluates the append-flow function, Kindling resolves the provider,
163
+ calls `read_entity_as_stream()` with the declarative-source option, passes
164
+ later inputs as static DataFrames, invokes the registered pipe transform, and
165
+ returns the streaming DataFrame to Lakeflow.
166
+
167
+ The output entity's declared `schema` is not forwarded to
168
+ `create_streaming_table` pending Lakeflow platform evidence; the target schema
169
+ is inferred from the append-flow DataFrame on this path.
170
+
171
+ Lakeflow owns query startup, checkpoint placement, retries, update scheduling,
172
+ and target persistence for this path. Kindling does not call `writeStream`,
173
+ does not pass `checkpointLocation`, and does not start a streaming query.
174
+
175
+ ## Incremental reads of external Delta inputs
176
+
177
+ A normal pipe whose driving input is an ordinary Delta table — one produced
178
+ by a separate ingestion pipeline, not by a streaming provider — can be read
179
+ incrementally by opting in per pipe:
180
+
181
+ ```yaml
182
+ datapipes:
183
+ silver.device_telemetry:
184
+ engine:
185
+ databricks_sdp:
186
+ streaming_inputs: ["bronze.device_telemetry"]
187
+ ```
188
+
189
+ The output is then declared as a streaming table plus one append flow, the
190
+ named input is read with `spark.readStream.table(...)`, and every other
191
+ input stays a batch read — so reference-table joins remain stream-static.
192
+ Everything else on this path (expectations, table properties, partitioning,
193
+ clustering, the deferred `schema` forwarding) behaves exactly as it does for
194
+ a provider-owned streaming source; both shapes reach the same emission.
195
+
196
+ **The opt-in is never inferred.** An incremental read changes what a pipe
197
+ means — append-only, with no reprocessing of revised rows — so eligibility
198
+ is the app's decision, not the adapter's. Nothing changes for a Delta input
199
+ that is not named here.
200
+
201
+ It is adapter-tier: `engine="sdp"` reports `capability_not_supported`.
202
+ Rejected shapes, all at declaration time:
203
+
204
+ | Code | Shape |
205
+ | --- | --- |
206
+ | `streaming_input_not_an_input` | names an entity the pipe does not read |
207
+ | `streaming_input_not_driving` | names a non-driving input |
208
+ | `streaming_inputs_partial` | does not name every driving input |
209
+ | `multiple_streaming_inputs` | names more than one input |
210
+ | `streaming_input_not_delta` | names a non-Delta entity |
211
+ | `streaming_input_internal` | names a dataset produced in this pipeline |
212
+ | `streaming_input_provider_owned` | names a provider-owned stream (already streamed) |
213
+ | `streaming_dataset_type_conflict` | output explicitly requests a materialized view |
214
+
215
+ `streaming_inputs_partial` exists because the append flow streams whatever
216
+ `driving_entity_ids` selects: a partial opt-in would silently stream an
217
+ input the app never opted in, so the sets must match.
218
+
219
+ ## Temporal chain execution mode
220
+
221
+ A temporal chain lowers its event strata `<events>__g0..gK` as streaming
222
+ tables fed by append flows, so each base-event transform runs over a
223
+ Structured Streaming DataFrame. That rejects ordered analytic windows —
224
+ `row_number`, `lag`, unbounded forward fill — even though the same transform
225
+ runs correctly under the runner engine. Set the mode to `batch` to lower
226
+ those strata as materialized views with batch reads instead:
227
+
228
+ ```yaml
229
+ kindling:
230
+ lakeflow:
231
+ temporal_mode: batch
232
+ ```
233
+
234
+ From a Databricks Asset Bundle, use the canonical SparkConf spelling in the
235
+ pipeline's `configuration:` mapping — it is point-looked-up by the selector,
236
+ so it also works on serverless and shared-access runtimes that cannot
237
+ enumerate Spark configuration:
238
+
239
+ ```yaml
240
+ configuration:
241
+ spark.kindling.lakeflow.temporal_mode: "batch"
242
+ ```
243
+
244
+ | Component | `streaming` (default, omitted) | `batch` |
245
+ | --- | --- | --- |
246
+ | `<events>__g0` | Streaming table, one append flow per base declaration | One materialized view, union of the transformed base inputs |
247
+ | Base reads | `spark.readStream.table(...)` | `spark.table(...)` |
248
+ | `<events>__g1..gK` | Streaming tables and append flows | Materialized view per generation, batch reads of lower strata |
249
+ | Empty generation | Empty streaming projection of `__g0` | Empty batch projection of `__g0`, same schema and dependency |
250
+ | Episode snapshot, episodes, determinations, higher-order strata, public events | unchanged | unchanged |
251
+
252
+ ### Unpersisted generations
253
+
254
+ In `batch` mode the numbered strata can be declared as pipeline-scoped
255
+ temporary views instead of materialized views, so no `__g*` tables are
256
+ created and nothing intermediate is written:
257
+
258
+ ```yaml
259
+ kindling:
260
+ lakeflow:
261
+ temporal_mode: batch
262
+ temporal_strata_materialization: view # default: table
263
+ ```
264
+
265
+ Only the numbered strata change. The determinations view, higher-order
266
+ boundaries, the episodes Auto CDC target and the public `events` union are
267
+ declared identically, so query results are unchanged.
268
+
269
+ The cost is recomputation. Generation `k` reads every stratum below it, so a
270
+ temporary view is re-expanded once per reference rather than read back from
271
+ storage: the plan behind `events` grows exponentially in
272
+ `kindling.temporal.max_generations`, on the order of `2^K` rescans of the
273
+ base stratum at ceiling `K`. That is fine at a ceiling of 1–3 and a serious
274
+ problem at the default of 10. You also give up per-generation observability —
275
+ there is no `__g*` table to query when working out which generation produced
276
+ an event, and no per-stratum metrics in the event log.
277
+
278
+ `view` requires `temporal_mode: batch`. A streaming stratum is an append-flow
279
+ target and a temporary view cannot be one, so the combination fails the
280
+ declaration instead of quietly downgrading to batch reads.
281
+
282
+ ### How many strata get declared
283
+
284
+ `kindling.temporal.max_generations` is a ceiling, not a target. A chain whose
285
+ conditions are all registry-declared (`DataConditions.register`) declares
286
+ exactly as many numbered strata as those rules reach — a single-generation
287
+ app gets `__g0` and `__g1`, nothing else — because both emission and
288
+ `validate()`'s name reservation can compute that depth in-process from the
289
+ condition registry. The ceiling still applies: a rule set reaching past it
290
+ fails the declaration rather than being truncated.
291
+
292
+ A chain with any table-sourced condition engine keeps the ceiling as its
293
+ topology, declaring `__g0..__gK` whether or not each generation holds rules.
294
+ Rule rows are data that changes between updates, and the reservation path
295
+ runs with no Spark session to read their depth with, so a fixed shape is the
296
+ only way emission and reservation can agree. Empty strata there are real
297
+ datasets with a flow that filters everything out, so set the ceiling to the
298
+ depth you actually use.
299
+
300
+ Shrinking the depth of a registry chain leaves the highest `__g*` dataset
301
+ behind, no longer updated. That is a deploy-time event; drop the stale table
302
+ once.
303
+
304
+ Multi-source fan-in is unchanged: every base declaration lands in the one
305
+ stratum-0 dataset, each input keeping its own transform. Base sources still
306
+ resolve to external physical names through `EntityNameMapper` in both modes,
307
+ so a producer selected in the same pipeline establishes no local dependency
308
+ edge — keep those sources in an upstream resource. A batch chain declares no
309
+ `readStream` and no append flow at all; a chain *with* episodes still calls
310
+ `create_streaming_table` for the snapshot-CDC target only, because that is
311
+ what the AUTO CDC FROM SNAPSHOT API requires.
312
+
313
+ Semantics to plan for:
314
+
315
+ - Batch strata carry batch-query semantics over the rows available at each
316
+ refresh. A refresh can revise or remove previously produced events, and
317
+ late arrivals can change window results across a whole subject partition.
318
+ They are not append-only event archives and they do not use runner
319
+ watermarks.
320
+ - Retain the source history the computation needs. Removing input rows can
321
+ remove derived events and change the episode snapshot; snapshot CDC keeps
322
+ its SCD2 version history but cannot reconstruct source history that was
323
+ never retained.
324
+ - Use stable event identity and deterministic window ordering, including a
325
+ tie breaker for equal timestamps. Never use a mutable row rank as event
326
+ identity.
327
+ - Changing the mode changes `__g0..gK` dataset types. It is not a hot toggle
328
+ with portable checkpoint state: select the mode for newly provisioned
329
+ pipeline outputs, and treat conversion of an existing pipeline as a
330
+ deployment operation. Nothing is dropped or reset automatically.
331
+ - The value ignores surrounding whitespace and case. Anything other than
332
+ `streaming` or `batch` fails the declaration before any Lakeflow object is
333
+ created. The setting is inert when no temporal chain-events pipe is
334
+ selected, and never changes general SDP pipe execution, Lakeflow triggered
335
+ versus continuous scheduling, or the runner engine's watermark behavior.
336
+
337
+ Batch mode enables batch analysis; it does not guarantee that arbitrary user
338
+ code is declarable, or that a query refreshes incrementally.
339
+
340
+
341
+ ## Deferred
342
+
343
+ - Multiple provider-owned streaming inputs and stream-stream joins.
344
+ - Bitemporal AUTO CDC (`stored_as_scd_type="bitemporal"`) — Beta;
345
+ tracked, not adopted (proposal decision).
346
+ - Current-view companion as a declared view.
347
+
348
+
349
+ ## Output dataset naming
350
+
351
+ Lakeflow uses the shared [SDP dataset naming configuration](https://github.com/sep/spark-kindling-framework/blob/main/packages/extensions/kindling_ext_sdp/README.md#output-dataset-naming):
352
+
353
+ ```yaml
354
+ kindling:
355
+ sdp:
356
+ dataset_naming: leaf
357
+ ```
358
+
359
+ With a pipeline destination of catalog `dev_silver`, schema `cwmdp`,
360
+ `silver.device_telemetry` is declared as `device_telemetry`, producing
361
+ `dev_silver.cwmdp.device_telemetry`. Set separate destinations and select
362
+ the appropriate pipes for each medallion resource. Same-leaf outputs in
363
+ one plan are rejected.
364
+
365
+ This also controls AUTO CDC targets and temporary sources, temporal
366
+ events/episodes, and their generated strata, snapshots, and internal
367
+ references. Cross-pipeline external table resolution remains independent,
368
+ including temporal reads through `EntityNameMapper` and entity
369
+ `provider.table_catalog` overrides. Omitting the setting preserves the
370
+ existing `silver_device_telemetry` convention.
371
+
372
+
373
+ When consuming these outputs, explicitly align external entity metadata with
374
+ the leaf table, for example
375
+ `provider.table_name: dev_silver.cwmdp.device_telemetry`. Otherwise a consumer
376
+ configured with catalog and schema still resolves the historical flattened
377
+ name. See [external reads and generated-name reservations](https://github.com/sep/spark-kindling-framework/blob/main/packages/extensions/kindling_ext_sdp/README.md#reading-leaf-named-outputs-from-elsewhere)
378
+ for the complete example and the temporal source limitation.
379
+
380
+ ## Canonical configuration from a Bundle
381
+
382
+ Lakeflow uses the same Kindling configuration hierarchy as jobs and notebooks.
383
+ Publish ordinary Kindling settings files and pass them through the canonical
384
+ bootstrap key `spark.kindling.bootstrap.config_files`, or publish them through
385
+ `spark.kindling.bootstrap.artifacts_storage_path` and let bootstrap discover
386
+ base, platform, workspace, environment, and app overlays.
387
+
388
+ The selector does not parse YAML, inspect files, or validate structured
389
+ sections. It bridges Lakeflow pipeline configuration into
390
+ `kindling.initialize(..., app_name=<selected>, engine="databricks_sdp")` and
391
+ sets `declaration_only`; the shared bootstrap/Dynaconf path owns transport,
392
+ parsing, validation, precedence, and overlays. Flat bridged pipeline keys still
393
+ act as bootstrap overrides and win over settings files.
394
+
395
+ Use Bundle sync to deploy config to a stable workspace or volume path. This
396
+ matches the transport pattern in the
397
+ [DAB config promotion guide](https://github.com/sep/spark-kindling-framework/blob/main/docs/guide/dab_config_promotion.md):
398
+
399
+ ```yaml
400
+ # databricks.yml
401
+ bundle:
402
+ name: telemetry-lakeflow
403
+
404
+ targets:
405
+ dev:
406
+ workspace:
407
+ host: https://adb-<dev>.azuredatabricks.net
408
+ file_path: /Workspace/Shared/kindling/dev
409
+
410
+ sync:
411
+ include:
412
+ - config/**
413
+ - data-apps/**
414
+
415
+ resources:
416
+ pipelines:
417
+ telemetry_bronze:
418
+ name: telemetry-bronze
419
+ catalog: dev_bronze
420
+ target: cwmdp
421
+ configuration:
422
+ "kindling.data_app": telemetry
423
+ "kindling.lakeflow.pipes": bronze.ingest_telemetry
424
+ "spark.kindling.bootstrap.environment": dev
425
+ "spark.kindling.bootstrap.workspace_id": adb-dev
426
+ "spark.kindling.bootstrap.config_files": '["/Workspace/Shared/kindling/dev/config/settings.yaml", "/Workspace/Shared/kindling/dev/config/settings.databricks.yaml", "/Workspace/Shared/kindling/dev/data-apps/telemetry/settings.yaml"]'
427
+ telemetry_silver:
428
+ name: telemetry-silver
429
+ catalog: dev_silver
430
+ target: cwmdp
431
+ configuration:
432
+ "kindling.data_app": telemetry
433
+ "kindling.lakeflow.pipes": silver.build_telemetry,silver.derive_events,silver.derive_episodes
434
+ "spark.kindling.bootstrap.environment": dev
435
+ "spark.kindling.bootstrap.workspace_id": adb-dev
436
+ "spark.kindling.bootstrap.config_files": '["/Workspace/Shared/kindling/dev/config/settings.yaml", "/Workspace/Shared/kindling/dev/config/settings.databricks.yaml", "/Workspace/Shared/kindling/dev/data-apps/telemetry/settings.yaml"]'
437
+ ```
438
+
439
+ ```yaml
440
+ # data-apps/telemetry/settings.yaml
441
+ kindling:
442
+ sdp:
443
+ dataset_naming: leaf
444
+
445
+ dataentities:
446
+ bronze.device_telemetry:
447
+ tags:
448
+ provider.table_name: dev_bronze.cwmdp.device_telemetry
449
+ silver.device_telemetry:
450
+ tags:
451
+ provider.table_name: dev_silver.cwmdp.device_telemetry
452
+ silver.events:
453
+ tags:
454
+ provider.table_name: dev_silver.cwmdp.events
455
+ silver.episodes:
456
+ tags:
457
+ provider.table_name: dev_silver.cwmdp.episodes
458
+ ```
459
+
460
+ With `dataset_naming: leaf`, dotted logical IDs stay unchanged in the Kindling
461
+ registry while Lakeflow emits single-part output names inside each pipeline
462
+ destination:
463
+
464
+ | Logical ID | Pipeline destination | Physical table |
465
+ |---|---|---|
466
+ | `bronze.device_telemetry` | `dev_bronze.cwmdp` | `dev_bronze.cwmdp.device_telemetry` |
467
+ | `silver.device_telemetry` | `dev_silver.cwmdp` | `dev_silver.cwmdp.device_telemetry` |
468
+ | `silver.events` | `dev_silver.cwmdp` | `dev_silver.cwmdp.events` |
469
+ | `silver.episodes` | `dev_silver.cwmdp` | `dev_silver.cwmdp.episodes` |
470
+
471
+ Pipeline-local outputs get those names from `dataset_naming: leaf` plus the
472
+ pipeline catalog/schema. External reads of the same tables still use the
473
+ ordinary entity resolver, so they need the fully qualified
474
+ `provider.table_name` overrides shown in the structured YAML. A catalog/schema
475
+ override alone is not enough: it would still resolve the default flattened
476
+ table name. Temporal chain base-event sources always resolve externally, so
477
+ keep their external table overrides in the same structured YAML.
478
+
479
+ Workspace files under `/Workspace/...` are documented as driver-readable for
480
+ clusters and jobs. Readability from a Lakeflow serverless pipeline has not yet
481
+ been confirmed; if that is required, deploy the same YAML to a Unity Catalog
482
+ volume path and reference that path in
483
+ `spark.kindling.bootstrap.config_files`.
484
+
485
+ `kindling.lakeflow.config_files`, the comma-separated alias that 0.12.35
486
+ deprecated, was removed in 0.12.47 and is now ignored. Bundle configuration
487
+ that still sets it loads no files from it: move the paths to
488
+ `spark.kindling.bootstrap.config_files` as a JSON array string (or a bare
489
+ string for one file).