fast2dev-dbt-runner 0.0.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (35) hide show
  1. fast2dev_dbt_runner-0.0.1/CHANGELOG.md +284 -0
  2. fast2dev_dbt_runner-0.0.1/LICENSE +21 -0
  3. fast2dev_dbt_runner-0.0.1/MANIFEST.in +15 -0
  4. fast2dev_dbt_runner-0.0.1/PKG-INFO +398 -0
  5. fast2dev_dbt_runner-0.0.1/README.md +343 -0
  6. fast2dev_dbt_runner-0.0.1/docs/README.md +84 -0
  7. fast2dev_dbt_runner-0.0.1/docs/examples.md +537 -0
  8. fast2dev_dbt_runner-0.0.1/docs/installation.md +304 -0
  9. fast2dev_dbt_runner-0.0.1/docs/operators.md +259 -0
  10. fast2dev_dbt_runner-0.0.1/docs/publishing.md +123 -0
  11. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/__init__.py +47 -0
  12. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/_compat.py +41 -0
  13. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/airbyte_task_group_builder.py +259 -0
  14. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/bash_operator/__init__.py +3 -0
  15. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/bash_operator/datawarehouse_secrets.py +490 -0
  16. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/bash_operator/dbt_hook.py +615 -0
  17. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/bash_operator/dbt_operator.py +248 -0
  18. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/cached_manifest_loader.py +275 -0
  19. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/dbt_manifest_parser_api_operator.py +452 -0
  20. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/dbt_manifest_parser_bash_operator.py +361 -0
  21. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/dbt_manifest_parser_gke_operator.py +401 -0
  22. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/dbt_manifest_parser_k8s_operator.py +448 -0
  23. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner/utils.py +451 -0
  24. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner.egg-info/PKG-INFO +398 -0
  25. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner.egg-info/SOURCES.txt +33 -0
  26. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner.egg-info/dependency_links.txt +1 -0
  27. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner.egg-info/requires.txt +26 -0
  28. fast2dev_dbt_runner-0.0.1/fast2dev_dbt_runner.egg-info/top_level.txt +1 -0
  29. fast2dev_dbt_runner-0.0.1/pyproject.toml +144 -0
  30. fast2dev_dbt_runner-0.0.1/requirements.txt +22 -0
  31. fast2dev_dbt_runner-0.0.1/setup.cfg +4 -0
  32. fast2dev_dbt_runner-0.0.1/setup.py +5 -0
  33. fast2dev_dbt_runner-0.0.1/tests/test_cached_manifest_loader.py +126 -0
  34. fast2dev_dbt_runner-0.0.1/tests/test_manifest_parser_bash.py +242 -0
  35. fast2dev_dbt_runner-0.0.1/tests/test_utils.py +324 -0
@@ -0,0 +1,284 @@
1
+ # Changelog
2
+
3
+ All notable changes to the Fast2.Dev DBT Runner package will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [0.0.1] - 2026-10-04
9
+
10
+ First public package under **M7XLab / Fast2.Dev**. Distribution name `fast2dev-dbt-runner`, import package `fast2dev_dbt_runner`. Prior calendar versions below were internal only and were never published to pypi.org under this name.
11
+
12
+ ### Added
13
+ - PyPI metadata for a 0.0.1 release: classifiers, keywords, `requires-python >=3.10,<3.13`, MIT license, Fast2.Dev / M7XLab author fields.
14
+ - Package-level import aliases (`DbtManifestParserK8sOperator`, `DbtManifestParserBashOperator`, `DbtManifestParserApiOperator`, `DbtManifestParserGkeOperator`).
15
+ - `docs/publishing.md` for pypi.org + twine (`__token__` / `TWINE_PASSWORD`). No token is stored in the repository.
16
+
17
+ ### Changed
18
+ - Documented current product targets: dbt 1.12, Airflow 3, private registry images at `registry.m7xlab.top/fast2dev/*`.
19
+ - Source and issue URLs now point at the Fast2.Dev monorepo on Gitea (`fast2dev/fast2dev-ddp`). Invented standalone GitHub project URLs were removed.
20
+ - Airflow extra now requires Airflow 3 (`apache-airflow>=3.0.0,<4`) plus `apache-airflow-providers-cncf-kubernetes`.
21
+ - Bash operators no longer use the removed Airflow 2 `@apply_defaults` decorator.
22
+ - **Airflow 3.2:** `days_ago` is implemented locally (UTC, timezone-aware). The package never imports `airflow.utils.dates.days_ago` (removed in Airflow 3; the module still exists and that import raises `ImportError`).
23
+ - Airflow types go through `fast2dev_dbt_runner._compat` (`airflow.sdk` / `apache-airflow-providers-standard`).
24
+ - `json_safe_param()` converts `datetime.date` / `datetime` to ISO strings for Airflow 3 DAG `params` (Param defaults must be JSON-serializable; `date.today()` is not).
25
+ - Removed the IntelliJ leftover `fast_bi_dbt_runner.iml`. `.gitignore` excludes `.pypirc`, `dist/`, `*.iml`, and secrets.
26
+
27
+ ---
28
+
29
+ ## [2026.1.0.9] - 2026-06-05
30
+
31
+ ### Added
32
+ - **`DBT_MODEL_DEPENDS_ON_SNAPSHOT` parity for the k8s, GKE, and API operators**: the snapshot-dependency handling shipped for the bash operator in `2026.1.0.8` (batch mode) and `2026.1.0.7` (sharded "weave") now behaves identically across all four operator variants.
33
+ - New parser methods on the k8s, GKE, and API `DbtManifestParser`: `create_dbt_build_batch_task()`, `get_model_names_for_resource_types()` (multi-type selection), and `manifest_has_model_depends_on_snapshot()`.
34
+ - k8s/GKE: the build batch runs in a single pod with `DBT_COMMAND=build` and `MODEL="<models snapshots>"`; the existing image entrypoint already runs `dbt build --select <...>`, so no dbt image change is required. `E2E_MODE_EMPTY` is now also forwarded for `build` (mirroring `run`).
35
+ - API: the build batch posts a single `dbt build --select "<models snapshots>"` command to the API server; `--empty` (E2E) is now honored for `build`.
36
+ - All three parsers now thread `DBT_MODEL_DEPENDS_ON_SNAPSHOT` into the cached manifest loader (enabling the sharded "weave"), fix the `dbt_command` leak in `create_dbt_task_groups`, run a woven snapshot as `dbt snapshot` (not `dbt run`), and drop an empty task group via the `None` guard — matching the bash operator.
37
+ - DAG templates `template_dbt_project_dag_k8s.py`, `template_dbt_project_dag_gke.py`, and `template_dbt_project_dag_api_server.py`: compute `use_build_batch` and, when true, emit the single `dbt build` batch and skip the separate snapshot batch (identical to the bash template).
38
+ - **Tests**: parametrized parity tests (`tests/test_manifest_parser_pod_api.py`) covering detection, multi-type selection, and the build batch task for the k8s/GKE/API parsers.
39
+
40
+ ### Notes
41
+ - This completes the "k8s/GKE/API parity is a follow-up" item noted in `2026.1.0.7` and `2026.1.0.8`. The feature now works in both sharded and batch modes for every operator.
42
+ - k8s/GKE E2E `--empty`: the env var is forwarded for `build` as it is for `run`; whether `--empty` is applied still depends on the dbt image entrypoint (unchanged here), so the behavior matches the existing `run` path on those operators.
43
+
44
+ ---
45
+
46
+ ## [2026.1.0.8] - 2026-06-05
47
+
48
+ ### Added
49
+ - **`DBT_MODEL_DEPENDS_ON_SNAPSHOT` now works in batch mode (`DBT_MODEL_SHARDING=False`)**: previously the flag only had an effect in sharded mode and was a no-op when models ran as a single batch (a plain `dbt run` cannot include snapshots, so a `model -> snapshot -> model` chain ran the downstream model before the snapshot). When the flag is enabled, `DBT_MODEL_SHARDING` is `False`, and a selected model actually depends on a snapshot, the separate model-run + snapshot batches are replaced with a single `dbt build` over the models **and** snapshots. `dbt build` orders models, snapshots, and tests by lineage in one process, so the chain (e.g. `marketing_daily_report -> marketing_daily_report_hourly (snapshot) -> marketing_hourly_report`) resolves correctly. Selection is the explicit node list already resolved in the manifest (tag + ancestors), so scope is identical to the run batch it replaces, just with snapshots interleaved. Tests defined on the selected nodes run inline (standard `dbt build` behavior). `--empty` (E2E) is honored for `build`.
50
+ - New `DbtBuildOperator` (bash) running `dbt build` with `--empty`/`--full-refresh`/`--vars` parity with `DbtRunOperator`.
51
+ - New parser helpers: `create_dbt_build_batch_task()`, `get_model_names_for_resource_types()` (multi-type selection), and `manifest_has_model_depends_on_snapshot()` detection.
52
+ - `dbt_hook.py`: `build` added to the `--target-path` and `--empty` whitelists.
53
+ - DAG template (`template_dbt_project_dag_bash.py`): computes `use_build_batch` and, when true, emits the single `dbt build` batch and skips the separate snapshot batch. Sharded mode and the non-snapshot batch path are unchanged.
54
+ - Bash operator only for now; k8s/GKE/API parity remains a follow-up.
55
+
56
+ ### Notes
57
+ - This supersedes the `2026.1.0.7` limitation that `DBT_MODEL_DEPENDS_ON_SNAPSHOT` was a no-op in batch mode. Sharded mode still uses the manifest "weave" introduced in `2026.1.0.7`; batch mode uses the new `dbt build` path.
58
+
59
+ ---
60
+
61
+ ## [2026.1.0.7] - 2026-06-04
62
+
63
+ ### Added
64
+ - **`DBT_MODEL_DEPENDS_ON_SNAPSHOT` flag**: New Airflow variable (default: `False`). When enabled, a snapshot that a selected model depends on (via `ref()`) is "woven" into the model task group instead of running in the separate snapshot phase. This lets a `model -> snapshot -> model` chain execute as real task-level dependencies inside a single DAG (e.g. `marketing_daily_report -> marketing_daily_report_hourly (snapshot) -> marketing_hourly_report`), so if the snapshot fails the downstream model is skipped rather than silently bypassed. Detection is a pure manifest read (a snapshot is "model-blocking" when a selected model has its node id in `depends_on`), evaluated on the raw `depends_on` before test-dependency rewriting. The snapshot's tests are moved with it, preserving `snapshot -> its tests -> downstream model` ordering (matching `dbt build` semantics). Implemented in the shared `utils`/cache layer (bash operator wired now; k8s/GKE/API parity is a follow-up). Only effective with `DBT_MODEL_SHARDING=True`; it is a documented no-op in batch mode.
65
+
66
+ ### Changed
67
+ - **Manifest cache hardening** (`cached_manifest_loader.py`): the cache key now includes the new flag and normalizes all boolean flags through `to_bool`, so semantically-equal values (e.g. `True` and `"true"`) no longer create duplicate or cross-contaminating entries. `load_dbt_manifest_cached` now returns a deep copy of the cached manifest so caller-side mutation can never corrupt the shared cached object.
68
+
69
+ ### Fixed
70
+ - **`dbt_command` leak in `create_dbt_task_groups`** (bash operator): the command was reassigned on the loop variable, so a model node iterated after a test/source node could inherit the wrong command. It now uses a per-node local command.
71
+ - **Empty task-group guard**: `create_dbt_task_groups` returns `None` when no tasks are created (e.g. all snapshots were woven into the model group), so the now-empty snapshot phase is dropped cleanly by the existing `filter(None, ...)`.
72
+
73
+ ---
74
+
75
+ ## [2026.1.0.6] - 2026-03-31
76
+
77
+ ### Added
78
+ - **`DBT_MODEL` flag**: New Airflow variable `DBT_MODEL` (default: `True`). Enables or disables model execution entirely, consistent with the existing `DBT_SEED`, `DBT_SNAPSHOT`, and `DBT_SOURCE` flags. When set to `False`, the model section is skipped completely even if models are present in the manifest. Implemented across all four operator variants (bash, k8s, GKE, API).
79
+
80
+ ### Changed
81
+ - **Model execution check is now 3-level**: The model section now follows the same 3-level guard pattern used by seeds, snapshots, and sources: `is_resource_type_in_manifest("model")` → `DBT_MODEL == "true"` → `DBT_MODEL_SHARDING == "true"` (full lineage) or batch mode (`--select`). Previously the model section had only 2 levels (no `DBT_MODEL` gate).
82
+
83
+ ---
84
+
85
+ ## [2026.1.0.5] - 2026-03-31
86
+
87
+ ### Added
88
+ - **`DBT_MODEL_SHARDING` flag**: New Airflow variable `DBT_MODEL_SHARDING` (default: `True`). When set to `False`, all tag-filtered models are run in a single Airflow task (`dbt run --select "model1 model2 ..."`) instead of one task per model with full dependency lineage. The model names are read from the already tag-filtered manifest, so `DBT_TAGS` filtering is fully respected. Implemented across all four operator variants (bash, k8s, GKE, API).
89
+ - **Pytest test suite**: 73 unit tests covering manifest loading, caching, tag filtering, utils, and `create_dbt_batch_task` / `get_model_names_for_resource_type`. Uses `tests/fixtures/jaffle_shop_manifest.json` — no full dbt project required.
90
+ - **CI test step**: `.github/workflows/test.yml` now runs `pytest tests/` as a required blocking step with JUnit XML artifact upload.
91
+
92
+ ### Fixed
93
+ - **Tag filtering ignored in non-sharded mode for seeds, snapshots, and source freshness**: When `DBT_SEED_SHARDING`, `DBT_SNAPSHOT_SHARDING`, or `DBT_SOURCE_SHARDING` is set to `False` and `DBT_TAGS` is configured, the single-batch task previously called `dbt seed` / `dbt snapshot` / `dbt source freshness` with no `--select`, running all nodes regardless of tag filters. These now use `create_dbt_batch_task` which extracts tag-filtered node names from the manifest and passes them via `--select`. Fixed across all four operator variants (bash, k8s, GKE, API).
94
+
95
+ ## [2026.1.0.4] - 2026-03-12
96
+
97
+ ### Added
98
+ - **Snapshot: GIT_BRANCH support**
99
+ - **Bash operator**: `DbtSnapshotOperator` now accepts and passes `git_branch` (from Airflow vars `GIT_BRANCH`), so snapshot tasks run with the same branch context as run/test/seed when the pipeline sets it.
100
+ - **K8s operator**: Snapshot tasks receive env var `GIT_BRANCH` in the pod when `GIT_BRANCH` is set in Airflow variables.
101
+ - **GKE operator**: Same as K8s — snapshot tasks get `GIT_BRANCH` env var from Airflow variables when set.
102
+ - **API operator: snapshot command support**
103
+ - Snapshot is now fully supported: default command list for `dbt snapshot` (no `--exclude package:re_data`), and correct command building in task groups for single-snapshot (`--select <name>`) and run-all-snapshots.
104
+ - Snapshot is excluded from the full-refresh (`-f`) flag, which does not apply to `dbt snapshot`.
105
+
106
+ ## [2026.1.0.3] - 2026-03-09
107
+
108
+ ### Added
109
+ - **E2E empty mode (`--empty` for dbt run)**: New Airflow variable `E2E_MODE_EMPTY` enables passing the `--empty` flag to `dbt run` only (for E2E empty runs).
110
+ - **Bash operator**: `DbtCliHook` and `DbtBaseOperator` accept an `empty` parameter; when `E2E_MODE_EMPTY` is set in `airflow_vars`, `DbtRunOperator` receives `empty=True` and the hook appends `--empty` to the run command.
111
+ - **API operator**: When building the command list for run tasks, appends `--empty` if `E2E_MODE_EMPTY` is set.
112
+ - **GKE / K8s operators**: When `E2E_MODE_EMPTY` is set, the pod receives env var `E2E_MODE_EMPTY=true` for run tasks; the runner image must read this and add `--empty` to the dbt run invocation.
113
+
114
+ ## [2026.1.0.2] - 2026-03-04
115
+
116
+ ### Added
117
+ - **Error handling (dbt hook)**: When a dbt command fails and dbt prints "compiled code at" paths (e.g. for `dbt run` or `dbt test`), the hook now logs the contents of those compiled/run SQL files into Airflow task logs before the "Command exited with return code" line. This makes it easier to debug failures without opening the worker or re-running locally. Commands that do not emit "compiled code at" (e.g. `dbt source freshness`, `dbt deps`, `dbt debug`) are unchanged.
118
+
119
+ ## [2026.1.0.0-.1] - 2026-02-18 (Release)
120
+
121
+ ### Changed
122
+ - **Target path**: Use last segment of task_id for per-task target path (e.g. `models.burga.base.foo.model_name` → `/tmp/target_model_name`) for shorter, cleaner paths.
123
+ - **dbt CLI**: Use `--select` instead of deprecated `--models` when passing model selection to dbt run/test.
124
+ - **Per-DAG+task target isolation**: Prefix per-task target path with a short, stable hash of `dag_id` when available (e.g. `/tmp/target_<dag_hash>_model_name`) so each DAG/task pair reuses its own consistent target directory without risking OS path length limits.
125
+
126
+ ## [2026.1.0.0b2] - 2026-02-18 (Pre-release)
127
+
128
+ ### Fixed
129
+ - **Per-task --target-path**: Only pass `--target-path` for dbt commands that use the target directory (`run`, `test`, `seed`, `snapshot`). Other commands (`deps`, `debug`, `source freshness`, etc.) no longer receive `--target-path`, avoiding unsupported flag usage per [dbt command reference](https://docs.getdbt.com/category/list-of-commands).
130
+
131
+ ## [2026.1.0.0b1] - 2026-02-18 (Pre-release)
132
+
133
+ ### Added
134
+ - **DBT_DEPS flag (DAG template)**: Optional `dbt deps` step at the start of the DAG
135
+ - New Airflow variable `DBT_DEPS` (default: `True`). When set to `False`, the `install_dbt_dependencies` task is not created, so DAGs can rely on packages vendored in the repo for stable, scaling-friendly runs.
136
+ - Implemented in `data-orchestrator-core` template and compatible generated DAGs (e.g. bash operator).
137
+ - **Per-task dbt target path (bash operator)**: Avoid concurrent dbt runs sharing the same target directory
138
+ - Commands that use the target directory (`run`, `test`, `seed`, `snapshot`) now use `--target-path /tmp/target_{task_id}/` by default so parallel tasks no longer conflict. Other commands (e.g. `deps`, `debug`, `source freshness`) do not pass `--target-path` as they do not use it per [dbt command reference](https://docs.getdbt.com/category/list-of-commands).
139
+ - Implemented in `DbtCliHook` and `DbtBaseOperator`; task ID is sanitized for use as a directory name.
140
+
141
+ ### Changed
142
+ - **DAG template (bash operator)**: Start-of-DAG flow now conditionally includes `dbt deps` and correctly chains Airbyte → (optional deps) → (optional debug) → `show_input_data` when `DBT_DEPS` is toggled.
143
+
144
+ ### Technical details
145
+ - `dbt_hook.py`: Added `task_id` and `target_path` to `DbtCliHook`; `run_cli()` appends `--target-path` when set.
146
+ - `dbt_operator.py`: Passes `task_id` into the hook so each task gets a unique target path.
147
+ - Version source: `setup.py` now reads version from `pyproject.toml` when `CI_COMMIT_TAG` is not set, keeping a single source of truth for local builds.
148
+
149
+ ## [2025.2.0.0b2] - 2025-12-09 (Pre-release)
150
+
151
+ ### Fixed
152
+ - **Circular Dependency Issue in Source Task Groups**: Fixed `AirflowDagCycleException` when tests reference sources
153
+ - **Root Cause**: Tests that reference sources (via `source()` function) were incorrectly included in the "sources" task group, causing circular dependencies
154
+ - **How it happened**:
155
+ - Tests get their `group_type` assigned based on what they depend on (from `utils.py` line 322-323)
156
+ - Tests referencing sources get `"source"` added to their `group_type` (e.g., `["model", "source"]`)
157
+ - When creating source freshness task groups, the parser included all nodes with `"source"` in `group_type`, including tests
158
+ - This caused cycles because tests depend on both models and sources, creating circular dependency chains
159
+ - **The Fix**:
160
+ - Excluded tests from source task group creation in all 4 parser implementations:
161
+ - `dbt_manifest_parser_bash_operator.py`
162
+ - `dbt_manifest_parser_k8s_operator.py`
163
+ - `dbt_manifest_parser_gke_operator.py`
164
+ - `dbt_manifest_parser_api_operator.py`
165
+ - Tests now only run with models (where they belong), not as part of source freshness checks
166
+ - Simplified `set_dependencies()` to only set dependencies for nodes that were actually created as tasks
167
+ - **Impact**:
168
+ - DAGs with tests referencing sources no longer fail with cycle detection errors
169
+ - Tests still execute correctly as part of the models task group
170
+ - Source freshness checks remain isolated and don't include tests
171
+ - **Example**: A test file that uses both `ref('stg_example_model')` and `source('example_schema', 'example_source_table')` no longer causes DAG cycle errors
172
+
173
+ ## [2025.2.0.0b1] - 2025-12-02 (Pre-release)
174
+
175
+ ### ⚠️ BREAKING CHANGES - Beta Release
176
+ This is a **PRE-RELEASE** version containing major performance optimizations. Please test thoroughly in development/staging environments before production deployment.
177
+
178
+ ### Added - Major Performance Enhancement
179
+ - **Manifest Caching System**: Implemented file hash-based caching for dbt manifest parsing
180
+ - New module: `cached_manifest_loader.py` with intelligent caching mechanism
181
+ - Reduces DAG import time by 99% for unchanged manifests (2-4s → <10ms)
182
+ - MD5 hash-based cache invalidation ensures accuracy
183
+ - Thread-safe module-level cache with LRU eviction
184
+ - Configurable via environment variables:
185
+ - `AIRFLOW__CORE__MANIFEST_CACHE_ENABLED` (default: True)
186
+ - `AIRFLOW__CORE__MANIFEST_CACHE_DEBUG` (default: False)
187
+ - `AIRFLOW__CORE__MANIFEST_CACHE_MAX_SIZE` (default: 50)
188
+ - Cache statistics and monitoring via `get_cache_stats()` function
189
+ - Manual cache clearing via `clear_cache()` function
190
+
191
+ ### Changed
192
+ - **All 4 Parser Implementations Updated**:
193
+ - `dbt_manifest_parser_bash_operator.py` - Now uses cached loader
194
+ - `dbt_manifest_parser_api_operator.py` - Now uses cached loader
195
+ - `dbt_manifest_parser_gke_operator.py` - Now uses cached loader
196
+ - `dbt_manifest_parser_k8s_operator.py` - Now uses cached loader
197
+ - **Package Exports**: Added `load_dbt_manifest_cached`, `get_cache_stats`, and `clear_cache` to public API
198
+
199
+ ### Performance Impact
200
+ - **Before**: ~480 manifest parsing operations per hour (with 2 schedulers)
201
+ - **After**: ~5-10 cache misses per hour (only on actual manifest changes)
202
+ - **Expected Cache Hit Rate**: >99% in production
203
+ - **DAG Import Time Reduction**: 200-400x faster for cache hits
204
+ - **dag-processor CPU Usage**: Expected 30-50% reduction
205
+
206
+ ### Technical Details
207
+ - Manifest files are hashed using MD5 for change detection
208
+ - Cache keys include file hash, DBT tags, and ancestor/descendant flags
209
+ - Different tag configurations maintain separate cache entries
210
+ - All parsers share the same cache for maximum efficiency
211
+ - Automatic cache eviction when size exceeds configured maximum
212
+
213
+ ### Testing Recommendations
214
+ 1. Deploy to development/staging environment first
215
+ 2. Monitor cache hit rates using `get_cache_stats()`
216
+ 3. Enable debug logging to verify cache behavior
217
+ 4. Validate DAG parsing correctness
218
+ 5. Monitor dag-processor CPU and memory usage
219
+
220
+ ### Upgrade Notes
221
+ - **Backward Compatible**: Non-breaking change, drop-in replacement
222
+ - **No Configuration Required**: Caching is enabled by default
223
+ - **Easy Rollback**: Can be disabled via `AIRFLOW__CORE__MANIFEST_CACHE_ENABLED=False`
224
+ - **No Data Changes**: Cached data is identical to non-cached parsing
225
+
226
+ ### Known Limitations
227
+ - Cache is process-local (not shared across pod restarts)
228
+ - Memory usage: ~5-10MB per cached manifest
229
+ - First parse after restart will be cache miss (expected behavior)
230
+
231
+ ## [2025.1.0.2] - 2025-01-15
232
+
233
+ ### Fixed
234
+ - Fixed datetime parsing issue in `get_valid_start_date()` function in `utils.py`
235
+ - Improved ISO datetime parsing to properly handle datetime objects from DAG configurations
236
+ - Resolved customer issues with datetime parsing from DAG start dates
237
+
238
+ ## [2025.1.0.1] - 2025-09-01
239
+
240
+ ### Added
241
+ - Initial launch of Fast2.Dev DBT Runner package
242
+ - Four execution operators: K8S, Bash, API, and GKE
243
+ - DBT manifest parsing capabilities
244
+ - Airbyte task group builder integration
245
+ - Airflow integration support
246
+ - Comprehensive configuration management
247
+ - Data quality integration support
248
+ - Debug and monitoring capabilities
249
+
250
+ ### Features
251
+ - **K8S Operator**: Cost-optimized Kubernetes pod execution
252
+ - **Bash Operator**: Balanced cost-speed execution within Airflow workers
253
+ - **API Operator**: High-performance dedicated machine execution
254
+ - **GKE Operator**: Isolated external cluster execution
255
+ - **Manifest Parser**: Dynamic DAG generation from DBT manifests
256
+ - **Airbyte Integration**: Seamless Airbyte task group building
257
+ - **Flexible Configuration**: Extensive configuration options for various deployment scenarios
258
+
259
+ ### Technical Details
260
+ - Python 3.9+ compatibility
261
+ - Apache Airflow integration
262
+ - Google Cloud Platform support
263
+ - Kubernetes orchestration
264
+ - DBT Core compatibility
265
+ - MIT License
266
+
267
+ ### Beta Release Notes
268
+ This is the initial beta release of the Fast2.Dev DBT Runner package. The package provides a comprehensive solution for managing DBT workloads within the Fast2.Dev data development platform with various cost-performance trade-offs.
269
+
270
+ **What's Included:**
271
+ - Core package with all four operator types
272
+ - Basic documentation and examples
273
+ - PyPI distribution ready
274
+ - GitHub Actions CI/CD pipeline
275
+
276
+ **Next Steps:**
277
+ - Community feedback and testing
278
+ - Performance optimization
279
+ - Additional operator types
280
+ - Enhanced documentation and examples
281
+
282
+ ---
283
+
284
+ For operator details see [docs/operators.md](docs/operators.md). Publish steps: [docs/publishing.md](docs/publishing.md).
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 M7XLab
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,15 @@
1
+ include README.md
2
+ include CHANGELOG.md
3
+ include LICENSE
4
+ include requirements.txt
5
+ include setup.py
6
+ include docs/publishing.md
7
+ recursive-include fast2dev_dbt_runner *.py
8
+ recursive-include docs *
9
+ global-exclude *.pyc
10
+ global-exclude *.pyo
11
+ global-exclude __pycache__
12
+ global-exclude .DS_Store
13
+ global-exclude *.iml
14
+ global-exclude .pypirc
15
+ global-exclude .git*