oh-my-customcode 0.92.0 → 0.94.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli/index.js +1 -1
- package/dist/index.js +1 -1
- package/package.json +1 -1
- package/templates/.claude/agents/de-airflow-expert.md +17 -5
- package/templates/.claude/skills/airflow-best-practices/SKILL.md +52 -15
- package/templates/guides/airflow/README.md +26 -11
- package/templates/manifest.json +1 -1
package/dist/cli/index.js
CHANGED
package/dist/index.js
CHANGED
package/package.json
CHANGED
|
@@ -17,19 +17,31 @@ tools:
|
|
|
17
17
|
permissionMode: bypassPermissions
|
|
18
18
|
---
|
|
19
19
|
|
|
20
|
-
You are an expert Apache Airflow developer for production-ready DAGs following official best practices.
|
|
20
|
+
You are an expert Apache Airflow developer for production-ready DAGs following official best practices, targeting **Airflow 3.1.8**.
|
|
21
21
|
|
|
22
22
|
## Capabilities
|
|
23
23
|
|
|
24
|
-
- DAG authoring
|
|
25
|
-
-
|
|
26
|
-
-
|
|
24
|
+
- DAG authoring with `airflow.sdk` namespace (`DAG`, `@task`, `Asset`)
|
|
25
|
+
- TaskFlow API patterns and dynamic task mapping (`expand()`)
|
|
26
|
+
- Task dependency design and scheduling (cron, timetables, data-aware with Assets)
|
|
27
|
+
- DAG and task testing (`dag.test()`, unit tests, integration tests)
|
|
27
28
|
- Connection/variable management and secret backend integration
|
|
28
29
|
- DAG parsing and execution optimization
|
|
30
|
+
- Airflow 2.x → 3.x migration guidance (import paths, deprecated context vars, AIP-72/AIP-44)
|
|
31
|
+
|
|
32
|
+
## Key Airflow 3.x Differences
|
|
33
|
+
|
|
34
|
+
| Area | Airflow 2.x | Airflow 3.x |
|
|
35
|
+
|------|-------------|-------------|
|
|
36
|
+
| Imports | `from airflow.models import DAG` | `from airflow.sdk import DAG, task` |
|
|
37
|
+
| Data-aware | `Dataset` | `Asset` |
|
|
38
|
+
| Context | `execution_date` | `dag_run.logical_date` |
|
|
39
|
+
| Architecture | Tight coupling | Task Execution Interface (AIP-72) |
|
|
40
|
+
| API | DB direct access | Internal API (AIP-44) |
|
|
29
41
|
|
|
30
42
|
## Skills
|
|
31
43
|
|
|
32
|
-
Apply **airflow-best-practices** for core Airflow guidelines.
|
|
44
|
+
Apply **airflow-best-practices** for core Airflow 3.1.8 guidelines.
|
|
33
45
|
|
|
34
46
|
## Reference Guides
|
|
35
47
|
|
|
@@ -5,29 +5,48 @@ scope: core
|
|
|
5
5
|
user-invocable: false
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# Apache Airflow Best Practices
|
|
8
|
+
# Apache Airflow Best Practices (3.1.8)
|
|
9
9
|
|
|
10
10
|
## DAG Authoring
|
|
11
11
|
|
|
12
|
+
### Imports (Airflow 3.x)
|
|
13
|
+
- Use `from airflow.sdk import DAG, task, Asset` — the stable public API
|
|
14
|
+
- Legacy `from airflow.models import DAG` and `from airflow.decorators import task` are deprecated
|
|
15
|
+
|
|
12
16
|
### Top-Level Code (CRITICAL)
|
|
13
17
|
- Avoid heavy computation at module level (executed on every DAG parse)
|
|
14
|
-
- Minimize imports at module level
|
|
15
|
-
-
|
|
16
|
-
-
|
|
18
|
+
- Minimize imports at module level — lazy-load inside `@task` functions
|
|
19
|
+
- Never call APIs, query databases, or access Variables at top level
|
|
20
|
+
- If Variables needed at top level, enable experimental cache with TTL
|
|
21
|
+
|
|
22
|
+
### TaskFlow API (Default Pattern)
|
|
23
|
+
- Use `@task` decorator for all Python tasks (preferred over classic operators)
|
|
24
|
+
- XCom serialization is automatic — return values become XCom
|
|
25
|
+
- Use `@task.branch` for branching logic
|
|
26
|
+
- Use `@task.sensor` for sensor tasks
|
|
27
|
+
|
|
28
|
+
### Dynamic Task Mapping
|
|
29
|
+
- Use `task.expand()` for runtime-determined task instances
|
|
30
|
+
- Combine with `.partial()` for fixed kwargs
|
|
31
|
+
- Map over lists, dicts, or XCom outputs from upstream tasks
|
|
17
32
|
|
|
18
33
|
### Scheduling
|
|
19
|
-
- Use cron expressions or timetables
|
|
20
|
-
- Set `catchup=False` for most
|
|
21
|
-
- Use data-aware scheduling (
|
|
34
|
+
- Use cron expressions or timetables for `schedule` parameter
|
|
35
|
+
- Set `catchup=False` for most DAGs
|
|
36
|
+
- Use data-aware scheduling with `Asset` (replaces `Dataset`) for dependencies
|
|
22
37
|
- Configure SLA monitoring
|
|
23
38
|
|
|
24
39
|
### Task Dependencies
|
|
25
|
-
- Use `>>` / `<<` for clarity
|
|
26
|
-
- Group related tasks with TaskGroup
|
|
40
|
+
- Use `>>` / `<<` operators for clarity
|
|
41
|
+
- Group related tasks with `TaskGroup`
|
|
27
42
|
- Avoid deep nesting (max 3 levels)
|
|
28
43
|
|
|
29
44
|
## Testing
|
|
30
45
|
|
|
46
|
+
### Local Testing
|
|
47
|
+
- Use `dag.test()` in `if __name__ == "__main__":` block for IDE debugging
|
|
48
|
+
- Runs all tasks in single serialized process without executor
|
|
49
|
+
|
|
31
50
|
### Unit Tests
|
|
32
51
|
- Test DAG import without errors
|
|
33
52
|
- Detect cycles in dependencies
|
|
@@ -42,16 +61,34 @@ user-invocable: false
|
|
|
42
61
|
## Production Deployment
|
|
43
62
|
|
|
44
63
|
### Performance
|
|
45
|
-
- Lazy-load heavy libraries inside
|
|
64
|
+
- Lazy-load heavy libraries inside `@task` functions
|
|
46
65
|
- Use connection pooling
|
|
47
|
-
- Minimize DAG parse time
|
|
48
|
-
- Enable parallelism
|
|
66
|
+
- Minimize DAG parse time (target < 30s for all DAGs)
|
|
67
|
+
- Enable parallelism appropriately
|
|
49
68
|
|
|
50
69
|
### Reliability
|
|
51
|
-
- Set appropriate retries and retry_delay
|
|
70
|
+
- Set appropriate `retries` and `retry_delay`
|
|
52
71
|
- Use SLA callbacks for monitoring
|
|
53
|
-
- Implement proper error handling
|
|
72
|
+
- Implement proper error handling with `on_failure_callback`
|
|
54
73
|
- Log important events
|
|
55
74
|
|
|
75
|
+
## Migration: 2.x → 3.x
|
|
76
|
+
|
|
77
|
+
### Deprecated (Remove or Replace)
|
|
78
|
+
| Deprecated | Replacement |
|
|
79
|
+
|-----------|-------------|
|
|
80
|
+
| `from airflow.models import DAG` | `from airflow.sdk import DAG` |
|
|
81
|
+
| `from airflow.decorators import task` | `from airflow.sdk import task` |
|
|
82
|
+
| `Dataset` | `Asset` |
|
|
83
|
+
| `execution_date` in context | `dag_run.logical_date` |
|
|
84
|
+
| `conf` in task context | Removed — use Variables or params |
|
|
85
|
+
|
|
86
|
+
### Architecture Changes
|
|
87
|
+
- **AIP-72**: Task Execution Interface — tasks run in isolated subprocesses via Execution API Server
|
|
88
|
+
- **AIP-44**: Internal API — components communicate via API, not direct DB access
|
|
89
|
+
- **New UI**: React-based web interface (replaces Flask-based UI)
|
|
90
|
+
|
|
56
91
|
## References
|
|
57
|
-
- [Airflow Best Practices](https://airflow.apache.org/docs/apache-airflow/
|
|
92
|
+
- [Airflow 3.1.8 Best Practices](https://airflow.apache.org/docs/apache-airflow/3.1.8/best-practices.html)
|
|
93
|
+
- [Airflow SDK (Task SDK)](https://airflow.apache.org/docs/apache-airflow/3.1.8/authoring-and-scheduling/index.html)
|
|
94
|
+
- [Migration Guide 2.x → 3.x](https://airflow.apache.org/docs/apache-airflow/3.1.8/migration-guide.html)
|
|
@@ -1,22 +1,35 @@
|
|
|
1
|
-
# Apache Airflow Guide
|
|
1
|
+
# Apache Airflow Guide (3.1.8)
|
|
2
2
|
|
|
3
|
-
Reference documentation for Apache Airflow DAG development best practices.
|
|
3
|
+
Reference documentation for Apache Airflow 3.1.8 DAG development best practices.
|
|
4
4
|
|
|
5
5
|
## Source
|
|
6
6
|
|
|
7
|
-
Based on [Apache Airflow official documentation](https://airflow.apache.org/docs/) and [Astronomer best practices](https://
|
|
7
|
+
Based on [Apache Airflow 3.1.8 official documentation](https://airflow.apache.org/docs/apache-airflow/3.1.8/) and [Astronomer best practices](https://docs.astronomer.io/).
|
|
8
|
+
|
|
9
|
+
## Airflow 3.x Key Changes
|
|
10
|
+
|
|
11
|
+
| Change | Details |
|
|
12
|
+
|--------|---------|
|
|
13
|
+
| SDK namespace | `from airflow.sdk import DAG, task, Asset` |
|
|
14
|
+
| Task Execution (AIP-72) | Isolated subprocess execution via Execution API Server |
|
|
15
|
+
| Internal API (AIP-44) | Components use API instead of direct DB access |
|
|
16
|
+
| Asset (was Dataset) | Data-aware scheduling primitive renamed |
|
|
17
|
+
| Context cleanup | `execution_date` → `dag_run.logical_date` |
|
|
18
|
+
| New UI | React-based web interface |
|
|
19
|
+
| TaskFlow API | `@task` is the default pattern for Python tasks |
|
|
8
20
|
|
|
9
21
|
## Categories
|
|
10
22
|
|
|
11
23
|
| Priority | Category | Impact |
|
|
12
24
|
|----------|----------|--------|
|
|
13
|
-
| 1 | DAG Authoring | CRITICAL |
|
|
14
|
-
| 2 |
|
|
15
|
-
| 3 | Testing | HIGH |
|
|
16
|
-
| 4 | Scheduling | HIGH |
|
|
25
|
+
| 1 | DAG Authoring (airflow.sdk) | CRITICAL |
|
|
26
|
+
| 2 | TaskFlow API & Dynamic Mapping | CRITICAL |
|
|
27
|
+
| 3 | Testing (dag.test()) | HIGH |
|
|
28
|
+
| 4 | Scheduling & Assets | HIGH |
|
|
17
29
|
| 5 | Connections & Variables | MEDIUM |
|
|
18
30
|
| 6 | Monitoring & SLA | MEDIUM |
|
|
19
31
|
| 7 | Performance Optimization | LOW-MEDIUM |
|
|
32
|
+
| 8 | 2.x → 3.x Migration | HIGH |
|
|
20
33
|
|
|
21
34
|
## Usage
|
|
22
35
|
|
|
@@ -26,7 +39,9 @@ This guide is referenced by:
|
|
|
26
39
|
|
|
27
40
|
## External Resources
|
|
28
41
|
|
|
29
|
-
- [Airflow Official Docs](https://airflow.apache.org/docs/)
|
|
30
|
-
- [Airflow Best Practices](https://airflow.apache.org/docs/apache-airflow/
|
|
31
|
-
- [
|
|
32
|
-
- [Airflow TaskFlow API](https://airflow.apache.org/docs/apache-airflow/
|
|
42
|
+
- [Airflow 3.1.8 Official Docs](https://airflow.apache.org/docs/apache-airflow/3.1.8/)
|
|
43
|
+
- [Airflow 3.1.8 Best Practices](https://airflow.apache.org/docs/apache-airflow/3.1.8/best-practices.html)
|
|
44
|
+
- [Airflow Task SDK](https://airflow.apache.org/docs/apache-airflow/3.1.8/authoring-and-scheduling/index.html)
|
|
45
|
+
- [Airflow TaskFlow API](https://airflow.apache.org/docs/apache-airflow/3.1.8/core-concepts/taskflow.html)
|
|
46
|
+
- [Migration Guide 2.x → 3.x](https://airflow.apache.org/docs/apache-airflow/3.1.8/migration-guide.html)
|
|
47
|
+
- [Astronomer Docs](https://docs.astronomer.io/)
|
package/templates/manifest.json
CHANGED