simple_flow 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.envrc +1 -0
- data/.github/workflows/deploy-github-pages.yml +52 -0
- data/.rubocop.yml +57 -0
- data/CHANGELOG.md +4 -0
- data/COMMITS.md +196 -0
- data/LICENSE +21 -0
- data/README.md +481 -0
- data/Rakefile +15 -0
- data/benchmarks/parallel_vs_sequential.rb +98 -0
- data/benchmarks/pipeline_overhead.rb +130 -0
- data/docs/api/middleware.md +468 -0
- data/docs/api/parallel-step.md +363 -0
- data/docs/api/pipeline.md +382 -0
- data/docs/api/result.md +375 -0
- data/docs/concurrent/best-practices.md +687 -0
- data/docs/concurrent/introduction.md +246 -0
- data/docs/concurrent/parallel-steps.md +418 -0
- data/docs/concurrent/performance.md +481 -0
- data/docs/core-concepts/flow-control.md +452 -0
- data/docs/core-concepts/middleware.md +389 -0
- data/docs/core-concepts/overview.md +219 -0
- data/docs/core-concepts/pipeline.md +315 -0
- data/docs/core-concepts/result.md +168 -0
- data/docs/core-concepts/steps.md +391 -0
- data/docs/development/benchmarking.md +443 -0
- data/docs/development/contributing.md +380 -0
- data/docs/development/dagwood-concepts.md +435 -0
- data/docs/development/testing.md +514 -0
- data/docs/getting-started/examples.md +197 -0
- data/docs/getting-started/installation.md +62 -0
- data/docs/getting-started/quick-start.md +218 -0
- data/docs/guides/choosing-concurrency-model.md +441 -0
- data/docs/guides/complex-workflows.md +440 -0
- data/docs/guides/data-fetching.md +478 -0
- data/docs/guides/error-handling.md +635 -0
- data/docs/guides/file-processing.md +505 -0
- data/docs/guides/validation-patterns.md +496 -0
- data/docs/index.md +169 -0
- data/examples/.gitignore +3 -0
- data/examples/01_basic_pipeline.rb +112 -0
- data/examples/02_error_handling.rb +178 -0
- data/examples/03_middleware.rb +186 -0
- data/examples/04_parallel_automatic.rb +221 -0
- data/examples/05_parallel_explicit.rb +279 -0
- data/examples/06_real_world_ecommerce.rb +288 -0
- data/examples/07_real_world_etl.rb +277 -0
- data/examples/08_graph_visualization.rb +246 -0
- data/examples/09_pipeline_visualization.rb +266 -0
- data/examples/10_concurrency_control.rb +235 -0
- data/examples/11_sequential_dependencies.rb +243 -0
- data/examples/12_none_constant.rb +161 -0
- data/examples/README.md +374 -0
- data/examples/regression_test/01_basic_pipeline.txt +38 -0
- data/examples/regression_test/02_error_handling.txt +92 -0
- data/examples/regression_test/03_middleware.txt +61 -0
- data/examples/regression_test/04_parallel_automatic.txt +86 -0
- data/examples/regression_test/05_parallel_explicit.txt +80 -0
- data/examples/regression_test/06_real_world_ecommerce.txt +53 -0
- data/examples/regression_test/07_real_world_etl.txt +58 -0
- data/examples/regression_test/08_graph_visualization.txt +429 -0
- data/examples/regression_test/09_pipeline_visualization.txt +305 -0
- data/examples/regression_test/10_concurrency_control.txt +96 -0
- data/examples/regression_test/11_sequential_dependencies.txt +86 -0
- data/examples/regression_test/12_none_constant.txt +64 -0
- data/examples/regression_test.rb +105 -0
- data/lib/simple_flow/dependency_graph.rb +120 -0
- data/lib/simple_flow/dependency_graph_visualizer.rb +326 -0
- data/lib/simple_flow/middleware.rb +36 -0
- data/lib/simple_flow/parallel_executor.rb +80 -0
- data/lib/simple_flow/pipeline.rb +405 -0
- data/lib/simple_flow/result.rb +88 -0
- data/lib/simple_flow/step_tracker.rb +58 -0
- data/lib/simple_flow/version.rb +5 -0
- data/lib/simple_flow.rb +41 -0
- data/mkdocs.yml +146 -0
- data/pipeline_graph.dot +51 -0
- data/pipeline_graph.html +60 -0
- data/pipeline_graph.mmd +19 -0
- metadata +127 -0
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
============================================================
|
|
2
|
+
Explicit Parallel Blocks
|
|
3
|
+
============================================================
|
|
4
|
+
|
|
5
|
+
✓ Async gem is available - will use fiber-based concurrency
|
|
6
|
+
|
|
7
|
+
Example 1: Basic Parallel Block
|
|
8
|
+
------------------------------------------------------------
|
|
9
|
+
|
|
10
|
+
[Sequential] Pre-processing input...
|
|
11
|
+
[Parallel A] Fetching from API...
|
|
12
|
+
[Parallel B] Fetching from cache...
|
|
13
|
+
[Parallel C] Fetching from database...
|
|
14
|
+
[Sequential] Post-processing results...
|
|
15
|
+
|
|
16
|
+
Result: {api: nil, cache: nil, db: {records: 10}}
|
|
17
|
+
Execution time: 101.17ms
|
|
18
|
+
(Should be ~100ms with parallel, ~300ms sequential)
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
Example 2: Multiple Parallel Blocks
|
|
22
|
+
------------------------------------------------------------
|
|
23
|
+
|
|
24
|
+
[Step 1] Initialize
|
|
25
|
+
[Block 1.A] Validate email
|
|
26
|
+
[Block 1.B] Validate phone
|
|
27
|
+
[Step 2] Process validations
|
|
28
|
+
[Block 2.A] Send email notification
|
|
29
|
+
[Block 2.B] Send SMS notification
|
|
30
|
+
[Block 2.C] Log to database
|
|
31
|
+
[Step 3] Finalize
|
|
32
|
+
|
|
33
|
+
Result: All notifications sent
|
|
34
|
+
Context: {phone_valid: true, logged: true}
|
|
35
|
+
Execution time: 102.68ms
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
Example 3: Mixed Execution Styles
|
|
39
|
+
------------------------------------------------------------
|
|
40
|
+
|
|
41
|
+
[Unnamed] Starting workflow...
|
|
42
|
+
[Named] Fetching configuration...
|
|
43
|
+
[Parallel] Processing batch 1...
|
|
44
|
+
[Parallel] Processing batch 2...
|
|
45
|
+
[Unnamed] Finalizing...
|
|
46
|
+
|
|
47
|
+
Result: Workflow complete
|
|
48
|
+
Context: {config: {timeout: 30}, batch2: :done}
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
Example 4: Error Handling in Parallel Blocks
|
|
52
|
+
------------------------------------------------------------
|
|
53
|
+
|
|
54
|
+
[Pre] Starting operations...
|
|
55
|
+
[Parallel A] Running successfully...
|
|
56
|
+
[Parallel B] Encountering error...
|
|
57
|
+
[Parallel C] Running successfully...
|
|
58
|
+
|
|
59
|
+
Result:
|
|
60
|
+
Continue? false
|
|
61
|
+
Value:
|
|
62
|
+
Errors: {task_b: ["Failed to process batch"]}
|
|
63
|
+
Context: {}
|
|
64
|
+
Note: Pipeline halted due to error in parallel block
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
Example 5: Performance Comparison
|
|
68
|
+
------------------------------------------------------------
|
|
69
|
+
|
|
70
|
+
Running sequential pipeline (4 steps @ 100ms each)...
|
|
71
|
+
Running parallel pipeline (4 steps @ 100ms each in parallel)...
|
|
72
|
+
|
|
73
|
+
Results:
|
|
74
|
+
Sequential: 418.79ms (expected ~400ms)
|
|
75
|
+
Parallel: 101.32ms (expected ~100ms)
|
|
76
|
+
Speedup: 4.13x
|
|
77
|
+
|
|
78
|
+
============================================================
|
|
79
|
+
Explicit parallel blocks examples completed!
|
|
80
|
+
============================================================
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
============================================================
|
|
2
|
+
Real-World Example: E-commerce Order Processing
|
|
3
|
+
============================================================
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
Processing order: ORD-712
|
|
7
|
+
Items: 2 items totaling $109.97
|
|
8
|
+
|
|
9
|
+
✓ Validating order...
|
|
10
|
+
✓ Calculating shipping...
|
|
11
|
+
✓ Checking inventory...
|
|
12
|
+
✓ Calculating totals...
|
|
13
|
+
✓ Processing payment...
|
|
14
|
+
✓ Reserving inventory...
|
|
15
|
+
✓ Creating shipment...
|
|
16
|
+
✓ Sending email confirmation...
|
|
17
|
+
✓ Sending SMS confirmation...
|
|
18
|
+
📧 Email sent to customer@example.com: Order Confirmed
|
|
19
|
+
📱 SMS sent to +1-555-0123: Order confirmed! Tracking: TRACK-8767
|
|
20
|
+
✓ Finalizing order...
|
|
21
|
+
|
|
22
|
+
============================================================
|
|
23
|
+
✅ Order processed successfully!
|
|
24
|
+
============================================================
|
|
25
|
+
|
|
26
|
+
Order Details:
|
|
27
|
+
Order ID: ORD-712
|
|
28
|
+
Status: confirmed
|
|
29
|
+
Total: $128.77
|
|
30
|
+
Transaction: TXN-9615
|
|
31
|
+
Tracking: TRACK-8767
|
|
32
|
+
Estimated Delivery: 3 days
|
|
33
|
+
|
|
34
|
+
Processing time: 339.24ms
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
============================================================
|
|
38
|
+
Testing with invalid order (missing payment)...
|
|
39
|
+
============================================================
|
|
40
|
+
|
|
41
|
+
✓ Validating order...
|
|
42
|
+
✓ Calculating shipping...
|
|
43
|
+
✓ Checking inventory...
|
|
44
|
+
✓ Calculating totals...
|
|
45
|
+
✓ Processing payment...
|
|
46
|
+
❌ Order failed (as expected)
|
|
47
|
+
|
|
48
|
+
Errors:
|
|
49
|
+
payment: Invalid card token
|
|
50
|
+
|
|
51
|
+
============================================================
|
|
52
|
+
E-commerce example completed!
|
|
53
|
+
============================================================
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
============================================================
|
|
2
|
+
Real-World Example: Data ETL Pipeline
|
|
3
|
+
============================================================
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
Starting ETL pipeline...
|
|
7
|
+
============================================================
|
|
8
|
+
|
|
9
|
+
📥 Extracting orders from JSON...
|
|
10
|
+
📥 Extracting products from API...
|
|
11
|
+
📥 Extracting users from CSV...
|
|
12
|
+
🔄 Transforming order data...
|
|
13
|
+
🔄 Transforming product data...
|
|
14
|
+
🔄 Transforming user data...
|
|
15
|
+
📊 Aggregating category statistics...
|
|
16
|
+
📊 Aggregating user statistics...
|
|
17
|
+
✅ Validating data quality...
|
|
18
|
+
💾 Preparing output...
|
|
19
|
+
|
|
20
|
+
============================================================
|
|
21
|
+
✅ ETL Pipeline completed successfully!
|
|
22
|
+
============================================================
|
|
23
|
+
|
|
24
|
+
Metadata:
|
|
25
|
+
Processed at: 2001-09-11 07:00:00 -0500
|
|
26
|
+
Pipeline version: 1.0
|
|
27
|
+
|
|
28
|
+
Data Summary:
|
|
29
|
+
Users processed: 3
|
|
30
|
+
Orders processed: 3
|
|
31
|
+
Products processed: 3
|
|
32
|
+
Total revenue: $459.54
|
|
33
|
+
|
|
34
|
+
User Statistics:
|
|
35
|
+
Alice Johnson (alice@example.com)
|
|
36
|
+
Orders: 2, Spent: $378.0, Avg: $189.0
|
|
37
|
+
Bob Smith (bob@example.com)
|
|
38
|
+
Orders: 1, Spent: $81.54, Avg: $81.54
|
|
39
|
+
Charlie Brown (charlie@example.com)
|
|
40
|
+
Orders: 0, Spent: $0, Avg: $0
|
|
41
|
+
|
|
42
|
+
Category Statistics:
|
|
43
|
+
tools: 2 products (Widget, Doohickey)
|
|
44
|
+
electronics: 1 products (Gadget)
|
|
45
|
+
|
|
46
|
+
Processing time: 101.43ms
|
|
47
|
+
|
|
48
|
+
Execution Flow:
|
|
49
|
+
Phase 1 (Extract): users, orders, products (parallel)
|
|
50
|
+
Phase 2 (Transform): transform_users, transform_orders, transform_products (parallel)
|
|
51
|
+
Phase 3 (Aggregate): user_stats, category_stats (parallel after transforms)
|
|
52
|
+
Phase 4 (Validate): data validation
|
|
53
|
+
Phase 5 (Load): prepare output
|
|
54
|
+
|
|
55
|
+
============================================================
|
|
56
|
+
ETL example completed!
|
|
57
|
+
Run with --save flag to save output to JSON file
|
|
58
|
+
============================================================
|
|
@@ -0,0 +1,429 @@
|
|
|
1
|
+
============================================================
|
|
2
|
+
Dependency Graph Visualization
|
|
3
|
+
============================================================
|
|
4
|
+
|
|
5
|
+
Example 1: Simple Dependency Graph
|
|
6
|
+
------------------------------------------------------------
|
|
7
|
+
|
|
8
|
+
Dependency Graph
|
|
9
|
+
============================================================
|
|
10
|
+
|
|
11
|
+
Dependencies:
|
|
12
|
+
:step_a
|
|
13
|
+
└─ depends on: (none)
|
|
14
|
+
:step_b
|
|
15
|
+
└─ depends on: :step_a
|
|
16
|
+
:step_c
|
|
17
|
+
└─ depends on: :step_a
|
|
18
|
+
:step_d
|
|
19
|
+
└─ depends on: :step_b, :step_c
|
|
20
|
+
|
|
21
|
+
Execution Order (sequential):
|
|
22
|
+
:step_a
|
|
23
|
+
↓ :step_b
|
|
24
|
+
↓ :step_c
|
|
25
|
+
↓ :step_d
|
|
26
|
+
|
|
27
|
+
Parallel Execution Groups:
|
|
28
|
+
Group 1:
|
|
29
|
+
└─ :step_a (sequential)
|
|
30
|
+
Group 2:
|
|
31
|
+
├─ Parallel execution of 2 steps:
|
|
32
|
+
├─ :step_b
|
|
33
|
+
└─ :step_c
|
|
34
|
+
Group 3:
|
|
35
|
+
└─ :step_d (sequential)
|
|
36
|
+
|
|
37
|
+
Execution Tree:
|
|
38
|
+
├─ :step_a
|
|
39
|
+
├─ [Parallel]
|
|
40
|
+
│ ├─ :step_b
|
|
41
|
+
│ └─ :step_c
|
|
42
|
+
└─ :step_d
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
============================================================
|
|
46
|
+
Example 2: E-commerce Order Processing Graph
|
|
47
|
+
============================================================
|
|
48
|
+
|
|
49
|
+
Dependency Graph
|
|
50
|
+
============================================================
|
|
51
|
+
|
|
52
|
+
Dependencies:
|
|
53
|
+
:validate_order
|
|
54
|
+
└─ depends on: (none)
|
|
55
|
+
:check_inventory
|
|
56
|
+
└─ depends on: :validate_order
|
|
57
|
+
:calculate_shipping
|
|
58
|
+
└─ depends on: :validate_order
|
|
59
|
+
:calculate_totals
|
|
60
|
+
└─ depends on: :calculate_shipping, :check_inventory
|
|
61
|
+
:process_payment
|
|
62
|
+
└─ depends on: :calculate_totals
|
|
63
|
+
:reserve_inventory
|
|
64
|
+
└─ depends on: :process_payment
|
|
65
|
+
:create_shipment
|
|
66
|
+
└─ depends on: :reserve_inventory
|
|
67
|
+
:send_email
|
|
68
|
+
└─ depends on: :create_shipment
|
|
69
|
+
:send_sms
|
|
70
|
+
└─ depends on: :create_shipment
|
|
71
|
+
:finalize_order
|
|
72
|
+
└─ depends on: :send_email, :send_sms
|
|
73
|
+
|
|
74
|
+
Execution Order (sequential):
|
|
75
|
+
:validate_order
|
|
76
|
+
↓ :check_inventory
|
|
77
|
+
↓ :calculate_shipping
|
|
78
|
+
↓ :calculate_totals
|
|
79
|
+
↓ :process_payment
|
|
80
|
+
↓ :reserve_inventory
|
|
81
|
+
↓ :create_shipment
|
|
82
|
+
↓ :send_email
|
|
83
|
+
↓ :send_sms
|
|
84
|
+
↓ :finalize_order
|
|
85
|
+
|
|
86
|
+
Parallel Execution Groups:
|
|
87
|
+
Group 1:
|
|
88
|
+
└─ :validate_order (sequential)
|
|
89
|
+
Group 2:
|
|
90
|
+
├─ Parallel execution of 2 steps:
|
|
91
|
+
├─ :calculate_shipping
|
|
92
|
+
└─ :check_inventory
|
|
93
|
+
Group 3:
|
|
94
|
+
└─ :calculate_totals (sequential)
|
|
95
|
+
Group 4:
|
|
96
|
+
└─ :process_payment (sequential)
|
|
97
|
+
Group 5:
|
|
98
|
+
└─ :reserve_inventory (sequential)
|
|
99
|
+
Group 6:
|
|
100
|
+
└─ :create_shipment (sequential)
|
|
101
|
+
Group 7:
|
|
102
|
+
├─ Parallel execution of 2 steps:
|
|
103
|
+
├─ :send_email
|
|
104
|
+
└─ :send_sms
|
|
105
|
+
Group 8:
|
|
106
|
+
└─ :finalize_order (sequential)
|
|
107
|
+
|
|
108
|
+
Execution Tree:
|
|
109
|
+
├─ :validate_order
|
|
110
|
+
├─ [Parallel]
|
|
111
|
+
│ ├─ :calculate_shipping
|
|
112
|
+
│ └─ :check_inventory
|
|
113
|
+
├─ :calculate_totals
|
|
114
|
+
├─ :process_payment
|
|
115
|
+
├─ :reserve_inventory
|
|
116
|
+
├─ :create_shipment
|
|
117
|
+
├─ [Parallel]
|
|
118
|
+
│ ├─ :send_email
|
|
119
|
+
│ └─ :send_sms
|
|
120
|
+
└─ :finalize_order
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
============================================================
|
|
124
|
+
Example 3: Execution Plan
|
|
125
|
+
============================================================
|
|
126
|
+
|
|
127
|
+
Execution Plan
|
|
128
|
+
============================================================
|
|
129
|
+
|
|
130
|
+
Total Steps: 10
|
|
131
|
+
Execution Phases: 8
|
|
132
|
+
|
|
133
|
+
Phase 1:
|
|
134
|
+
→ Execute :validate_order
|
|
135
|
+
|
|
136
|
+
Phase 2:
|
|
137
|
+
⚡ Execute in parallel:
|
|
138
|
+
• :calculate_shipping (after :validate_order)
|
|
139
|
+
• :check_inventory (after :validate_order)
|
|
140
|
+
|
|
141
|
+
Phase 3:
|
|
142
|
+
→ Execute :calculate_totals
|
|
143
|
+
|
|
144
|
+
Phase 4:
|
|
145
|
+
→ Execute :process_payment
|
|
146
|
+
|
|
147
|
+
Phase 5:
|
|
148
|
+
→ Execute :reserve_inventory
|
|
149
|
+
|
|
150
|
+
Phase 6:
|
|
151
|
+
→ Execute :create_shipment
|
|
152
|
+
|
|
153
|
+
Phase 7:
|
|
154
|
+
⚡ Execute in parallel:
|
|
155
|
+
• :send_email (after :create_shipment)
|
|
156
|
+
• :send_sms (after :create_shipment)
|
|
157
|
+
|
|
158
|
+
Phase 8:
|
|
159
|
+
→ Execute :finalize_order
|
|
160
|
+
|
|
161
|
+
Performance Estimate:
|
|
162
|
+
Sequential execution: 10 time units
|
|
163
|
+
Parallel execution: 8 time units
|
|
164
|
+
Potential speedup: 1.25x
|
|
165
|
+
|
|
166
|
+
|
|
167
|
+
============================================================
|
|
168
|
+
Example 4: ETL Pipeline Graph
|
|
169
|
+
============================================================
|
|
170
|
+
|
|
171
|
+
Execution Plan
|
|
172
|
+
============================================================
|
|
173
|
+
|
|
174
|
+
Total Steps: 10
|
|
175
|
+
Execution Phases: 5
|
|
176
|
+
|
|
177
|
+
Phase 1:
|
|
178
|
+
⚡ Execute in parallel:
|
|
179
|
+
• :extract_orders (no dependencies)
|
|
180
|
+
• :extract_products (no dependencies)
|
|
181
|
+
• :extract_users (no dependencies)
|
|
182
|
+
|
|
183
|
+
Phase 2:
|
|
184
|
+
⚡ Execute in parallel:
|
|
185
|
+
• :transform_orders (after :extract_orders)
|
|
186
|
+
• :transform_products (after :extract_products)
|
|
187
|
+
• :transform_users (after :extract_users)
|
|
188
|
+
|
|
189
|
+
Phase 3:
|
|
190
|
+
⚡ Execute in parallel:
|
|
191
|
+
• :aggregate_category_stats (after :transform_products)
|
|
192
|
+
• :aggregate_user_stats (after :transform_orders, :transform_users)
|
|
193
|
+
|
|
194
|
+
Phase 4:
|
|
195
|
+
→ Execute :validate_data
|
|
196
|
+
|
|
197
|
+
Phase 5:
|
|
198
|
+
→ Execute :prepare_output
|
|
199
|
+
|
|
200
|
+
Performance Estimate:
|
|
201
|
+
Sequential execution: 10 time units
|
|
202
|
+
Parallel execution: 5 time units
|
|
203
|
+
Potential speedup: 2.0x
|
|
204
|
+
|
|
205
|
+
|
|
206
|
+
============================================================
|
|
207
|
+
Example 5: Exporting to Different Formats
|
|
208
|
+
============================================================
|
|
209
|
+
|
|
210
|
+
✓ Exported to Graphviz DOT format: ecommerce_graph.dot
|
|
211
|
+
To generate PNG: dot -Tpng ecommerce_graph.dot -o ecommerce_graph.png
|
|
212
|
+
To generate SVG: dot -Tsvg ecommerce_graph.dot -o ecommerce_graph.svg
|
|
213
|
+
|
|
214
|
+
✓ Exported to Mermaid format: ecommerce_graph.mmd
|
|
215
|
+
View at: https://mermaid.live/
|
|
216
|
+
|
|
217
|
+
✓ Exported to interactive HTML: ecommerce_graph.html
|
|
218
|
+
Open in browser to view interactive graph
|
|
219
|
+
|
|
220
|
+
|
|
221
|
+
Graphviz DOT Format Preview:
|
|
222
|
+
------------------------------------------------------------
|
|
223
|
+
digraph DependencyGraph {
|
|
224
|
+
rankdir=TB;
|
|
225
|
+
node [shape=box, style=rounded];
|
|
226
|
+
|
|
227
|
+
// Group 1
|
|
228
|
+
validate_order [fillcolor=lightblue, style="rounded,filled"];
|
|
229
|
+
// Group 2
|
|
230
|
+
calculate_shipping [fillcolor=lightgreen, style="rounded,filled"];
|
|
231
|
+
check_inventory [fillcolor=lightgreen, style="rounded,filled"];
|
|
232
|
+
// Group 3
|
|
233
|
+
calculate_totals [fillcolor=lightyellow, style="rounded,filled"];
|
|
234
|
+
// Group 4
|
|
235
|
+
process_payment [fillcolor=lightpink, style="rounded,filled"];
|
|
236
|
+
// Group 5
|
|
237
|
+
reserve_inventory [fillcolor=lightgray, style="rounded,filled"];
|
|
238
|
+
// Group 6
|
|
239
|
+
create_shipment [fillcolor=lightblue, style="rounded,filled"];
|
|
240
|
+
// Group 7
|
|
241
|
+
send_email [fillcolor=lightgreen, style="rounded,filled"];
|
|
242
|
+
send_sms [fillcolor=lightgreen, style="rounded,filled"];
|
|
243
|
+
... (truncated)
|
|
244
|
+
|
|
245
|
+
|
|
246
|
+
Mermaid Format Preview:
|
|
247
|
+
------------------------------------------------------------
|
|
248
|
+
graph TD
|
|
249
|
+
validate_order[validate_order]
|
|
250
|
+
validate_order[validate_order] --> check_inventory[check_inventory]
|
|
251
|
+
validate_order[validate_order] --> calculate_shipping[calculate_shipping]
|
|
252
|
+
calculate_shipping[calculate_shipping] --> calculate_totals[calculate_totals]
|
|
253
|
+
check_inventory[check_inventory] --> calculate_totals[calculate_totals]
|
|
254
|
+
calculate_totals[calculate_totals] --> process_payment[process_payment]
|
|
255
|
+
process_payment[process_payment] --> reserve_inventory[reserve_inventory]
|
|
256
|
+
reserve_inventory[reserve_inventory] --> create_shipment[create_shipment]
|
|
257
|
+
create_shipment[create_shipment] --> send_email[send_email]
|
|
258
|
+
create_shipment[create_shipment] --> send_sms[send_sms]
|
|
259
|
+
send_email[send_email] --> finalize_order[finalize_order]
|
|
260
|
+
send_sms[send_sms] --> finalize_order[finalize_order]
|
|
261
|
+
classDef group1 fill:#9f9
|
|
262
|
+
class calculate_shipping group1
|
|
263
|
+
... (truncated)
|
|
264
|
+
|
|
265
|
+
|
|
266
|
+
============================================================
|
|
267
|
+
Example 6: Visualizing a Pipeline Directly
|
|
268
|
+
============================================================
|
|
269
|
+
|
|
270
|
+
Pipeline Dependency Graph:
|
|
271
|
+
|
|
272
|
+
Dependency Graph
|
|
273
|
+
============================================================
|
|
274
|
+
|
|
275
|
+
Dependencies:
|
|
276
|
+
:fetch_config
|
|
277
|
+
└─ depends on: (none)
|
|
278
|
+
:load_data
|
|
279
|
+
└─ depends on: :fetch_config
|
|
280
|
+
:validate_schema
|
|
281
|
+
└─ depends on: :load_data
|
|
282
|
+
:enrich_data
|
|
283
|
+
└─ depends on: :load_data
|
|
284
|
+
:save_results
|
|
285
|
+
└─ depends on: :enrich_data, :validate_schema
|
|
286
|
+
|
|
287
|
+
Execution Order (sequential):
|
|
288
|
+
:fetch_config
|
|
289
|
+
↓ :load_data
|
|
290
|
+
↓ :validate_schema
|
|
291
|
+
↓ :enrich_data
|
|
292
|
+
↓ :save_results
|
|
293
|
+
|
|
294
|
+
Parallel Execution Groups:
|
|
295
|
+
Group 1:
|
|
296
|
+
└─ :fetch_config (sequential)
|
|
297
|
+
Group 2:
|
|
298
|
+
└─ :load_data (sequential)
|
|
299
|
+
Group 3:
|
|
300
|
+
├─ Parallel execution of 2 steps:
|
|
301
|
+
├─ :enrich_data
|
|
302
|
+
└─ :validate_schema
|
|
303
|
+
Group 4:
|
|
304
|
+
└─ :save_results (sequential)
|
|
305
|
+
|
|
306
|
+
Execution Tree:
|
|
307
|
+
├─ :fetch_config
|
|
308
|
+
├─ :load_data
|
|
309
|
+
├─ [Parallel]
|
|
310
|
+
│ ├─ :enrich_data
|
|
311
|
+
│ └─ :validate_schema
|
|
312
|
+
└─ :save_results
|
|
313
|
+
|
|
314
|
+
|
|
315
|
+
============================================================
|
|
316
|
+
Example 7: Graph Structure Comparison
|
|
317
|
+
============================================================
|
|
318
|
+
|
|
319
|
+
Linear Pipeline (Sequential):
|
|
320
|
+
Execution Plan
|
|
321
|
+
============================================================
|
|
322
|
+
|
|
323
|
+
Total Steps: 5
|
|
324
|
+
Execution Phases: 5
|
|
325
|
+
|
|
326
|
+
Phase 1:
|
|
327
|
+
→ Execute :step1
|
|
328
|
+
|
|
329
|
+
Phase 2:
|
|
330
|
+
→ Execute :step2
|
|
331
|
+
|
|
332
|
+
Phase 3:
|
|
333
|
+
→ Execute :step3
|
|
334
|
+
|
|
335
|
+
Phase 4:
|
|
336
|
+
→ Execute :step4
|
|
337
|
+
|
|
338
|
+
Phase 5:
|
|
339
|
+
→ Execute :step5
|
|
340
|
+
|
|
341
|
+
Performance Estimate:
|
|
342
|
+
Sequential execution: 5 time units
|
|
343
|
+
Parallel execution: 5 time units
|
|
344
|
+
Potential speedup: 1.0x
|
|
345
|
+
|
|
346
|
+
|
|
347
|
+
Fan-out/Fan-in Pipeline (Parallel):
|
|
348
|
+
Execution Plan
|
|
349
|
+
============================================================
|
|
350
|
+
|
|
351
|
+
Total Steps: 6
|
|
352
|
+
Execution Phases: 3
|
|
353
|
+
|
|
354
|
+
Phase 1:
|
|
355
|
+
→ Execute :start
|
|
356
|
+
|
|
357
|
+
Phase 2:
|
|
358
|
+
⚡ Execute in parallel:
|
|
359
|
+
• :task1 (after :start)
|
|
360
|
+
• :task2 (after :start)
|
|
361
|
+
• :task3 (after :start)
|
|
362
|
+
• :task4 (after :start)
|
|
363
|
+
|
|
364
|
+
Phase 3:
|
|
365
|
+
→ Execute :end
|
|
366
|
+
|
|
367
|
+
Performance Estimate:
|
|
368
|
+
Sequential execution: 6 time units
|
|
369
|
+
Parallel execution: 3 time units
|
|
370
|
+
Potential speedup: 2.0x
|
|
371
|
+
|
|
372
|
+
|
|
373
|
+
============================================================
|
|
374
|
+
Example 8: Graph Analytics
|
|
375
|
+
============================================================
|
|
376
|
+
|
|
377
|
+
Simple Graph:
|
|
378
|
+
Total steps: 4
|
|
379
|
+
Execution phases: 3
|
|
380
|
+
Max parallel steps: 2
|
|
381
|
+
Theoretical speedup: 1.33x
|
|
382
|
+
Parallelization ratio: 50.0%
|
|
383
|
+
|
|
384
|
+
E-commerce Graph:
|
|
385
|
+
Total steps: 10
|
|
386
|
+
Execution phases: 8
|
|
387
|
+
Max parallel steps: 2
|
|
388
|
+
Theoretical speedup: 1.25x
|
|
389
|
+
Parallelization ratio: 20.0%
|
|
390
|
+
|
|
391
|
+
ETL Graph:
|
|
392
|
+
Total steps: 10
|
|
393
|
+
Execution phases: 5
|
|
394
|
+
Max parallel steps: 3
|
|
395
|
+
Theoretical speedup: 2.0x
|
|
396
|
+
Parallelization ratio: 30.0%
|
|
397
|
+
|
|
398
|
+
Linear Graph:
|
|
399
|
+
Total steps: 5
|
|
400
|
+
Execution phases: 5
|
|
401
|
+
Max parallel steps: 1
|
|
402
|
+
Theoretical speedup: 1.0x
|
|
403
|
+
Parallelization ratio: 20.0%
|
|
404
|
+
|
|
405
|
+
Fan-out/Fan-in Graph:
|
|
406
|
+
Total steps: 6
|
|
407
|
+
Execution phases: 3
|
|
408
|
+
Max parallel steps: 4
|
|
409
|
+
Theoretical speedup: 2.0x
|
|
410
|
+
Parallelization ratio: 66.7%
|
|
411
|
+
|
|
412
|
+
============================================================
|
|
413
|
+
Graph visualization examples completed!
|
|
414
|
+
|
|
415
|
+
Generated files:
|
|
416
|
+
- ecommerce_graph.dot (Graphviz format)
|
|
417
|
+
- ecommerce_graph.mmd (Mermaid format)
|
|
418
|
+
- ecommerce_graph.html (Interactive HTML)
|
|
419
|
+
|
|
420
|
+
To generate images with Graphviz:
|
|
421
|
+
$ dot -Tpng ecommerce_graph.dot -o ecommerce_graph.png
|
|
422
|
+
$ dot -Tsvg ecommerce_graph.dot -o ecommerce_graph.svg
|
|
423
|
+
$ dot -Tpdf ecommerce_graph.dot -o ecommerce_graph.pdf
|
|
424
|
+
|
|
425
|
+
To view Mermaid diagram:
|
|
426
|
+
1. Visit https://mermaid.live/
|
|
427
|
+
2. Paste contents of ecommerce_graph.mmd
|
|
428
|
+
3. Or use Mermaid CLI: mmdc -i ecommerce_graph.mmd -o graph.png
|
|
429
|
+
============================================================
|