fractor 0.1.4 → 0.1.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.rubocop-https---raw-githubusercontent-com-riboseinc-oss-guides-main-ci-rubocop-yml +552 -0
- data/.rubocop.yml +14 -8
- data/.rubocop_todo.yml +284 -43
- data/README.adoc +111 -950
- data/docs/.lycheeignore +16 -0
- data/docs/Gemfile +24 -0
- data/docs/README.md +157 -0
- data/docs/_config.yml +151 -0
- data/docs/_features/error-handling.adoc +1192 -0
- data/docs/_features/index.adoc +80 -0
- data/docs/_features/monitoring.adoc +589 -0
- data/docs/_features/signal-handling.adoc +202 -0
- data/docs/_features/workflows.adoc +1235 -0
- data/docs/_guides/continuous-mode.adoc +736 -0
- data/docs/_guides/cookbook.adoc +1133 -0
- data/docs/_guides/index.adoc +55 -0
- data/docs/_guides/pipeline-mode.adoc +730 -0
- data/docs/_guides/troubleshooting.adoc +358 -0
- data/docs/_pages/architecture.adoc +1390 -0
- data/docs/_pages/core-concepts.adoc +1392 -0
- data/docs/_pages/design-principles.adoc +862 -0
- data/docs/_pages/getting-started.adoc +290 -0
- data/docs/_pages/installation.adoc +143 -0
- data/docs/_reference/api.adoc +1080 -0
- data/docs/_reference/error-reporting.adoc +670 -0
- data/docs/_reference/examples.adoc +181 -0
- data/docs/_reference/index.adoc +96 -0
- data/docs/_reference/troubleshooting.adoc +862 -0
- data/docs/_tutorials/complex-workflows.adoc +1022 -0
- data/docs/_tutorials/data-processing-pipeline.adoc +740 -0
- data/docs/_tutorials/first-application.adoc +384 -0
- data/docs/_tutorials/index.adoc +48 -0
- data/docs/_tutorials/long-running-services.adoc +931 -0
- data/docs/assets/images/favicon-16.png +0 -0
- data/docs/assets/images/favicon-32.png +0 -0
- data/docs/assets/images/favicon-48.png +0 -0
- data/docs/assets/images/favicon.ico +0 -0
- data/docs/assets/images/favicon.png +0 -0
- data/docs/assets/images/favicon.svg +45 -0
- data/docs/assets/images/fractor-icon.svg +49 -0
- data/docs/assets/images/fractor-logo.svg +61 -0
- data/docs/index.adoc +131 -0
- data/docs/lychee.toml +39 -0
- data/examples/api_aggregator/README.adoc +627 -0
- data/examples/api_aggregator/api_aggregator.rb +376 -0
- data/examples/auto_detection/README.adoc +407 -29
- data/examples/auto_detection/auto_detection.rb +9 -9
- data/examples/continuous_chat_common/message_protocol.rb +53 -0
- data/examples/continuous_chat_fractor/README.adoc +217 -0
- data/examples/continuous_chat_fractor/chat_client.rb +303 -0
- data/examples/continuous_chat_fractor/chat_common.rb +83 -0
- data/examples/continuous_chat_fractor/chat_server.rb +167 -0
- data/examples/continuous_chat_fractor/simulate.rb +345 -0
- data/examples/continuous_chat_server/README.adoc +135 -0
- data/examples/continuous_chat_server/chat_client.rb +303 -0
- data/examples/continuous_chat_server/chat_server.rb +359 -0
- data/examples/continuous_chat_server/simulate.rb +343 -0
- data/examples/error_reporting.rb +207 -0
- data/examples/file_processor/README.adoc +170 -0
- data/examples/file_processor/file_processor.rb +615 -0
- data/examples/file_processor/sample_files/invalid.csv +1 -0
- data/examples/file_processor/sample_files/orders.xml +24 -0
- data/examples/file_processor/sample_files/products.json +23 -0
- data/examples/file_processor/sample_files/users.csv +6 -0
- data/examples/hierarchical_hasher/README.adoc +629 -41
- data/examples/hierarchical_hasher/hierarchical_hasher.rb +12 -8
- data/examples/image_processor/README.adoc +610 -0
- data/examples/image_processor/image_processor.rb +349 -0
- data/examples/image_processor/processed_images/sample_10_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_1_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_2_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_3_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_4_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_5_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_6_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_7_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_8_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_9_processed.jpg.json +12 -0
- data/examples/image_processor/test_images/sample_1.png +1 -0
- data/examples/image_processor/test_images/sample_10.png +1 -0
- data/examples/image_processor/test_images/sample_2.png +1 -0
- data/examples/image_processor/test_images/sample_3.png +1 -0
- data/examples/image_processor/test_images/sample_4.png +1 -0
- data/examples/image_processor/test_images/sample_5.png +1 -0
- data/examples/image_processor/test_images/sample_6.png +1 -0
- data/examples/image_processor/test_images/sample_7.png +1 -0
- data/examples/image_processor/test_images/sample_8.png +1 -0
- data/examples/image_processor/test_images/sample_9.png +1 -0
- data/examples/log_analyzer/README.adoc +662 -0
- data/examples/log_analyzer/log_analyzer.rb +579 -0
- data/examples/log_analyzer/sample_logs/apache.log +20 -0
- data/examples/log_analyzer/sample_logs/json.log +15 -0
- data/examples/log_analyzer/sample_logs/nginx.log +15 -0
- data/examples/log_analyzer/sample_logs/rails.log +29 -0
- data/examples/multi_work_type/README.adoc +576 -26
- data/examples/multi_work_type/multi_work_type.rb +30 -29
- data/examples/performance_monitoring.rb +120 -0
- data/examples/pipeline_processing/README.adoc +740 -26
- data/examples/pipeline_processing/pipeline_processing.rb +16 -16
- data/examples/priority_work_example.rb +155 -0
- data/examples/producer_subscriber/README.adoc +889 -46
- data/examples/producer_subscriber/producer_subscriber.rb +20 -16
- data/examples/scatter_gather/README.adoc +829 -27
- data/examples/scatter_gather/scatter_gather.rb +29 -28
- data/examples/simple/README.adoc +347 -0
- data/examples/simple/sample.rb +5 -5
- data/examples/specialized_workers/README.adoc +622 -26
- data/examples/specialized_workers/specialized_workers.rb +88 -45
- data/examples/stream_processor/README.adoc +206 -0
- data/examples/stream_processor/stream_processor.rb +284 -0
- data/examples/web_scraper/README.adoc +625 -0
- data/examples/web_scraper/web_scraper.rb +285 -0
- data/examples/workflow/README.adoc +406 -0
- data/examples/workflow/circuit_breaker/README.adoc +360 -0
- data/examples/workflow/circuit_breaker/circuit_breaker_workflow.rb +225 -0
- data/examples/workflow/conditional/README.adoc +483 -0
- data/examples/workflow/conditional/conditional_workflow.rb +215 -0
- data/examples/workflow/dead_letter_queue/README.adoc +374 -0
- data/examples/workflow/dead_letter_queue/dead_letter_queue_workflow.rb +217 -0
- data/examples/workflow/fan_out/README.adoc +381 -0
- data/examples/workflow/fan_out/fan_out_workflow.rb +202 -0
- data/examples/workflow/retry/README.adoc +248 -0
- data/examples/workflow/retry/retry_workflow.rb +195 -0
- data/examples/workflow/simple_linear/README.adoc +267 -0
- data/examples/workflow/simple_linear/simple_linear_workflow.rb +175 -0
- data/examples/workflow/simplified/README.adoc +329 -0
- data/examples/workflow/simplified/simplified_workflow.rb +222 -0
- data/exe/fractor +10 -0
- data/lib/fractor/cli.rb +288 -0
- data/lib/fractor/configuration.rb +307 -0
- data/lib/fractor/continuous_server.rb +183 -0
- data/lib/fractor/error_formatter.rb +72 -0
- data/lib/fractor/error_report_generator.rb +152 -0
- data/lib/fractor/error_reporter.rb +244 -0
- data/lib/fractor/error_statistics.rb +147 -0
- data/lib/fractor/execution_tracer.rb +162 -0
- data/lib/fractor/logger.rb +230 -0
- data/lib/fractor/main_loop_handler.rb +406 -0
- data/lib/fractor/main_loop_handler3.rb +135 -0
- data/lib/fractor/main_loop_handler4.rb +299 -0
- data/lib/fractor/performance_metrics_collector.rb +181 -0
- data/lib/fractor/performance_monitor.rb +215 -0
- data/lib/fractor/performance_report_generator.rb +202 -0
- data/lib/fractor/priority_work.rb +93 -0
- data/lib/fractor/priority_work_queue.rb +189 -0
- data/lib/fractor/result_aggregator.rb +33 -1
- data/lib/fractor/shutdown_handler.rb +168 -0
- data/lib/fractor/signal_handler.rb +80 -0
- data/lib/fractor/supervisor.rb +430 -144
- data/lib/fractor/supervisor_logger.rb +88 -0
- data/lib/fractor/version.rb +1 -1
- data/lib/fractor/work.rb +12 -0
- data/lib/fractor/work_distribution_manager.rb +151 -0
- data/lib/fractor/work_queue.rb +88 -0
- data/lib/fractor/work_result.rb +181 -9
- data/lib/fractor/worker.rb +75 -1
- data/lib/fractor/workflow/builder.rb +210 -0
- data/lib/fractor/workflow/chain_builder.rb +169 -0
- data/lib/fractor/workflow/circuit_breaker.rb +183 -0
- data/lib/fractor/workflow/circuit_breaker_orchestrator.rb +208 -0
- data/lib/fractor/workflow/circuit_breaker_registry.rb +112 -0
- data/lib/fractor/workflow/dead_letter_queue.rb +334 -0
- data/lib/fractor/workflow/execution_hooks.rb +39 -0
- data/lib/fractor/workflow/execution_strategy.rb +225 -0
- data/lib/fractor/workflow/execution_trace.rb +134 -0
- data/lib/fractor/workflow/helpers.rb +191 -0
- data/lib/fractor/workflow/job.rb +290 -0
- data/lib/fractor/workflow/job_dependency_validator.rb +120 -0
- data/lib/fractor/workflow/logger.rb +110 -0
- data/lib/fractor/workflow/pre_execution_context.rb +193 -0
- data/lib/fractor/workflow/retry_config.rb +156 -0
- data/lib/fractor/workflow/retry_orchestrator.rb +184 -0
- data/lib/fractor/workflow/retry_strategy.rb +93 -0
- data/lib/fractor/workflow/structured_logger.rb +30 -0
- data/lib/fractor/workflow/type_compatibility_validator.rb +222 -0
- data/lib/fractor/workflow/visualizer.rb +211 -0
- data/lib/fractor/workflow/workflow_context.rb +132 -0
- data/lib/fractor/workflow/workflow_executor.rb +669 -0
- data/lib/fractor/workflow/workflow_result.rb +55 -0
- data/lib/fractor/workflow/workflow_validator.rb +295 -0
- data/lib/fractor/workflow.rb +333 -0
- data/lib/fractor/wrapped_ractor.rb +66 -91
- data/lib/fractor/wrapped_ractor3.rb +161 -0
- data/lib/fractor/wrapped_ractor4.rb +242 -0
- data/lib/fractor.rb +93 -3
- metadata +192 -6
- data/tests/sample.rb.bak +0 -309
- data/tests/sample_working.rb.bak +0 -209
data/README.adoc
CHANGED
|
@@ -1,1061 +1,222 @@
|
|
|
1
1
|
= Fractor: Function-driven Ractors framework
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
image:https://img.shields.io/gem/v/fractor.svg[RubyGems Version, link=https://rubygems.org/gems/fractor]
|
|
4
|
+
image:https://img.shields.io/github/license/metanorma/fractor.svg[License, link=https://github.com/metanorma/fractor/blob/main/LICENSE]
|
|
5
|
+
image:https://github.com/metanorma/fractor/actions/workflows/rake.yml/badge.svg[Build Status, link=https://github.com/metanorma/fractor/actions/workflows/rake.yml]
|
|
6
|
+
image:https://img.shields.io/badge/ruby-3.0%20%E2%86%92%204.0%2B-ruby.svg[Ruby 3.0 to 4.0+, link=https://www.ruby-lang.org]
|
|
5
7
|
|
|
6
|
-
|
|
8
|
+
Fractor is a lightweight Ruby framework for parallel processing using Ractors
|
|
9
|
+
(Ruby Actors).
|
|
7
10
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
across multiple Ractors (Ruby's actor-like concurrency model).
|
|
11
|
+
It provides a structured way to distribute computational work across multiple
|
|
12
|
+
Ractors with minimal boilerplate.
|
|
11
13
|
|
|
12
|
-
|
|
13
|
-
process it in parallel using Ractors, and aggregate the results, while
|
|
14
|
-
abstracting away much of the boilerplate code involved in Ractor management and
|
|
15
|
-
communication.
|
|
14
|
+
== Ruby version support
|
|
16
15
|
|
|
17
|
-
|
|
16
|
+
Fractor fully supports both **Ruby 3.x** and **Ruby 4.0+**:
|
|
18
17
|
|
|
19
|
-
|
|
18
|
+
* *Ruby 3.0+*: Uses `Ractor.yield` for message passing from workers
|
|
20
19
|
|
|
21
|
-
|
|
22
|
-
----
|
|
23
|
-
gem install fractor
|
|
24
|
-
----
|
|
20
|
+
* *Ruby 4.0+*: Uses `Ractor::Port` for more efficient communication patterns
|
|
25
21
|
|
|
26
|
-
|
|
22
|
+
Fractor automatically detects your Ruby version and uses the appropriate
|
|
23
|
+
internal implementation. The user-facing API is identical across versions --
|
|
24
|
+
write your code once, and Fractor handles the differences internally.
|
|
27
25
|
|
|
28
|
-
|
|
26
|
+
See
|
|
27
|
+
link:docs/_pages/architecture.adoc#ruby-version-compatibility[Architecture: Ruby Version Compatibility]
|
|
28
|
+
for details on the internal differences.
|
|
29
29
|
|
|
30
|
-
|
|
31
|
-
----
|
|
32
|
-
gem 'fractor'
|
|
33
|
-
----
|
|
30
|
+
== Quick start
|
|
34
31
|
|
|
35
|
-
|
|
32
|
+
=== Installation
|
|
36
33
|
|
|
37
34
|
[source,sh]
|
|
38
35
|
----
|
|
39
|
-
|
|
36
|
+
gem install fractor
|
|
40
37
|
----
|
|
41
38
|
|
|
39
|
+
See link:docs/_pages/installation.adoc[Installation Guide] for more options.
|
|
42
40
|
|
|
43
|
-
===
|
|
44
|
-
|
|
45
|
-
* *Function-driven:* You define the core processing logic by subclassing
|
|
46
|
-
`Fractor::Worker` and implementing the `process` method.
|
|
47
|
-
|
|
48
|
-
* *Parallel execution:* Work items are automatically distributed to available
|
|
49
|
-
worker Ractors for concurrent processing.
|
|
50
|
-
|
|
51
|
-
* *Result aggregation:* The framework collects both successful results and
|
|
52
|
-
errors from the workers.
|
|
53
|
-
|
|
54
|
-
* *Separation of concerns:* Keeps the framework logic (`fractor.rb`) separate
|
|
55
|
-
from the client's specific implementation (`sample.rb`).
|
|
56
|
-
|
|
57
|
-
== Scope
|
|
58
|
-
|
|
59
|
-
This document describes the design, implementation, and usage of the Fractor
|
|
60
|
-
framework. It provides detailed information about the framework's components,
|
|
61
|
-
their interactions, and how to use them to implement parallel processing in Ruby
|
|
62
|
-
applications.
|
|
63
|
-
|
|
64
|
-
[bibliography]
|
|
65
|
-
== Normative references
|
|
66
|
-
|
|
67
|
-
* [[[ruby-ractor,Ruby Ractor Documentation]]], https://docs.ruby-lang.org/en/master/Ractor.html
|
|
68
|
-
|
|
69
|
-
== Terms and definitions
|
|
70
|
-
|
|
71
|
-
=== ractor
|
|
72
|
-
|
|
73
|
-
concurrent programming abstraction in Ruby that enables parallel execution
|
|
74
|
-
with thread safety
|
|
75
|
-
|
|
76
|
-
[.source]
|
|
77
|
-
<<ruby>>
|
|
78
|
-
|
|
79
|
-
=== worker
|
|
80
|
-
|
|
81
|
-
component that processes work items to produce work results
|
|
82
|
-
|
|
83
|
-
=== work item
|
|
84
|
-
|
|
85
|
-
unit of computation to be processed by a ractor
|
|
86
|
-
|
|
87
|
-
=== work result
|
|
88
|
-
|
|
89
|
-
result of processing a work item, either successful or an error
|
|
90
|
-
|
|
91
|
-
=== work item class
|
|
92
|
-
|
|
93
|
-
class that represents a work item, typically subclassing `Fractor::Work`
|
|
94
|
-
|
|
95
|
-
=== worker class
|
|
96
|
-
|
|
97
|
-
class that represents a worker, typically subclassing `Fractor::Worker`
|
|
98
|
-
|
|
99
|
-
=== wrapped ractor
|
|
100
|
-
|
|
101
|
-
component that manages a single ractor and its associated worker
|
|
102
|
-
|
|
103
|
-
=== supervisor
|
|
104
|
-
|
|
105
|
-
component that manages the pool of workers and distributes work items
|
|
106
|
-
|
|
107
|
-
=== result aggregator
|
|
108
|
-
|
|
109
|
-
component that collects and organizes work results from workers
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
== Core components
|
|
115
|
-
|
|
116
|
-
=== General
|
|
117
|
-
|
|
118
|
-
The Fractor framework consists of the following main classes, all residing
|
|
119
|
-
within the `Fractor` module.
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
=== Fractor::Worker
|
|
123
|
-
|
|
124
|
-
The abstract base class for defining how work should be processed.
|
|
125
|
-
|
|
126
|
-
Client code must subclass this and implement the `process(work)` method.
|
|
127
|
-
|
|
128
|
-
The `process` method receives a `Fractor::Work` object (or a subclass) and
|
|
129
|
-
should return a `Fractor::WorkResult` object.
|
|
130
|
-
|
|
131
|
-
=== Fractor::Work
|
|
132
|
-
|
|
133
|
-
The abstract base class for representing a unit of work.
|
|
134
|
-
|
|
135
|
-
Typically holds the input data needed by the `Worker`.
|
|
136
|
-
|
|
137
|
-
Client code should subclass this to define specific types of work items.
|
|
138
|
-
|
|
139
|
-
=== Fractor::WorkResult
|
|
140
|
-
|
|
141
|
-
A container object returned by the `Worker#process` method.
|
|
142
|
-
|
|
143
|
-
Holds either the successful `:result` of the computation or an `:error`
|
|
144
|
-
message if processing failed.
|
|
145
|
-
|
|
146
|
-
Includes a reference back to the original `:work` item.
|
|
147
|
-
|
|
148
|
-
Provides a `success?` method.
|
|
149
|
-
|
|
150
|
-
=== Fractor::ResultAggregator
|
|
151
|
-
|
|
152
|
-
Collects and stores all `WorkResult` objects generated by the workers.
|
|
153
|
-
|
|
154
|
-
Separates results into `results` (successful) and `errors` arrays.
|
|
155
|
-
|
|
156
|
-
=== Fractor::WrappedRactor
|
|
157
|
-
|
|
158
|
-
Manages an individual Ruby `Ractor`.
|
|
159
|
-
|
|
160
|
-
Instantiates the client-provided `Worker` subclass within the Ractor.
|
|
161
|
-
|
|
162
|
-
Handles receiving `Work` items, calling the `Worker#process` method, and
|
|
163
|
-
yielding `WorkResult` objects (or errors) back to the `Supervisor`.
|
|
164
|
-
|
|
165
|
-
=== Fractor::Supervisor
|
|
166
|
-
|
|
167
|
-
The main orchestrator of the framework.
|
|
168
|
-
|
|
169
|
-
Initializes and manages a pool of `WrappedRactor` instances.
|
|
170
|
-
|
|
171
|
-
Manages a `work_queue` of input data.
|
|
172
|
-
|
|
173
|
-
Distributes work items (wrapped in the client's `Work` subclass) to available
|
|
174
|
-
Ractors.
|
|
175
|
-
|
|
176
|
-
Listens for results and errors from Ractors using `Ractor.select`.
|
|
177
|
-
|
|
178
|
-
Uses `ResultAggregator` to store outcomes.
|
|
179
|
-
|
|
180
|
-
Handles graceful shutdown on `SIGINT` (Ctrl+C).
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
== Quick start
|
|
185
|
-
|
|
186
|
-
=== General
|
|
187
|
-
|
|
188
|
-
This quick start guide shows the minimum steps needed to get a simple parallel
|
|
189
|
-
execution working with Fractor.
|
|
190
|
-
|
|
191
|
-
=== Step 1: Create a minimal Work class
|
|
192
|
-
|
|
193
|
-
The Work class represents a unit of work to be processed by a Worker. It
|
|
194
|
-
encapsulates the input data needed for processing.
|
|
41
|
+
=== 30-second example
|
|
195
42
|
|
|
196
43
|
[source,ruby]
|
|
197
44
|
----
|
|
198
45
|
require 'fractor'
|
|
199
46
|
|
|
47
|
+
# Define your work
|
|
200
48
|
class MyWork < Fractor::Work
|
|
201
|
-
# Store all properties in the input hash
|
|
202
49
|
def initialize(value)
|
|
203
50
|
super({ value: value })
|
|
204
51
|
end
|
|
205
52
|
|
|
206
|
-
# Accessor method for the stored value
|
|
207
53
|
def value
|
|
208
54
|
input[:value]
|
|
209
55
|
end
|
|
210
|
-
|
|
211
|
-
def to_s
|
|
212
|
-
"MyWork: #{value}"
|
|
213
|
-
end
|
|
214
56
|
end
|
|
215
|
-
----
|
|
216
|
-
|
|
217
|
-
A Work is instantiated with the input data it will process this way:
|
|
218
57
|
|
|
219
|
-
|
|
220
|
-
----
|
|
221
|
-
work_item = MyWork.new(42)
|
|
222
|
-
puts work_item.to_s # Output: MyWork: 42
|
|
223
|
-
----
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
=== Step 2: Create a minimal Worker class
|
|
227
|
-
|
|
228
|
-
The Worker class defines the processing logic for work items. Each Worker
|
|
229
|
-
instance runs within its own Ractor and processes Work objects sent to it.
|
|
230
|
-
|
|
231
|
-
It must implement the `process(work)` method, which takes a Work object as
|
|
232
|
-
input and returns a `Fractor::WorkResult` object.
|
|
233
|
-
|
|
234
|
-
The `process` method should handle both successful processing and error
|
|
235
|
-
conditions.
|
|
236
|
-
|
|
237
|
-
[source,ruby]
|
|
238
|
-
----
|
|
58
|
+
# Define your worker
|
|
239
59
|
class MyWorker < Fractor::Worker
|
|
240
60
|
def process(work)
|
|
241
|
-
|
|
242
|
-
result = work.input * 2
|
|
243
|
-
|
|
244
|
-
# Return a success result
|
|
61
|
+
result = work.value * 2
|
|
245
62
|
Fractor::WorkResult.new(result: result, work: work)
|
|
246
63
|
rescue => e
|
|
247
|
-
# Return an error result if something goes wrong
|
|
248
64
|
Fractor::WorkResult.new(error: e.message, work: work)
|
|
249
65
|
end
|
|
250
66
|
end
|
|
251
|
-
----
|
|
252
|
-
|
|
253
|
-
The `process` method can perform any computation you need. In this example, it
|
|
254
|
-
multiplies the input by 2. If an error occurs, it catches the exception and
|
|
255
|
-
returns an error result.
|
|
256
|
-
|
|
257
|
-
=== Step 3: Set up and run the Supervisor
|
|
258
|
-
|
|
259
|
-
The Supervisor class orchestrates the entire framework, managing worker Ractors,
|
|
260
|
-
distributing work, and collecting results.
|
|
261
|
-
|
|
262
|
-
It initializes pools of Ractors, each running an instance of a Worker
|
|
263
|
-
class. The Supervisor handles the communication between the main thread and
|
|
264
|
-
the Ractors, including sending work items and receiving results.
|
|
265
|
-
|
|
266
|
-
The Supervisor also manages the work queue and the ResultAggregator, which
|
|
267
|
-
collects and organizes all results from the workers.
|
|
268
|
-
|
|
269
|
-
To set up the Supervisor, you specify worker pools, each containing a Worker class
|
|
270
|
-
and optionally the number of workers to create. If you don't specify `num_workers`,
|
|
271
|
-
Fractor will automatically detect the number of available processors on your system
|
|
272
|
-
and use that value. You can create multiple worker pools with different worker types
|
|
273
|
-
to handle different kinds of work. Each worker pool can process any type of Work
|
|
274
|
-
object that inherits from Fractor::Work.
|
|
275
|
-
|
|
276
|
-
[source,ruby]
|
|
277
|
-
----
|
|
278
|
-
# Create the supervisor with auto-detected number of workers
|
|
279
|
-
supervisor = Fractor::Supervisor.new(
|
|
280
|
-
worker_pools: [
|
|
281
|
-
{ worker_class: MyWorker } # Number of workers auto-detected
|
|
282
|
-
]
|
|
283
|
-
)
|
|
284
67
|
|
|
285
|
-
#
|
|
68
|
+
# Create supervisor and process work
|
|
286
69
|
supervisor = Fractor::Supervisor.new(
|
|
287
|
-
worker_pools: [
|
|
288
|
-
{ worker_class: MyWorker, num_workers: 4 } # Explicitly use 4 workers
|
|
289
|
-
]
|
|
70
|
+
worker_pools: [{ worker_class: MyWorker }]
|
|
290
71
|
)
|
|
291
72
|
|
|
292
|
-
# Add individual work items (instances of Work subclasses)
|
|
293
|
-
supervisor.add_work_item(MyWork.new(1))
|
|
294
|
-
|
|
295
|
-
# Add multiple work items
|
|
296
73
|
supervisor.add_work_items([
|
|
74
|
+
MyWork.new(1),
|
|
297
75
|
MyWork.new(2),
|
|
298
|
-
MyWork.new(3)
|
|
299
|
-
MyWork.new(4),
|
|
300
|
-
MyWork.new(5)
|
|
301
|
-
])
|
|
302
|
-
|
|
303
|
-
# You can add different types of Work objects to the same supervisor
|
|
304
|
-
supervisor.add_work_items([
|
|
305
|
-
MyWork.new(6),
|
|
306
|
-
OtherWork.new("data")
|
|
76
|
+
MyWork.new(3)
|
|
307
77
|
])
|
|
308
78
|
|
|
309
|
-
# Run the processing
|
|
310
79
|
supervisor.run
|
|
311
80
|
|
|
312
|
-
# Access results
|
|
313
81
|
puts "Results: #{supervisor.results.results.map(&:result)}"
|
|
314
|
-
|
|
315
|
-
----
|
|
316
|
-
|
|
317
|
-
That's it! With these three simple steps, you have a working parallel processing
|
|
318
|
-
system using Fractor.
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
== Usage
|
|
322
|
-
|
|
323
|
-
=== Work class
|
|
324
|
-
|
|
325
|
-
==== Purpose and responsibilities
|
|
326
|
-
|
|
327
|
-
The `Fractor::Work` class represents a unit of work to be processed by a Worker.
|
|
328
|
-
Its primary responsibility is to encapsulate the input data needed for
|
|
329
|
-
processing.
|
|
330
|
-
|
|
331
|
-
==== Implementation requirements
|
|
332
|
-
|
|
333
|
-
At minimum, your Work subclass should:
|
|
334
|
-
|
|
335
|
-
. Inherit from `Fractor::Work`
|
|
336
|
-
. Pass the input data to the superclass constructor
|
|
337
|
-
|
|
338
|
-
[source,ruby]
|
|
339
|
-
----
|
|
340
|
-
class MyWork < Fractor::Work
|
|
341
|
-
def initialize(input)
|
|
342
|
-
super(input) # This stores input in @input
|
|
343
|
-
# Add any additional initialization if needed
|
|
344
|
-
end
|
|
345
|
-
end
|
|
346
|
-
----
|
|
347
|
-
|
|
348
|
-
==== Advanced usage
|
|
349
|
-
|
|
350
|
-
You can extend your Work class to include additional data or methods:
|
|
351
|
-
|
|
352
|
-
[source,ruby]
|
|
353
|
-
----
|
|
354
|
-
class ComplexWork < Fractor::Work
|
|
355
|
-
attr_reader :options
|
|
356
|
-
|
|
357
|
-
def initialize(input, options = {})
|
|
358
|
-
super(input)
|
|
359
|
-
@options = options
|
|
360
|
-
end
|
|
361
|
-
|
|
362
|
-
def high_priority?
|
|
363
|
-
@options[:priority] == :high
|
|
364
|
-
end
|
|
365
|
-
|
|
366
|
-
def to_s
|
|
367
|
-
"ComplexWork: #{@input} (#{@options[:priority]} priority)"
|
|
368
|
-
end
|
|
369
|
-
end
|
|
370
|
-
----
|
|
371
|
-
|
|
372
|
-
[TIP]
|
|
373
|
-
====
|
|
374
|
-
====
|
|
375
|
-
* Keep Work objects lightweight and serializable since they will be passed
|
|
376
|
-
between Ractors
|
|
377
|
-
* Implement a meaningful `to_s` method for better debugging
|
|
378
|
-
====
|
|
379
|
-
* Consider adding validation in the initializer to catch issues early
|
|
380
|
-
====
|
|
381
|
-
|
|
382
|
-
=== Worker class
|
|
383
|
-
|
|
384
|
-
==== Purpose and responsibilities
|
|
385
|
-
|
|
386
|
-
The `Fractor::Worker` class defines the processing logic for work items. Each
|
|
387
|
-
Worker instance runs within its own Ractor and processes Work objects sent to
|
|
388
|
-
it.
|
|
389
|
-
|
|
390
|
-
==== Implementation requirements
|
|
391
|
-
|
|
392
|
-
Your Worker subclass must:
|
|
393
|
-
|
|
394
|
-
. Inherit from `Fractor::Worker`
|
|
395
|
-
. Implement the `process(work)` method
|
|
396
|
-
. Return a `Fractor::WorkResult` object from the `process` method
|
|
397
|
-
. Handle both successful processing and error conditions
|
|
398
|
-
|
|
399
|
-
[source,ruby]
|
|
400
|
-
----
|
|
401
|
-
class MyWorker < Fractor::Worker
|
|
402
|
-
def process(work)
|
|
403
|
-
# Process the work
|
|
404
|
-
|
|
405
|
-
if work.input < 0
|
|
406
|
-
return Fractor::WorkResult.new(
|
|
407
|
-
error: "Cannot process negative numbers",
|
|
408
|
-
work: work
|
|
409
|
-
)
|
|
410
|
-
end
|
|
411
|
-
|
|
412
|
-
# Normal processing...
|
|
413
|
-
result = work.input * 2
|
|
414
|
-
|
|
415
|
-
# Return a WorkResult
|
|
416
|
-
Fractor::WorkResult.new(result: result, work: work)
|
|
417
|
-
end
|
|
418
|
-
end
|
|
419
|
-
----
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
==== Error handling
|
|
423
|
-
|
|
424
|
-
The Worker class should handle two types of errors.
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
===== Handled errors
|
|
428
|
-
|
|
429
|
-
These are expected error conditions that your code explicitly checks for.
|
|
430
|
-
|
|
431
|
-
[source,ruby]
|
|
432
|
-
----
|
|
433
|
-
def process(work)
|
|
434
|
-
if work.input < 0
|
|
435
|
-
return Fractor::WorkResult.new(
|
|
436
|
-
error: "Cannot process negative numbers",
|
|
437
|
-
work: work
|
|
438
|
-
)
|
|
439
|
-
end
|
|
440
|
-
|
|
441
|
-
# Normal processing...
|
|
442
|
-
Fractor::WorkResult.new(result: calculated_value, work: work)
|
|
443
|
-
end
|
|
82
|
+
# => Results: [2, 4, 6]
|
|
444
83
|
----
|
|
445
84
|
|
|
446
|
-
|
|
85
|
+
== Key features
|
|
447
86
|
|
|
448
|
-
|
|
449
|
-
|
|
87
|
+
* *Function-driven*: Define processing logic by subclassing `Fractor::Worker`
|
|
88
|
+
* *Parallel execution*: Work automatically distributed across Ractor workers
|
|
89
|
+
* *Two operating modes*:
|
|
90
|
+
** **Pipeline mode** for batch processing
|
|
91
|
+
** **Continuous mode** for long-running servers
|
|
92
|
+
* *Workflow system*: GitHub Actions-style declarative workflows
|
|
93
|
+
* *Error handling*: Retry logic, circuit breakers, dead letter queues, error reporting
|
|
94
|
+
* *Production-ready*: Signal handling, logging, monitoring, graceful shutdown
|
|
95
|
+
* *Performance tools*: Built-in monitoring, benchmarking, and error analytics
|
|
96
|
+
* *High-level primitives*: WorkQueue and ContinuousServer eliminate boilerplate
|
|
450
97
|
|
|
451
|
-
|
|
452
|
-
----
|
|
453
|
-
def process(work)
|
|
454
|
-
# Processing that might raise exceptions
|
|
455
|
-
result = complex_calculation(work.input)
|
|
456
|
-
|
|
457
|
-
Fractor::WorkResult.new(result: result, work: work)
|
|
458
|
-
rescue StandardError => e
|
|
459
|
-
# Catch and convert any unexpected exceptions to error results
|
|
460
|
-
Fractor::WorkResult.new(error: "An unexpected error occurred: #{e.message}", work: work)
|
|
461
|
-
end
|
|
462
|
-
----
|
|
463
|
-
|
|
464
|
-
[TIP]
|
|
465
|
-
====
|
|
466
|
-
* Keep the `process` method focused on a single responsibility
|
|
467
|
-
* Use meaningful error messages that help diagnose issues
|
|
468
|
-
* Consider adding logging within the `process` method for debugging
|
|
469
|
-
* Ensure all paths return a valid `WorkResult` object
|
|
470
|
-
====
|
|
98
|
+
== Documentation
|
|
471
99
|
|
|
472
|
-
===
|
|
100
|
+
=== Getting started
|
|
473
101
|
|
|
474
|
-
|
|
102
|
+
* link:docs/_pages/installation.adoc[Installation] - System requirements and installation methods
|
|
103
|
+
* link:docs/_pages/getting-started.adoc[Getting Started] - Quick start guides for both modes
|
|
104
|
+
* link:docs/_pages/core-concepts.adoc[Core Concepts] - Understanding Fractor components
|
|
475
105
|
|
|
476
|
-
|
|
477
|
-
result of processing or an error message, along with a reference to the original
|
|
478
|
-
work item.
|
|
106
|
+
=== Operating modes
|
|
479
107
|
|
|
480
|
-
|
|
108
|
+
* link:docs/_guides/pipeline-mode.adoc[Pipeline Mode] - Batch processing with predefined work
|
|
109
|
+
* link:docs/_guides/continuous-mode.adoc[Continuous Mode] - Long-running servers and streaming
|
|
481
110
|
|
|
482
|
-
|
|
111
|
+
=== Advanced features
|
|
483
112
|
|
|
484
|
-
[
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
----
|
|
113
|
+
* link:docs/_features/workflows.adoc[Workflows] - Declarative workflow system for complex pipelines
|
|
114
|
+
* link:docs/_features/error-handling.adoc[Error Handling] - Retry logic, circuit breakers, and dead letter queues
|
|
115
|
+
* link:docs/_features/monitoring.adoc[Monitoring] - Performance monitoring and metrics
|
|
116
|
+
* link:docs/_features/signal-handling.adoc[Signal Handling] - Process monitoring and graceful shutdown
|
|
489
117
|
|
|
490
|
-
|
|
118
|
+
=== Reference
|
|
491
119
|
|
|
492
|
-
[
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
Fractor::WorkResult.new(error: "Error message", work: work_object)
|
|
496
|
-
----
|
|
120
|
+
* link:docs/_reference/api.adoc[API Reference] - Complete API documentation
|
|
121
|
+
* link:docs/_reference/examples.adoc[Examples] - Complete examples for all patterns
|
|
122
|
+
* link:docs/_reference/troubleshooting.adoc[Troubleshooting] - Common issues and solutions
|
|
497
123
|
|
|
498
|
-
|
|
124
|
+
== Operating modes
|
|
499
125
|
|
|
500
|
-
|
|
126
|
+
Fractor supports two distinct modes:
|
|
501
127
|
|
|
502
|
-
[
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
# Handle successful result
|
|
506
|
-
processed_value = work_result.result
|
|
507
|
-
else
|
|
508
|
-
# Handle error
|
|
509
|
-
error_message = work_result.error
|
|
510
|
-
end
|
|
511
|
-
----
|
|
128
|
+
[cols="1,2,2",options="header"]
|
|
129
|
+
|===
|
|
130
|
+
|Mode |Best for |Example use cases
|
|
512
131
|
|
|
513
|
-
|
|
132
|
+
|*Pipeline Mode*
|
|
133
|
+
|Batch processing, one-time jobs
|
|
134
|
+
|File processing, ETL pipelines, data transformations
|
|
514
135
|
|
|
515
|
-
|
|
136
|
+
|*Continuous Mode*
|
|
137
|
+
|Long-running servers, streaming
|
|
138
|
+
|Chat servers, job processors, event streams
|
|
139
|
+
|===
|
|
516
140
|
|
|
517
|
-
[
|
|
518
|
-
----
|
|
519
|
-
original_work = work_result.work
|
|
520
|
-
input_value = original_work.input
|
|
521
|
-
----
|
|
141
|
+
See link:docs/getting-started.adoc#choosing-your-mode[Choosing Your Mode] for detailed guidance.
|
|
522
142
|
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
==== Purpose and responsibilities
|
|
526
|
-
|
|
527
|
-
The `Fractor::ResultAggregator` collects and organizes all results from the
|
|
528
|
-
workers, separating successful results from errors.
|
|
529
|
-
|
|
530
|
-
Completed work results may be order independent or order dependent.
|
|
531
|
-
|
|
532
|
-
* For order independent results, the results may be utilized (popped) as they
|
|
533
|
-
are received.
|
|
534
|
-
|
|
535
|
-
* For order dependent results, the results are aggregated in the order they
|
|
536
|
-
are received. The order of results is important for re-assembly or
|
|
537
|
-
further processing.
|
|
538
|
-
|
|
539
|
-
* For results that require aggregation, the `ResultsAggregator` is used to determine
|
|
540
|
-
whether the results are completed, which signify that all work items have
|
|
541
|
-
been processed and ready for further processing.
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
==== Accessing results
|
|
545
|
-
|
|
546
|
-
To access successful results:
|
|
547
|
-
|
|
548
|
-
[source,ruby]
|
|
549
|
-
----
|
|
550
|
-
# Get all successful results
|
|
551
|
-
successful_results = supervisor.results.results
|
|
552
|
-
|
|
553
|
-
# Extract just the result values
|
|
554
|
-
result_values = successful_results.map(&:result)
|
|
555
|
-
----
|
|
556
|
-
|
|
557
|
-
To access errors:
|
|
558
|
-
|
|
559
|
-
[source,ruby]
|
|
560
|
-
----
|
|
561
|
-
# Get all error results
|
|
562
|
-
error_results = supervisor.results.errors
|
|
563
|
-
|
|
564
|
-
# Extract error messages
|
|
565
|
-
error_messages = error_results.map(&:error)
|
|
566
|
-
|
|
567
|
-
# Get the work items that failed
|
|
568
|
-
failed_work_items = error_results.map(&:work)
|
|
569
|
-
----
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
[TIP]
|
|
573
|
-
====
|
|
574
|
-
* Check both successful results and errors after processing completes
|
|
575
|
-
* Consider implementing custom reporting based on the aggregated results
|
|
576
|
-
====
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
=== WrappedRactor class
|
|
580
|
-
|
|
581
|
-
==== Purpose and responsibilities
|
|
582
|
-
|
|
583
|
-
The `Fractor::WrappedRactor` class manages an individual Ruby Ractor, handling
|
|
584
|
-
the communication between the Supervisor and the Worker instance running inside
|
|
585
|
-
the Ractor.
|
|
586
|
-
|
|
587
|
-
==== Usage notes
|
|
588
|
-
|
|
589
|
-
This class is primarily used internally by the Supervisor, but understanding its
|
|
590
|
-
role helps with debugging:
|
|
591
|
-
|
|
592
|
-
* Each WrappedRactor creates and manages one Ractor
|
|
593
|
-
* The Worker instance lives inside the Ractor
|
|
594
|
-
* Work items are sent to the Ractor via the WrappedRactor's `send` method
|
|
595
|
-
* Results are yielded back to the Supervisor
|
|
596
|
-
|
|
597
|
-
==== Error propagation
|
|
598
|
-
|
|
599
|
-
The WrappedRactor handles error propagation in two ways:
|
|
600
|
-
|
|
601
|
-
. Errors from the Worker's `process` method are wrapped in a WorkResult and
|
|
602
|
-
yielded back
|
|
603
|
-
. Unexpected errors in the Ractor itself are caught and logged
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
=== Supervisor class
|
|
607
|
-
|
|
608
|
-
==== Purpose and responsibilities
|
|
609
|
-
|
|
610
|
-
The `Fractor::Supervisor` class orchestrates the entire framework, managing
|
|
611
|
-
worker Ractors, distributing work, and collecting results.
|
|
612
|
-
|
|
613
|
-
==== Configuration options
|
|
614
|
-
|
|
615
|
-
When creating a Supervisor, you can configure:
|
|
616
|
-
|
|
617
|
-
[source,ruby]
|
|
618
|
-
----
|
|
619
|
-
supervisor = Fractor::Supervisor.new(
|
|
620
|
-
worker_pools: [
|
|
621
|
-
# Pool 1 - for general data processing
|
|
622
|
-
{ worker_class: MyWorker, num_workers: 4 },
|
|
623
|
-
|
|
624
|
-
# Pool 2 - for specialized image processing
|
|
625
|
-
{ worker_class: ImageWorker, num_workers: 2 }
|
|
626
|
-
],
|
|
627
|
-
continuous_mode: false # Optional: Run in continuous mode (default: false)
|
|
628
|
-
)
|
|
629
|
-
----
|
|
630
|
-
|
|
631
|
-
==== Worker auto-detection
|
|
143
|
+
== Example applications
|
|
632
144
|
|
|
633
|
-
|
|
634
|
-
and uses that value when `num_workers` is not specified. This provides optimal
|
|
635
|
-
resource utilization across different deployment environments without requiring
|
|
636
|
-
manual configuration.
|
|
145
|
+
=== Pipeline mode
|
|
637
146
|
|
|
638
147
|
[source,ruby]
|
|
639
148
|
----
|
|
640
|
-
# Auto-detect number of workers (recommended for most cases)
|
|
641
|
-
supervisor = Fractor::Supervisor.new(
|
|
642
|
-
worker_pools: [
|
|
643
|
-
{ worker_class: MyWorker } # Will use number of available processors
|
|
644
|
-
]
|
|
645
|
-
)
|
|
646
|
-
|
|
647
|
-
# Explicitly set number of workers (useful for specific requirements)
|
|
648
|
-
supervisor = Fractor::Supervisor.new(
|
|
649
|
-
worker_pools: [
|
|
650
|
-
{ worker_class: MyWorker, num_workers: 4 } # Always use exactly 4 workers
|
|
651
|
-
]
|
|
652
|
-
)
|
|
653
|
-
|
|
654
|
-
# Mix auto-detection and explicit configuration
|
|
655
149
|
supervisor = Fractor::Supervisor.new(
|
|
656
|
-
worker_pools: [
|
|
657
|
-
{ worker_class: FastWorker }, # Auto-detected
|
|
658
|
-
{ worker_class: HeavyWorker, num_workers: 2 } # Explicitly 2 workers
|
|
659
|
-
]
|
|
150
|
+
worker_pools: [{ worker_class: DataWorker }]
|
|
660
151
|
)
|
|
661
|
-
----
|
|
662
|
-
|
|
663
|
-
The auto-detection uses Ruby's `Etc.nprocessors` which returns the number of
|
|
664
|
-
available processors. If detection fails for any reason, it falls back to 2
|
|
665
|
-
workers.
|
|
666
|
-
|
|
667
|
-
[TIP]
|
|
668
|
-
* Use auto-detection for portable code that adapts to different environments
|
|
669
|
-
* Explicitly set `num_workers` when you need precise control over resource usage
|
|
670
|
-
* Consider system load and other factors when choosing explicit values
|
|
671
|
-
|
|
672
|
-
==== Adding work
|
|
673
|
-
|
|
674
|
-
You can add work items individually or in batches:
|
|
675
|
-
|
|
676
|
-
[source,ruby]
|
|
677
|
-
----
|
|
678
|
-
# Add a single item
|
|
679
|
-
supervisor.add_work_item(MyWork.new(42))
|
|
680
|
-
|
|
681
|
-
# Add multiple items
|
|
682
|
-
supervisor.add_work_items([
|
|
683
|
-
MyWork.new(1),
|
|
684
|
-
MyWork.new(2),
|
|
685
|
-
MyWork.new(3),
|
|
686
|
-
MyWork.new(4),
|
|
687
|
-
MyWork.new(5)
|
|
688
|
-
])
|
|
689
|
-
|
|
690
|
-
# Add items of different work types
|
|
691
|
-
supervisor.add_work_items([
|
|
692
|
-
TextWork.new("Process this text"),
|
|
693
|
-
ImageWork.new({ width: 800, height: 600 })
|
|
694
|
-
])
|
|
695
|
-
----
|
|
696
|
-
|
|
697
|
-
The Supervisor can handle any Work object that inherits from Fractor::Work.
|
|
698
|
-
Workers must check the type of Work they receive and process it accordingly.
|
|
699
|
-
|
|
700
|
-
==== Running and monitoring
|
|
701
152
|
|
|
702
|
-
|
|
703
|
-
|
|
704
|
-
[source,ruby]
|
|
705
|
-
----
|
|
706
|
-
# Start processing and block until complete
|
|
153
|
+
supervisor.add_work_items(dataset.map { |item| DataWork.new(item) })
|
|
707
154
|
supervisor.run
|
|
708
|
-
----
|
|
709
|
-
|
|
710
|
-
The Supervisor automatically handles:
|
|
711
|
-
|
|
712
|
-
* Starting the worker Ractors
|
|
713
|
-
* Distributing work items to available workers
|
|
714
|
-
* Collecting results and errors
|
|
715
|
-
* Graceful shutdown on completion or interruption (Ctrl+C)
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
==== Accessing results
|
|
719
|
-
|
|
720
|
-
After processing completes:
|
|
721
|
-
|
|
722
|
-
[source,ruby]
|
|
723
|
-
----
|
|
724
|
-
# Get the ResultAggregator
|
|
725
|
-
aggregator = supervisor.results
|
|
726
155
|
|
|
727
|
-
#
|
|
728
|
-
puts "Processed #{aggregator.results.size} items successfully"
|
|
729
|
-
puts "Encountered #{aggregator.errors.size} errors"
|
|
730
|
-
|
|
731
|
-
# Access successful results
|
|
732
|
-
aggregator.results.each do |result|
|
|
733
|
-
puts "Work item #{result.work.input} produced #{result.result}"
|
|
734
|
-
end
|
|
735
|
-
|
|
736
|
-
# Access errors
|
|
737
|
-
aggregator.errors.each do |error_result|
|
|
738
|
-
puts "Work item #{error_result.work.input} failed: #{error_result.error}"
|
|
739
|
-
end
|
|
156
|
+
puts "Processed #{supervisor.results.results.size} items"
|
|
740
157
|
----
|
|
741
158
|
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
=== Custom work distribution
|
|
159
|
+
See link:examples/simple/[Simple Example] and link:docs/examples.adoc#pipeline-mode-examples[more examples].
|
|
745
160
|
|
|
746
|
-
|
|
161
|
+
=== Continuous mode
|
|
747
162
|
|
|
748
163
|
[source,ruby]
|
|
749
164
|
----
|
|
750
|
-
|
|
751
|
-
high_priority_works = high_priority_items.map { |item| MyWork.new(item) }
|
|
752
|
-
|
|
753
|
-
# Add high-priority items first
|
|
754
|
-
supervisor.add_work_items(high_priority_works)
|
|
755
|
-
|
|
756
|
-
# Run with just enough workers for high-priority items
|
|
757
|
-
supervisor.run
|
|
758
|
-
|
|
759
|
-
# Create Work objects for lower priority items
|
|
760
|
-
low_priority_works = low_priority_items.map { |item| MyWork.new(item) }
|
|
761
|
-
|
|
762
|
-
# Add and process lower-priority items
|
|
763
|
-
supervisor.add_work_items(low_priority_works)
|
|
764
|
-
supervisor.run
|
|
765
|
-
----
|
|
165
|
+
work_queue = Fractor::WorkQueue.new
|
|
766
166
|
|
|
767
|
-
|
|
768
|
-
|
|
769
|
-
|
|
770
|
-
|
|
771
|
-
[source,ruby]
|
|
772
|
-
----
|
|
773
|
-
large_dataset.each_slice(1000) do |batch|
|
|
774
|
-
# Convert batch items to Work objects
|
|
775
|
-
work_batch = batch.map { |item| MyWork.new(item) }
|
|
776
|
-
|
|
777
|
-
supervisor.add_work_items(work_batch)
|
|
778
|
-
supervisor.run
|
|
779
|
-
|
|
780
|
-
# Process this batch's results before continuing
|
|
781
|
-
process_batch_results(supervisor.results)
|
|
782
|
-
end
|
|
783
|
-
----
|
|
784
|
-
|
|
785
|
-
|
|
786
|
-
== Running a basic example
|
|
787
|
-
|
|
788
|
-
. Install the gem as described in the Installation section.
|
|
789
|
-
|
|
790
|
-
. Create a new Ruby file (e.g., `my_fractor_example.rb`) with your
|
|
791
|
-
implementation:
|
|
792
|
-
|
|
793
|
-
[source,ruby]
|
|
794
|
-
----
|
|
795
|
-
require 'fractor'
|
|
796
|
-
|
|
797
|
-
# Define your Work class
|
|
798
|
-
class MyWork < Fractor::Work
|
|
799
|
-
def to_s
|
|
800
|
-
"MyWork: #{@input}"
|
|
801
|
-
end
|
|
802
|
-
end
|
|
803
|
-
|
|
804
|
-
# Define your Worker class
|
|
805
|
-
class MyWorker < Fractor::Worker
|
|
806
|
-
def process(work)
|
|
807
|
-
if work.input == 5
|
|
808
|
-
# Return a Fractor::WorkResult for errors
|
|
809
|
-
return Fractor::WorkResult.new(error: "Error processing work #{work.input}", work: work)
|
|
810
|
-
end
|
|
811
|
-
|
|
812
|
-
calculated = work.input * 2
|
|
813
|
-
# Return a Fractor::WorkResult for success
|
|
814
|
-
Fractor::WorkResult.new(result: calculated, work: work)
|
|
815
|
-
end
|
|
816
|
-
end
|
|
817
|
-
|
|
818
|
-
# Create supervisor with a worker pool
|
|
819
|
-
supervisor = Fractor::Supervisor.new(
|
|
820
|
-
worker_pools: [
|
|
821
|
-
{ worker_class: MyWorker, num_workers: 2 }
|
|
822
|
-
]
|
|
167
|
+
server = Fractor::ContinuousServer.new(
|
|
168
|
+
worker_pools: [{ worker_class: MessageWorker, num_workers: 4 }],
|
|
169
|
+
work_queue: work_queue
|
|
823
170
|
)
|
|
824
171
|
|
|
825
|
-
|
|
826
|
-
|
|
827
|
-
|
|
828
|
-
# Add work items
|
|
829
|
-
supervisor.add_work_items(work_items)
|
|
830
|
-
|
|
831
|
-
# Run processing
|
|
832
|
-
supervisor.run
|
|
833
|
-
|
|
834
|
-
# Display results
|
|
835
|
-
puts "Results: #{supervisor.results.results.map(&:result).join(', ')}"
|
|
836
|
-
puts "Errors: #{supervisor.results.errors.map { |e| e.work.input }.join(', ')}"
|
|
837
|
-
----
|
|
172
|
+
server.on_result { |result| puts "Processed: #{result.result}" }
|
|
173
|
+
server.on_error { |error| puts "Error: #{error.error}" }
|
|
838
174
|
|
|
839
|
-
.
|
|
175
|
+
Thread.new { server.run }
|
|
840
176
|
|
|
841
|
-
|
|
177
|
+
# Add work dynamically
|
|
178
|
+
work_queue << MessageWork.new(client_id: 1, message: "Hello")
|
|
842
179
|
----
|
|
843
|
-
ruby my_fractor_example.rb
|
|
844
|
-
----
|
|
845
|
-
|
|
846
|
-
You will see output showing Ractors starting, receiving work, processing it, and
|
|
847
|
-
the final aggregated results, including any errors encountered. Press `Ctrl+C`
|
|
848
|
-
during execution to test the graceful shutdown.
|
|
849
|
-
|
|
850
180
|
|
|
851
|
-
|
|
181
|
+
See link:examples/continuous_chat_fractor/[Chat Server Example] and link:docs/examples.adoc#continuous-mode-examples[more examples].
|
|
852
182
|
|
|
853
|
-
===
|
|
854
|
-
|
|
855
|
-
Fractor provides a powerful feature called "continuous mode" that allows
|
|
856
|
-
supervisors to run indefinitely, processing work items as they arrive without
|
|
857
|
-
stopping after the initial work queue is empty.
|
|
858
|
-
|
|
859
|
-
=== Features
|
|
860
|
-
|
|
861
|
-
* *Non-stopping Execution*: Supervisors run indefinitely until explicitly stopped
|
|
862
|
-
* *On-demand Work*: Workers only process work when it's available
|
|
863
|
-
* *Resource Efficiency*: Workers idle when no work is available, without consuming excessive resources
|
|
864
|
-
* *Dynamic Work Addition*: New work can be added at any time through the work source callback
|
|
865
|
-
* *Graceful Shutdown*: Resources are properly cleaned up when the supervisor is stopped
|
|
866
|
-
|
|
867
|
-
Continuous mode is particularly useful for:
|
|
868
|
-
|
|
869
|
-
* *Chat servers*: Processing incoming messages as they arrive
|
|
870
|
-
* *Background job processors*: Handling tasks from a job queue
|
|
871
|
-
* *Real-time data processing*: Analyzing data streams as they come in
|
|
872
|
-
* *Web servers*: Responding to incoming requests in parallel
|
|
873
|
-
* *Monitoring systems*: Continuously checking system statuses
|
|
874
|
-
|
|
875
|
-
See the Chat Server example in the examples directory for a complete implementation of continuous mode.
|
|
876
|
-
|
|
877
|
-
|
|
878
|
-
=== Using continuous mode
|
|
879
|
-
|
|
880
|
-
==== Step 1. Create a supervisor with the `continuous_mode: true` option
|
|
183
|
+
=== Workflows
|
|
881
184
|
|
|
882
185
|
[source,ruby]
|
|
883
186
|
----
|
|
884
|
-
|
|
885
|
-
|
|
886
|
-
|
|
887
|
-
|
|
888
|
-
|
|
889
|
-
)
|
|
890
|
-
----
|
|
891
|
-
|
|
892
|
-
==== Step 2. Register a work source callback that provides new work on demand
|
|
893
|
-
|
|
894
|
-
[source,ruby]
|
|
895
|
-
----
|
|
896
|
-
supervisor.register_work_source do
|
|
897
|
-
# Return nil or empty array if no work is available
|
|
898
|
-
# Return a work item or array of work items when available
|
|
899
|
-
items = get_next_work_items
|
|
900
|
-
if items && !items.empty?
|
|
901
|
-
# Convert to Work objects if needed
|
|
902
|
-
items.map { |item| MyWork.new(item) }
|
|
903
|
-
else
|
|
904
|
-
nil
|
|
905
|
-
end
|
|
187
|
+
# Define workflow with simplified syntax
|
|
188
|
+
workflow = Fractor::Workflow.define("data-pipeline") do
|
|
189
|
+
job :extract, ExtractWorker
|
|
190
|
+
job :transform, TransformWorker, needs: :extract
|
|
191
|
+
job :load, LoadWorker, needs: :transform
|
|
906
192
|
end
|
|
907
|
-
----
|
|
908
|
-
|
|
909
|
-
==== Step 4. Run the supervisor in a non-blocking way
|
|
910
|
-
|
|
911
|
-
Typically in a background thread.
|
|
912
|
-
|
|
913
|
-
[source,ruby]
|
|
914
|
-
----
|
|
915
|
-
supervisor_thread = Thread.new { supervisor.run }
|
|
916
|
-
----
|
|
917
193
|
|
|
918
|
-
|
|
919
|
-
|
|
920
|
-
[source,ruby]
|
|
921
|
-
----
|
|
922
|
-
supervisor.stop
|
|
923
|
-
supervisor_thread.join # Wait for the supervisor thread to finish
|
|
194
|
+
# Execute workflow
|
|
195
|
+
result = workflow.new.execute(input_data)
|
|
924
196
|
----
|
|
925
197
|
|
|
198
|
+
See link:examples/workflow/simplified/README.adoc[Simplified Workflows] and link:docs/workflows.adoc[Workflow Guide].
|
|
926
199
|
|
|
200
|
+
== Production deployment
|
|
927
201
|
|
|
928
|
-
|
|
929
|
-
|
|
930
|
-
=== General
|
|
931
|
-
|
|
932
|
-
The Fractor gem comes with several example applications that demonstrate various
|
|
933
|
-
patterns and use cases. Each example can be found in the `examples` directory of
|
|
934
|
-
the gem repository. Detailed descriptions for these are provided below.
|
|
935
|
-
|
|
936
|
-
=== Simple example
|
|
937
|
-
|
|
938
|
-
The Simple Example (link:examples/simple/[examples/simple/]) demonstrates the
|
|
939
|
-
basic usage of the Fractor framework. It shows how to create a simple Work
|
|
940
|
-
class, a Worker class, and a Supervisor to manage the processing of work items
|
|
941
|
-
in parallel. This example serves as a starting point for understanding how to
|
|
942
|
-
use Fractor.
|
|
943
|
-
|
|
944
|
-
Key features:
|
|
945
|
-
|
|
946
|
-
* Basic Work and Worker class implementation
|
|
947
|
-
* Simple Supervisor setup
|
|
948
|
-
* Parallel processing of work items
|
|
949
|
-
* Error handling and result aggregation
|
|
950
|
-
* Auto-detection of available processors
|
|
951
|
-
* Graceful shutdown on completion
|
|
952
|
-
|
|
953
|
-
=== Auto-detection example
|
|
954
|
-
|
|
955
|
-
The Auto-Detection Example (link:examples/auto_detection/[examples/auto_detection/])
|
|
956
|
-
demonstrates Fractor's automatic worker detection feature. It shows how to use
|
|
957
|
-
auto-detection, explicit configuration, and mixed approaches for controlling
|
|
958
|
-
the number of workers.
|
|
202
|
+
Fractor includes production-ready features:
|
|
959
203
|
|
|
960
|
-
|
|
204
|
+
* **Signal handling**: SIGTERM, SIGINT, SIGUSR1/SIGBREAK
|
|
205
|
+
* **Graceful shutdown**: Complete in-progress work before exit
|
|
206
|
+
* **Process monitoring**: Runtime status via signals
|
|
207
|
+
* **Structured logging**: JSON logging with correlation IDs
|
|
208
|
+
* **Workflow visualization**: Mermaid, DOT, ASCII diagrams
|
|
961
209
|
|
|
962
|
-
|
|
963
|
-
* Comparison of auto-detection vs explicit configuration
|
|
964
|
-
* Mixed configuration with multiple worker pools
|
|
965
|
-
* Best practices for worker configuration
|
|
966
|
-
* Portable code that adapts to different environments
|
|
210
|
+
See link:docs/signal-handling.adoc[Signal Handling Guide] for deployment patterns.
|
|
967
211
|
|
|
968
|
-
|
|
212
|
+
== Contributing
|
|
969
213
|
|
|
970
|
-
|
|
971
|
-
(link:examples/hierarchical_hasher/[examples/hierarchical_hasher/]) demonstrates
|
|
972
|
-
how to use the Fractor framework to process a file in parallel by breaking it
|
|
973
|
-
into chunks, hashing each chunk independently, and then combining the results
|
|
974
|
-
into a final hash. This approach is useful for processing large files
|
|
975
|
-
efficiently.
|
|
214
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/metanorma/fractor.
|
|
976
215
|
|
|
977
|
-
|
|
216
|
+
== License
|
|
978
217
|
|
|
979
|
-
|
|
980
|
-
* Independent processing of data segments
|
|
981
|
-
* Aggregation of results to form a final output
|
|
218
|
+
The gem is available as open source under the terms of the Ribose BSD 2-Clause License.
|
|
982
219
|
|
|
983
|
-
|
|
984
|
-
|
|
985
|
-
The Multi-Work Type example
|
|
986
|
-
(link:examples/multi_work_type/[examples/multi_work_type/]) demonstrates how a
|
|
987
|
-
single Fractor supervisor and worker can handle multiple types of work items
|
|
988
|
-
(e.g., `TextWork` and `ImageWork`). The worker intelligently adapts its
|
|
989
|
-
processing strategy based on the class of the incoming work item.
|
|
990
|
-
|
|
991
|
-
Key features:
|
|
992
|
-
|
|
993
|
-
* Support for multiple `Fractor::Work` subclasses
|
|
994
|
-
* Polymorphic worker processing based on work type
|
|
995
|
-
* Unified workflow for diverse tasks
|
|
996
|
-
|
|
997
|
-
=== Pipeline processing
|
|
998
|
-
|
|
999
|
-
The Pipeline Processing example
|
|
1000
|
-
(link:examples/pipeline_processing/[examples/pipeline_processing/]) implements a
|
|
1001
|
-
multi-stage processing pipeline where data flows sequentially through a series
|
|
1002
|
-
of transformations. The output of one stage becomes the input for the next, and
|
|
1003
|
-
different stages can operate concurrently on different data items.
|
|
1004
|
-
|
|
1005
|
-
Key features:
|
|
1006
|
-
|
|
1007
|
-
* Sequential data flow through multiple processing stages
|
|
1008
|
-
* Concurrent execution of different pipeline stages
|
|
1009
|
-
* Data transformation at each step of the pipeline
|
|
1010
|
-
|
|
1011
|
-
=== Producer/subscriber
|
|
1012
|
-
|
|
1013
|
-
The Producer/Subscriber example
|
|
1014
|
-
(link:examples/producer_subscriber/[examples/producer_subscriber/]) showcases a
|
|
1015
|
-
multi-stage document processing system where initial work (processing a
|
|
1016
|
-
document) can generate additional sub-work items (processing sections of the
|
|
1017
|
-
document). This creates a hierarchical processing pattern.
|
|
1018
|
-
|
|
1019
|
-
Key features:
|
|
1020
|
-
|
|
1021
|
-
* Implementation of producer-consumer patterns
|
|
1022
|
-
* Dynamic generation of sub-work based on initial processing
|
|
1023
|
-
* Construction of hierarchical result structures
|
|
1024
|
-
|
|
1025
|
-
=== Scatter/gather
|
|
1026
|
-
|
|
1027
|
-
The Scatter/Gather example
|
|
1028
|
-
(link:examples/scatter_gather/[examples/scatter_gather/]) illustrates how a
|
|
1029
|
-
large task or dataset is broken down (scattered) into smaller, independent
|
|
1030
|
-
subtasks. These subtasks are processed in parallel by multiple workers, and
|
|
1031
|
-
their results are then collected (gathered) and combined to produce the final
|
|
1032
|
-
output.
|
|
1033
|
-
|
|
1034
|
-
Key features:
|
|
1035
|
-
|
|
1036
|
-
* Distribution of a large task into smaller, parallelizable subtasks
|
|
1037
|
-
* Concurrent processing of subtasks
|
|
1038
|
-
* Aggregation of partial results into a final result
|
|
1039
|
-
|
|
1040
|
-
=== Specialized workers
|
|
1041
|
-
|
|
1042
|
-
The Specialized Workers example
|
|
1043
|
-
(link:examples/specialized_workers/[examples/specialized_workers/]) demonstrates
|
|
1044
|
-
creating distinct worker types, each tailored to handle specific kinds of tasks
|
|
1045
|
-
(e.g., `ComputeWorker` for CPU-intensive operations and `DatabaseWorker` for
|
|
1046
|
-
I/O-bound database interactions). This allows for optimized resource utilization
|
|
1047
|
-
and domain-specific logic.
|
|
1048
|
-
|
|
1049
|
-
Key features:
|
|
1050
|
-
|
|
1051
|
-
* Creation of worker classes for specific processing domains
|
|
1052
|
-
* Routing of work items to appropriately specialized workers
|
|
1053
|
-
* Optimization of resources and logic per task type
|
|
1054
|
-
|
|
1055
|
-
|
|
1056
|
-
|
|
1057
|
-
== Copyright and license
|
|
220
|
+
== Copyright
|
|
1058
221
|
|
|
1059
222
|
Copyright Ribose.
|
|
1060
|
-
|
|
1061
|
-
Licensed under the MIT License.
|