fractor 0.1.6 → 0.1.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.rubocop_todo.yml +227 -102
- data/README.adoc +113 -1940
- data/docs/.lycheeignore +16 -0
- data/docs/Gemfile +24 -0
- data/docs/README.md +157 -0
- data/docs/_config.yml +151 -0
- data/docs/_features/error-handling.adoc +1192 -0
- data/docs/_features/index.adoc +80 -0
- data/docs/_features/monitoring.adoc +589 -0
- data/docs/_features/signal-handling.adoc +202 -0
- data/docs/_features/workflows.adoc +1235 -0
- data/docs/_guides/continuous-mode.adoc +736 -0
- data/docs/_guides/cookbook.adoc +1133 -0
- data/docs/_guides/index.adoc +55 -0
- data/docs/_guides/pipeline-mode.adoc +730 -0
- data/docs/_guides/troubleshooting.adoc +358 -0
- data/docs/_pages/architecture.adoc +1390 -0
- data/docs/_pages/core-concepts.adoc +1392 -0
- data/docs/_pages/design-principles.adoc +862 -0
- data/docs/_pages/getting-started.adoc +290 -0
- data/docs/_pages/installation.adoc +143 -0
- data/docs/_reference/api.adoc +1080 -0
- data/docs/_reference/error-reporting.adoc +670 -0
- data/docs/_reference/examples.adoc +181 -0
- data/docs/_reference/index.adoc +96 -0
- data/docs/_reference/troubleshooting.adoc +862 -0
- data/docs/_tutorials/complex-workflows.adoc +1022 -0
- data/docs/_tutorials/data-processing-pipeline.adoc +740 -0
- data/docs/_tutorials/first-application.adoc +384 -0
- data/docs/_tutorials/index.adoc +48 -0
- data/docs/_tutorials/long-running-services.adoc +931 -0
- data/docs/assets/images/favicon-16.png +0 -0
- data/docs/assets/images/favicon-32.png +0 -0
- data/docs/assets/images/favicon-48.png +0 -0
- data/docs/assets/images/favicon.ico +0 -0
- data/docs/assets/images/favicon.png +0 -0
- data/docs/assets/images/favicon.svg +45 -0
- data/docs/assets/images/fractor-icon.svg +49 -0
- data/docs/assets/images/fractor-logo.svg +61 -0
- data/docs/index.adoc +131 -0
- data/docs/lychee.toml +39 -0
- data/examples/api_aggregator/README.adoc +627 -0
- data/examples/api_aggregator/api_aggregator.rb +376 -0
- data/examples/auto_detection/README.adoc +407 -29
- data/examples/continuous_chat_common/message_protocol.rb +1 -1
- data/examples/error_reporting.rb +207 -0
- data/examples/file_processor/README.adoc +170 -0
- data/examples/file_processor/file_processor.rb +615 -0
- data/examples/file_processor/sample_files/invalid.csv +1 -0
- data/examples/file_processor/sample_files/orders.xml +24 -0
- data/examples/file_processor/sample_files/products.json +23 -0
- data/examples/file_processor/sample_files/users.csv +6 -0
- data/examples/hierarchical_hasher/README.adoc +629 -41
- data/examples/image_processor/README.adoc +610 -0
- data/examples/image_processor/image_processor.rb +349 -0
- data/examples/image_processor/processed_images/sample_10_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_1_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_2_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_3_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_4_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_5_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_6_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_7_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_8_processed.jpg.json +12 -0
- data/examples/image_processor/processed_images/sample_9_processed.jpg.json +12 -0
- data/examples/image_processor/test_images/sample_1.png +1 -0
- data/examples/image_processor/test_images/sample_10.png +1 -0
- data/examples/image_processor/test_images/sample_2.png +1 -0
- data/examples/image_processor/test_images/sample_3.png +1 -0
- data/examples/image_processor/test_images/sample_4.png +1 -0
- data/examples/image_processor/test_images/sample_5.png +1 -0
- data/examples/image_processor/test_images/sample_6.png +1 -0
- data/examples/image_processor/test_images/sample_7.png +1 -0
- data/examples/image_processor/test_images/sample_8.png +1 -0
- data/examples/image_processor/test_images/sample_9.png +1 -0
- data/examples/log_analyzer/README.adoc +662 -0
- data/examples/log_analyzer/log_analyzer.rb +579 -0
- data/examples/log_analyzer/sample_logs/apache.log +20 -0
- data/examples/log_analyzer/sample_logs/json.log +15 -0
- data/examples/log_analyzer/sample_logs/nginx.log +15 -0
- data/examples/log_analyzer/sample_logs/rails.log +29 -0
- data/examples/multi_work_type/README.adoc +576 -26
- data/examples/performance_monitoring.rb +120 -0
- data/examples/pipeline_processing/README.adoc +740 -26
- data/examples/pipeline_processing/pipeline_processing.rb +2 -2
- data/examples/priority_work_example.rb +155 -0
- data/examples/producer_subscriber/README.adoc +889 -46
- data/examples/scatter_gather/README.adoc +829 -27
- data/examples/simple/README.adoc +347 -0
- data/examples/specialized_workers/README.adoc +622 -26
- data/examples/specialized_workers/specialized_workers.rb +44 -8
- data/examples/stream_processor/README.adoc +206 -0
- data/examples/stream_processor/stream_processor.rb +284 -0
- data/examples/web_scraper/README.adoc +625 -0
- data/examples/web_scraper/web_scraper.rb +285 -0
- data/examples/workflow/README.adoc +406 -0
- data/examples/workflow/circuit_breaker/README.adoc +360 -0
- data/examples/workflow/circuit_breaker/circuit_breaker_workflow.rb +225 -0
- data/examples/workflow/conditional/README.adoc +483 -0
- data/examples/workflow/conditional/conditional_workflow.rb +215 -0
- data/examples/workflow/dead_letter_queue/README.adoc +374 -0
- data/examples/workflow/dead_letter_queue/dead_letter_queue_workflow.rb +217 -0
- data/examples/workflow/fan_out/README.adoc +381 -0
- data/examples/workflow/fan_out/fan_out_workflow.rb +202 -0
- data/examples/workflow/retry/README.adoc +248 -0
- data/examples/workflow/retry/retry_workflow.rb +195 -0
- data/examples/workflow/simple_linear/README.adoc +267 -0
- data/examples/workflow/simple_linear/simple_linear_workflow.rb +175 -0
- data/examples/workflow/simplified/README.adoc +329 -0
- data/examples/workflow/simplified/simplified_workflow.rb +222 -0
- data/exe/fractor +10 -0
- data/lib/fractor/cli.rb +288 -0
- data/lib/fractor/configuration.rb +307 -0
- data/lib/fractor/continuous_server.rb +60 -65
- data/lib/fractor/error_formatter.rb +72 -0
- data/lib/fractor/error_report_generator.rb +152 -0
- data/lib/fractor/error_reporter.rb +244 -0
- data/lib/fractor/error_statistics.rb +147 -0
- data/lib/fractor/execution_tracer.rb +162 -0
- data/lib/fractor/logger.rb +230 -0
- data/lib/fractor/main_loop_handler.rb +406 -0
- data/lib/fractor/main_loop_handler3.rb +135 -0
- data/lib/fractor/main_loop_handler4.rb +299 -0
- data/lib/fractor/performance_metrics_collector.rb +181 -0
- data/lib/fractor/performance_monitor.rb +215 -0
- data/lib/fractor/performance_report_generator.rb +202 -0
- data/lib/fractor/priority_work.rb +93 -0
- data/lib/fractor/priority_work_queue.rb +189 -0
- data/lib/fractor/result_aggregator.rb +32 -0
- data/lib/fractor/shutdown_handler.rb +168 -0
- data/lib/fractor/signal_handler.rb +80 -0
- data/lib/fractor/supervisor.rb +382 -269
- data/lib/fractor/supervisor_logger.rb +88 -0
- data/lib/fractor/version.rb +1 -1
- data/lib/fractor/work.rb +12 -0
- data/lib/fractor/work_distribution_manager.rb +151 -0
- data/lib/fractor/work_queue.rb +20 -0
- data/lib/fractor/work_result.rb +181 -9
- data/lib/fractor/worker.rb +73 -0
- data/lib/fractor/workflow/builder.rb +210 -0
- data/lib/fractor/workflow/chain_builder.rb +169 -0
- data/lib/fractor/workflow/circuit_breaker.rb +183 -0
- data/lib/fractor/workflow/circuit_breaker_orchestrator.rb +208 -0
- data/lib/fractor/workflow/circuit_breaker_registry.rb +112 -0
- data/lib/fractor/workflow/dead_letter_queue.rb +334 -0
- data/lib/fractor/workflow/execution_hooks.rb +39 -0
- data/lib/fractor/workflow/execution_strategy.rb +225 -0
- data/lib/fractor/workflow/execution_trace.rb +134 -0
- data/lib/fractor/workflow/helpers.rb +191 -0
- data/lib/fractor/workflow/job.rb +290 -0
- data/lib/fractor/workflow/job_dependency_validator.rb +120 -0
- data/lib/fractor/workflow/logger.rb +110 -0
- data/lib/fractor/workflow/pre_execution_context.rb +193 -0
- data/lib/fractor/workflow/retry_config.rb +156 -0
- data/lib/fractor/workflow/retry_orchestrator.rb +184 -0
- data/lib/fractor/workflow/retry_strategy.rb +93 -0
- data/lib/fractor/workflow/structured_logger.rb +30 -0
- data/lib/fractor/workflow/type_compatibility_validator.rb +222 -0
- data/lib/fractor/workflow/visualizer.rb +211 -0
- data/lib/fractor/workflow/workflow_context.rb +132 -0
- data/lib/fractor/workflow/workflow_executor.rb +669 -0
- data/lib/fractor/workflow/workflow_result.rb +55 -0
- data/lib/fractor/workflow/workflow_validator.rb +295 -0
- data/lib/fractor/workflow.rb +333 -0
- data/lib/fractor/wrapped_ractor.rb +66 -101
- data/lib/fractor/wrapped_ractor3.rb +161 -0
- data/lib/fractor/wrapped_ractor4.rb +242 -0
- data/lib/fractor.rb +92 -4
- metadata +179 -6
- data/tests/sample.rb.bak +0 -309
- data/tests/sample_working.rb.bak +0 -209
data/README.adoc
CHANGED
|
@@ -1,2049 +1,222 @@
|
|
|
1
1
|
= Fractor: Function-driven Ractors framework
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
image:https://img.shields.io/gem/v/fractor.svg[RubyGems Version, link=https://rubygems.org/gems/fractor]
|
|
4
|
+
image:https://img.shields.io/github/license/metanorma/fractor.svg[License, link=https://github.com/metanorma/fractor/blob/main/LICENSE]
|
|
5
|
+
image:https://github.com/metanorma/fractor/actions/workflows/rake.yml/badge.svg[Build Status, link=https://github.com/metanorma/fractor/actions/workflows/rake.yml]
|
|
6
|
+
image:https://img.shields.io/badge/ruby-3.0%20%E2%86%92%204.0%2B-ruby.svg[Ruby 3.0 to 4.0+, link=https://www.ruby-lang.org]
|
|
5
7
|
|
|
6
|
-
|
|
8
|
+
Fractor is a lightweight Ruby framework for parallel processing using Ractors
|
|
9
|
+
(Ruby Actors).
|
|
7
10
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
work across multiple Ractors (Ruby's actor-like concurrency model).
|
|
11
|
+
It provides a structured way to distribute computational work across multiple
|
|
12
|
+
Ractors with minimal boilerplate.
|
|
11
13
|
|
|
12
|
-
|
|
13
|
-
process it in parallel using Ractors, and aggregate the results, while
|
|
14
|
-
abstracting away much of the boilerplate code involved in Ractor management and
|
|
15
|
-
communication.
|
|
14
|
+
== Ruby version support
|
|
16
15
|
|
|
17
|
-
|
|
16
|
+
Fractor fully supports both **Ruby 3.x** and **Ruby 4.0+**:
|
|
18
17
|
|
|
19
|
-
|
|
18
|
+
* *Ruby 3.0+*: Uses `Ractor.yield` for message passing from workers
|
|
20
19
|
|
|
21
|
-
|
|
22
|
-
----
|
|
23
|
-
gem install fractor
|
|
24
|
-
----
|
|
20
|
+
* *Ruby 4.0+*: Uses `Ractor::Port` for more efficient communication patterns
|
|
25
21
|
|
|
26
|
-
|
|
22
|
+
Fractor automatically detects your Ruby version and uses the appropriate
|
|
23
|
+
internal implementation. The user-facing API is identical across versions --
|
|
24
|
+
write your code once, and Fractor handles the differences internally.
|
|
27
25
|
|
|
28
|
-
|
|
26
|
+
See
|
|
27
|
+
link:docs/_pages/architecture.adoc#ruby-version-compatibility[Architecture: Ruby Version Compatibility]
|
|
28
|
+
for details on the internal differences.
|
|
29
29
|
|
|
30
|
-
|
|
31
|
-
----
|
|
32
|
-
gem 'fractor'
|
|
33
|
-
----
|
|
30
|
+
== Quick start
|
|
34
31
|
|
|
35
|
-
|
|
32
|
+
=== Installation
|
|
36
33
|
|
|
37
34
|
[source,sh]
|
|
38
35
|
----
|
|
39
|
-
|
|
36
|
+
gem install fractor
|
|
40
37
|
----
|
|
41
38
|
|
|
39
|
+
See link:docs/_pages/installation.adoc[Installation Guide] for more options.
|
|
42
40
|
|
|
43
|
-
===
|
|
44
|
-
|
|
45
|
-
* *Function-driven:* You define the core processing logic by subclassing
|
|
46
|
-
`Fractor::Worker` and implementing the `process` method.
|
|
47
|
-
|
|
48
|
-
* *Parallel execution:* Work items are automatically distributed to available
|
|
49
|
-
worker Ractors for concurrent processing.
|
|
50
|
-
|
|
51
|
-
* *Result aggregation:* The framework collects both successful results and
|
|
52
|
-
errors from the workers.
|
|
53
|
-
|
|
54
|
-
* *Separation of concerns:* Keeps the framework logic (`fractor.rb`) separate
|
|
55
|
-
from the client's specific implementation (`sample.rb`).
|
|
56
|
-
|
|
57
|
-
== Scope
|
|
58
|
-
|
|
59
|
-
This document describes the design, implementation, and usage of the Fractor
|
|
60
|
-
framework. It provides detailed information about the framework's components,
|
|
61
|
-
their interactions, and how to use them to implement parallel processing in Ruby
|
|
62
|
-
applications.
|
|
63
|
-
|
|
64
|
-
[bibliography]
|
|
65
|
-
== Normative references
|
|
66
|
-
|
|
67
|
-
* [[[ruby-ractor,Ruby Ractor Documentation]]], https://docs.ruby-lang.org/en/master/Ractor.html
|
|
68
|
-
|
|
69
|
-
== Terms and definitions
|
|
70
|
-
|
|
71
|
-
=== ractor
|
|
72
|
-
|
|
73
|
-
concurrent programming abstraction in Ruby that enables parallel execution
|
|
74
|
-
with thread safety
|
|
75
|
-
|
|
76
|
-
[.source]
|
|
77
|
-
<<ruby>>
|
|
78
|
-
|
|
79
|
-
=== worker
|
|
80
|
-
|
|
81
|
-
component that processes work items to produce work results
|
|
82
|
-
|
|
83
|
-
=== work item
|
|
84
|
-
|
|
85
|
-
unit of computation to be processed by a ractor
|
|
86
|
-
|
|
87
|
-
=== work result
|
|
88
|
-
|
|
89
|
-
result of processing a work item, either successful or an error
|
|
90
|
-
|
|
91
|
-
=== work item class
|
|
92
|
-
|
|
93
|
-
class that represents a work item, typically subclassing `Fractor::Work`
|
|
94
|
-
|
|
95
|
-
=== worker class
|
|
96
|
-
|
|
97
|
-
class that represents a worker, typically subclassing `Fractor::Worker`
|
|
98
|
-
|
|
99
|
-
=== wrapped ractor
|
|
100
|
-
|
|
101
|
-
component that manages a single ractor and its associated worker
|
|
102
|
-
|
|
103
|
-
=== supervisor
|
|
104
|
-
|
|
105
|
-
component that manages the pool of workers and distributes work items
|
|
106
|
-
|
|
107
|
-
=== result aggregator
|
|
108
|
-
|
|
109
|
-
component that collects and organizes work results from workers
|
|
110
|
-
|
|
111
|
-
=== pipeline mode
|
|
112
|
-
|
|
113
|
-
operating mode where Fractor processes a defined set of work items and then
|
|
114
|
-
stops
|
|
115
|
-
|
|
116
|
-
=== continuous mode
|
|
117
|
-
|
|
118
|
-
operating mode where Fractor runs indefinitely, processing work items as they
|
|
119
|
-
arrive
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
== Understanding Fractor operating modes
|
|
125
|
-
|
|
126
|
-
=== General
|
|
127
|
-
|
|
128
|
-
Fractor supports two distinct operating modes, each optimized for different use
|
|
129
|
-
cases. Understanding these modes is essential for choosing the right approach
|
|
130
|
-
for your application.
|
|
131
|
-
|
|
132
|
-
=== Pipeline mode (batch processing)
|
|
133
|
-
|
|
134
|
-
Pipeline mode is designed for processing a defined set of work items with a
|
|
135
|
-
clear beginning and end.
|
|
136
|
-
|
|
137
|
-
Characteristics:
|
|
138
|
-
|
|
139
|
-
* Processes a predetermined batch of work items
|
|
140
|
-
* Stops automatically when all work is completed
|
|
141
|
-
* Results are collected and accessed after processing completes
|
|
142
|
-
* Ideal for one-time computations or periodic batch jobs
|
|
143
|
-
|
|
144
|
-
Common use cases:
|
|
145
|
-
|
|
146
|
-
* Processing a file or dataset
|
|
147
|
-
* Batch data transformations
|
|
148
|
-
* One-time parallel computations
|
|
149
|
-
* Scheduled batch jobs
|
|
150
|
-
* Hierarchical or multi-stage processing
|
|
151
|
-
|
|
152
|
-
=== Continuous mode (long-running servers)
|
|
153
|
-
|
|
154
|
-
Continuous mode is designed for applications that need to run indefinitely,
|
|
155
|
-
processing work items as they arrive.
|
|
156
|
-
|
|
157
|
-
Characteristics:
|
|
158
|
-
|
|
159
|
-
* Runs continuously without a predetermined end
|
|
160
|
-
* Processes work items dynamically as they become available
|
|
161
|
-
* Workers idle efficiently when no work is available
|
|
162
|
-
* Results are processed via callbacks, not batch collection
|
|
163
|
-
* Supports graceful shutdown and runtime monitoring
|
|
164
|
-
|
|
165
|
-
Common use cases:
|
|
166
|
-
|
|
167
|
-
* Chat servers and messaging systems
|
|
168
|
-
* Background job processors
|
|
169
|
-
* Real-time data stream processing
|
|
170
|
-
* Web servers handling concurrent requests
|
|
171
|
-
* Monitoring and alerting systems
|
|
172
|
-
* Event-driven architectures
|
|
173
|
-
|
|
174
|
-
=== Comparison
|
|
175
|
-
|
|
176
|
-
[cols="1,2,2",options="header"]
|
|
177
|
-
|===
|
|
178
|
-
|Aspect |Pipeline Mode |Continuous Mode
|
|
179
|
-
|
|
180
|
-
|Duration
|
|
181
|
-
|Finite (stops when done)
|
|
182
|
-
|Indefinite (runs until stopped)
|
|
183
|
-
|
|
184
|
-
|Work arrival
|
|
185
|
-
|All work known upfront
|
|
186
|
-
|Work arrives dynamically
|
|
187
|
-
|
|
188
|
-
|Result handling
|
|
189
|
-
|Batch collection after completion
|
|
190
|
-
|Callback-based processing
|
|
191
|
-
|
|
192
|
-
|Typical lifetime
|
|
193
|
-
|Seconds to minutes
|
|
194
|
-
|Hours to days/weeks
|
|
195
|
-
|
|
196
|
-
|Shutdown
|
|
197
|
-
|Automatic on completion
|
|
198
|
-
|Manual or signal-based
|
|
199
|
-
|
|
200
|
-
|Best for
|
|
201
|
-
|Batch jobs, file processing
|
|
202
|
-
|Servers, streams, job queues
|
|
203
|
-
|===
|
|
204
|
-
|
|
205
|
-
=== Decision guide
|
|
206
|
-
|
|
207
|
-
Choose *Pipeline mode* when:
|
|
208
|
-
|
|
209
|
-
* You have a complete dataset to process
|
|
210
|
-
* Processing has a clear start and end
|
|
211
|
-
* You need all results aggregated after completion
|
|
212
|
-
* The task is one-time or scheduled periodically
|
|
213
|
-
|
|
214
|
-
Choose *Continuous mode* when:
|
|
215
|
-
|
|
216
|
-
* Work arrives over time from external sources
|
|
217
|
-
* Your application runs as a long-lived server
|
|
218
|
-
* You need to process items as they arrive
|
|
219
|
-
* Results should be handled immediately via callbacks
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
== Quick start: Pipeline mode
|
|
225
|
-
|
|
226
|
-
=== General
|
|
227
|
-
|
|
228
|
-
This quick start guide shows the minimum steps needed to get parallel batch
|
|
229
|
-
processing working with Fractor.
|
|
230
|
-
|
|
231
|
-
=== Step 1: Create a minimal Work class
|
|
232
|
-
|
|
233
|
-
The Work class represents a unit of work to be processed by a Worker. It
|
|
234
|
-
encapsulates the input data needed for processing.
|
|
41
|
+
=== 30-second example
|
|
235
42
|
|
|
236
43
|
[source,ruby]
|
|
237
44
|
----
|
|
238
45
|
require 'fractor'
|
|
239
46
|
|
|
47
|
+
# Define your work
|
|
240
48
|
class MyWork < Fractor::Work
|
|
241
|
-
# Store all properties in the input hash
|
|
242
49
|
def initialize(value)
|
|
243
50
|
super({ value: value })
|
|
244
51
|
end
|
|
245
52
|
|
|
246
|
-
# Accessor method for the stored value
|
|
247
53
|
def value
|
|
248
54
|
input[:value]
|
|
249
55
|
end
|
|
250
|
-
|
|
251
|
-
def to_s
|
|
252
|
-
"MyWork: #{value}"
|
|
253
|
-
end
|
|
254
56
|
end
|
|
255
|
-
----
|
|
256
|
-
|
|
257
|
-
A Work is instantiated with the input data it will process this way:
|
|
258
|
-
|
|
259
|
-
[source,ruby]
|
|
260
|
-
----
|
|
261
|
-
work_item = MyWork.new(42)
|
|
262
|
-
puts work_item.to_s # Output: MyWork: 42
|
|
263
|
-
----
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
=== Step 2: Create a minimal Worker class
|
|
267
57
|
|
|
268
|
-
|
|
269
|
-
instance runs within its own Ractor and processes Work objects sent to it.
|
|
270
|
-
|
|
271
|
-
It must implement the `process(work)` method, which takes a Work object as
|
|
272
|
-
input and returns a `Fractor::WorkResult` object.
|
|
273
|
-
|
|
274
|
-
The `process` method should handle both successful processing and error
|
|
275
|
-
conditions.
|
|
276
|
-
|
|
277
|
-
[source,ruby]
|
|
278
|
-
----
|
|
58
|
+
# Define your worker
|
|
279
59
|
class MyWorker < Fractor::Worker
|
|
280
60
|
def process(work)
|
|
281
|
-
|
|
282
|
-
result = work.input * 2
|
|
283
|
-
|
|
284
|
-
# Return a success result
|
|
61
|
+
result = work.value * 2
|
|
285
62
|
Fractor::WorkResult.new(result: result, work: work)
|
|
286
63
|
rescue => e
|
|
287
|
-
# Return an error result if something goes wrong
|
|
288
64
|
Fractor::WorkResult.new(error: e.message, work: work)
|
|
289
65
|
end
|
|
290
66
|
end
|
|
291
|
-
----
|
|
292
|
-
|
|
293
|
-
The `process` method can perform any computation you need. In this example, it
|
|
294
|
-
multiplies the input by 2. If an error occurs, it catches the exception and
|
|
295
|
-
returns an error result.
|
|
296
67
|
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
The Supervisor class orchestrates the entire framework, managing worker Ractors,
|
|
300
|
-
distributing work, and collecting results.
|
|
301
|
-
|
|
302
|
-
[source,ruby]
|
|
303
|
-
----
|
|
304
|
-
# Create the supervisor with auto-detected number of workers
|
|
68
|
+
# Create supervisor and process work
|
|
305
69
|
supervisor = Fractor::Supervisor.new(
|
|
306
|
-
worker_pools: [
|
|
307
|
-
{ worker_class: MyWorker } # Number of workers auto-detected
|
|
308
|
-
]
|
|
70
|
+
worker_pools: [{ worker_class: MyWorker }]
|
|
309
71
|
)
|
|
310
72
|
|
|
311
|
-
# Add work items (instances of Work subclasses)
|
|
312
73
|
supervisor.add_work_items([
|
|
313
74
|
MyWork.new(1),
|
|
314
75
|
MyWork.new(2),
|
|
315
|
-
MyWork.new(3)
|
|
316
|
-
MyWork.new(4),
|
|
317
|
-
MyWork.new(5)
|
|
76
|
+
MyWork.new(3)
|
|
318
77
|
])
|
|
319
78
|
|
|
320
|
-
# Run the processing
|
|
321
79
|
supervisor.run
|
|
322
80
|
|
|
323
|
-
# Access results after completion
|
|
324
81
|
puts "Results: #{supervisor.results.results.map(&:result)}"
|
|
325
|
-
|
|
82
|
+
# => Results: [2, 4, 6]
|
|
326
83
|
----
|
|
327
84
|
|
|
328
|
-
|
|
329
|
-
system using Fractor in pipeline mode.
|
|
85
|
+
== Key features
|
|
330
86
|
|
|
87
|
+
* *Function-driven*: Define processing logic by subclassing `Fractor::Worker`
|
|
88
|
+
* *Parallel execution*: Work automatically distributed across Ractor workers
|
|
89
|
+
* *Two operating modes*:
|
|
90
|
+
** **Pipeline mode** for batch processing
|
|
91
|
+
** **Continuous mode** for long-running servers
|
|
92
|
+
* *Workflow system*: GitHub Actions-style declarative workflows
|
|
93
|
+
* *Error handling*: Retry logic, circuit breakers, dead letter queues, error reporting
|
|
94
|
+
* *Production-ready*: Signal handling, logging, monitoring, graceful shutdown
|
|
95
|
+
* *Performance tools*: Built-in monitoring, benchmarking, and error analytics
|
|
96
|
+
* *High-level primitives*: WorkQueue and ContinuousServer eliminate boilerplate
|
|
331
97
|
|
|
98
|
+
== Documentation
|
|
332
99
|
|
|
100
|
+
=== Getting started
|
|
333
101
|
|
|
334
|
-
|
|
102
|
+
* link:docs/_pages/installation.adoc[Installation] - System requirements and installation methods
|
|
103
|
+
* link:docs/_pages/getting-started.adoc[Getting Started] - Quick start guides for both modes
|
|
104
|
+
* link:docs/_pages/core-concepts.adoc[Core Concepts] - Understanding Fractor components
|
|
335
105
|
|
|
336
|
-
===
|
|
106
|
+
=== Operating modes
|
|
337
107
|
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
code for thread management, queuing, and results processing.
|
|
108
|
+
* link:docs/_guides/pipeline-mode.adoc[Pipeline Mode] - Batch processing with predefined work
|
|
109
|
+
* link:docs/_guides/continuous-mode.adoc[Continuous Mode] - Long-running servers and streaming
|
|
341
110
|
|
|
342
|
-
===
|
|
111
|
+
=== Advanced features
|
|
343
112
|
|
|
344
|
-
|
|
113
|
+
* link:docs/_features/workflows.adoc[Workflows] - Declarative workflow system for complex pipelines
|
|
114
|
+
* link:docs/_features/error-handling.adoc[Error Handling] - Retry logic, circuit breakers, and dead letter queues
|
|
115
|
+
* link:docs/_features/monitoring.adoc[Monitoring] - Performance monitoring and metrics
|
|
116
|
+
* link:docs/_features/signal-handling.adoc[Signal Handling] - Process monitoring and graceful shutdown
|
|
345
117
|
|
|
346
|
-
|
|
347
|
-
----
|
|
348
|
-
require 'fractor'
|
|
349
|
-
|
|
350
|
-
class MessageWork < Fractor::Work
|
|
351
|
-
def initialize(client_id, message)
|
|
352
|
-
super({ client_id: client_id, message: message })
|
|
353
|
-
end
|
|
118
|
+
=== Reference
|
|
354
119
|
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
120
|
+
* link:docs/_reference/api.adoc[API Reference] - Complete API documentation
|
|
121
|
+
* link:docs/_reference/examples.adoc[Examples] - Complete examples for all patterns
|
|
122
|
+
* link:docs/_reference/troubleshooting.adoc[Troubleshooting] - Common issues and solutions
|
|
358
123
|
|
|
359
|
-
|
|
360
|
-
input[:message]
|
|
361
|
-
end
|
|
362
|
-
end
|
|
124
|
+
== Operating modes
|
|
363
125
|
|
|
364
|
-
|
|
365
|
-
def process(work)
|
|
366
|
-
# Process the message
|
|
367
|
-
processed = "Echo: #{work.message}"
|
|
126
|
+
Fractor supports two distinct modes:
|
|
368
127
|
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
)
|
|
373
|
-
rescue => e
|
|
374
|
-
Fractor::WorkResult.new(error: e.message, work: work)
|
|
375
|
-
end
|
|
376
|
-
end
|
|
377
|
-
----
|
|
128
|
+
[cols="1,2,2",options="header"]
|
|
129
|
+
|===
|
|
130
|
+
|Mode |Best for |Example use cases
|
|
378
131
|
|
|
379
|
-
|
|
132
|
+
|*Pipeline Mode*
|
|
133
|
+
|Batch processing, one-time jobs
|
|
134
|
+
|File processing, ETL pipelines, data transformations
|
|
380
135
|
|
|
381
|
-
|
|
136
|
+
|*Continuous Mode*
|
|
137
|
+
|Long-running servers, streaming
|
|
138
|
+
|Chat servers, job processors, event streams
|
|
139
|
+
|===
|
|
382
140
|
|
|
383
|
-
[
|
|
384
|
-
----
|
|
385
|
-
# Create a thread-safe work queue
|
|
386
|
-
work_queue = Fractor::WorkQueue.new
|
|
387
|
-
----
|
|
141
|
+
See link:docs/getting-started.adoc#choosing-your-mode[Choosing Your Mode] for detailed guidance.
|
|
388
142
|
|
|
389
|
-
|
|
143
|
+
== Example applications
|
|
390
144
|
|
|
391
|
-
|
|
392
|
-
handling, and results processing.
|
|
145
|
+
=== Pipeline mode
|
|
393
146
|
|
|
394
147
|
[source,ruby]
|
|
395
148
|
----
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
worker_pools: [
|
|
399
|
-
{ worker_class: MessageWorker, num_workers: 4 }
|
|
400
|
-
],
|
|
401
|
-
work_queue: work_queue, # Auto-registers as work source
|
|
402
|
-
log_file: 'logs/server.log' # Optional logging
|
|
149
|
+
supervisor = Fractor::Supervisor.new(
|
|
150
|
+
worker_pools: [{ worker_class: DataWorker }]
|
|
403
151
|
)
|
|
404
152
|
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
client_id = result.result[:client_id]
|
|
408
|
-
response = result.result[:response]
|
|
409
|
-
puts "Sending to client #{client_id}: #{response}"
|
|
410
|
-
# Send response to client here
|
|
411
|
-
end
|
|
412
|
-
|
|
413
|
-
# Define how to handle errors
|
|
414
|
-
server.on_error do |error_result|
|
|
415
|
-
puts "Error processing work: #{error_result.error}"
|
|
416
|
-
end
|
|
417
|
-
----
|
|
418
|
-
|
|
419
|
-
=== Step 4: Run and add work dynamically
|
|
420
|
-
|
|
421
|
-
Start the server and add work items as they arrive:
|
|
422
|
-
|
|
423
|
-
[source,ruby]
|
|
424
|
-
----
|
|
425
|
-
# Start the server in a background thread
|
|
426
|
-
server_thread = Thread.new { server.run }
|
|
427
|
-
|
|
428
|
-
# Your application can now push work items dynamically
|
|
429
|
-
# For example, when a client sends a message:
|
|
430
|
-
work_queue << MessageWork.new(client_id: 1, message: "Hello")
|
|
431
|
-
work_queue << MessageWork.new(client_id: 2, message: "World")
|
|
432
|
-
|
|
433
|
-
# The server runs indefinitely, processing work as it arrives
|
|
434
|
-
# Use Ctrl+C or send SIGTERM for graceful shutdown
|
|
435
|
-
|
|
436
|
-
# Or stop programmatically
|
|
437
|
-
sleep 10
|
|
438
|
-
server.stop
|
|
439
|
-
server_thread.join
|
|
440
|
-
----
|
|
441
|
-
|
|
442
|
-
That's it! The ContinuousServer handles all thread management, signal handling,
|
|
443
|
-
and graceful shutdown automatically.
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
== Core components
|
|
449
|
-
|
|
450
|
-
=== General
|
|
451
|
-
|
|
452
|
-
The Fractor framework consists of the following main classes, all residing
|
|
453
|
-
within the `Fractor` module. These core components are used by both pipeline
|
|
454
|
-
mode and continuous mode.
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
=== Fractor::Worker
|
|
458
|
-
|
|
459
|
-
The abstract base class for defining how work should be processed.
|
|
460
|
-
|
|
461
|
-
Client code must subclass this and implement the `process(work)` method.
|
|
462
|
-
|
|
463
|
-
The `process` method receives a `Fractor::Work` object (or a subclass) and
|
|
464
|
-
should return a `Fractor::WorkResult` object.
|
|
465
|
-
|
|
466
|
-
=== Fractor::Work
|
|
467
|
-
|
|
468
|
-
The abstract base class for representing a unit of work.
|
|
469
|
-
|
|
470
|
-
Typically holds the input data needed by the `Worker`.
|
|
471
|
-
|
|
472
|
-
Client code should subclass this to define specific types of work items.
|
|
473
|
-
|
|
474
|
-
=== Fractor::WorkResult
|
|
475
|
-
|
|
476
|
-
A container object returned by the `Worker#process` method.
|
|
477
|
-
|
|
478
|
-
Holds either the successful `:result` of the computation or an `:error`
|
|
479
|
-
message if processing failed.
|
|
480
|
-
|
|
481
|
-
Includes a reference back to the original `:work` item.
|
|
482
|
-
|
|
483
|
-
Provides a `success?` method.
|
|
484
|
-
|
|
485
|
-
=== Fractor::ResultAggregator
|
|
486
|
-
|
|
487
|
-
Collects and stores all `WorkResult` objects generated by the workers.
|
|
488
|
-
|
|
489
|
-
Separates results into `results` (successful) and `errors` arrays.
|
|
490
|
-
|
|
491
|
-
=== Fractor::WrappedRactor
|
|
492
|
-
|
|
493
|
-
Manages an individual Ruby `Ractor`.
|
|
494
|
-
|
|
495
|
-
Instantiates the client-provided `Worker` subclass within the Ractor.
|
|
496
|
-
|
|
497
|
-
Handles receiving `Work` items, calling the `Worker#process` method, and
|
|
498
|
-
yielding `WorkResult` objects (or errors) back to the `Supervisor`.
|
|
499
|
-
|
|
500
|
-
=== Fractor::Supervisor
|
|
501
|
-
|
|
502
|
-
The main orchestrator of the framework.
|
|
503
|
-
|
|
504
|
-
Initializes and manages a pool of `WrappedRactor` instances.
|
|
505
|
-
|
|
506
|
-
Manages a `work_queue` of input data.
|
|
507
|
-
|
|
508
|
-
Distributes work items (wrapped in the client's `Work` subclass) to available
|
|
509
|
-
Ractors.
|
|
510
|
-
|
|
511
|
-
Listens for results and errors from Ractors using `Ractor.select`.
|
|
512
|
-
|
|
513
|
-
Uses `ResultAggregator` to store outcomes.
|
|
514
|
-
|
|
515
|
-
Handles graceful shutdown on `SIGINT` (Ctrl+C).
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
== Pipeline mode components
|
|
521
|
-
|
|
522
|
-
=== General
|
|
523
|
-
|
|
524
|
-
This section describes the components and their detailed usage specifically for
|
|
525
|
-
pipeline mode (batch processing). For continuous mode, see the Continuous mode
|
|
526
|
-
components section.
|
|
527
|
-
|
|
528
|
-
Pipeline mode uses only the core components without any additional primitives.
|
|
529
|
-
|
|
530
|
-
=== Work class
|
|
531
|
-
|
|
532
|
-
==== Purpose and responsibilities
|
|
533
|
-
|
|
534
|
-
The `Fractor::Work` class represents a unit of work to be processed by a Worker.
|
|
535
|
-
Its primary responsibility is to encapsulate the input data needed for
|
|
536
|
-
processing.
|
|
537
|
-
|
|
538
|
-
==== Implementation requirements
|
|
539
|
-
|
|
540
|
-
At minimum, your Work subclass should:
|
|
541
|
-
|
|
542
|
-
. Inherit from `Fractor::Work`
|
|
543
|
-
. Pass the input data to the superclass constructor
|
|
153
|
+
supervisor.add_work_items(dataset.map { |item| DataWork.new(item) })
|
|
154
|
+
supervisor.run
|
|
544
155
|
|
|
545
|
-
|
|
546
|
-
----
|
|
547
|
-
class MyWork < Fractor::Work
|
|
548
|
-
def initialize(input)
|
|
549
|
-
super(input) # This stores input in @input
|
|
550
|
-
# Add any additional initialization if needed
|
|
551
|
-
end
|
|
552
|
-
end
|
|
156
|
+
puts "Processed #{supervisor.results.results.size} items"
|
|
553
157
|
----
|
|
554
158
|
|
|
555
|
-
|
|
159
|
+
See link:examples/simple/[Simple Example] and link:docs/examples.adoc#pipeline-mode-examples[more examples].
|
|
556
160
|
|
|
557
|
-
|
|
161
|
+
=== Continuous mode
|
|
558
162
|
|
|
559
163
|
[source,ruby]
|
|
560
164
|
----
|
|
561
|
-
|
|
562
|
-
attr_reader :options
|
|
563
|
-
|
|
564
|
-
def initialize(input, options = {})
|
|
565
|
-
super(input)
|
|
566
|
-
@options = options
|
|
567
|
-
end
|
|
568
|
-
|
|
569
|
-
def high_priority?
|
|
570
|
-
@options[:priority] == :high
|
|
571
|
-
end
|
|
572
|
-
|
|
573
|
-
def to_s
|
|
574
|
-
"ComplexWork: #{@input} (#{@options[:priority]} priority)"
|
|
575
|
-
end
|
|
576
|
-
end
|
|
577
|
-
----
|
|
578
|
-
|
|
579
|
-
[TIP]
|
|
580
|
-
====
|
|
581
|
-
* Keep Work objects lightweight and serializable since they will be passed
|
|
582
|
-
between Ractors
|
|
583
|
-
* Implement a meaningful `to_s` method for better debugging
|
|
584
|
-
* Consider adding validation in the initializer to catch issues early
|
|
585
|
-
====
|
|
586
|
-
|
|
587
|
-
=== Worker class
|
|
588
|
-
|
|
589
|
-
==== Purpose and responsibilities
|
|
590
|
-
|
|
591
|
-
The `Fractor::Worker` class defines the processing logic for work items. Each
|
|
592
|
-
Worker instance runs within its own Ractor and processes Work objects sent to
|
|
593
|
-
it.
|
|
594
|
-
|
|
595
|
-
==== Implementation requirements
|
|
596
|
-
|
|
597
|
-
Your Worker subclass must:
|
|
598
|
-
|
|
599
|
-
. Inherit from `Fractor::Worker`
|
|
600
|
-
. Implement the `process(work)` method
|
|
601
|
-
. Return a `Fractor::WorkResult` object from the `process` method
|
|
602
|
-
. Handle both successful processing and error conditions
|
|
165
|
+
work_queue = Fractor::WorkQueue.new
|
|
603
166
|
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
# Process the work
|
|
167
|
+
server = Fractor::ContinuousServer.new(
|
|
168
|
+
worker_pools: [{ worker_class: MessageWorker, num_workers: 4 }],
|
|
169
|
+
work_queue: work_queue
|
|
170
|
+
)
|
|
609
171
|
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
error: "Cannot process negative numbers",
|
|
613
|
-
work: work
|
|
614
|
-
)
|
|
615
|
-
end
|
|
172
|
+
server.on_result { |result| puts "Processed: #{result.result}" }
|
|
173
|
+
server.on_error { |error| puts "Error: #{error.error}" }
|
|
616
174
|
|
|
617
|
-
|
|
618
|
-
result = work.input * 2
|
|
175
|
+
Thread.new { server.run }
|
|
619
176
|
|
|
620
|
-
|
|
621
|
-
|
|
622
|
-
end
|
|
623
|
-
end
|
|
177
|
+
# Add work dynamically
|
|
178
|
+
work_queue << MessageWork.new(client_id: 1, message: "Hello")
|
|
624
179
|
----
|
|
625
180
|
|
|
181
|
+
See link:examples/continuous_chat_fractor/[Chat Server Example] and link:docs/examples.adoc#continuous-mode-examples[more examples].
|
|
626
182
|
|
|
627
|
-
|
|
628
|
-
|
|
629
|
-
The Worker class should handle two types of errors.
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
===== Handled errors
|
|
633
|
-
|
|
634
|
-
These are expected error conditions that your code explicitly checks for.
|
|
183
|
+
=== Workflows
|
|
635
184
|
|
|
636
185
|
[source,ruby]
|
|
637
186
|
----
|
|
638
|
-
|
|
639
|
-
|
|
640
|
-
|
|
641
|
-
|
|
642
|
-
|
|
643
|
-
)
|
|
644
|
-
end
|
|
645
|
-
|
|
646
|
-
# Normal processing...
|
|
647
|
-
Fractor::WorkResult.new(result: calculated_value, work: work)
|
|
187
|
+
# Define workflow with simplified syntax
|
|
188
|
+
workflow = Fractor::Workflow.define("data-pipeline") do
|
|
189
|
+
job :extract, ExtractWorker
|
|
190
|
+
job :transform, TransformWorker, needs: :extract
|
|
191
|
+
job :load, LoadWorker, needs: :transform
|
|
648
192
|
end
|
|
649
|
-
----
|
|
650
|
-
|
|
651
|
-
===== Unexpected errors caught by rescue
|
|
652
193
|
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
[source,ruby]
|
|
194
|
+
# Execute workflow
|
|
195
|
+
result = workflow.new.execute(input_data)
|
|
657
196
|
----
|
|
658
|
-
def process(work)
|
|
659
|
-
# Processing that might raise exceptions
|
|
660
|
-
result = complex_calculation(work.input)
|
|
661
|
-
|
|
662
|
-
Fractor::WorkResult.new(result: result, work: work)
|
|
663
|
-
rescue StandardError => e
|
|
664
|
-
# Catch and convert any unexpected exceptions to error results
|
|
665
|
-
Fractor::WorkResult.new(
|
|
666
|
-
error: "An unexpected error occurred: #{e.message}",
|
|
667
|
-
work: work
|
|
668
|
-
)
|
|
669
|
-
end
|
|
670
|
-
----
|
|
671
|
-
|
|
672
|
-
[TIP]
|
|
673
|
-
====
|
|
674
|
-
* Keep the `process` method focused on a single responsibility
|
|
675
|
-
* Use meaningful error messages that help diagnose issues
|
|
676
|
-
* Consider adding logging within the `process` method for debugging
|
|
677
|
-
* Ensure all paths return a valid `WorkResult` object
|
|
678
|
-
====
|
|
679
|
-
|
|
680
|
-
=== Supervisor class for pipeline mode
|
|
681
197
|
|
|
682
|
-
|
|
198
|
+
See link:examples/workflow/simplified/README.adoc[Simplified Workflows] and link:docs/workflows.adoc[Workflow Guide].
|
|
683
199
|
|
|
684
|
-
|
|
685
|
-
worker Ractors, distributing work, and collecting results.
|
|
200
|
+
== Production deployment
|
|
686
201
|
|
|
687
|
-
|
|
202
|
+
Fractor includes production-ready features:
|
|
688
203
|
|
|
689
|
-
|
|
204
|
+
* **Signal handling**: SIGTERM, SIGINT, SIGUSR1/SIGBREAK
|
|
205
|
+
* **Graceful shutdown**: Complete in-progress work before exit
|
|
206
|
+
* **Process monitoring**: Runtime status via signals
|
|
207
|
+
* **Structured logging**: JSON logging with correlation IDs
|
|
208
|
+
* **Workflow visualization**: Mermaid, DOT, ASCII diagrams
|
|
690
209
|
|
|
691
|
-
[
|
|
692
|
-
----
|
|
693
|
-
supervisor = Fractor::Supervisor.new(
|
|
694
|
-
worker_pools: [
|
|
695
|
-
# Pool 1 - for general data processing
|
|
696
|
-
{ worker_class: MyWorker, num_workers: 4 },
|
|
697
|
-
|
|
698
|
-
# Pool 2 - for specialized image processing
|
|
699
|
-
{ worker_class: ImageWorker, num_workers: 2 }
|
|
700
|
-
]
|
|
701
|
-
# Note: continuous_mode defaults to false for pipeline mode
|
|
702
|
-
)
|
|
703
|
-
----
|
|
704
|
-
|
|
705
|
-
==== Worker auto-detection
|
|
210
|
+
See link:docs/signal-handling.adoc[Signal Handling Guide] for deployment patterns.
|
|
706
211
|
|
|
707
|
-
|
|
708
|
-
and uses that value when `num_workers` is not specified. This provides optimal
|
|
709
|
-
resource utilization across different deployment environments without requiring
|
|
710
|
-
manual configuration.
|
|
711
|
-
|
|
712
|
-
[source,ruby]
|
|
713
|
-
----
|
|
714
|
-
# Auto-detect number of workers (recommended for most cases)
|
|
715
|
-
supervisor = Fractor::Supervisor.new(
|
|
716
|
-
worker_pools: [
|
|
717
|
-
{ worker_class: MyWorker } # Will use number of available processors
|
|
718
|
-
]
|
|
719
|
-
)
|
|
212
|
+
== Contributing
|
|
720
213
|
|
|
721
|
-
|
|
722
|
-
supervisor = Fractor::Supervisor.new(
|
|
723
|
-
worker_pools: [
|
|
724
|
-
{ worker_class: MyWorker, num_workers: 4 } # Always use exactly 4 workers
|
|
725
|
-
]
|
|
726
|
-
)
|
|
214
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/metanorma/fractor.
|
|
727
215
|
|
|
728
|
-
|
|
729
|
-
supervisor = Fractor::Supervisor.new(
|
|
730
|
-
worker_pools: [
|
|
731
|
-
{ worker_class: FastWorker }, # Auto-detected
|
|
732
|
-
{ worker_class: HeavyWorker, num_workers: 2 } # Explicitly 2 workers
|
|
733
|
-
]
|
|
734
|
-
)
|
|
735
|
-
----
|
|
216
|
+
== License
|
|
736
217
|
|
|
737
|
-
The
|
|
738
|
-
available processors. If detection fails for any reason, it falls back to 2
|
|
739
|
-
workers.
|
|
218
|
+
The gem is available as open source under the terms of the Ribose BSD 2-Clause License.
|
|
740
219
|
|
|
741
|
-
|
|
742
|
-
====
|
|
743
|
-
* Use auto-detection for portable code that adapts to different environments
|
|
744
|
-
* Explicitly set `num_workers` when you need precise control over resource usage
|
|
745
|
-
* Consider system load and other factors when choosing explicit values
|
|
746
|
-
====
|
|
747
|
-
|
|
748
|
-
==== Adding work
|
|
749
|
-
|
|
750
|
-
You can add work items individually or in batches:
|
|
751
|
-
|
|
752
|
-
[source,ruby]
|
|
753
|
-
----
|
|
754
|
-
# Add a single item
|
|
755
|
-
supervisor.add_work_item(MyWork.new(42))
|
|
756
|
-
|
|
757
|
-
# Add multiple items
|
|
758
|
-
supervisor.add_work_items([
|
|
759
|
-
MyWork.new(1),
|
|
760
|
-
MyWork.new(2),
|
|
761
|
-
MyWork.new(3),
|
|
762
|
-
MyWork.new(4),
|
|
763
|
-
MyWork.new(5)
|
|
764
|
-
])
|
|
765
|
-
|
|
766
|
-
# Add items of different work types
|
|
767
|
-
supervisor.add_work_items([
|
|
768
|
-
TextWork.new("Process this text"),
|
|
769
|
-
ImageWork.new({ width: 800, height: 600 })
|
|
770
|
-
])
|
|
771
|
-
----
|
|
772
|
-
|
|
773
|
-
The Supervisor can handle any Work object that inherits from Fractor::Work.
|
|
774
|
-
Workers must check the type of Work they receive and process it accordingly.
|
|
775
|
-
|
|
776
|
-
==== Running and monitoring
|
|
777
|
-
|
|
778
|
-
To start processing:
|
|
779
|
-
|
|
780
|
-
[source,ruby]
|
|
781
|
-
----
|
|
782
|
-
# Start processing and block until complete
|
|
783
|
-
supervisor.run
|
|
784
|
-
----
|
|
785
|
-
|
|
786
|
-
The Supervisor automatically handles:
|
|
787
|
-
|
|
788
|
-
* Starting the worker Ractors
|
|
789
|
-
* Distributing work items to available workers
|
|
790
|
-
* Collecting results and errors
|
|
791
|
-
* Graceful shutdown on completion or interruption (Ctrl+C)
|
|
792
|
-
|
|
793
|
-
=== ResultAggregator for pipeline mode
|
|
794
|
-
|
|
795
|
-
==== Purpose and responsibilities
|
|
796
|
-
|
|
797
|
-
The `Fractor::ResultAggregator` collects and organizes all results from the
|
|
798
|
-
workers, separating successful results from errors.
|
|
799
|
-
|
|
800
|
-
In pipeline mode, results are collected throughout processing and accessed
|
|
801
|
-
after the supervisor finishes running.
|
|
802
|
-
|
|
803
|
-
==== Accessing results
|
|
804
|
-
|
|
805
|
-
After processing completes:
|
|
806
|
-
|
|
807
|
-
[source,ruby]
|
|
808
|
-
----
|
|
809
|
-
# Get the ResultAggregator
|
|
810
|
-
aggregator = supervisor.results
|
|
811
|
-
|
|
812
|
-
# Check counts
|
|
813
|
-
puts "Processed #{aggregator.results.size} items successfully"
|
|
814
|
-
puts "Encountered #{aggregator.errors.size} errors"
|
|
815
|
-
|
|
816
|
-
# Access successful results
|
|
817
|
-
aggregator.results.each do |result|
|
|
818
|
-
puts "Work item #{result.work.input} produced #{result.result}"
|
|
819
|
-
end
|
|
820
|
-
|
|
821
|
-
# Access errors
|
|
822
|
-
aggregator.errors.each do |error_result|
|
|
823
|
-
puts "Work item #{error_result.work.input} failed: #{error_result.error}"
|
|
824
|
-
end
|
|
825
|
-
----
|
|
826
|
-
|
|
827
|
-
To access successful results:
|
|
828
|
-
|
|
829
|
-
[source,ruby]
|
|
830
|
-
----
|
|
831
|
-
# Get all successful results
|
|
832
|
-
successful_results = supervisor.results.results
|
|
833
|
-
|
|
834
|
-
# Extract just the result values
|
|
835
|
-
result_values = successful_results.map(&:result)
|
|
836
|
-
----
|
|
837
|
-
|
|
838
|
-
To access errors:
|
|
839
|
-
|
|
840
|
-
[source,ruby]
|
|
841
|
-
----
|
|
842
|
-
# Get all error results
|
|
843
|
-
error_results = supervisor.results.errors
|
|
844
|
-
|
|
845
|
-
# Extract error messages
|
|
846
|
-
error_messages = error_results.map(&:error)
|
|
847
|
-
|
|
848
|
-
# Get the work items that failed
|
|
849
|
-
failed_work_items = error_results.map(&:work)
|
|
850
|
-
----
|
|
851
|
-
|
|
852
|
-
|
|
853
|
-
[TIP]
|
|
854
|
-
====
|
|
855
|
-
* Check both successful results and errors after processing completes
|
|
856
|
-
* Consider implementing custom reporting based on the aggregated results
|
|
857
|
-
====
|
|
858
|
-
|
|
859
|
-
|
|
860
|
-
|
|
861
|
-
|
|
862
|
-
== Pipeline mode patterns
|
|
863
|
-
|
|
864
|
-
=== Custom work distribution
|
|
865
|
-
|
|
866
|
-
For more complex scenarios, you might want to prioritize certain work items:
|
|
867
|
-
|
|
868
|
-
[source,ruby]
|
|
869
|
-
----
|
|
870
|
-
# Create Work objects for high priority items
|
|
871
|
-
high_priority_works = high_priority_items.map { |item| MyWork.new(item) }
|
|
872
|
-
|
|
873
|
-
# Add high-priority items first
|
|
874
|
-
supervisor.add_work_items(high_priority_works)
|
|
875
|
-
|
|
876
|
-
# Run with just enough workers for high-priority items
|
|
877
|
-
supervisor.run
|
|
878
|
-
|
|
879
|
-
# Create Work objects for lower priority items
|
|
880
|
-
low_priority_works = low_priority_items.map { |item| MyWork.new(item) }
|
|
881
|
-
|
|
882
|
-
# Add and process lower-priority items
|
|
883
|
-
supervisor.add_work_items(low_priority_works)
|
|
884
|
-
supervisor.run
|
|
885
|
-
----
|
|
886
|
-
|
|
887
|
-
=== Handling large datasets
|
|
888
|
-
|
|
889
|
-
For very large datasets, consider processing in batches:
|
|
890
|
-
|
|
891
|
-
[source,ruby]
|
|
892
|
-
----
|
|
893
|
-
large_dataset.each_slice(1000) do |batch|
|
|
894
|
-
# Convert batch items to Work objects
|
|
895
|
-
work_batch = batch.map { |item| MyWork.new(item) }
|
|
896
|
-
|
|
897
|
-
supervisor.add_work_items(work_batch)
|
|
898
|
-
supervisor.run
|
|
899
|
-
|
|
900
|
-
# Process this batch's results before continuing
|
|
901
|
-
process_batch_results(supervisor.results)
|
|
902
|
-
end
|
|
903
|
-
----
|
|
904
|
-
|
|
905
|
-
=== Multi-work type processing
|
|
906
|
-
|
|
907
|
-
The Multi-Work Type pattern demonstrates how a single supervisor and worker can
|
|
908
|
-
handle multiple types of work items.
|
|
909
|
-
|
|
910
|
-
[source,ruby]
|
|
911
|
-
----
|
|
912
|
-
class UniversalWorker < Fractor::Worker
|
|
913
|
-
def process(work)
|
|
914
|
-
case work
|
|
915
|
-
when TextWork
|
|
916
|
-
process_text(work)
|
|
917
|
-
when ImageWork
|
|
918
|
-
process_image(work)
|
|
919
|
-
else
|
|
920
|
-
Fractor::WorkResult.new(
|
|
921
|
-
error: "Unknown work type: #{work.class}",
|
|
922
|
-
work: work
|
|
923
|
-
)
|
|
924
|
-
end
|
|
925
|
-
end
|
|
926
|
-
|
|
927
|
-
private
|
|
928
|
-
|
|
929
|
-
def process_text(work)
|
|
930
|
-
result = work.text.upcase
|
|
931
|
-
Fractor::WorkResult.new(result: result, work: work)
|
|
932
|
-
end
|
|
933
|
-
|
|
934
|
-
def process_image(work)
|
|
935
|
-
result = { width: work.width * 2, height: work.height * 2 }
|
|
936
|
-
Fractor::WorkResult.new(result: result, work: work)
|
|
937
|
-
end
|
|
938
|
-
end
|
|
939
|
-
|
|
940
|
-
# Add different types of work
|
|
941
|
-
supervisor.add_work_items([
|
|
942
|
-
TextWork.new("hello"),
|
|
943
|
-
ImageWork.new(width: 100, height: 100),
|
|
944
|
-
TextWork.new("world")
|
|
945
|
-
])
|
|
946
|
-
----
|
|
947
|
-
|
|
948
|
-
=== Hierarchical work processing
|
|
949
|
-
|
|
950
|
-
The Producer/Subscriber pattern showcases processing that generates sub-work:
|
|
951
|
-
|
|
952
|
-
[source,ruby]
|
|
953
|
-
----
|
|
954
|
-
# First pass: Process documents
|
|
955
|
-
supervisor.add_work_items(documents.map { |doc| DocumentWork.new(doc) })
|
|
956
|
-
supervisor.run
|
|
957
|
-
|
|
958
|
-
# Collect sections generated from documents
|
|
959
|
-
sections = supervisor.results.results.flat_map do |result|
|
|
960
|
-
result.result[:sections]
|
|
961
|
-
end
|
|
962
|
-
|
|
963
|
-
# Second pass: Process sections
|
|
964
|
-
supervisor.add_work_items(sections.map { |section| SectionWork.new(section) })
|
|
965
|
-
supervisor.run
|
|
966
|
-
----
|
|
967
|
-
|
|
968
|
-
=== Pipeline stages
|
|
969
|
-
|
|
970
|
-
The Pipeline Processing pattern implements multi-stage transformation:
|
|
971
|
-
|
|
972
|
-
[source,ruby]
|
|
973
|
-
----
|
|
974
|
-
# Stage 1: Extract data
|
|
975
|
-
supervisor1 = Fractor::Supervisor.new(
|
|
976
|
-
worker_pools: [{ worker_class: ExtractionWorker }]
|
|
977
|
-
)
|
|
978
|
-
supervisor1.add_work_items(raw_data.map { |d| ExtractionWork.new(d) })
|
|
979
|
-
supervisor1.run
|
|
980
|
-
extracted = supervisor1.results.results.map(&:result)
|
|
981
|
-
|
|
982
|
-
# Stage 2: Transform data
|
|
983
|
-
supervisor2 = Fractor::Supervisor.new(
|
|
984
|
-
worker_pools: [{ worker_class: TransformWorker }]
|
|
985
|
-
)
|
|
986
|
-
supervisor2.add_work_items(extracted.map { |e| TransformWork.new(e) })
|
|
987
|
-
supervisor2.run
|
|
988
|
-
transformed = supervisor2.results.results.map(&:result)
|
|
989
|
-
|
|
990
|
-
# Stage 3: Load data
|
|
991
|
-
supervisor3 = Fractor::Supervisor.new(
|
|
992
|
-
worker_pools: [{ worker_class: LoadWorker }]
|
|
993
|
-
)
|
|
994
|
-
supervisor3.add_work_items(transformed.map { |t| LoadWork.new(t) })
|
|
995
|
-
supervisor3.run
|
|
996
|
-
----
|
|
997
|
-
|
|
998
|
-
|
|
999
|
-
|
|
1000
|
-
|
|
1001
|
-
== Continuous mode components
|
|
1002
|
-
|
|
1003
|
-
=== General
|
|
1004
|
-
|
|
1005
|
-
This section describes the components and their detailed usage specifically for
|
|
1006
|
-
continuous mode (long-running servers). For pipeline mode, see the Pipeline mode
|
|
1007
|
-
components section.
|
|
1008
|
-
|
|
1009
|
-
Continuous mode offers two approaches: a low-level API for manual control, and
|
|
1010
|
-
high-level primitives that eliminate boilerplate code.
|
|
1011
|
-
|
|
1012
|
-
=== Low-level components
|
|
1013
|
-
|
|
1014
|
-
==== General
|
|
1015
|
-
|
|
1016
|
-
The low-level API provides manual control over continuous mode operation. This
|
|
1017
|
-
approach is useful when you need fine-grained control over threading, work
|
|
1018
|
-
sources, or results processing.
|
|
1019
|
-
|
|
1020
|
-
Use the low-level API when:
|
|
1021
|
-
|
|
1022
|
-
* You need custom thread management
|
|
1023
|
-
* Your work source logic is complex
|
|
1024
|
-
* You require precise control over the supervisor lifecycle
|
|
1025
|
-
* You're integrating with existing thread pools or event loops
|
|
1026
|
-
|
|
1027
|
-
For most applications, the high-level primitives (described in the next section)
|
|
1028
|
-
are recommended as they eliminate significant boilerplate code.
|
|
1029
|
-
|
|
1030
|
-
==== Supervisor with continuous_mode: true
|
|
1031
|
-
|
|
1032
|
-
To enable continuous mode, set the `continuous_mode` option:
|
|
1033
|
-
|
|
1034
|
-
[source,ruby]
|
|
1035
|
-
----
|
|
1036
|
-
supervisor = Fractor::Supervisor.new(
|
|
1037
|
-
worker_pools: [
|
|
1038
|
-
{ worker_class: MyWorker, num_workers: 2 }
|
|
1039
|
-
],
|
|
1040
|
-
continuous_mode: true # Enable continuous mode
|
|
1041
|
-
)
|
|
1042
|
-
----
|
|
1043
|
-
|
|
1044
|
-
==== Work source callbacks
|
|
1045
|
-
|
|
1046
|
-
Register a callback that provides new work on demand:
|
|
1047
|
-
|
|
1048
|
-
[source,ruby]
|
|
1049
|
-
----
|
|
1050
|
-
supervisor.register_work_source do
|
|
1051
|
-
# Return nil or empty array if no work is available
|
|
1052
|
-
# Return a work item or array of work items when available
|
|
1053
|
-
items = get_next_work_items
|
|
1054
|
-
if items && !items.empty?
|
|
1055
|
-
# Convert to Work objects if needed
|
|
1056
|
-
items.map { |item| MyWork.new(item) }
|
|
1057
|
-
else
|
|
1058
|
-
nil
|
|
1059
|
-
end
|
|
1060
|
-
end
|
|
1061
|
-
----
|
|
1062
|
-
|
|
1063
|
-
The callback is polled every 100ms by an internal timer thread.
|
|
1064
|
-
|
|
1065
|
-
==== Manual thread management
|
|
1066
|
-
|
|
1067
|
-
You must manually manage threads and results processing:
|
|
1068
|
-
|
|
1069
|
-
[source,ruby]
|
|
1070
|
-
----
|
|
1071
|
-
# Start supervisor in a background thread
|
|
1072
|
-
supervisor_thread = Thread.new { supervisor.run }
|
|
1073
|
-
|
|
1074
|
-
# Start results processing thread
|
|
1075
|
-
results_thread = Thread.new do
|
|
1076
|
-
loop do
|
|
1077
|
-
# Process results
|
|
1078
|
-
while (result = supervisor.results.results.shift)
|
|
1079
|
-
handle_result(result)
|
|
1080
|
-
end
|
|
1081
|
-
|
|
1082
|
-
# Process errors
|
|
1083
|
-
while (error = supervisor.results.errors.shift)
|
|
1084
|
-
handle_error(error)
|
|
1085
|
-
end
|
|
1086
|
-
|
|
1087
|
-
sleep 0.1
|
|
1088
|
-
end
|
|
1089
|
-
end
|
|
1090
|
-
|
|
1091
|
-
# Ensure cleanup on shutdown
|
|
1092
|
-
begin
|
|
1093
|
-
supervisor_thread.join
|
|
1094
|
-
rescue Interrupt
|
|
1095
|
-
supervisor.stop
|
|
1096
|
-
ensure
|
|
1097
|
-
results_thread.kill
|
|
1098
|
-
supervisor_thread.join
|
|
1099
|
-
end
|
|
1100
|
-
----
|
|
1101
|
-
|
|
1102
|
-
=== High-level components
|
|
1103
|
-
|
|
1104
|
-
==== General
|
|
1105
|
-
|
|
1106
|
-
Fractor provides high-level primitives that dramatically simplify continuous
|
|
1107
|
-
mode applications by eliminating boilerplate code.
|
|
1108
|
-
|
|
1109
|
-
These primitives solve common problems:
|
|
1110
|
-
|
|
1111
|
-
* *Thread management*: Automatic supervisor and results processing threads
|
|
1112
|
-
* *Queue synchronization*: Thread-safe work queue with automatic integration
|
|
1113
|
-
* *Results processing*: Callback-based handling instead of manual loops
|
|
1114
|
-
* *Signal handling*: Built-in support for SIGINT, SIGTERM, SIGUSR1/SIGBREAK
|
|
1115
|
-
* *Graceful shutdown*: Coordinated cleanup across all threads
|
|
1116
|
-
|
|
1117
|
-
Real-world benefits:
|
|
1118
|
-
|
|
1119
|
-
* The chat server example reduced from 279 lines to 167 lines (40% reduction)
|
|
1120
|
-
* Eliminates ~112 lines of thread, queue, and signal handling boilerplate
|
|
1121
|
-
* Simpler, more maintainable code with fewer error-prone details
|
|
1122
|
-
|
|
1123
|
-
==== Fractor::WorkQueue
|
|
1124
|
-
|
|
1125
|
-
===== Purpose and responsibilities
|
|
1126
|
-
|
|
1127
|
-
`Fractor::WorkQueue` provides a thread-safe queue for continuous mode
|
|
1128
|
-
applications. It handles work item storage and integrates automatically with the
|
|
1129
|
-
supervisor's work source mechanism.
|
|
1130
|
-
|
|
1131
|
-
===== Thread-safety
|
|
1132
|
-
|
|
1133
|
-
The WorkQueue is *thread-safe* but not *Ractor-safe*:
|
|
1134
|
-
|
|
1135
|
-
* *Thread-safe*: Multiple threads can safely push work items concurrently
|
|
1136
|
-
* *Not Ractor-safe*: The queue lives in the main process and cannot be shared
|
|
1137
|
-
across Ractor boundaries
|
|
1138
|
-
|
|
1139
|
-
This design is intentional. The WorkQueue operates in the main process where
|
|
1140
|
-
your application code runs. Work items are retrieved by the Supervisor (also in
|
|
1141
|
-
the main process) and then sent to worker Ractors.
|
|
1142
|
-
|
|
1143
|
-
.WorkQueue architecture
|
|
1144
|
-
[source]
|
|
1145
|
-
----
|
|
1146
|
-
Main Process
|
|
1147
|
-
├─→ Your application threads (push to WorkQueue)
|
|
1148
|
-
├─→ WorkQueue (thread-safe, lives here)
|
|
1149
|
-
├─→ Supervisor (polls WorkQueue)
|
|
1150
|
-
│ └─→ Sends work to Worker Ractors
|
|
1151
|
-
└─→ Worker Ractors (receive frozen/shareable work items)
|
|
1152
|
-
----
|
|
1153
|
-
|
|
1154
|
-
===== Creating a WorkQueue
|
|
1155
|
-
|
|
1156
|
-
[source,ruby]
|
|
1157
|
-
----
|
|
1158
|
-
work_queue = Fractor::WorkQueue.new
|
|
1159
|
-
----
|
|
1160
|
-
|
|
1161
|
-
===== Adding work items
|
|
1162
|
-
|
|
1163
|
-
Use the `<<` operator for thread-safe push operations:
|
|
1164
|
-
|
|
1165
|
-
[source,ruby]
|
|
1166
|
-
----
|
|
1167
|
-
# From any thread in your application
|
|
1168
|
-
work_queue << MyWork.new(data)
|
|
1169
|
-
|
|
1170
|
-
# Thread-safe even from multiple threads
|
|
1171
|
-
threads = 10.times.map do |i|
|
|
1172
|
-
Thread.new do
|
|
1173
|
-
100.times do |j|
|
|
1174
|
-
work_queue << MyWork.new("thread-#{i}-item-#{j}")
|
|
1175
|
-
end
|
|
1176
|
-
end
|
|
1177
|
-
end
|
|
1178
|
-
threads.each(&:join)
|
|
1179
|
-
----
|
|
1180
|
-
|
|
1181
|
-
===== Checking queue status
|
|
1182
|
-
|
|
1183
|
-
[source,ruby]
|
|
1184
|
-
----
|
|
1185
|
-
# Check if queue is empty
|
|
1186
|
-
if work_queue.empty?
|
|
1187
|
-
puts "No work available"
|
|
1188
|
-
end
|
|
1189
|
-
|
|
1190
|
-
# Get current queue size
|
|
1191
|
-
puts "Queue has #{work_queue.size} items"
|
|
1192
|
-
----
|
|
1193
|
-
|
|
1194
|
-
===== Integration with Supervisor
|
|
1195
|
-
|
|
1196
|
-
The WorkQueue integrates automatically with ContinuousServer (see next section).
|
|
1197
|
-
For manual integration with a Supervisor:
|
|
1198
|
-
|
|
1199
|
-
[source,ruby]
|
|
1200
|
-
----
|
|
1201
|
-
supervisor = Fractor::Supervisor.new(
|
|
1202
|
-
worker_pools: [{ worker_class: MyWorker }],
|
|
1203
|
-
continuous_mode: true
|
|
1204
|
-
)
|
|
1205
|
-
|
|
1206
|
-
# Register the work queue as a work source
|
|
1207
|
-
work_queue.register_with_supervisor(supervisor)
|
|
1208
|
-
|
|
1209
|
-
# Now the supervisor will automatically poll the queue for work
|
|
1210
|
-
----
|
|
1211
|
-
|
|
1212
|
-
==== Fractor::ContinuousServer
|
|
1213
|
-
|
|
1214
|
-
===== Purpose and responsibilities
|
|
1215
|
-
|
|
1216
|
-
`Fractor::ContinuousServer` is a high-level wrapper that handles all the
|
|
1217
|
-
complexity of running a continuous mode application. It manages:
|
|
1218
|
-
|
|
1219
|
-
* Supervisor thread lifecycle
|
|
1220
|
-
* Results processing thread with callback system
|
|
1221
|
-
* Signal handling (SIGINT, SIGTERM, SIGUSR1/SIGBREAK)
|
|
1222
|
-
* Graceful shutdown coordination
|
|
1223
|
-
* Optional logging
|
|
1224
|
-
|
|
1225
|
-
===== Creating a ContinuousServer
|
|
1226
|
-
|
|
1227
|
-
[source,ruby]
|
|
1228
|
-
----
|
|
1229
|
-
server = Fractor::ContinuousServer.new(
|
|
1230
|
-
worker_pools: [
|
|
1231
|
-
{ worker_class: MessageWorker, num_workers: 4 }
|
|
1232
|
-
],
|
|
1233
|
-
work_queue: work_queue, # Optional, auto-registers if provided
|
|
1234
|
-
log_file: 'logs/server.log' # Optional
|
|
1235
|
-
)
|
|
1236
|
-
----
|
|
1237
|
-
|
|
1238
|
-
Parameters:
|
|
1239
|
-
|
|
1240
|
-
* `worker_pools` (required): Array of worker pool configurations
|
|
1241
|
-
* `work_queue` (optional): A Fractor::WorkQueue instance to auto-register
|
|
1242
|
-
* `log_file` (optional): Path for log output
|
|
1243
|
-
|
|
1244
|
-
===== Registering callbacks
|
|
1245
|
-
|
|
1246
|
-
Define how to handle results and errors:
|
|
1247
|
-
|
|
1248
|
-
[source,ruby]
|
|
1249
|
-
----
|
|
1250
|
-
# Handle successful results
|
|
1251
|
-
server.on_result do |result|
|
|
1252
|
-
# result is a Fractor::WorkResult with result.result containing your data
|
|
1253
|
-
puts "Success: #{result.result}"
|
|
1254
|
-
# Send response to client, update database, etc.
|
|
1255
|
-
end
|
|
1256
|
-
|
|
1257
|
-
# Handle errors
|
|
1258
|
-
server.on_error do |error_result|
|
|
1259
|
-
# error_result is a Fractor::WorkResult with error_result.error containing the message
|
|
1260
|
-
puts "Error: #{error_result.error}"
|
|
1261
|
-
# Log error, send notification, etc.
|
|
1262
|
-
end
|
|
1263
|
-
----
|
|
1264
|
-
|
|
1265
|
-
===== Running the server
|
|
1266
|
-
|
|
1267
|
-
[source,ruby]
|
|
1268
|
-
----
|
|
1269
|
-
# Blocking: Run the server (blocks until shutdown signal)
|
|
1270
|
-
server.run
|
|
1271
|
-
|
|
1272
|
-
# Non-blocking: Run in background thread
|
|
1273
|
-
server_thread = Thread.new { server.run }
|
|
1274
|
-
|
|
1275
|
-
# Your application continues here...
|
|
1276
|
-
# Add work to queue as needed
|
|
1277
|
-
work_queue << MyWork.new(data)
|
|
1278
|
-
|
|
1279
|
-
# Later, stop the server
|
|
1280
|
-
server.stop
|
|
1281
|
-
server_thread.join
|
|
1282
|
-
----
|
|
1283
|
-
|
|
1284
|
-
===== Signal handling
|
|
1285
|
-
|
|
1286
|
-
The ContinuousServer automatically handles:
|
|
1287
|
-
|
|
1288
|
-
* *SIGINT* (Ctrl+C): Graceful shutdown
|
|
1289
|
-
* *SIGTERM*: Graceful shutdown (production deployment)
|
|
1290
|
-
* *SIGUSR1* (Unix) / *SIGBREAK* (Windows): Status output
|
|
1291
|
-
|
|
1292
|
-
No additional code needed - signals work automatically.
|
|
1293
|
-
|
|
1294
|
-
===== Graceful shutdown
|
|
1295
|
-
|
|
1296
|
-
When a shutdown signal is received:
|
|
1297
|
-
|
|
1298
|
-
. Stops accepting new work from the work queue
|
|
1299
|
-
. Allows in-progress work to complete (within ~2 seconds)
|
|
1300
|
-
. Processes remaining results through callbacks
|
|
1301
|
-
. Cleans up all threads and resources
|
|
1302
|
-
. Returns from the `run` method
|
|
1303
|
-
|
|
1304
|
-
===== Programmatic shutdown
|
|
1305
|
-
|
|
1306
|
-
[source,ruby]
|
|
1307
|
-
----
|
|
1308
|
-
# Stop the server programmatically
|
|
1309
|
-
server.stop
|
|
1310
|
-
|
|
1311
|
-
# The run method will return shortly after
|
|
1312
|
-
----
|
|
1313
|
-
|
|
1314
|
-
==== Integration architecture
|
|
1315
|
-
|
|
1316
|
-
The high-level components work together seamlessly:
|
|
1317
|
-
|
|
1318
|
-
.Complete architecture diagram
|
|
1319
|
-
[source]
|
|
1320
|
-
----
|
|
1321
|
-
┌───────────────────────────────────────────────────────────┐
|
|
1322
|
-
│ Main Process │
|
|
1323
|
-
│ │
|
|
1324
|
-
│ ┌──────────────┐ ┌──────────────────────────────┐ │
|
|
1325
|
-
│ │ Your App │────>│ WorkQueue (thread-safe) │ │
|
|
1326
|
-
│ │ (any thread) │ │ - Thread::Queue internally │ │
|
|
1327
|
-
│ └──────────────┘ └──────────────────────────────┘ │
|
|
1328
|
-
│ │ │
|
|
1329
|
-
│ │ polled every 100ms │
|
|
1330
|
-
│ ▼ │
|
|
1331
|
-
│ ┌────────────────────────────────────────────────────┐ │
|
|
1332
|
-
│ │ ContinuousServer │ │
|
|
1333
|
-
│ │ ┌─────────────────────────────────────────────┐ │ │
|
|
1334
|
-
│ │ │ Supervisor Thread │ │ │
|
|
1335
|
-
│ │ │ - Manages worker Ractors │ │ │
|
|
1336
|
-
│ │ │ - Distributes work │ │ │
|
|
1337
|
-
│ │ │ - Coordinates shutdown │ │ │
|
|
1338
|
-
│ │ └─────────────────────────────────────────────┘ │ │
|
|
1339
|
-
│ │ │ │ │
|
|
1340
|
-
│ │ ▼ │ │
|
|
1341
|
-
│ │ ┌─────────────────────────────────────────────┐ │ │
|
|
1342
|
-
│ │ │ Worker Ractors (parallel execution) │ │ │
|
|
1343
|
-
│ │ │ - Ractor 1: WorkerInstance.process(work) │ │ │
|
|
1344
|
-
│ │ │ - Ractor 2: WorkerInstance.process(work) │ │ │
|
|
1345
|
-
│ │ │ - Ractor N: WorkerInstance.process(work) │ │ │
|
|
1346
|
-
│ │ └─────────────────────────────────────────────┘ │ │
|
|
1347
|
-
│ │ │ │ │
|
|
1348
|
-
│ │ ▼ (WorkResults) │ │
|
|
1349
|
-
│ │ ┌─────────────────────────────────────────────┐ │ │
|
|
1350
|
-
│ │ │ Results Processing Thread │ │ │
|
|
1351
|
-
│ │ │ - on_result callback for successes │ │ │
|
|
1352
|
-
│ │ │ - on_error callback for failures │ │ │
|
|
1353
|
-
│ │ └─────────────────────────────────────────────┘ │ │
|
|
1354
|
-
│ │ │ │
|
|
1355
|
-
│ │ ┌─────────────────────────────────────────────┐ │ │
|
|
1356
|
-
│ │ │ Signal Handler Thread │ │ │
|
|
1357
|
-
│ │ │ - SIGINT/SIGTERM: Shutdown │ │ │
|
|
1358
|
-
│ │ │ - SIGUSR1/SIGBREAK: Status │ │ │
|
|
1359
|
-
│ │ └─────────────────────────────────────────────┘ │ │
|
|
1360
|
-
│ └────────────────────────────────────────────────────┘ │
|
|
1361
|
-
└───────────────────────────────────────────────────────────┘
|
|
1362
|
-
----
|
|
1363
|
-
|
|
1364
|
-
Key points:
|
|
1365
|
-
|
|
1366
|
-
* WorkQueue lives in main process (thread-safe, not Ractor-safe)
|
|
1367
|
-
* Supervisor polls WorkQueue and distributes to Ractors
|
|
1368
|
-
* Work items must be frozen/shareable to cross Ractor boundary
|
|
1369
|
-
* Results come back through callbacks, not batch collection
|
|
1370
|
-
* All thread management is automatic
|
|
1371
|
-
|
|
1372
|
-
|
|
1373
|
-
|
|
1374
|
-
|
|
1375
|
-
== Continuous mode patterns
|
|
1376
|
-
|
|
1377
|
-
=== Basic server with callbacks
|
|
1378
|
-
|
|
1379
|
-
The most common pattern uses WorkQueue + ContinuousServer:
|
|
1380
|
-
|
|
1381
|
-
[source,ruby]
|
|
1382
|
-
----
|
|
1383
|
-
require 'fractor'
|
|
1384
|
-
|
|
1385
|
-
# Define work and worker
|
|
1386
|
-
class RequestWork < Fractor::Work
|
|
1387
|
-
def initialize(request_id, data)
|
|
1388
|
-
super({ request_id: request_id, data: data })
|
|
1389
|
-
end
|
|
1390
|
-
end
|
|
1391
|
-
|
|
1392
|
-
class RequestWorker < Fractor::Worker
|
|
1393
|
-
def process(work)
|
|
1394
|
-
# Process the request
|
|
1395
|
-
result = perform_computation(work.input[:data])
|
|
1396
|
-
|
|
1397
|
-
Fractor::WorkResult.new(
|
|
1398
|
-
result: { request_id: work.input[:request_id], response: result },
|
|
1399
|
-
work: work
|
|
1400
|
-
)
|
|
1401
|
-
rescue => e
|
|
1402
|
-
Fractor::WorkResult.new(error: e.message, work: work)
|
|
1403
|
-
end
|
|
1404
|
-
|
|
1405
|
-
private
|
|
1406
|
-
|
|
1407
|
-
def perform_computation(data)
|
|
1408
|
-
# Your business logic here
|
|
1409
|
-
data.upcase
|
|
1410
|
-
end
|
|
1411
|
-
end
|
|
1412
|
-
|
|
1413
|
-
# Set up server
|
|
1414
|
-
work_queue = Fractor::WorkQueue.new
|
|
1415
|
-
|
|
1416
|
-
server = Fractor::ContinuousServer.new(
|
|
1417
|
-
worker_pools: [{ worker_class: RequestWorker, num_workers: 4 }],
|
|
1418
|
-
work_queue: work_queue
|
|
1419
|
-
)
|
|
1420
|
-
|
|
1421
|
-
server.on_result { |result| puts "Success: #{result.result}" }
|
|
1422
|
-
server.on_error { |error| puts "Error: #{error.error}" }
|
|
1423
|
-
|
|
1424
|
-
# Run server (blocks until shutdown)
|
|
1425
|
-
Thread.new { server.run }
|
|
1426
|
-
|
|
1427
|
-
# Application logic adds work as needed
|
|
1428
|
-
work_queue << RequestWork.new(1, "hello")
|
|
1429
|
-
work_queue << RequestWork.new(2, "world")
|
|
1430
|
-
|
|
1431
|
-
sleep # Keep main thread alive
|
|
1432
|
-
----
|
|
1433
|
-
|
|
1434
|
-
=== Event-driven processing
|
|
1435
|
-
|
|
1436
|
-
Process events from external sources as they arrive:
|
|
1437
|
-
|
|
1438
|
-
[source,ruby]
|
|
1439
|
-
----
|
|
1440
|
-
# Event source (could be webhooks, message queue, etc.)
|
|
1441
|
-
event_source = EventSource.new
|
|
1442
|
-
|
|
1443
|
-
# Set up work queue and server
|
|
1444
|
-
work_queue = Fractor::WorkQueue.new
|
|
1445
|
-
server = Fractor::ContinuousServer.new(
|
|
1446
|
-
worker_pools: [{ worker_class: EventWorker, num_workers: 8 }],
|
|
1447
|
-
work_queue: work_queue
|
|
1448
|
-
)
|
|
1449
|
-
|
|
1450
|
-
server.on_result do |result|
|
|
1451
|
-
# Publish result to subscribers
|
|
1452
|
-
publish_event(result.result)
|
|
1453
|
-
end
|
|
1454
|
-
|
|
1455
|
-
# Event loop adds work to queue
|
|
1456
|
-
event_source.on_event do |event|
|
|
1457
|
-
work_queue << EventWork.new(event)
|
|
1458
|
-
end
|
|
1459
|
-
|
|
1460
|
-
# Start server
|
|
1461
|
-
server.run
|
|
1462
|
-
----
|
|
1463
|
-
|
|
1464
|
-
=== Dynamic work sources
|
|
1465
|
-
|
|
1466
|
-
Combine multiple work sources:
|
|
1467
|
-
|
|
1468
|
-
[source,ruby]
|
|
1469
|
-
----
|
|
1470
|
-
work_queue = Fractor::WorkQueue.new
|
|
1471
|
-
|
|
1472
|
-
# Source 1: HTTP requests
|
|
1473
|
-
http_server.on_request do |request|
|
|
1474
|
-
work_queue << HttpWork.new(request)
|
|
1475
|
-
end
|
|
1476
|
-
|
|
1477
|
-
# Source 2: Message queue
|
|
1478
|
-
message_queue.subscribe do |message|
|
|
1479
|
-
work_queue << MessageWork.new(message)
|
|
1480
|
-
end
|
|
1481
|
-
|
|
1482
|
-
# Source 3: Scheduled tasks
|
|
1483
|
-
scheduler.every('1m') do
|
|
1484
|
-
work_queue << ScheduledWork.new(Time.now)
|
|
1485
|
-
end
|
|
1486
|
-
|
|
1487
|
-
# Single server processes all work types
|
|
1488
|
-
server = Fractor::ContinuousServer.new(
|
|
1489
|
-
worker_pools: [
|
|
1490
|
-
{ worker_class: HttpWorker, num_workers: 4 },
|
|
1491
|
-
{ worker_class: MessageWorker, num_workers: 2 },
|
|
1492
|
-
{ worker_class: ScheduledWorker, num_workers: 1 }
|
|
1493
|
-
],
|
|
1494
|
-
work_queue: work_queue
|
|
1495
|
-
)
|
|
1496
|
-
|
|
1497
|
-
server.run
|
|
1498
|
-
----
|
|
1499
|
-
|
|
1500
|
-
=== Graceful shutdown strategies
|
|
1501
|
-
|
|
1502
|
-
==== Signal-based shutdown (production)
|
|
1503
|
-
|
|
1504
|
-
[source,ruby]
|
|
1505
|
-
----
|
|
1506
|
-
# Server automatically handles SIGTERM
|
|
1507
|
-
server = Fractor::ContinuousServer.new(
|
|
1508
|
-
worker_pools: [{ worker_class: MyWorker }],
|
|
1509
|
-
work_queue: work_queue,
|
|
1510
|
-
log_file: '/var/log/myapp/server.log'
|
|
1511
|
-
)
|
|
1512
|
-
|
|
1513
|
-
# Just run the server - signals handled automatically
|
|
1514
|
-
server.run
|
|
1515
|
-
|
|
1516
|
-
# In production:
|
|
1517
|
-
# systemctl stop myapp # Sends SIGTERM
|
|
1518
|
-
# docker stop container # Sends SIGTERM
|
|
1519
|
-
# kill -TERM <pid> # Manual SIGTERM
|
|
1520
|
-
----
|
|
1521
|
-
|
|
1522
|
-
==== Time-based shutdown
|
|
1523
|
-
|
|
1524
|
-
[source,ruby]
|
|
1525
|
-
----
|
|
1526
|
-
server_thread = Thread.new { server.run }
|
|
1527
|
-
|
|
1528
|
-
# Run for specific duration
|
|
1529
|
-
sleep 3600 # Run for 1 hour
|
|
1530
|
-
server.stop
|
|
1531
|
-
server_thread.join
|
|
1532
|
-
----
|
|
1533
|
-
|
|
1534
|
-
==== Condition-based shutdown
|
|
1535
|
-
|
|
1536
|
-
[source,ruby]
|
|
1537
|
-
----
|
|
1538
|
-
server_thread = Thread.new { server.run }
|
|
1539
|
-
|
|
1540
|
-
# Monitor thread checks conditions
|
|
1541
|
-
monitor = Thread.new do
|
|
1542
|
-
loop do
|
|
1543
|
-
if should_shutdown?
|
|
1544
|
-
server.stop
|
|
1545
|
-
break
|
|
1546
|
-
end
|
|
1547
|
-
sleep 10
|
|
1548
|
-
end
|
|
1549
|
-
end
|
|
1550
|
-
|
|
1551
|
-
server_thread.join
|
|
1552
|
-
monitor.kill
|
|
1553
|
-
----
|
|
1554
|
-
|
|
1555
|
-
=== Before/after comparison
|
|
1556
|
-
|
|
1557
|
-
The chat server example demonstrates the real-world impact of using the
|
|
1558
|
-
high-level primitives.
|
|
1559
|
-
|
|
1560
|
-
==== Before: Low-level API (279 lines)
|
|
1561
|
-
|
|
1562
|
-
Required manual management of:
|
|
1563
|
-
|
|
1564
|
-
* Supervisor thread creation and lifecycle (~15 lines)
|
|
1565
|
-
* Results processing thread with loops (~50 lines)
|
|
1566
|
-
* Queue creation and synchronization (~10 lines)
|
|
1567
|
-
* Signal handling setup (~15 lines)
|
|
1568
|
-
* Thread coordination and shutdown (~20 lines)
|
|
1569
|
-
* IO.select event loop (~110 lines)
|
|
1570
|
-
* Manual error handling throughout (~59 lines)
|
|
1571
|
-
|
|
1572
|
-
==== After: High-level primitives (167 lines)
|
|
1573
|
-
|
|
1574
|
-
Eliminated boilerplate:
|
|
1575
|
-
|
|
1576
|
-
* WorkQueue handles queue and synchronization (automatic)
|
|
1577
|
-
* ContinuousServer manages all threads (automatic)
|
|
1578
|
-
* Callbacks replace manual results loops (automatic)
|
|
1579
|
-
* Signal handling built-in (automatic)
|
|
1580
|
-
* Graceful shutdown coordinated (automatic)
|
|
1581
|
-
|
|
1582
|
-
Result: **40% code reduction** (112 fewer lines), simpler architecture, fewer
|
|
1583
|
-
error-prone details.
|
|
1584
|
-
|
|
1585
|
-
See link:examples/continuous_chat_fractor/chat_server.rb[the refactored chat
|
|
1586
|
-
server] for the complete example.
|
|
1587
|
-
|
|
1588
|
-
|
|
1589
|
-
|
|
1590
|
-
|
|
1591
|
-
== Process monitoring and logging
|
|
1592
|
-
|
|
1593
|
-
=== Status monitoring and health checks
|
|
1594
|
-
|
|
1595
|
-
The signals SIGUSR1 (or SIGBREAK on Windows) can be used for health checks.
|
|
1596
|
-
|
|
1597
|
-
When the signal is received, the supervisor prints its current status to
|
|
1598
|
-
standard output.
|
|
1599
|
-
|
|
1600
|
-
[example]
|
|
1601
|
-
Sending the signal:
|
|
1602
|
-
|
|
1603
|
-
Unix:
|
|
1604
|
-
|
|
1605
|
-
[source,sh]
|
|
1606
|
-
----
|
|
1607
|
-
# Send SIGUSR1 to the supervisor process
|
|
1608
|
-
kill -USR1 <pid>
|
|
1609
|
-
----
|
|
1610
|
-
|
|
1611
|
-
Windows:
|
|
1612
|
-
|
|
1613
|
-
[source,sh]
|
|
1614
|
-
----
|
|
1615
|
-
# Send SIGBREAK to the supervisor process
|
|
1616
|
-
kill -BREAK <pid>
|
|
1617
|
-
----
|
|
1618
|
-
|
|
1619
|
-
Output:
|
|
1620
|
-
|
|
1621
|
-
[source]
|
|
1622
|
-
----
|
|
1623
|
-
=== Fractor Supervisor Status ===
|
|
1624
|
-
Mode: Continuous
|
|
1625
|
-
Running: true
|
|
1626
|
-
Workers: 4
|
|
1627
|
-
Idle workers: 2
|
|
1628
|
-
Queue size: 15
|
|
1629
|
-
Results: 127
|
|
1630
|
-
Errors: 3
|
|
1631
|
-
----
|
|
1632
|
-
|
|
1633
|
-
=== Logging
|
|
1634
|
-
|
|
1635
|
-
Fractor supports logging of its operations to a specified log file.
|
|
1636
|
-
|
|
1637
|
-
For ContinuousServer, pass the `log_file` parameter:
|
|
1638
|
-
|
|
1639
|
-
[source,ruby]
|
|
1640
|
-
----
|
|
1641
|
-
server = Fractor::ContinuousServer.new(
|
|
1642
|
-
worker_pools: [{ worker_class: MyWorker }],
|
|
1643
|
-
work_queue: work_queue,
|
|
1644
|
-
log_file: 'logs/server.log'
|
|
1645
|
-
)
|
|
1646
|
-
----
|
|
1647
|
-
|
|
1648
|
-
For manual Supervisor usage, set the `FRACTOR_LOG_FILE` environment variable
|
|
1649
|
-
before starting your application:
|
|
1650
|
-
|
|
1651
|
-
[source,sh]
|
|
1652
|
-
----
|
|
1653
|
-
export FRACTOR_LOG_FILE=/path/to/logs/server.log
|
|
1654
|
-
ruby my_fractor_app.rb
|
|
1655
|
-
----
|
|
1656
|
-
|
|
1657
|
-
The log file will contain detailed information about the supervisor's
|
|
1658
|
-
operations, including worker activity, work distribution, results, and errors.
|
|
1659
|
-
|
|
1660
|
-
.Examples of accessing logs
|
|
1661
|
-
[example]
|
|
1662
|
-
[source,sh]
|
|
1663
|
-
----
|
|
1664
|
-
# Check if server is responsive (Unix/Linux/macOS)
|
|
1665
|
-
kill -USR1 <pid> && tail -f /path/to/logs/server.log
|
|
1666
|
-
|
|
1667
|
-
# Monitor with systemd
|
|
1668
|
-
systemctl status fractor-server
|
|
1669
|
-
journalctl -u fractor-server -f
|
|
1670
|
-
|
|
1671
|
-
# Monitor with Docker
|
|
1672
|
-
docker logs -f <container_id>
|
|
1673
|
-
----
|
|
1674
|
-
|
|
1675
|
-
|
|
1676
|
-
|
|
1677
|
-
|
|
1678
|
-
== Signal handling
|
|
1679
|
-
|
|
1680
|
-
=== General
|
|
1681
|
-
|
|
1682
|
-
Fractor provides production-ready signal handling for process control and
|
|
1683
|
-
monitoring. The framework supports different signals depending on the operating
|
|
1684
|
-
system, enabling graceful shutdown and runtime status monitoring.
|
|
1685
|
-
|
|
1686
|
-
=== Unix signals (Linux, macOS, Unix)
|
|
1687
|
-
|
|
1688
|
-
==== SIGINT (Ctrl+C)
|
|
1689
|
-
|
|
1690
|
-
Interactive interrupt signal for graceful shutdown.
|
|
1691
|
-
|
|
1692
|
-
Usage:
|
|
1693
|
-
|
|
1694
|
-
* Press `Ctrl+C` in the terminal running Fractor
|
|
1695
|
-
* Behavior depends on mode:
|
|
1696
|
-
** *Batch mode*: Stops immediately after current work completes
|
|
1697
|
-
** *Continuous mode*: Initiates graceful shutdown
|
|
1698
|
-
|
|
1699
|
-
==== SIGTERM
|
|
1700
|
-
|
|
1701
|
-
Standard Unix termination signal, preferred for production deployments.
|
|
1702
|
-
|
|
1703
|
-
This ensures a graceful shutdown of the Fractor supervisor and its workers.
|
|
1704
|
-
|
|
1705
|
-
Usage:
|
|
1706
|
-
|
|
1707
|
-
[source,sh]
|
|
1708
|
-
----
|
|
1709
|
-
kill -TERM <pid>
|
|
1710
|
-
# or simply
|
|
1711
|
-
kill <pid> # SIGTERM is the default
|
|
1712
|
-
----
|
|
1713
|
-
|
|
1714
|
-
Typical signals from service managers:
|
|
1715
|
-
|
|
1716
|
-
* Systemd sends SIGTERM on `systemctl stop`
|
|
1717
|
-
* Docker sends SIGTERM on `docker stop`
|
|
1718
|
-
* Kubernetes sends SIGTERM during pod termination
|
|
1719
|
-
|
|
1720
|
-
[source,ini]
|
|
1721
|
-
----
|
|
1722
|
-
# Example systemd service
|
|
1723
|
-
[Service]
|
|
1724
|
-
ExecStart=/usr/bin/ruby /path/to/fractor_server.rb
|
|
1725
|
-
KillMode=process
|
|
1726
|
-
KillSignal=SIGTERM
|
|
1727
|
-
TimeoutStopSec=30
|
|
1728
|
-
----
|
|
1729
|
-
|
|
1730
|
-
==== SIGUSR1
|
|
1731
|
-
|
|
1732
|
-
Real-time status monitoring without stopping the process.
|
|
1733
|
-
|
|
1734
|
-
Usage:
|
|
1735
|
-
|
|
1736
|
-
[source,sh]
|
|
1737
|
-
----
|
|
1738
|
-
kill -USR1 <pid>
|
|
1739
|
-
----
|
|
1740
|
-
|
|
1741
|
-
Output example:
|
|
1742
|
-
|
|
1743
|
-
[example]
|
|
1744
|
-
[source]
|
|
1745
|
-
----
|
|
1746
|
-
=== Fractor Supervisor Status ===
|
|
1747
|
-
Mode: Continuous
|
|
1748
|
-
Running: true
|
|
1749
|
-
Workers: 4
|
|
1750
|
-
Idle workers: 2
|
|
1751
|
-
Queue size: 15
|
|
1752
|
-
Results: 127
|
|
1753
|
-
Errors: 3
|
|
1754
|
-
----
|
|
1755
|
-
|
|
1756
|
-
=== Windows signals
|
|
1757
|
-
|
|
1758
|
-
==== SIGBREAK (Ctrl+Break)
|
|
1759
|
-
|
|
1760
|
-
Windows alternative to SIGUSR1 for status monitoring.
|
|
1761
|
-
|
|
1762
|
-
Usage:
|
|
1763
|
-
|
|
1764
|
-
* Press `Ctrl+Break` in the terminal running Fractor
|
|
1765
|
-
* Same output as SIGUSR1 on Unix
|
|
1766
|
-
|
|
1767
|
-
[NOTE]
|
|
1768
|
-
SIGUSR1 is not available on Windows. Use `Ctrl+Break` instead for status
|
|
1769
|
-
monitoring on Windows platforms.
|
|
1770
|
-
|
|
1771
|
-
|
|
1772
|
-
=== Signal behavior by mode
|
|
1773
|
-
|
|
1774
|
-
==== Batch mode
|
|
1775
|
-
|
|
1776
|
-
In batch processing mode:
|
|
1777
|
-
|
|
1778
|
-
* SIGINT/SIGTERM: Stops immediately after current work completes
|
|
1779
|
-
* SIGUSR1/SIGBREAK: Displays current status
|
|
1780
|
-
|
|
1781
|
-
==== Continuous mode
|
|
1782
|
-
|
|
1783
|
-
In continuous mode (long-running servers):
|
|
1784
|
-
|
|
1785
|
-
* SIGINT/SIGTERM: Graceful shutdown within ~2 seconds
|
|
1786
|
-
** Stops accepting new work
|
|
1787
|
-
** Completes in-progress work
|
|
1788
|
-
** Cleans up resources
|
|
1789
|
-
* SIGUSR1/SIGBREAK: Displays current status
|
|
1790
|
-
|
|
1791
|
-
|
|
1792
|
-
|
|
1793
|
-
|
|
1794
|
-
== Running a basic example
|
|
1795
|
-
|
|
1796
|
-
. Install the gem as described in the Installation section.
|
|
1797
|
-
|
|
1798
|
-
. Create a new Ruby file (e.g., `my_fractor_example.rb`) with your
|
|
1799
|
-
implementation:
|
|
1800
|
-
|
|
1801
|
-
[source,ruby]
|
|
1802
|
-
----
|
|
1803
|
-
require 'fractor'
|
|
1804
|
-
|
|
1805
|
-
# Define your Work class
|
|
1806
|
-
class MyWork < Fractor::Work
|
|
1807
|
-
def to_s
|
|
1808
|
-
"MyWork: #{@input}"
|
|
1809
|
-
end
|
|
1810
|
-
end
|
|
1811
|
-
|
|
1812
|
-
# Define your Worker class
|
|
1813
|
-
class MyWorker < Fractor::Worker
|
|
1814
|
-
def process(work)
|
|
1815
|
-
if work.input == 5
|
|
1816
|
-
# Return a Fractor::WorkResult for errors
|
|
1817
|
-
return Fractor::WorkResult.new(
|
|
1818
|
-
error: "Error processing work #{work.input}",
|
|
1819
|
-
work: work
|
|
1820
|
-
)
|
|
1821
|
-
end
|
|
1822
|
-
|
|
1823
|
-
calculated = work.input * 2
|
|
1824
|
-
# Return a Fractor::WorkResult for success
|
|
1825
|
-
Fractor::WorkResult.new(result: calculated, work: work)
|
|
1826
|
-
end
|
|
1827
|
-
end
|
|
1828
|
-
|
|
1829
|
-
# Create supervisor with a worker pool
|
|
1830
|
-
supervisor = Fractor::Supervisor.new(
|
|
1831
|
-
worker_pools: [
|
|
1832
|
-
{ worker_class: MyWorker, num_workers: 2 }
|
|
1833
|
-
]
|
|
1834
|
-
)
|
|
1835
|
-
|
|
1836
|
-
# Create Work objects
|
|
1837
|
-
work_items = (1..10).map { |i| MyWork.new(i) }
|
|
1838
|
-
|
|
1839
|
-
# Add work items
|
|
1840
|
-
supervisor.add_work_items(work_items)
|
|
1841
|
-
|
|
1842
|
-
# Run processing
|
|
1843
|
-
supervisor.run
|
|
1844
|
-
|
|
1845
|
-
# Display results
|
|
1846
|
-
puts "Results: #{supervisor.results.results.map(&:result).join(', ')}"
|
|
1847
|
-
puts "Errors: #{supervisor.results.errors.map { |e| e.work.input }.join(', ')}"
|
|
1848
|
-
----
|
|
1849
|
-
|
|
1850
|
-
. Run the example from your terminal:
|
|
1851
|
-
|
|
1852
|
-
[source,sh]
|
|
1853
|
-
----
|
|
1854
|
-
ruby my_fractor_example.rb
|
|
1855
|
-
----
|
|
1856
|
-
|
|
1857
|
-
You will see output showing Ractors starting, receiving work, processing it, and
|
|
1858
|
-
the final aggregated results, including any errors encountered. Press `Ctrl+C`
|
|
1859
|
-
during execution to test the graceful shutdown.
|
|
1860
|
-
|
|
1861
|
-
|
|
1862
|
-
|
|
1863
|
-
|
|
1864
|
-
== Example applications
|
|
1865
|
-
|
|
1866
|
-
=== General
|
|
1867
|
-
|
|
1868
|
-
The Fractor gem comes with several example applications that demonstrate various
|
|
1869
|
-
patterns and use cases. Each example can be found in the `examples` directory of
|
|
1870
|
-
the gem repository. Detailed descriptions for these are provided below.
|
|
1871
|
-
|
|
1872
|
-
=== Pipeline mode examples
|
|
1873
|
-
|
|
1874
|
-
==== Simple example
|
|
1875
|
-
|
|
1876
|
-
The Simple Example (link:examples/simple/[examples/simple/]) demonstrates the
|
|
1877
|
-
basic usage of the Fractor framework. It shows how to create a simple Work
|
|
1878
|
-
class, a Worker class, and a Supervisor to manage the processing of work items
|
|
1879
|
-
in parallel. This example serves as a starting point for understanding how to
|
|
1880
|
-
use Fractor.
|
|
1881
|
-
|
|
1882
|
-
Key features:
|
|
1883
|
-
|
|
1884
|
-
* Basic Work and Worker class implementation
|
|
1885
|
-
* Simple Supervisor setup
|
|
1886
|
-
* Parallel processing of work items
|
|
1887
|
-
* Error handling and result aggregation
|
|
1888
|
-
* Auto-detection of available processors
|
|
1889
|
-
* Graceful shutdown on completion
|
|
1890
|
-
|
|
1891
|
-
==== Auto-detection example
|
|
1892
|
-
|
|
1893
|
-
The Auto-Detection Example
|
|
1894
|
-
(link:examples/auto_detection/[examples/auto_detection/]) demonstrates
|
|
1895
|
-
Fractor's automatic worker detection feature. It shows how to use
|
|
1896
|
-
auto-detection, explicit configuration, and mixed approaches for controlling
|
|
1897
|
-
the number of workers.
|
|
1898
|
-
|
|
1899
|
-
Key features:
|
|
1900
|
-
|
|
1901
|
-
* Automatic detection of available processors
|
|
1902
|
-
* Comparison of auto-detection vs explicit configuration
|
|
1903
|
-
* Mixed configuration with multiple worker pools
|
|
1904
|
-
* Best practices for worker configuration
|
|
1905
|
-
* Portable code that adapts to different environments
|
|
1906
|
-
|
|
1907
|
-
==== Hierarchical hasher
|
|
1908
|
-
|
|
1909
|
-
The Hierarchical Hasher example
|
|
1910
|
-
(link:examples/hierarchical_hasher/[examples/hierarchical_hasher/]) demonstrates
|
|
1911
|
-
how to use the Fractor framework to process a file in parallel by breaking it
|
|
1912
|
-
into chunks, hashing each chunk independently, and then combining the results
|
|
1913
|
-
into a final hash. This approach is useful for processing large files
|
|
1914
|
-
efficiently.
|
|
1915
|
-
|
|
1916
|
-
Key features:
|
|
1917
|
-
|
|
1918
|
-
* Parallel data chunking for large files
|
|
1919
|
-
* Independent processing of data segments
|
|
1920
|
-
* Aggregation of results to form a final output
|
|
1921
|
-
|
|
1922
|
-
==== Multi-work type
|
|
1923
|
-
|
|
1924
|
-
The Multi-Work Type example
|
|
1925
|
-
(link:examples/multi_work_type/[examples/multi_work_type/]) demonstrates how a
|
|
1926
|
-
single Fractor supervisor and worker can handle multiple types of work items
|
|
1927
|
-
(e.g., `TextWork` and `ImageWork`). The worker intelligently adapts its
|
|
1928
|
-
processing strategy based on the class of the incoming work item.
|
|
1929
|
-
|
|
1930
|
-
Key features:
|
|
1931
|
-
|
|
1932
|
-
* Support for multiple `Fractor::Work` subclasses
|
|
1933
|
-
* Polymorphic worker processing based on work type
|
|
1934
|
-
* Unified workflow for diverse tasks
|
|
1935
|
-
|
|
1936
|
-
==== Pipeline processing
|
|
1937
|
-
|
|
1938
|
-
The Pipeline Processing example
|
|
1939
|
-
(link:examples/pipeline_processing/[examples/pipeline_processing/]) implements a
|
|
1940
|
-
multi-stage processing pipeline where data flows sequentially through a series
|
|
1941
|
-
of transformations. The output of one stage becomes the input for the next, and
|
|
1942
|
-
different stages can operate concurrently on different data items.
|
|
1943
|
-
|
|
1944
|
-
Key features:
|
|
1945
|
-
|
|
1946
|
-
* Sequential data flow through multiple processing stages
|
|
1947
|
-
* Concurrent execution of different pipeline stages
|
|
1948
|
-
* Data transformation at each step of the pipeline
|
|
1949
|
-
|
|
1950
|
-
==== Producer/subscriber
|
|
1951
|
-
|
|
1952
|
-
The Producer/Subscriber example
|
|
1953
|
-
(link:examples/producer_subscriber/[examples/producer_subscriber/]) showcases a
|
|
1954
|
-
multi-stage document processing system where initial work (processing a
|
|
1955
|
-
document) can generate additional sub-work items (processing sections of the
|
|
1956
|
-
document). This creates a hierarchical processing pattern.
|
|
1957
|
-
|
|
1958
|
-
Key features:
|
|
1959
|
-
|
|
1960
|
-
* Implementation of producer-consumer patterns
|
|
1961
|
-
* Dynamic generation of sub-work based on initial processing
|
|
1962
|
-
* Construction of hierarchical result structures
|
|
1963
|
-
|
|
1964
|
-
==== Scatter/gather
|
|
1965
|
-
|
|
1966
|
-
The Scatter/Gather example
|
|
1967
|
-
(link:examples/scatter_gather/[examples/scatter_gather/]) illustrates how a
|
|
1968
|
-
large task or dataset is broken down (scattered) into smaller, independent
|
|
1969
|
-
subtasks. These subtasks are processed in parallel by multiple workers, and
|
|
1970
|
-
their results are then collected (gathered) and combined to produce the final
|
|
1971
|
-
output.
|
|
1972
|
-
|
|
1973
|
-
Key features:
|
|
1974
|
-
|
|
1975
|
-
* Distribution of a large task into smaller, parallelizable subtasks
|
|
1976
|
-
* Concurrent processing of subtasks
|
|
1977
|
-
* Aggregation of partial results into a final result
|
|
1978
|
-
|
|
1979
|
-
==== Specialized workers
|
|
1980
|
-
|
|
1981
|
-
The Specialized Workers example
|
|
1982
|
-
(link:examples/specialized_workers/[examples/specialized_workers/]) demonstrates
|
|
1983
|
-
creating distinct worker types, each tailored to handle specific kinds of tasks
|
|
1984
|
-
(e.g., `ComputeWorker` for CPU-intensive operations and `DatabaseWorker` for
|
|
1985
|
-
I/O-bound database interactions). This allows for optimized resource utilization
|
|
1986
|
-
and domain-specific logic.
|
|
1987
|
-
|
|
1988
|
-
Key features:
|
|
1989
|
-
|
|
1990
|
-
* Creation of worker classes for specific processing domains
|
|
1991
|
-
* Routing of work items to appropriately specialized workers
|
|
1992
|
-
* Optimization of resources and logic per task type
|
|
1993
|
-
|
|
1994
|
-
=== Continuous mode examples
|
|
1995
|
-
|
|
1996
|
-
==== Plain socket implementation
|
|
1997
|
-
|
|
1998
|
-
The plain socket implementation
|
|
1999
|
-
(link:examples/continuous_chat_server/[examples/continuous_chat_server/])
|
|
2000
|
-
provides a baseline chat server using plain TCP sockets without Fractor. This
|
|
2001
|
-
serves as a comparison point to understand the benefits of using Fractor for
|
|
2002
|
-
continuous processing.
|
|
2003
|
-
|
|
2004
|
-
==== Fractor-based implementation
|
|
2005
|
-
|
|
2006
|
-
The Fractor-based implementation
|
|
2007
|
-
(link:examples/continuous_chat_fractor/[examples/continuous_chat_fractor/])
|
|
2008
|
-
demonstrates how to build a production-ready chat server using Fractor's
|
|
2009
|
-
continuous mode with high-level primitives.
|
|
2010
|
-
|
|
2011
|
-
Key features:
|
|
2012
|
-
|
|
2013
|
-
* *Continuous mode operation*: Server runs indefinitely processing messages as
|
|
2014
|
-
they arrive
|
|
2015
|
-
* *High-level primitives*: Uses WorkQueue and ContinuousServer to eliminate
|
|
2016
|
-
boilerplate
|
|
2017
|
-
* *Graceful shutdown*: Production-ready signal handling (SIGINT, SIGTERM,
|
|
2018
|
-
SIGUSR1/SIGBREAK)
|
|
2019
|
-
* *Callback-based results*: Clean separation of concerns with on_result and
|
|
2020
|
-
on_error callbacks
|
|
2021
|
-
* *Cross-platform support*: Works on Unix/Linux/macOS and Windows
|
|
2022
|
-
* *Process monitoring*: Runtime status checking via signals
|
|
2023
|
-
* *40% code reduction*: 167 lines vs 279 lines with low-level API
|
|
2024
|
-
|
|
2025
|
-
The implementation includes:
|
|
2026
|
-
|
|
2027
|
-
* `chat_common.rb`: Work and Worker class definitions for chat message
|
|
2028
|
-
processing
|
|
2029
|
-
* `chat_server.rb`: Main server using high-level primitives
|
|
2030
|
-
* `simulate.rb`: Test client simulator
|
|
2031
|
-
|
|
2032
|
-
This example demonstrates production deployment patterns including:
|
|
2033
|
-
|
|
2034
|
-
* Systemd service integration
|
|
2035
|
-
* Docker container deployment
|
|
2036
|
-
* Process monitoring and health checks
|
|
2037
|
-
* Graceful restart procedures
|
|
2038
|
-
|
|
2039
|
-
See link:examples/continuous_chat_fractor/README.adoc[the chat server README]
|
|
2040
|
-
for detailed implementation documentation.
|
|
2041
|
-
|
|
2042
|
-
|
|
2043
|
-
|
|
2044
|
-
|
|
2045
|
-
== Copyright and license
|
|
220
|
+
== Copyright
|
|
2046
221
|
|
|
2047
222
|
Copyright Ribose.
|
|
2048
|
-
|
|
2049
|
-
Licensed under the Ribose BSD 2-Clause License.
|