cadence-ruby 0.0.0 → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (83) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +456 -0
  3. data/cadence.gemspec +9 -2
  4. data/lib/cadence-ruby.rb +1 -0
  5. data/lib/cadence.rb +176 -0
  6. data/lib/cadence/activity.rb +33 -0
  7. data/lib/cadence/activity/async_token.rb +34 -0
  8. data/lib/cadence/activity/context.rb +64 -0
  9. data/lib/cadence/activity/poller.rb +89 -0
  10. data/lib/cadence/activity/task_processor.rb +73 -0
  11. data/lib/cadence/activity/workflow_convenience_methods.rb +41 -0
  12. data/lib/cadence/client.rb +21 -0
  13. data/lib/cadence/client/errors.rb +8 -0
  14. data/lib/cadence/client/thrift_client.rb +380 -0
  15. data/lib/cadence/concerns/executable.rb +33 -0
  16. data/lib/cadence/concerns/typed.rb +40 -0
  17. data/lib/cadence/configuration.rb +36 -0
  18. data/lib/cadence/errors.rb +21 -0
  19. data/lib/cadence/executable_lookup.rb +25 -0
  20. data/lib/cadence/execution_options.rb +32 -0
  21. data/lib/cadence/json.rb +18 -0
  22. data/lib/cadence/metadata.rb +73 -0
  23. data/lib/cadence/metadata/activity.rb +28 -0
  24. data/lib/cadence/metadata/base.rb +17 -0
  25. data/lib/cadence/metadata/decision.rb +25 -0
  26. data/lib/cadence/metadata/workflow.rb +23 -0
  27. data/lib/cadence/metrics.rb +37 -0
  28. data/lib/cadence/metrics_adapters/log.rb +33 -0
  29. data/lib/cadence/metrics_adapters/null.rb +9 -0
  30. data/lib/cadence/middleware/chain.rb +30 -0
  31. data/lib/cadence/middleware/entry.rb +9 -0
  32. data/lib/cadence/retry_policy.rb +27 -0
  33. data/lib/cadence/saga/concern.rb +37 -0
  34. data/lib/cadence/saga/result.rb +22 -0
  35. data/lib/cadence/saga/saga.rb +24 -0
  36. data/lib/cadence/testing.rb +50 -0
  37. data/lib/cadence/testing/cadence_override.rb +112 -0
  38. data/lib/cadence/testing/future_registry.rb +27 -0
  39. data/lib/cadence/testing/local_activity_context.rb +17 -0
  40. data/lib/cadence/testing/local_workflow_context.rb +207 -0
  41. data/lib/cadence/testing/workflow_execution.rb +44 -0
  42. data/lib/cadence/testing/workflow_override.rb +36 -0
  43. data/lib/cadence/thread_local_context.rb +14 -0
  44. data/lib/cadence/thread_pool.rb +68 -0
  45. data/lib/cadence/types.rb +7 -0
  46. data/lib/cadence/utils.rb +17 -0
  47. data/lib/cadence/uuid.rb +19 -0
  48. data/lib/cadence/version.rb +1 -1
  49. data/lib/cadence/worker.rb +91 -0
  50. data/lib/cadence/workflow.rb +42 -0
  51. data/lib/cadence/workflow/context.rb +266 -0
  52. data/lib/cadence/workflow/convenience_methods.rb +34 -0
  53. data/lib/cadence/workflow/decision.rb +39 -0
  54. data/lib/cadence/workflow/decision_state_machine.rb +48 -0
  55. data/lib/cadence/workflow/decision_task_processor.rb +105 -0
  56. data/lib/cadence/workflow/dispatcher.rb +31 -0
  57. data/lib/cadence/workflow/execution_info.rb +45 -0
  58. data/lib/cadence/workflow/executor.rb +45 -0
  59. data/lib/cadence/workflow/future.rb +75 -0
  60. data/lib/cadence/workflow/history.rb +76 -0
  61. data/lib/cadence/workflow/history/event.rb +71 -0
  62. data/lib/cadence/workflow/history/event_target.rb +79 -0
  63. data/lib/cadence/workflow/history/window.rb +40 -0
  64. data/lib/cadence/workflow/poller.rb +74 -0
  65. data/lib/cadence/workflow/replay_aware_logger.rb +36 -0
  66. data/lib/cadence/workflow/serializer.rb +31 -0
  67. data/lib/cadence/workflow/serializer/base.rb +22 -0
  68. data/lib/cadence/workflow/serializer/cancel_timer.rb +19 -0
  69. data/lib/cadence/workflow/serializer/complete_workflow.rb +20 -0
  70. data/lib/cadence/workflow/serializer/fail_workflow.rb +21 -0
  71. data/lib/cadence/workflow/serializer/record_marker.rb +21 -0
  72. data/lib/cadence/workflow/serializer/request_activity_cancellation.rb +19 -0
  73. data/lib/cadence/workflow/serializer/schedule_activity.rb +54 -0
  74. data/lib/cadence/workflow/serializer/start_child_workflow.rb +52 -0
  75. data/lib/cadence/workflow/serializer/start_timer.rb +20 -0
  76. data/lib/cadence/workflow/state_manager.rb +324 -0
  77. data/lib/gen/thrift/cadence_constants.rb +11 -0
  78. data/lib/gen/thrift/cadence_types.rb +11 -0
  79. data/lib/gen/thrift/shared_constants.rb +11 -0
  80. data/lib/gen/thrift/shared_types.rb +4600 -0
  81. data/lib/gen/thrift/workflow_service.rb +3142 -0
  82. data/rbi/cadence-ruby.rbi +39 -0
  83. metadata +152 -5
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: df8e7fe5d4ae9875c0ecc1bd549ecaeec06c618b62036814173346370c7aad9c
4
- data.tar.gz: 3bf69fa228a085934a11620110962d93b587a8768f72ce3369613fac93d0ccf6
3
+ metadata.gz: 630ceaf2df3bdaf72836bc6efd65bb753967a65b867269a3cff7c9479252bd72
4
+ data.tar.gz: cd8645ae092493325f25071257be7c87fcb5a4e43376b19d79c7273260966ffc
5
5
  SHA512:
6
- metadata.gz: 4f5f5567301de8d5db1538c29fbbf012640cd93dfaf22eaaf16f9dfdbf373e61e5c31a2fb6bd1bccc2b740f61ed8b21b79e358568d37f1ddc0418ab11f01ab82
7
- data.tar.gz: 7a76002bb67315fd5817100db3d1479f0c632a5d44510715af5a71078379b658b608d66107f55341892291a91349b3754cd677fbb39cafb2b40e0042c5ee546e
6
+ metadata.gz: dc61fa5be6a36cf0dab33000d1ff3d5a2d1be22574074dc1a6b59598ce8f0332dbcfe268a2edd649d9e24d064a4c467fff448a17d8508d96cef3adb5e6328779
7
+ data.tar.gz: 5bef6a80f0b1ecee69d8e08d1fb6b746f2a98bc5dec8a388fd53738ef2e60e9c2f943bd4a81ef9254574baae3c0ee3b7f3fdc7c031b2f6ceb69c43e66eabdc98
data/README.md CHANGED
@@ -1,5 +1,461 @@
1
1
  # Ruby worker for Cadence
2
2
 
3
+ <img src="./assets/cadence_logo_2.png" width="250" align="right" alt="Cadence" />
4
+
3
5
  A pure Ruby library for defining and running Cadence workflows and activities.
4
6
 
5
7
  To find more about Cadence please visit <https://cadenceworkflow.io/>.
8
+
9
+
10
+ ## Getting Started
11
+
12
+ *NOTE: Make sure you have both Cadence and TChannel Proxy up and running. Head over to
13
+ [this section](#installing-dependencies) for installation instructions.*
14
+
15
+ Clone this repository:
16
+
17
+ ```sh
18
+ > git clone git@github.com:coinbase/cadence-ruby.git
19
+ ```
20
+
21
+ Include this gem to your `Gemfile`:
22
+
23
+ ```ruby
24
+ gem 'cadence-ruby', path: 'path/to/a/cloned/cadence-ruby/'
25
+ ```
26
+
27
+ Define an activity:
28
+
29
+ ```ruby
30
+ class HelloActivity < Cadence::Activity
31
+ def execute(name)
32
+ puts "Hello #{name}!"
33
+
34
+ return
35
+ end
36
+ end
37
+ ```
38
+
39
+ Define a workflow:
40
+
41
+ ```ruby
42
+ require 'path/to/hello_activity'
43
+
44
+ class HelloWorldWorkflow < Cadence::Workflow
45
+ def execute
46
+ HelloActivity.execute!('World')
47
+
48
+ return
49
+ end
50
+ end
51
+ ```
52
+
53
+ Configure your Cadence connection:
54
+
55
+ ```ruby
56
+ Cadence.configure do |config|
57
+ config.host = 'localhost'
58
+ config.port = 6666 # this should point to the tchannel proxy
59
+ config.domain = 'ruby-samples'
60
+ config.task_list = 'hello-world'
61
+ end
62
+ ```
63
+
64
+ Register domain with the Cadence service:
65
+
66
+ ```ruby
67
+ Cadence.register_domain('ruby-samples', 'A safe space for playing with Cadence Ruby')
68
+ ```
69
+
70
+ Configure and start your worker process:
71
+
72
+ ```ruby
73
+ require 'cadence/worker'
74
+
75
+ worker = Cadence::Worker.new
76
+ worker.register_workflow(HelloWorldWorkflow)
77
+ worker.register_activity(HelloActivity)
78
+ worker.start
79
+ ```
80
+
81
+ And finally start your workflow:
82
+
83
+ ```ruby
84
+ require 'path/to/hello_world_workflow'
85
+
86
+ Cadence.start_workflow(HelloWorldWorkflow)
87
+ ```
88
+
89
+ Congratulation you've just created and executed a distributed workflow!
90
+
91
+ To view more details about your execution, point your browser to
92
+ <http://localhost:8088/domain/ruby-samples/workflows?range=last-3-hours&status=CLOSED>.
93
+
94
+ There are plenty of [runnable examples](examples/) demonstrating various features of this library
95
+ available, make sure to check them out.
96
+
97
+
98
+ ## Installing dependencies
99
+
100
+ In order to run your Ruby workers you need to have the Cadence service and the TChannel Proxy
101
+ running. Below are the instructions on setting these up:
102
+
103
+ ### Cadence
104
+
105
+ Cadence service handles all the persistence, fault tolerance and coordination of your workflows and
106
+ activities. To set it up locally, download and boot the Docker Compose file from the official repo:
107
+
108
+ ```sh
109
+ > curl -O https://raw.githubusercontent.com/uber/cadence/master/docker/docker-compose.yml
110
+
111
+ > docker-compose up
112
+ ```
113
+
114
+ ### TChannel Proxy
115
+
116
+ Right now the Cadence service only communicates with the workers using Thrift over TChannel.
117
+ Unfortunately there isn't a working TChannel protocol implementation for Ruby, so in order to
118
+ connect to the Cadence service a simple proxy was created. You can run it using:
119
+
120
+ ```sh
121
+ > cd proxy
122
+
123
+ > bin/proxy
124
+ ```
125
+
126
+ The code and detailed instructions can be found [here](proxy/).
127
+
128
+
129
+ ## Workflows
130
+
131
+ A workflow is defined using pure Ruby code, however it should contain only a high-level
132
+ deterministic outline of the steps (their composition) that need to be executed to complete a
133
+ workflow. The actual work should be defined in your activities.
134
+
135
+ *NOTE: Keep in mind that your workflow code can get run multiple times (replayed) during the same
136
+ execution, which is why it must NOT contain any non-deterministic code (network requests, DB
137
+ queries, etc) as it can break your workflows.*
138
+
139
+ Here's an example workflow:
140
+
141
+ ```ruby
142
+ class RenewSubscriptionWorkflow < Cadence::Workflow
143
+ def execute(user_id)
144
+ subscription = FetchUserSubscriptionActivity.execute!(user_id)
145
+ subscription ||= CreateUserSubscriptionActivity.execute!(user_id)
146
+
147
+ return if subscription[:active]
148
+
149
+ ChargeCreditCardActivity.execute!(subscription[:price], subscription[:card_token])
150
+
151
+ RenewedSubscriptionActivity.execute!(subscription[:id])
152
+ SendSubscriptionRenewalEmailActivity.execute!(user_id, subscription[:id])
153
+ rescue CreditCardNotChargedError => e
154
+ CancelSubscriptionActivity.execute!(subscription[:id])
155
+ SendSubscriptionCancellationEmailActivity.execute!(user_id, subscription[:id])
156
+ end
157
+ end
158
+ ```
159
+
160
+ In this simple workflow we are checking if a user has an active subscription and then attempt to
161
+ charge their credit card to renew an expired subscription, notifying the user of the outcome. All
162
+ the work is encapsulated in activities, while the workflow itself is responsible for calling the
163
+ activities in the right order, passing values between them and handling failures.
164
+
165
+ There is a couple of ways to execute an activity from your workflow:
166
+
167
+ ```ruby
168
+ # Calls the activity by its class and blocks the execution until activity is
169
+ # finished. The return value of your activity will get assigned to the result
170
+ result = MyActivity.execute!(arg1, arg2)
171
+
172
+ # Here's a non-blocking version of the execute, returning back the future that
173
+ # will get fulfilled when activity completes. This approach allows modelling
174
+ # asynchronous workflows with activities executed in parallel
175
+ future = MyActivity.execute(arg1, arg2)
176
+ result = future.get
177
+
178
+ # Full versions of the calls from above, but has more flexibility (shown below)
179
+ result = workflow.execute_activity!(MyActivity, arg1, arg2)
180
+ future = workflow.execute_activity(MyActivity, arg1, arg2)
181
+
182
+ # In case your workflow code does not have access to activity classes (separate
183
+ # process, activities implemented in a different language, etc), you can
184
+ # simply reference them by their names
185
+ workflow.execute_activity('MyActivity', arg1, arg2, options: { domain: 'my-domain', task_list: 'my-task-list' })
186
+ ```
187
+
188
+ Besides calling activities workflows can:
189
+
190
+ - Use timers
191
+ - Receive signals
192
+ - Execute other (child) workflows [not yet implemented]
193
+ - Respond to queries [not yet implemented]
194
+
195
+
196
+ ## Activities
197
+
198
+ An activity is a basic unit of work that performs the desired action (potentially causing
199
+ side-effects). It can return a result or raise an error. It is defined like so:
200
+
201
+ ```ruby
202
+ class CloseUserAccountActivity < Cadence::Activity
203
+ class UserNotFound < Cadence::ActivityException; end
204
+
205
+ def execute(user_id)
206
+ user = User.find_by(id: user_id)
207
+
208
+ raise UserNotFound, 'User with specified ID does not exist' unless user
209
+
210
+ user.close_account
211
+ user.save
212
+
213
+ AccountClosureEmail.deliver(user)
214
+
215
+ return
216
+ end
217
+ end
218
+ ```
219
+
220
+ It is important to make your activities **idempotent**, because they can get retried by Cadence (in
221
+ case a timeout is reached or your activity has thrown an error). You normally want to avoid
222
+ generating additional side effects during subsequent activity execution.
223
+
224
+ To achieve this there are two methods (returning a UUID token) available from your activity class:
225
+
226
+ - `activity.run_idem` — unique within for the current workflow execution (scoped to run_id)
227
+ - `activity.workflow_idem` — unique across all execution of the workflow (scoped to workflow_id)
228
+
229
+ Both tokens will remain the same across multiple retry attempts of the activity.
230
+
231
+ ### Asynchronous completion
232
+
233
+ When dealing with asynchronous business logic in your activities, you might need to wait for an
234
+ external event to complete your activity (e.g. a callback or a webhook). This can be achieved by
235
+ manually completing your activity using a provided `async_token` from activity's context:
236
+
237
+ ```ruby
238
+ class AsyncActivity < Cadence::Activity
239
+ def execute(user_id)
240
+ user = User.find_by(id: user_id)
241
+
242
+ # Pass the async_token to complete your activity later
243
+ ExternalSystem.verify_user(user, activity.async_token)
244
+
245
+ activity.async # prevents activity from completing immediately
246
+ end
247
+ end
248
+ ```
249
+
250
+ Later when a confirmation is received you'll need to complete your activity manually using the token
251
+ provided:
252
+
253
+ ```ruby
254
+ Cadence.complete_activity(async_token, result)
255
+ ```
256
+
257
+ Similarly you can fail the activity by calling:
258
+
259
+ ```ruby
260
+ Cadence.fail_activity(async_token, MyError.new('Something went wrong'))
261
+ ```
262
+
263
+ This doesn't change the behaviour from the workflow's perspective — as any other activity the result
264
+ will be returned or an error raised.
265
+
266
+ *NOTE: Make sure to configure your timeouts accordingly and not to set heartbeat timeout (off by
267
+ default) since you won't be able to emit heartbeats and your async activities will keep timing out.*
268
+
269
+ Similar behaviour can also be achieved in other ways (one which might be more preferable in your
270
+ specific use-case), e.g.:
271
+
272
+ - by polling for a result within your activity (long-running activities with heartbeat)
273
+ - using retry policy to keep retrying activity until a result is available
274
+ - completing your activity after the initial call is made, but then waiting on a completion signal
275
+ from your workflow
276
+
277
+
278
+ ## Worker
279
+
280
+ Worker is a process that communicates with the Cadence server and manages Workflow and Activity
281
+ execution. To start a worker:
282
+
283
+ ```ruby
284
+ require 'cadence/worker'
285
+
286
+ worker = Cadence::Worker.new
287
+ worker.register_workflow(HelloWorldWorkflow)
288
+ worker.register_activity(SomeActivity)
289
+ worker.register_activity(SomeOtherActivity)
290
+ worker.start
291
+ ```
292
+
293
+ A call to `worker.start` will take over the current process and will keep it unning until a `TERM`
294
+ or `INT` signal is received. By only registering a subset of your workflows/activities with a given
295
+ worker you can split processing across as many workers as you need.
296
+
297
+
298
+ ## Starting a workflow
299
+
300
+ All communication is handled via Cadence service, so in order to start a workflow you need to send a
301
+ message to Cadence:
302
+
303
+ ```ruby
304
+ Cadence.start_workflow(HelloWorldWorkflow)
305
+ ```
306
+
307
+ Optionally you can pass input and other options to the workflow:
308
+
309
+ ```ruby
310
+ Cadence.start_workflow(RenewSubscriptionWorkflow, user_id, options: { workflow_id: user_id })
311
+ ```
312
+
313
+ Passing in a `workflow_id` allows you to prevent concurrent execution of a workflow — a subsequent
314
+ call with the same `workflow_id` will always get rejected while it is still running, raising
315
+ `CadenceThrift::WorkflowExecutionAlreadyStartedError`. You can adjust the behaviour for finished
316
+ workflows by supplying the `workflow_id_reuse_policy:` argument with one of these options:
317
+
318
+ - `:allow_failed` will allow re-running workflows that have failed (terminated, cancelled, timed out or failed)
319
+ - `:allow` will allow re-running any finished workflows both failed and completed
320
+ - `:reject` will reject any subsequent attempt to run a workflow
321
+
322
+
323
+ ## Execution Options
324
+
325
+ There are lots of ways in which you can configure your Workflows and Activities. The common ones
326
+ (domain, task_list, timeouts and retry policy) can be defined in one of these places (in the order
327
+ of precedence):
328
+
329
+ 1. Inline when starting or registering a workflow/activity (use `options:` argument)
330
+ 2. In your workflow/activity class definitions by calling a class method (e.g. `domain 'my-domain'`)
331
+ 3. Globally, when configuring your Cadence library via `Cadence.configure`
332
+
333
+
334
+ ## Breaking Changes
335
+
336
+ Since the workflow execution has to be deterministic, breaking changes can not be simply added and
337
+ deployed — this will undermine the consistency of running workflows and might lead to unexpected
338
+ behaviour. However, breaking changes are often needed and these include:
339
+
340
+ - Adding new activities, timers, child workflows, etc.
341
+ - Remove existing activities, timers, child workflows, etc.
342
+ - Rearranging existing activities, timers, child workflows, etc.
343
+ - Adding/removing signal handlers
344
+
345
+ In order to add a breaking change you can use `workflow.has_release?(release_name)` method in your
346
+ workflows, which is guaranteed to return a consistent result whether or not it was called prior to
347
+ shipping the new release. It is also consistent for all the subsequent calls with the same
348
+ `release_name` — all of them will return the original result. Consider the following example:
349
+
350
+ ```ruby
351
+ class MyWorkflow < Cadence::Workflow
352
+ def execute
353
+ ActivityOld1.execute!
354
+
355
+ workflow.sleep(10)
356
+
357
+ ActivityOld2.execute!
358
+
359
+ return
360
+ end
361
+ end
362
+ ```
363
+
364
+ which got updated to:
365
+
366
+ ```ruby
367
+ class MyWorkflow < Cadence::Workflow
368
+ def execute
369
+ Activity1.execute!
370
+
371
+ if workflow.has_release?(:fix_1)
372
+ ActivityNew1.execute!
373
+ end
374
+
375
+ workflow.sleep(10)
376
+
377
+ if workflow.has_release?(:fix_1)
378
+ ActivityNew2.execute!
379
+ else
380
+ ActivityOld.execute!
381
+ end
382
+
383
+ if workflow.has_release?(:fix_2)
384
+ ActivityNew3.execute!
385
+ end
386
+
387
+ return
388
+ end
389
+ end
390
+ ```
391
+
392
+ If the release got deployed while the original workflow was waiting on a timer, `ActivityNew1` and
393
+ `ActivityNew2` won't get executed, because they are part of the same change (same release_name),
394
+ however `ActivityNew3` will get executed, since the release wasn't yet checked at the time. And for
395
+ every new execution of the workflow — all new activities will get executed, while `ActivityOld` will
396
+ not.
397
+
398
+ Later on you can clean it up and drop all the checks if you don't have any older workflows running
399
+ or expect them to ever be executed (e.g. reset).
400
+
401
+ *NOTE: Releases with different names do not depend on each other in any way.*
402
+
403
+ ## Testing
404
+
405
+ It is crucial to properly test your workflows and activities before running them in production. The
406
+ provided testing framework is still limited in functionality, but will allow you to test basic
407
+ use-cases.
408
+
409
+ The testing framework is not required automatically when you require `cadence-ruby`, so you have to
410
+ do this yourself (it is strongly recommended to only include this in your test environment,
411
+ `spec_helper.rb` or similar):
412
+
413
+ ```ruby
414
+ require 'cadence/testing'
415
+ ```
416
+
417
+ This will allow you to execute workflows locally by running `HelloWorldWorkflow.execute_locally`.
418
+ Any arguments provided will forwarded to your `#execute` method.
419
+
420
+ In case of a higher level end-to-end integration specs, where you need to execute a Cadence workflow
421
+ as part of your code, you can enable local testing:
422
+
423
+ ```ruby
424
+ Cadence::Testing.local!
425
+ ```
426
+
427
+ This will treat every `Cadence.start_workflow` call as local and perform your workflows inline. It
428
+ also works with a block, restoring the original mode back after the execution:
429
+
430
+ ```ruby
431
+ Cadence::Testing.local! do
432
+ Cadence.start_workflow(HelloWorldWorkflow)
433
+ end
434
+ ```
435
+
436
+ Make sure to check out [example integration specs](examples/specs/integration) for more details.
437
+
438
+
439
+ ## TODO
440
+
441
+ There's plenty of work to be done, but most importanly we need:
442
+
443
+ - Write specs for everything
444
+ - Implement support for missing features
445
+
446
+
447
+ ## LICENSE
448
+
449
+ Copyright 2020 Coinbase, Inc.
450
+
451
+ Licensed under the Apache License, Version 2.0 (the "License");
452
+ you may not use this file except in compliance with the License.
453
+ You may obtain a copy of the License at
454
+
455
+ http://www.apache.org/licenses/LICENSE-2.0
456
+
457
+ Unless required by applicable law or agreed to in writing, software
458
+ distributed under the License is distributed on an "AS IS" BASIS,
459
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
460
+ See the License for the specific language governing permissions and
461
+ limitations under the License.