redis_queued_locks 1.8.0 → 1.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. checksums.yaml +4 -4
  2. data/.ruby-version +1 -1
  3. data/CHANGELOG.md +66 -0
  4. data/README.md +528 -51
  5. data/lib/redis_queued_locks/acquier/acquire_lock/dequeue_from_lock_queue/log_visitor.rb +4 -0
  6. data/lib/redis_queued_locks/acquier/acquire_lock/dequeue_from_lock_queue.rb +4 -1
  7. data/lib/redis_queued_locks/acquier/acquire_lock/instr_visitor.rb +20 -5
  8. data/lib/redis_queued_locks/acquier/acquire_lock/log_visitor.rb +24 -0
  9. data/lib/redis_queued_locks/acquier/acquire_lock/try_to_lock/log_visitor.rb +56 -0
  10. data/lib/redis_queued_locks/acquier/acquire_lock/try_to_lock.rb +37 -30
  11. data/lib/redis_queued_locks/acquier/acquire_lock/yield_expire/log_visitor.rb +8 -0
  12. data/lib/redis_queued_locks/acquier/acquire_lock/yield_expire.rb +24 -12
  13. data/lib/redis_queued_locks/acquier/acquire_lock.rb +44 -20
  14. data/lib/redis_queued_locks/acquier/clear_dead_requests.rb +5 -1
  15. data/lib/redis_queued_locks/acquier/extend_lock_ttl.rb +5 -1
  16. data/lib/redis_queued_locks/acquier/lock_info.rb +4 -3
  17. data/lib/redis_queued_locks/acquier/locks.rb +2 -2
  18. data/lib/redis_queued_locks/acquier/queue_info.rb +2 -2
  19. data/lib/redis_queued_locks/acquier/release_all_locks.rb +12 -2
  20. data/lib/redis_queued_locks/acquier/release_lock.rb +12 -2
  21. data/lib/redis_queued_locks/client.rb +284 -34
  22. data/lib/redis_queued_locks/errors.rb +8 -0
  23. data/lib/redis_queued_locks/instrument.rb +8 -1
  24. data/lib/redis_queued_locks/logging.rb +8 -1
  25. data/lib/redis_queued_locks/resource.rb +59 -1
  26. data/lib/redis_queued_locks/swarm/acquirers.rb +44 -0
  27. data/lib/redis_queued_locks/swarm/flush_zombies.rb +133 -0
  28. data/lib/redis_queued_locks/swarm/probe_hosts.rb +69 -0
  29. data/lib/redis_queued_locks/swarm/redis_client_builder.rb +67 -0
  30. data/lib/redis_queued_locks/swarm/supervisor.rb +83 -0
  31. data/lib/redis_queued_locks/swarm/swarm_element/isolated.rb +287 -0
  32. data/lib/redis_queued_locks/swarm/swarm_element/threaded.rb +351 -0
  33. data/lib/redis_queued_locks/swarm/swarm_element.rb +8 -0
  34. data/lib/redis_queued_locks/swarm/zombie_info.rb +145 -0
  35. data/lib/redis_queued_locks/swarm.rb +246 -0
  36. data/lib/redis_queued_locks/utilities/lock.rb +22 -0
  37. data/lib/redis_queued_locks/utilities.rb +75 -0
  38. data/lib/redis_queued_locks/version.rb +2 -2
  39. data/lib/redis_queued_locks.rb +2 -0
  40. metadata +14 -4
  41. data/lib/redis_queued_locks/watcher.rb +0 -1
data/README.md CHANGED
@@ -35,12 +35,30 @@ Provides flexible invocation flow, parametrized limits (lock request ttl, lock t
35
35
  - [queues_info](#queues_info---get-list-of-queues-with-their-info)
36
36
  - [clear_dead_requests](#clear_dead_requests)
37
37
  - [current_acquirer_id](#current_acquirer_id)
38
+ - [current_host_id](#current_host_id)
39
+ - [possible_host_ids](#possible_host_ids)
40
+ - [Swarm Mode and Zombie Locks](#swarm-mode-and-zombie-locks)
41
+ - [work and usage preview (temporary example-based docs)](#work-and-usage-preview-temporary-example-based-docs)
42
+ - [How to Swarm](#how-to-swarm)
43
+ - [configuration](#)
44
+ - [swarm_status](#swarm_status)
45
+ - [swarm_info](#swarm_info)
46
+ - [swarmize!](#swarmize!)
47
+ - [deswarmize!](#deswarmize!)
48
+ - [probe_hosts](#probe_hosts)
49
+ - [flush_zobmies](#flush_zombies)
50
+ - [zombies_info](#zombies_info)
51
+ - [zombie_locks](#zombie_locks)
52
+ - [zombie_hosts](#zombie_hosts)
53
+ - [zombie_acquiers](#zombie_acquiers)
38
54
  - [Lock Access Strategies](#lock-access-strategies)
39
55
  - [queued](#lock-access-strategies)
40
56
  - [random](#lock-access-strategies)
41
- - [Dead locks and Reentrant locks](#dead-locks-and-reentrant-locks)
57
+ - [Deadlocks and Reentrant locks](#deadlocks-and-reentrant-locks)
42
58
  - [Logging](#logging)
59
+ - [Logging Configuration](#logging-configuration)
43
60
  - [Instrumentation](#instrumentation)
61
+ - [Instrumentation Configuration](#instrumentation-configuration)
44
62
  - [Instrumentation Events](#instrumentation-events)
45
63
  - [Roadmap](#roadmap)
46
64
  - [Contributing](#contributing)
@@ -244,35 +262,35 @@ clinet = RedisQueuedLocks::Client.new(redis_client) do |config|
244
262
  # - should implement `debug(progname = nil, &block)` (minimal requirement) or be an instance of Ruby's `::Logger` class/subclass;
245
263
  # - supports `SemanticLogger::Logger` (see "semantic_logger" gem)
246
264
  # - at this moment the only debug logs are realised in following cases:
247
- # - "[redis_queued_locks.start_lock_obtaining]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
248
- # - "[redis_queued_locks.start_try_to_lock_cycle]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
249
- # - "[redis_queued_locks.dead_score_reached__reset_acquier_position]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
250
- # - "[redis_queued_locks.lock_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "acq_time", "acs_strat");
251
- # - "[redis_queued_locks.extendable_reentrant_lock_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "acq_time", "acs_strat");
252
- # - "[redis_queued_locks.reentrant_lock_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "acq_time", "acs_strat");
253
- # - "[redis_queued_locks.fail_fast_or_limits_reached_or_deadlock__dequeue]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
254
- # - "[redis_queued_locks.expire_lock]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
255
- # - "[redis_queued_locks.decrease_lock]" (logs "lock_key", "decreased_ttl", "queue_ttl", "acq_id", "acs_strat");
265
+ # - "[redis_queued_locks.start_lock_obtaining]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
266
+ # - "[redis_queued_locks.start_try_to_lock_cycle]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
267
+ # - "[redis_queued_locks.dead_score_reached__reset_acquier_position]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
268
+ # - "[redis_queued_locks.lock_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acq_time", "acs_strat");
269
+ # - "[redis_queued_locks.extendable_reentrant_lock_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acq_time", "acs_strat");
270
+ # - "[redis_queued_locks.reentrant_lock_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acq_time", "acs_strat");
271
+ # - "[redis_queued_locks.fail_fast_or_limits_reached_or_deadlock__dequeue]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
272
+ # - "[redis_queued_locks.expire_lock]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
273
+ # - "[redis_queued_locks.decrease_lock]" (logs "lock_key", "decreased_ttl", "queue_ttl", "acq_id", "hst_id", "acs_strat");
256
274
  # - by default uses VoidLogger that does nothing;
257
275
  config.logger = RedisQueuedLocks::Logging::VoidLogger
258
276
 
259
277
  # (default: false)
260
278
  # - adds additional debug logs;
261
279
  # - enables additional logs for each internal try-retry lock acquiring (a lot of logs can be generated depending on your retry configurations);
262
- # - it adds following logs in addition to the existing:
263
- # - "[redis_queued_locks.try_lock.start]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
264
- # - "[redis_queued_locks.try_lock.rconn_fetched]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
265
- # - "[redis_queued_locks.try_lock.same_process_conflict_detected]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
266
- # - "[redis_queued_locks.try_lock.same_process_conflict_analyzed]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "spc_status");
267
- # - "[redis_queued_locks.try_lock.reentrant_lock__extend_and_work_through]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "spc_status", "last_ext_ttl", "last_ext_ts");
268
- # - "[redis_queued_locks.try_lock.reentrant_lock__work_through]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "spc_status", last_spc_ts);
269
- # - "[redis_queued_locks.try_lock.acq_added_to_queue]" (logs "lock_key", "queue_ttl", "acq_id, "acs_strat")";
270
- # - "[redis_queued_locks.try_lock.remove_expired_acqs]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
271
- # - "[redis_queued_locks.try_lock.get_first_from_queue]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "first_acq_id_in_queue");
272
- # - "[redis_queued_locks.try_lock.exit__queue_ttl_reached]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
273
- # - "[redis_queued_locks.try_lock.exit__no_first]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "first_acq_id_in_queue", "<current_lock_data>");
274
- # - "[redis_queued_locks.try_lock.exit__lock_still_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "first_acq_id_in_queue", "locked_by_acq_id", "<current_lock_data>");
275
- # - "[redis_queued_locks.try_lock.obtain__free_to_acquire]" (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
280
+ # - it adds following debug logs in addition to the existing:
281
+ # - "[redis_queued_locks.try_lock.start]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
282
+ # - "[redis_queued_locks.try_lock.rconn_fetched]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
283
+ # - "[redis_queued_locks.try_lock.same_process_conflict_detected]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
284
+ # - "[redis_queued_locks.try_lock.same_process_conflict_analyzed]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status");
285
+ # - "[redis_queued_locks.try_lock.reentrant_lock__extend_and_work_through]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status", "last_ext_ttl", "last_ext_ts");
286
+ # - "[redis_queued_locks.try_lock.reentrant_lock__work_through]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status", last_spc_ts);
287
+ # - "[redis_queued_locks.try_lock.acq_added_to_queue]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat")";
288
+ # - "[redis_queued_locks.try_lock.remove_expired_acqs]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
289
+ # - "[redis_queued_locks.try_lock.get_first_from_queue]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "first_acq_id_in_queue");
290
+ # - "[redis_queued_locks.try_lock.exit__queue_ttl_reached]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
291
+ # - "[redis_queued_locks.try_lock.exit__no_first]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "first_acq_id_in_queue", "<current_lock_data>");
292
+ # - "[redis_queued_locks.try_lock.exit__lock_still_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "first_acq_id_in_queue", "locked_by_acq_id", "<current_lock_data>");
293
+ # - "[redis_queued_locks.try_lock.obtain__free_to_acquire]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
276
294
  config.log_lock_try = false
277
295
 
278
296
  # (default: false)
@@ -341,6 +359,8 @@ end
341
359
  - [queues_info](#queues_info---get-list-of-queues-with-their-info)
342
360
  - [clear_dead_requests](#clear_dead_requests)
343
361
  - [current_acquirer_id](#current_acquirer_id)
362
+ - [current_host_id](#current_host_id)
363
+ - [possible_host_ids](#possible_host_ids)
344
364
 
345
365
  ---
346
366
 
@@ -381,9 +401,11 @@ def lock(
381
401
  log_sampling_enabled: config[:log_sampling_enabled],
382
402
  log_sampling_percent: config[:log_sampling_percent],
383
403
  log_sampler: config[:log_sampler],
404
+ log_sample_this: false,
384
405
  instr_sampling_enabled: config[:instr_sampling_enabled],
385
406
  instr_sampling_percent: config[:instr_sampling_percent],
386
407
  instr_sampler: config[:instr_sampler],
408
+ instr_sample_this: false,
387
409
  &block
388
410
  )
389
411
  ```
@@ -447,7 +469,7 @@ def lock(
447
469
  - `:extendable_work_through` - continue working under the lock **with** lock's TTL extension;
448
470
  - `:wait_for_lock` - (default) - work in classic way (with timeouts, retry delays, retry limits, etc - in classic way :));
449
471
  - `:dead_locking` - fail with deadlock exception;
450
- - See [Dead locks and Reentrant locks](#dead-locks-and-reentrant-locks) documentation section for details;
472
+ - See [Deadlocks and Reentrant locks](#deadlocks-and-reentrant-locks) documentation section for details;
451
473
  - `identity` - (optional) `[String]`
452
474
  - An unique string that is unique per `RedisQueuedLock::Client` instance. Resolves the
453
475
  collisions between the same process_id/thread_id/fiber_id/ractor_id identifiers on different
@@ -503,6 +525,10 @@ def lock(
503
525
  - you can provide your own log sampler with bettter algorithm that should realize
504
526
  `sampling_happened?(percent) => boolean` interface (see `RedisQueuedLocks::Logging::Sampler` for example);
505
527
  - pre-configured in `config[:log_sampler]`;
528
+ - `log_sample_this` - (optional) `[Boolean]`
529
+ - marks the method that everything should be logged despite the enabled log sampling;
530
+ - makes sense when log sampling is enabled;
531
+ - `false` by default;
506
532
  - `instr_sampling_enabled` - (optional) `[Boolean]`
507
533
  - enables **instrumentaion sampling**: only the configured percent of RQL cases will be instrumented;
508
534
  - disabled by default;
@@ -521,6 +547,10 @@ def lock(
521
547
  - you can provide your own log sampler with bettter algorithm that should realize
522
548
  `sampling_happened?(percent) => boolean` interface (see `RedisQueuedLocks::Instrument::Sampler` for example);
523
549
  - pre-configured in `config[:instr_sampler]`;
550
+ - `instr_sample_this` - (optional) `[Boolean]`
551
+ - marks the method that everything should be instrumneted despite the enabled instrumentation sampling;
552
+ - makes sense when instrumentation sampling is enabled;
553
+ - `false` by default;
524
554
  - `block` - (optional) `[Block]`
525
555
  - A block of code that should be executed after the successfully acquired lock.
526
556
  - If block is **passed** the obtained lock will be released after the block execution or it's ttl (what will happen first);
@@ -558,7 +588,7 @@ Return value:
558
588
  - `:extendable_conflict_work_through` - reentrant lock acquiring process with lock's TTL extension. Suitable for `conflict_strategy: :extendable_work_through`;
559
589
  - `:conflict_work_through` - reentrant lock acquiring process without lock's TTL extension. Suitable for `conflict_strategy: :work_through`;
560
590
  - `:dead_locking` - current process tries to acquire a lock that is already acquired by them. Suitalbe for `conflict_startegy: :dead_locking`;
561
- - For more details see [Dead locks and Reentrant locks](#dead-locks-and-reentrant-locks) readme section;
591
+ - For more details see [Deadlocks and Reentrant locks](#deadlocks-and-reentrant-locks) readme section;
562
592
  - For successful lock obtaining:
563
593
  ```ruby
564
594
  {
@@ -566,6 +596,7 @@ Return value:
566
596
  result: {
567
597
  lock_key: String, # acquierd lock key ("rql:lock:your_lock_name")
568
598
  acq_id: String, # acquier identifier ("process_id/thread_id/fiber_id/ractor_id/identity")
599
+ hst_id: String, # host identifier ("process_id/thread_id/ractor_id/identity")
569
600
  ts: Float, # time (epoch) when lock was obtained (float, Time#to_f)
570
601
  ttl: Integer, # lock's time to live in milliseconds (integer)
571
602
  process: Symbol # which logical process has acquired the lock (:lock_obtaining, :extendable_conflict_work_through, :conflict_work_through, :conflict_dead_lock)
@@ -580,6 +611,7 @@ Return value:
580
611
  result: {
581
612
  lock_key: "rql:lock:my_lock",
582
613
  acq_id: "rql:acq:26672/2280/2300/2320/70ea5dbf10ea1056",
614
+ acq_id: "rql:acq:26672/2280/2320/70ea5dbf10ea1056",
583
615
  ts: 1711909612.653696,
584
616
  ttl: 10000,
585
617
  process: :lock_obtaining # for custom conflict strategies may be: :conflict_dead_lock, :conflict_work_through, :extendable_conflict_work_through
@@ -655,6 +687,7 @@ rql.lock_info("my_lock")
655
687
  {
656
688
  "lock_key" => "rql:lock:my_lock",
657
689
  "acq_id" => "rql:acq:123/456/567/678/374dd74324",
690
+ "hst_id" => "rql:acq:123/456/678/374dd74324",
658
691
  "ts" => 123456789,
659
692
  "ini_ttl" => 123456,
660
693
  "rem_ttl" => 123440,
@@ -755,7 +788,7 @@ rql.lock('my_lock', retry_delay: 3000, ttl: 3000, access_strategy: :random)
755
788
  - (`RedisQueuedLocks::LockAlreadyObtainedError`) when `fail_fast` is `true` and lock is already obtained;
756
789
  - (`RedisQueuedLocks::LockAcquiermentTimeoutError`) `timeout` limit reached before lock is obtained;
757
790
  - (`RedisQueuedLocks::LockAcquiermentRetryLimitError`) `retry_count` limit reached before lock is obtained;
758
- - (`RedisQueuedLocks::ConflictLockObtainError`) when `conflict_strategy: :dead_locking` is used and the "same-process-dead-lock" is happened (see [Dead locks and Reentrant locks](#dead-locks-and-reentrant-locks) for details);
791
+ - (`RedisQueuedLocks::ConflictLockObtainError`) when `conflict_strategy: :dead_locking` is used and the "same-process-dead-lock" is happened (see [Deadlocks and Reentrant locks](#deadlocks-and-reentrant-locks) for details);
759
792
 
760
793
  ```ruby
761
794
  def lock!(
@@ -780,9 +813,11 @@ def lock!(
780
813
  log_sampling_enabled: config[:log_sampling_enabled],
781
814
  log_sampling_percent: config[:log_sampling_percent],
782
815
  log_sampler: config[:log_sampler],
816
+ log_sample_this: false,
783
817
  instr_sampling_enabled: config[:instr_sampling_enabled],
784
818
  instr_sampling_percent: config[:instr_sampling_percent],
785
819
  instr_sampler: config[:instr_sampler],
820
+ instr_sample_this: false,
786
821
  &block
787
822
  )
788
823
  ```
@@ -800,6 +835,7 @@ See `#lock` method [documentation](#lock---obtain-a-lock).
800
835
  - lock data (`Hash<String,String|Integer>`):
801
836
  - `"lock_key"` - `string` - lock key in redis;
802
837
  - `"acq_id"` - `string` - acquier identifier (process_id/thread_id/fiber_id/ractor_id/identity);
838
+ - `"hst_id"` - `string` - host identifier (process_id/thread_id/ractor_id/identity);
803
839
  - `"ts"` - `numeric`/`epoch` - the time when lock was obtained;
804
840
  - `"init_ttl"` - `integer` - (milliseconds) initial lock key ttl;
805
841
  - `"rem_ttl"` - `integer` - (milliseconds) remaining lock key ttl;
@@ -822,6 +858,7 @@ rql.lock_info("your_lock_name")
822
858
  {
823
859
  "lock_key" => "rql:lock:your_lock_name",
824
860
  "acq_id" => "rql:acq:123/456/567/678/374dd74324",
861
+ "hst_id" => "rql:acq:123/456/678/374dd74324",
825
862
  "ts" => 123456789.12345,
826
863
  "ini_ttl" => 5_000,
827
864
  "rem_ttl" => 4_999
@@ -837,6 +874,7 @@ rql.lock_info("your_lock_name")
837
874
  {
838
875
  "lock_key" => "rql:lock:your_lock_name",
839
876
  "acq_id" => "rql:acq:123/456/567/678/374dd74324",
877
+ "hst_id" => "rql:acq:123/456/678/374dd74324",
840
878
  "ts" => 123456789.12345,
841
879
  "ini_ttl" => 5_000,
842
880
  "rem_ttl" => 4_999,
@@ -858,6 +896,7 @@ rql.lock_info("your_lock_name")
858
896
  {
859
897
  "lock_key" => "rql:lock:your_lock_name",
860
898
  "acq_id" => "rql:acq:123/456/567/678/374dd74324",
899
+ "hst_id" => "rql:acq:123/456/678/374dd74324",
861
900
  "ts" => 123456789.12345,
862
901
  "ini_ttl" => 5_000,
863
902
  "rem_ttl" => 9_444,
@@ -943,6 +982,7 @@ rql.queued?("your_lock_name") # => true/false
943
982
 
944
983
  - release the concrete lock with lock request queue;
945
984
  - queue will be relased first;
985
+ - has an alias: `#release_lock`;
946
986
  - accepts:
947
987
  - `lock_name` - (required) `[String]` - the lock name that should be released.
948
988
  - `:logger` - (optional) `[::Logger,#debug]`
@@ -963,6 +1003,10 @@ rql.queued?("your_lock_name") # => true/false
963
1003
  - `:log_sampler` - (optional) `[#sampling_happened?,Module<RedisQueuedLocks::Logging::Sampler>]`
964
1004
  - **log sampling**: percent-based log sampler that decides should be RQL case logged or not;
965
1005
  - pre-configured in `config[:log_sampler]`;
1006
+ - `log_sample_this` - (optional) `[Boolean]`
1007
+ - marks the method that everything should be logged despite the enabled log sampling;
1008
+ - makes sense when log sampling is enabled;
1009
+ - `false` by default;
966
1010
  - `:instr_sampling_enabled` - (optional) `[Boolean]`
967
1011
  - enables **instrumentaion sampling**;
968
1012
  - pre-configured in `config[:instr_sampling_enabled]`;
@@ -972,6 +1016,10 @@ rql.queued?("your_lock_name") # => true/false
972
1016
  - `instr_sampler` - (optional) `[#sampling_happened?,Module<RedisQueuedLocks::Instrument::Sampler>]`
973
1017
  - percent-based log sampler that decides should be RQL case instrumented or not;
974
1018
  - pre-configured in `config[:instr_sampler]`;
1019
+ - `instr_sample_this` - (optional) `[Boolean]`
1020
+ - marks the method that everything should be instrumneted despite the enabled instrumentation sampling;
1021
+ - makes sense when instrumentation sampling is enabled;
1022
+ - `false` by default;
975
1023
  - if you try to unlock non-existent lock you will receive `ok: true` result with operation timings
976
1024
  and `:nothing_to_release` result factor inside;
977
1025
 
@@ -1010,6 +1058,7 @@ rql.unlock("your_lock_name")
1010
1058
 
1011
1059
  - release all obtained locks and related lock request queues;
1012
1060
  - queues will be released first;
1061
+ - has an alias: `#release_locks`;
1013
1062
  - accepts:
1014
1063
  - `:batch_size` - (optional) `[Integer]`
1015
1064
  - the size of batch of locks and lock queus that should be cleared under the one pipelined redis command at once;
@@ -1031,6 +1080,10 @@ rql.unlock("your_lock_name")
1031
1080
  - `:log_sampler` - (optional) `[#sampling_happened?,Module<RedisQueuedLocks::Logging::Sampler>]`
1032
1081
  - **log sampling**: percent-based log sampler that decides should be RQL case logged or not;
1033
1082
  - pre-configured in `config[:log_sampler]`;
1083
+ - `log_sample_this` - (optional) `[Boolean]`
1084
+ - marks the method that everything should be logged despite the enabled log sampling;
1085
+ - makes sense when log sampling is enabled;
1086
+ - `false` by default;
1034
1087
  - `:instr_sampling_enabled` - (optional) `[Boolean]`
1035
1088
  - enables **instrumentaion sampling**;
1036
1089
  - pre-configured in `config[:instr_sampling_enabled]`;
@@ -1040,6 +1093,10 @@ rql.unlock("your_lock_name")
1040
1093
  - `instr_sampler` - (optional) `[#sampling_happened?,Module<RedisQueuedLocks::Instrument::Sampler>]`
1041
1094
  - percent-based log sampler that decides should be RQL case instrumented or not;
1042
1095
  - pre-configured in `config[:instr_sampler]`;
1096
+ - `instr_sample_this` - (optional) `[Boolean]`
1097
+ - marks the method that everything should be instrumneted despite the enabled instrumentation sampling;
1098
+ - makes sense when instrumentation sampling is enabled;
1099
+ - `false` by default;
1043
1100
  - returns:
1044
1101
  - `[Hash<Symbol,Numeric>]` - Format: `{ ok: true, result: Hash<Symbol,Numeric> }`;
1045
1102
  - result data:
@@ -1090,6 +1147,10 @@ rql.clear_locks
1090
1147
  - `:log_sampler` - (optional) `[#sampling_happened?,Module<RedisQueuedLocks::Logging::Sampler>]`
1091
1148
  - **log sampling**: percent-based log sampler that decides should be RQL case logged or not;
1092
1149
  - pre-configured in `config[:log_sampler]`;
1150
+ - `log_sample_this` - (optional) `[Boolean]`
1151
+ - marks the method that everything should be logged despite the enabled log sampling;
1152
+ - makes sense when log sampling is enabled;
1153
+ - `false` by default;
1093
1154
  - `:instr_sampling_enabled` - (optional) `[Boolean]`
1094
1155
  - enables **instrumentaion sampling**;
1095
1156
  - pre-configured in `config[:instr_sampling_enabled]`;
@@ -1099,6 +1160,10 @@ rql.clear_locks
1099
1160
  - `instr_sampler` - (optional) `[#sampling_happened?,Module<RedisQueuedLocks::Instrument::Sampler>]`
1100
1161
  - percent-based log sampler that decides should be RQL case instrumented or not;
1101
1162
  - pre-configured in `config[:instr_sampler]`;
1163
+ - `instr_sample_this` - (optional) `[Boolean]`
1164
+ - marks the method that everything should be instrumneted despite the enabled instrumentation sampling;
1165
+ - makes sense when instrumentation sampling is enabled;
1166
+ - `false` by default;
1102
1167
  - returns `{ ok: true, result: :ttl_extended }` when ttl is extended;
1103
1168
  - returns `{ ok: false, result: :async_expire_or_no_lock }` when a lock not found or a lock is already expired during
1104
1169
  some steps of invocation (see **Important** section below);
@@ -1253,6 +1318,7 @@ rql.locks_info # or rql.locks_info(scan_size: 123)
1253
1318
  :status=>:alive,
1254
1319
  :info=>{
1255
1320
  "acq_id"=>"rql:acq:41478/4320/4340/4360/848818f09d8c3420",
1321
+ "hst_id"=>"rql:hst:41478/4320/4360/848818f09d8c3420"
1256
1322
  "ts"=>1711607112.670343,
1257
1323
  "ini_ttl"=>15000,
1258
1324
  "rem_ttl"=>13998}},
@@ -1336,6 +1402,10 @@ Accepts:
1336
1402
  - `:log_sampler` - (optional) `[#sampling_happened?,Module<RedisQueuedLocks::Logging::Sampler>]`
1337
1403
  - **log sampling**: percent-based log sampler that decides should be RQL case logged or not;
1338
1404
  - pre-configured in `config[:log_sampler]`;
1405
+ - `log_sample_this` - (optional) `[Boolean]`
1406
+ - marks the method that everything should be logged despite the enabled log sampling;
1407
+ - makes sense when log sampling is enabled;
1408
+ - `false` by default;
1339
1409
  - `:instr_sampling_enabled` - (optional) `[Boolean]`
1340
1410
  - enables **instrumentaion sampling**;
1341
1411
  - pre-configured in `config[:instr_sampling_enabled]`;
@@ -1345,6 +1415,10 @@ Accepts:
1345
1415
  - `instr_sampler` - (optional) `[#sampling_happened?,Module<RedisQueuedLocks::Instrument::Sampler>]`
1346
1416
  - percent-based log sampler that decides should be RQL case instrumented or not;
1347
1417
  - pre-configured in `config[:instr_sampler]`;
1418
+ - `instr_sample_this` - (optional) `[Boolean]`
1419
+ - marks the method that everything should be instrumneted despite the enabled instrumentation sampling;
1420
+ - makes sense when instrumentation sampling is enabled;
1421
+ - `false` by default;
1348
1422
 
1349
1423
  Returns: `{ ok: true, processed_queues: Set<String> }` returns the list of processed lock queues;
1350
1424
 
@@ -1367,6 +1441,8 @@ rql.clear_dead_requests(dead_ttl: 60 * 60 * 1000) # 1 hour in milliseconds
1367
1441
 
1368
1442
  #### #current_acquirer_id
1369
1443
 
1444
+ <sup>\[[back to top](#usage)\]</sup>
1445
+
1370
1446
  - get the current acquirer identifier in RQL notation that you can use for debugging purposes during the lock analyzation;
1371
1447
  - acquirer identifier format:
1372
1448
  ```ruby
@@ -1400,6 +1476,278 @@ rql.current_acquirer_id
1400
1476
 
1401
1477
  ---
1402
1478
 
1479
+ #### #current_host_id
1480
+
1481
+ <sup>\[[back to top](#usage)\]</sup>
1482
+
1483
+ - get a current host identifier in RQL notation that you can use for debugging purposes during the lock analyzis;
1484
+ - the host is a ruby worker (a combination of process/thread/ractor/identity) that is alive and can obtain locks;
1485
+ - the host is limited to `process`/`thread`/`ractor` (without `fiber`) combination cuz we have no abilities to extract
1486
+ all fiber objects from the current ruby process when at least one ractor object is defined (**ObjectSpace** loses
1487
+ abilities to extract `Fiber` and `Thread` objects after the any ractor is created) (`Thread` objects are analyzed
1488
+ via `Thread.list` API which does not lose their abilites);
1489
+ - host identifier format:
1490
+ ```ruby
1491
+ "rql:hst:#{process_id}/#{thread_id}/#{ractor_id}/#{uniq_identity}"
1492
+ ```
1493
+ - because of the moment that `#lock`/`#lock!` gives you a possibility to customize `process_id`,
1494
+ `fiber_id`, `thread_id`, `ractor_id` and `unique identity` identifiers the `#current_host_id` method provides this possibility too
1495
+ (except the `fiber_id` correspondingly);
1496
+
1497
+ Accepts:
1498
+
1499
+ - `process_id:` - (optional) `[Integer,Any]`
1500
+ - `::Process.pid` by default;
1501
+ - `thread_id:` - (optional) `[Integer,Any]`;
1502
+ - `::Thread.current.object_id` by default;
1503
+ - `ractor_id:` - (optional) `[Integer,Any]`;
1504
+ - `::Ractor.current.object_id` by default;
1505
+ - `identity:` - (optional) `[String]`;
1506
+ - this value is calculated once during `RedisQueuedLock::Client` instantiation and stored in `@uniq_identity`;
1507
+ - this value can be accessed from `RedisQueuedLock::Client#uniq_identity`;
1508
+ - [Configuration](#configuration) documentation: see `config[:uniq_identifier]`;
1509
+ - [#lock](#lock---obtain-a-lock) method documentation: see `uniq_identifier`;
1510
+
1511
+ ```ruby
1512
+ rql.current_host_id
1513
+
1514
+ # =>
1515
+ "rql:acq:38529/4500/4360/66093702f24a3129"
1516
+ ```
1517
+
1518
+ ---
1519
+
1520
+ #### #possible_host_ids
1521
+
1522
+ <sup>\[[back to top](#usage)\]</sup>
1523
+
1524
+ - return the list (`Array<String>`) of possible host identifiers that can be reached from the current ractor;
1525
+ - the host is a ruby worker (a combination of process/thread/ractor/identity) that is alive and can obtain locks;
1526
+ - the host is limited to `process`/`thread`/`ractor` (without `fiber`) combination cuz we have no abilities to extract
1527
+ all fiber objects from the current ruby process when at least one ractor object is defined (**ObjectSpace** loses
1528
+ abilities to extract `Fiber` and `Thread` objects after the any ractor is created) (`Thread` objects are analyzed
1529
+ via `Thread.list` API which does not lose their abilites);
1530
+ - host identifier format:
1531
+ ```ruby
1532
+ "rql:hst:#{process_id}/#{thread_id}/#{ractor_id}/#{uniq_identity}"
1533
+ ```
1534
+
1535
+ Accepts:
1536
+
1537
+ - `identity` - (optional) `[String]`;
1538
+ - this value is calculated once during `RedisQueuedLock::Client` instantiation and stored in `@uniq_identity`;
1539
+ - this value can be accessed from `RedisQueuedLock::Client#uniq_identity`;
1540
+ - [Configuration](#configuration) documentation: see `config[:uniq_identifier]`;
1541
+ - [#lock](#lock---obtain-a-lock) method documentation: see `uniq_identifier`;
1542
+
1543
+ ```ruby
1544
+ rql.possible_host_ids
1545
+
1546
+ # =>
1547
+ [
1548
+ "rql:hst:18814/2300/2280/5ce0c4582fc59c06", # process id / thread id / ractor id / uniq identity
1549
+ "rql:hst:18814/2320/2280/5ce0c4582fc59c06", # ...
1550
+ "rql:hst:18814/2340/2280/5ce0c4582fc59c06", # ...
1551
+ "rql:hst:18814/2360/2280/5ce0c4582fc59c06", # ...
1552
+ "rql:hst:18814/2380/2280/5ce0c4582fc59c06", # ...
1553
+ "rql:hst:18814/2400/2280/5ce0c4582fc59c06"
1554
+ ]
1555
+ ```
1556
+ ---
1557
+
1558
+ ## Swarm Mode and Zombie Locks
1559
+
1560
+ <sup>\[[back to top](#table-of-contents)\]</sup>
1561
+
1562
+ > Eliminate zombie locks with a swarm.
1563
+
1564
+ **This documentation section is in progress!** (see the changelog and the usage preview for details at this moment)
1565
+
1566
+ [(work and usage preview (temporary example-based docs))](#work-and-usage-preview-temporary-example-based-docs)
1567
+
1568
+ - [How to Swarm](#how-to-swarm)
1569
+ - [configuration](#)
1570
+ - [swarm_status](#swarm_status)
1571
+ - [swarm_info](#swarm_info)
1572
+ - [swarmize!](#swarmize!)
1573
+ - [deswarmize!](#deswarmize!)
1574
+ - [probe_hosts](#probe_hosts)
1575
+ - [flush_zobmies](#flush_zombies)
1576
+ - [zombies_info](#zombies_info)
1577
+ - [zombie_locks](#zombie_locks)
1578
+ - [zombie_hosts](#zombie_hosts)
1579
+ - [zombie_acquiers](#zombie_acquiers)
1580
+
1581
+ <hr>
1582
+
1583
+ #### Work and Usage Preview (temporary example-based docs)
1584
+
1585
+ <sup>\[[back to top](#swarm-mode-and-zombie-locks)\]</sup>
1586
+
1587
+ <details>
1588
+ <summary>configuration</summary>
1589
+
1590
+ ```ruby
1591
+ redis_client = RedisClient.config.new_pool # NOTE: provide your own RedisClient instance
1592
+
1593
+ clinet = RedisQueuedLocks::Client.new(redis_client) do |config|
1594
+ # NOTE: auto-swarm your RQL client after initalization (run swarm elements and their supervisor)
1595
+ config.swarm.auto_swarm = false
1596
+
1597
+ # supervisor configs
1598
+ config.swarm.supervisor.liveness_probing_period = 2 # NOTE: in seconds
1599
+
1600
+ # (probe_hosts) host probing configuration
1601
+ config.swarm.probe_hosts.enabled_for_swarm = true # NOTE: run host-probing from or not
1602
+ config.swarm.probe_hosts.probe_period = 2 # NOTE: (in seconds) the period of time when the probing process is triggered
1603
+ # (probe_hosts) individual redis config
1604
+ config.swarm.probe_hosts.redis_config.sentinel = false # NOTE: individual redis config
1605
+ config.swarm.probe_hosts.redis_config.pooled = false # NOTE: individual redis config
1606
+ config.swarm.probe_hosts.redis_config.config = {} # NOTE: individual redis config
1607
+ config.swarm.probe_hosts.redis_config.pool_config = {} # NOTE: individual redis config
1608
+
1609
+ # (flush_zombies) zombie flushing configuration
1610
+ config.swarm.flush_zombies.enabled_for_swarm = true # NOTE: run zombie flushing or not
1611
+ config.swarm.flush_zombies.zombie_flush_period = 10 # NOTE: (in seconds) period of time when the zombie flusher is triggered
1612
+ config.swarm.flush_zombies.zombie_ttl = 15_000 # NOTE: (in milliseconds) when the lock/host/acquier is considered a zombie
1613
+ config.swarm.flush_zombies.zombie_lock_scan_size = 500 # NOTE: scan sizec during zombie flushing
1614
+ config.swarm.flush_zombies.zombie_queue_scan_size = 500 # NOTE: scan sizec during zombie flushing
1615
+ # (flush_zombies) individual redis config
1616
+ config.swarm.flush_zombies.redis_config.sentinel = false # NOTE: individual redis config
1617
+ config.swarm.flush_zombies.redis_config.pooled = false # NOTE: individual redis config
1618
+ config.swarm.flush_zombies.redis_config.config = {} # NOTE: individual redis config
1619
+ config.swarm.flush_zombies.redis_config.pool_config = {} # NOTE: individual redis config
1620
+ end
1621
+ ```
1622
+ </details>
1623
+
1624
+ <details>
1625
+ <summary>seed a zombie</summary>
1626
+
1627
+ - obtain some long living lock and kill the host process which will lead the lock becoming a zombie:
1628
+
1629
+ ```ruby
1630
+ daiver => ~/Projects/redis_queued_locks  master [$]
1631
+ ➜ bin/console
1632
+ [1] pry(main)> rql = RedisQueuedLocks::Client.new(RedisClient.new);
1633
+ [2] pry(main)> rql.swarmize!
1634
+ /Users/daiver/Projects/redis_queued_locks/lib/redis_queued_locks/swarm/flush_zombies.rb:107: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues.
1635
+ => {:ok=>true, :result=>:swarming}
1636
+ [3] pry(main)> rql.lock('kekpek', ttl: 1111111111)
1637
+ => {:ok=>true,
1638
+ :result=>
1639
+ {:lock_key=>"rql:lock:kekpek",
1640
+ :acq_id=>"rql:acq:17580/2260/2380/2280/3f16b93973612580",
1641
+ :hst_id=>"rql:hst:17580/2260/2280/3f16b93973612580",
1642
+ :ts=>1720305351.069259,
1643
+ :ttl=>1111111111,
1644
+ :process=>:lock_obtaining}}
1645
+ [4] pry(main)> exit
1646
+ ```
1647
+ </details>
1648
+
1649
+ <details>
1650
+ <summary>find zombies</summary>
1651
+
1652
+ - start another process, fetch the swarm info, see that our last process is a zombie now and their hosted lock is a zombie too:
1653
+
1654
+ ```ruby
1655
+ daiver => ~/Projects/redis_queued_locks  master [$] took 27.2s
1656
+ ➜ bin/console
1657
+ [1] pry(main)> rql = RedisQueuedLocks::Client.new(RedisClient.new);
1658
+ [2] pry(main)> rql.swarm_info
1659
+ => {"rql:hst:17580/2260/2280/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 12897/262144 +0300, :last_probe_score=>1720305353.0491982},
1660
+ "rql:hst:17580/2300/2280/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 211107/4194304 +0300, :last_probe_score=>1720305353.0503318},
1661
+ "rql:hst:17580/2320/2280/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 106615/2097152 +0300, :last_probe_score=>1720305353.050838},
1662
+ "rql:hst:17580/2260/2340/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 26239/524288 +0300, :last_probe_score=>1720305353.050047},
1663
+ "rql:hst:17580/2300/2340/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 106359/2097152 +0300, :last_probe_score=>1720305353.050716},
1664
+ "rql:hst:17580/2320/2340/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 213633/4194304 +0300, :last_probe_score=>1720305353.050934},
1665
+ "rql:hst:17580/2360/2280/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 214077/4194304 +0300, :last_probe_score=>1720305353.05104},
1666
+ "rql:hst:17580/2360/2340/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 214505/4194304 +0300, :last_probe_score=>1720305353.051142},
1667
+ "rql:hst:17580/2400/2280/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 53729/1048576 +0300, :last_probe_score=>1720305353.05124},
1668
+ "rql:hst:17580/2400/2340/3f16b93973612580"=>{:zombie=>true, :last_probe_time=>2024-07-07 01:35:53 3365/65536 +0300, :last_probe_score=>1720305353.0513458}}
1669
+ [3] pry(main)> rql.swarm_status
1670
+ => {:auto_swarm=>false,
1671
+ :supervisor=>{:running=>false, :state=>"non_initialized", :observable=>"non_initialized"},
1672
+ :probe_hosts=>{:enabled=>true, :thread=>{:running=>false, :state=>"non_initialized"}, :main_loop=>{:running=>false, :state=>"non_initialized"}},
1673
+ :flush_zombies=>{:enabled=>true, :ractor=>{:running=>false, :state=>"non_initialized"}, :main_loop=>{:running=>false, :state=>"non_initialized"}}}
1674
+ [4] pry(main)> rql.zombies_info
1675
+ => {:zombie_hosts=>
1676
+ #<Set:
1677
+ {"rql:hst:17580/2260/2280/3f16b93973612580",
1678
+ "rql:hst:17580/2300/2280/3f16b93973612580",
1679
+ "rql:hst:17580/2320/2280/3f16b93973612580",
1680
+ "rql:hst:17580/2260/2340/3f16b93973612580",
1681
+ "rql:hst:17580/2300/2340/3f16b93973612580",
1682
+ "rql:hst:17580/2320/2340/3f16b93973612580",
1683
+ "rql:hst:17580/2360/2280/3f16b93973612580",
1684
+ "rql:hst:17580/2360/2340/3f16b93973612580",
1685
+ "rql:hst:17580/2400/2280/3f16b93973612580",
1686
+ "rql:hst:17580/2400/2340/3f16b93973612580"}>,
1687
+ :zombie_acquirers=>#<Set: {"rql:acq:17580/2260/2380/2280/3f16b93973612580"}>,
1688
+ :zombie_locks=>#<Set: {"rql:lock:kekpek"}>}
1689
+ [5] pry(main)> rql.zombie_locks
1690
+ => #<Set: {"rql:lock:kekpek"}>
1691
+ [6] pry(main)> rql.zombie_acquiers
1692
+ => #<Set: {"rql:acq:17580/2260/2380/2280/3f16b93973612580"}>
1693
+ [7] pry(main)> rql.zombie_hosts
1694
+ => #<Set:
1695
+ {"rql:hst:17580/2260/2280/3f16b93973612580",
1696
+ "rql:hst:17580/2300/2280/3f16b93973612580",
1697
+ "rql:hst:17580/2320/2280/3f16b93973612580",
1698
+ "rql:hst:17580/2260/2340/3f16b93973612580",
1699
+ "rql:hst:17580/2300/2340/3f16b93973612580",
1700
+ "rql:hst:17580/2320/2340/3f16b93973612580",
1701
+ "rql:hst:17580/2360/2280/3f16b93973612580",
1702
+ "rql:hst:17580/2360/2340/3f16b93973612580",
1703
+ "rql:hst:17580/2400/2280/3f16b93973612580",
1704
+ "rql:hst:17580/2400/2340/3f16b93973612580"}>
1705
+ ```
1706
+ </details>
1707
+
1708
+ <details>
1709
+ <summary>kill zombies in a background</summary>
1710
+
1711
+ - swarmize the new current ruby process that should run the flush zombies element that will drop zombie locks, zombie hosts and their lock requests in a background:
1712
+
1713
+ ```ruby
1714
+ [8] pry(main)> rql.swarmize!
1715
+ /Users/daiver/Projects/redis_queued_locks/lib/redis_queued_locks/swarm/flush_zombies.rb:107: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues.
1716
+ => {:ok=>true, :result=>:swarming}
1717
+ [9] pry(main)> rql.swarm_info
1718
+ => {"rql:hst:17752/2260/2280/89beef198021f16d"=>{:zombie=>false, :last_probe_time=>2024-07-07 01:36:39 4012577/4194304 +0300, :last_probe_score=>1720305399.956673},
1719
+ "rql:hst:17752/2300/2280/89beef198021f16d"=>{:zombie=>false, :last_probe_time=>2024-07-07 01:36:39 4015233/4194304 +0300, :last_probe_score=>1720305399.9573061},
1720
+ "rql:hst:17752/2320/2280/89beef198021f16d"=>{:zombie=>false, :last_probe_time=>2024-07-07 01:36:39 4016755/4194304 +0300, :last_probe_score=>1720305399.957669},
1721
+ "rql:hst:17752/2260/2340/89beef198021f16d"=>{:zombie=>false, :last_probe_time=>2024-07-07 01:36:39 1003611/1048576 +0300, :last_probe_score=>1720305399.957118},
1722
+ "rql:hst:17752/2300/2340/89beef198021f16d"=>{:zombie=>false, :last_probe_time=>2024-07-07 01:36:39 2008027/2097152 +0300, :last_probe_score=>1720305399.957502},
1723
+ "rql:hst:17752/2320/2340/89beef198021f16d"=>{:zombie=>false, :last_probe_time=>2024-07-07 01:36:39 2008715/2097152 +0300, :last_probe_score=>1720305399.95783},
1724
+ "rql:hst:17752/2360/2280/89beef198021f16d"=>{:zombie=>false, :last_probe_time=>2024-07-07 01:36:39 4018063/4194304 +0300, :last_probe_score=>1720305399.9579809},
1725
+ "rql:hst:17752/2360/2340/89beef198021f16d"=>{:zombie=>false, :last_probe_time=>2024-07-07 01:36:39 1004673/1048576 +0300, :last_probe_score=>1720305399.9581308}}
1726
+ [10] pry(main)> rql.swarm_status
1727
+ => {:auto_swarm=>false,
1728
+ :supervisor=>{:running=>true, :state=>"sleep", :observable=>"initialized"},
1729
+ :probe_hosts=>{:enabled=>true, :thread=>{:running=>true, :state=>"sleep"}, :main_loop=>{:running=>true, :state=>"sleep"}},
1730
+ :flush_zombies=>{:enabled=>true, :ractor=>{:running=>true, :state=>"running"}, :main_loop=>{:running=>true, :state=>"sleep"}}}
1731
+ [11] pry(main)> rql.zombies_info
1732
+ => {:zombie_hosts=>#<Set: {}>, :zombie_acquirers=>#<Set: {}>, :zombie_locks=>#<Set: {}>}
1733
+ [12] pry(main)> rql.zombie_acquiers
1734
+ => #<Set: {}>
1735
+ [13] pry(main)> rql.zombie_hosts
1736
+ => #<Set: {}>
1737
+ [14] pry(main)>
1738
+ ```
1739
+ </details>
1740
+
1741
+ <details>
1742
+ <summary>swarm hosts key in Redis</summary>
1743
+
1744
+ ```ruby
1745
+ "rql:swarm:hsts"
1746
+ ```
1747
+ </details>
1748
+
1749
+ ---
1750
+
1403
1751
  ## Lock Access Strategies
1404
1752
 
1405
1753
  <sup>\[[back to top](#table-of-contents)\]</sup>
@@ -1418,7 +1766,7 @@ rql.current_acquirer_id
1418
1766
 
1419
1767
  ---
1420
1768
 
1421
- ## Dead locks and Reentrant locks
1769
+ ## Deadlocks and Reentrant locks
1422
1770
 
1423
1771
  <sup>\[[back to top](#table-of-contents)\]</sup>
1424
1772
 
@@ -1440,37 +1788,110 @@ rql.current_acquirer_id
1440
1788
 
1441
1789
  <sup>\[[back to top](#table-of-contents)\]</sup>
1442
1790
 
1791
+ - [Logging Configuration](#logging-configuration)
1792
+
1793
+
1443
1794
  - default logs (raised from `#lock`/`#lock!`):
1444
1795
 
1445
1796
  ```ruby
1446
- "[redis_queued_locks.start_lock_obtaining]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1447
- "[redis_queued_locks.start_try_to_lock_cycle]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1448
- "[redis_queued_locks.dead_score_reached__reset_acquier_position]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1449
- "[redis_queued_locks.lock_obtained]" # (logs "lock_key", "queue_ttl", "acq_id", "acq_time");
1450
- "[redis_queued_locks.extendable_reentrant_lock_obtained]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "acq_time");
1451
- "[redis_queued_locks.reentrant_lock_obtained]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "acq_time");
1452
- "[redis_queued_locks.fail_fast_or_limits_reached_or_deadlock__dequeue]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1453
- "[redis_queued_locks.expire_lock]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1454
- "[redis_queued_locks.decrease_lock]" # (logs "lock_key", "decreased_ttl", "queue_ttl", "acq_id", "acs_strat");
1797
+ "[redis_queued_locks.start_lock_obtaining]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1798
+ "[redis_queued_locks.start_try_to_lock_cycle]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1799
+ "[redis_queued_locks.dead_score_reached__reset_acquier_position]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1800
+ "[redis_queued_locks.lock_obtained]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acq_time");
1801
+ "[redis_queued_locks.extendable_reentrant_lock_obtained]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "acq_time");
1802
+ "[redis_queued_locks.reentrant_lock_obtained]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "acq_time");
1803
+ "[redis_queued_locks.fail_fast_or_limits_reached_or_deadlock__dequeue]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1804
+ "[redis_queued_locks.expire_lock]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1805
+ "[redis_queued_locks.decrease_lock]" # (logs "lock_key", "decreased_ttl", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1455
1806
  ```
1456
1807
 
1457
1808
  - additional logs (raised from `#lock`/`#lock!` with `confg[:log_lock_try] == true`):
1458
1809
 
1459
1810
  ```ruby
1460
- "[redis_queued_locks.try_lock.start]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1461
- "[redis_queued_locks.try_lock.rconn_fetched]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1462
- "[redis_queued_locks.try_lock.same_process_conflict_detected]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1463
- "[redis_queued_locks.try_lock.same_process_conflict_analyzed]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "spc_status");
1464
- "[redis_queued_locks.try_lock.reentrant_lock__extend_and_work_through]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "spc_status", "last_ext_ttl", "last_ext_ts");
1465
- "[redis_queued_locks.try_lock.reentrant_lock__work_through]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "spc_status", last_spc_ts);
1466
- "[redis_queued_locks.try_lock.single_process_lock_conflict__dead_lock]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "spc_status", "last_spc_ts");
1467
- "[redis_queued_locks.try_lock.acq_added_to_queue]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1468
- "[redis_queued_locks.try_lock.remove_expired_acqs]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1469
- "[redis_queued_locks.try_lock.get_first_from_queue]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "first_acq_id_in_queue");
1470
- "[redis_queued_locks.try_lock.exit__queue_ttl_reached]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1471
- "[redis_queued_locks.try_lock.exit__no_first]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "first_acq_id_in_queue", "<current_lock_data>");
1472
- "[redis_queued_locks.try_lock.exit__lock_still_obtained]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat", "first_acq_id_in_queue", "locked_by_acq_id", "<current_lock_data>");
1473
- "[redis_queued_locks.try_lock.obtain__free_to_acquire]" # (logs "lock_key", "queue_ttl", "acq_id", "acs_strat");
1811
+ "[redis_queued_locks.try_lock.start]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1812
+ "[redis_queued_locks.try_lock.rconn_fetched]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1813
+ "[redis_queued_locks.try_lock.same_process_conflict_detected]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1814
+ "[redis_queued_locks.try_lock.same_process_conflict_analyzed]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status");
1815
+ "[redis_queued_locks.try_lock.reentrant_lock__extend_and_work_through]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status", "last_ext_ttl", "last_ext_ts");
1816
+ "[redis_queued_locks.try_lock.reentrant_lock__work_through]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status", last_spc_ts);
1817
+ "[redis_queued_locks.try_lock.single_process_lock_conflict__dead_lock]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status", "last_spc_ts");
1818
+ "[redis_queued_locks.try_lock.acq_added_to_queue]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1819
+ "[redis_queued_locks.try_lock.remove_expired_acqs]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1820
+ "[redis_queued_locks.try_lock.get_first_from_queue]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "first_acq_id_in_queue");
1821
+ "[redis_queued_locks.try_lock.exit__queue_ttl_reached]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1822
+ "[redis_queued_locks.try_lock.exit__no_first]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "first_acq_id_in_queue", "<current_lock_data>");
1823
+ "[redis_queued_locks.try_lock.exit__lock_still_obtained]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "first_acq_id_in_queue", "locked_by_acq_id", "<current_lock_data>");
1824
+ "[redis_queued_locks.try_lock.obtain__free_to_acquire]" # (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1825
+ ```
1826
+
1827
+ ---
1828
+
1829
+ ### Logging Configuration
1830
+
1831
+ <sup>\[[back to top](#table-of-contents)\]</sup>
1832
+
1833
+ **NOTICE**: logging can be sampled via:
1834
+ - `config.log_samplign_enabled = true` (**false** by default);
1835
+ - `config.log_sampler = RedisQueuedLocks::Logging::Sampler` (used by default);
1836
+ - see **RedisQueuedLocks::Logging::Sampler** implementation in source code for customization details;
1837
+
1838
+ ```ruby
1839
+ # (default: RedisQueuedLocks::Logging::VoidLogger)
1840
+ # - the logger object;
1841
+ # - should implement `debug(progname = nil, &block)` (minimal requirement) or be an instance of Ruby's `::Logger` class/subclass;
1842
+ # - supports `SemanticLogger::Logger` (see "semantic_logger" gem)
1843
+ # - at this moment the only debug logs are realised in following cases:
1844
+ # - "[redis_queued_locks.start_lock_obtaining]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1845
+ # - "[redis_queued_locks.start_try_to_lock_cycle]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1846
+ # - "[redis_queued_locks.dead_score_reached__reset_acquier_position]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1847
+ # - "[redis_queued_locks.lock_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acq_time", "acs_strat");
1848
+ # - "[redis_queued_locks.extendable_reentrant_lock_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acq_time", "acs_strat");
1849
+ # - "[redis_queued_locks.reentrant_lock_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acq_time", "acs_strat");
1850
+ # - "[redis_queued_locks.fail_fast_or_limits_reached_or_deadlock__dequeue]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1851
+ # - "[redis_queued_locks.expire_lock]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1852
+ # - "[redis_queued_locks.decrease_lock]" (logs "lock_key", "decreased_ttl", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1853
+ # - by default uses VoidLogger that does nothing;
1854
+ config.logger = RedisQueuedLocks::Logging::VoidLogger
1855
+
1856
+ # (default: false)
1857
+ # - adds additional debug logs;
1858
+ # - enables additional logs for each internal try-retry lock acquiring (a lot of logs can be generated depending on your retry configurations);
1859
+ # - it adds following debug logs in addition to the existing:
1860
+ # - "[redis_queued_locks.try_lock.start]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1861
+ # - "[redis_queued_locks.try_lock.rconn_fetched]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1862
+ # - "[redis_queued_locks.try_lock.same_process_conflict_detected]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1863
+ # - "[redis_queued_locks.try_lock.same_process_conflict_analyzed]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status");
1864
+ # - "[redis_queued_locks.try_lock.reentrant_lock__extend_and_work_through]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status", "last_ext_ttl", "last_ext_ts");
1865
+ # - "[redis_queued_locks.try_lock.reentrant_lock__work_through]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "spc_status", last_spc_ts);
1866
+ # - "[redis_queued_locks.try_lock.acq_added_to_queue]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat")";
1867
+ # - "[redis_queued_locks.try_lock.remove_expired_acqs]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1868
+ # - "[redis_queued_locks.try_lock.get_first_from_queue]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "first_acq_id_in_queue");
1869
+ # - "[redis_queued_locks.try_lock.exit__queue_ttl_reached]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1870
+ # - "[redis_queued_locks.try_lock.exit__no_first]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "first_acq_id_in_queue", "<current_lock_data>");
1871
+ # - "[redis_queued_locks.try_lock.exit__lock_still_obtained]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat", "first_acq_id_in_queue", "locked_by_acq_id", "<current_lock_data>");
1872
+ # - "[redis_queued_locks.try_lock.obtain__free_to_acquire]" (logs "lock_key", "queue_ttl", "acq_id", "hst_id", "acs_strat");
1873
+ config.log_lock_try = false
1874
+
1875
+ # (default: false)
1876
+ # - enables <log sampling>: only the configured percent of RQL cases will be logged;
1877
+ # - disabled by default;
1878
+ # - works in tandem with <config.log_sampling_percent> and <log.sampler> configs;
1879
+ config.log_sampling_enabled = false
1880
+
1881
+ # (default: 15)
1882
+ # - the percent of cases that should be logged;
1883
+ # - take an effect when <config.log_sampling_enalbed> is true;
1884
+ # - works in tandem with <config.log_sampling_enabled> and <config.log_sampler> configs;
1885
+ config.log_sampling_percent = 15
1886
+
1887
+ # (default: RedisQueuedLocks::Logging::Sampler)
1888
+ # - percent-based log sampler that decides should be RQL case logged or not;
1889
+ # - works in tandem with <config.log_sampling_enabled> and <config.log_sampling_percent> configs;
1890
+ # - based on the ultra simple percent-based (weight-based) algorithm that uses SecureRandom.rand
1891
+ # method so the algorithm error is ~(0%..13%);
1892
+ # - you can provide your own log sampler with bettter algorithm that should realize
1893
+ # `sampling_happened?(percent) => boolean` interface (see `RedisQueuedLocks::Logging::Sampler` for example);
1894
+ config.log_sampler = RedisQueuedLocks::Logging::Sampler
1474
1895
  ```
1475
1896
 
1476
1897
  ---
@@ -1480,9 +1901,12 @@ rql.current_acquirer_id
1480
1901
  <sup>\[[back to top](#table-of-contents)\]</sup>
1481
1902
 
1482
1903
  - [Instrumentation Events](#instrumentation-events)
1904
+ - [Instrumentation Configuration](#instrumentation-configuration)
1483
1905
 
1484
1906
  An instrumentation layer is incapsulated in `instrumenter` object stored in [config](#configuration) (`RedisQueuedLocks::Client#config[:instrumenter]`).
1485
1907
 
1908
+ Instrumentation can be sampled. See [Instrumentation Configuration](#instrumentation-configuration) section for details.
1909
+
1486
1910
  Instrumenter object should provide `notify(event, payload)` method with the following signarue:
1487
1911
 
1488
1912
  - `event` - `string`;
@@ -1498,6 +1922,48 @@ By default `RedisQueuedLocks::Client` is configured with the void notifier (whic
1498
1922
 
1499
1923
  ---
1500
1924
 
1925
+ ### Instrumentation Configuration
1926
+
1927
+ <sup>\[[back to top](#table-of-contents)\]</sup>
1928
+
1929
+ **NOTICE**: instrumentation can be sampled via:
1930
+ - `config.instr_sampling_enabled = true` (**false** by default);
1931
+ - `config.instr_sampler = RedisQueuedLocks::Instrument::Sampler` (used by default);
1932
+ - see **RedisQueuedLocks::Instrument::Sampler** implementation in source code for customization details;
1933
+
1934
+ ```ruby
1935
+ # (default: RedisQueuedLocks::Instrument::VoidNotifier)
1936
+ # - instrumentation layer;
1937
+ # - you can provide your own instrumenter that should realize `#notify(event, payload = {})` interface:
1938
+ # - event: <string> requried;
1939
+ # - payload: <hash> requried;
1940
+ # - disabled by default via `VoidNotifier`;
1941
+ config.instrumenter = RedisQueuedLocks::Instrument::ActiveSupport
1942
+
1943
+ # (default: false)
1944
+ # - enables <instrumentaion sampling>: only the configured percent of RQL cases will be instrumented;
1945
+ # - disabled by default;
1946
+ # - works in tandem with <config.instr_sampling_percent and <log.instr_sampler>;
1947
+ config.instr_sampling_enabled = false
1948
+
1949
+ # (default: 15)
1950
+ # - the percent of cases that should be instrumented;
1951
+ # - take an effect when <config.instr_sampling_enalbed> is true;
1952
+ # - works in tandem with <config.instr_sampling_enabled> and <config.instr_sampler> configs;
1953
+ config.instr_sampling_percent = 15
1954
+
1955
+ # (default: RedisQueuedLocks::Instrument::Sampler)
1956
+ # - percent-based log sampler that decides should be RQL case instrumented or not;
1957
+ # - works in tandem with <config.instr_sampling_enabled> and <config.instr_sampling_percent> configs;
1958
+ # - based on the ultra simple percent-based (weight-based) algorithm that uses SecureRandom.rand
1959
+ # method so the algorithm error is ~(0%..13%);
1960
+ # - you can provide your own log sampler with bettter algorithm that should realize
1961
+ # `sampling_happened?(percent) => boolean` interface (see `RedisQueuedLocks::Instrument::Sampler` for example);
1962
+ config.instr_sampler = RedisQueuedLocks::Instrument::Sampler
1963
+ ```
1964
+
1965
+ ---
1966
+
1501
1967
  ### Instrumentation Events
1502
1968
 
1503
1969
  <sup>\[[back to top](#instrumentation)\]</sup>
@@ -1520,6 +1986,7 @@ Detalized event semantics and payload structure:
1520
1986
  - payload:
1521
1987
  - `:ttl` - `integer`/`milliseconds` - lock ttl;
1522
1988
  - `:acq_id` - `string` - lock acquier identifier;
1989
+ - `:hst_id` - `string` - lock's host identifier;
1523
1990
  - `:lock_key` - `string` - lock name;
1524
1991
  - `:ts` - `numeric`/`epoch` - the time when the lock was obtaiend;
1525
1992
  - `:acq_time` - `float`/`milliseconds` - time spent on lock acquiring;
@@ -1532,6 +1999,7 @@ Detalized event semantics and payload structure:
1532
1999
  - `:lock_key` - `string` - lock name;
1533
2000
  - `:ttl` - `integer`/`milliseconds` - last lock ttl by reentrant locking;
1534
2001
  - `:acq_id` - `string` - lock acquier identifier;
2002
+ - `:hst_id` - `string` - lock's host identifier;
1535
2003
  - `:ts` - `numeric`/`epoch` - the time when the lock was obtaiend as extendable reentrant lock;
1536
2004
  - `:acq_time` - `float`/`milliseconds` - time spent on lock acquiring;
1537
2005
  - `:instrument` - `nil`/`Any` - custom data passed to the `#lock`/`#lock!` method as `:instrument` attribute;
@@ -1543,6 +2011,7 @@ Detalized event semantics and payload structure:
1543
2011
  - `:lock_key` - `string` - lock name;
1544
2012
  - `:ttl` - `integer`/`milliseconds` - last lock ttl by reentrant locking;
1545
2013
  - `:acq_id` - `string` - lock acquier identifier;
2014
+ - `:hst_id` - `string` - lock's host identifier;
1546
2015
  - `:ts` - `numeric`/`epoch` - the time when the lock was obtaiend as reentrant lock;
1547
2016
  - `:acq_time` - `float`/`milliseconds` - time spent on lock acquiring;
1548
2017
  - `:instrument` - `nil`/`Any` - custom data passed to the `#lock`/`#lock!` method as `:instrument` attribute;
@@ -1554,6 +2023,7 @@ Detalized event semantics and payload structure:
1554
2023
  - `:hold_time` - `float`/`milliseconds` - lock hold time;
1555
2024
  - `:ttl` - `integer`/`milliseconds` - lock ttl;
1556
2025
  - `:acq_id` - `string` - lock acquier identifier;
2026
+ - `:hst_id` - `string` - lock's host identifier;
1557
2027
  - `:lock_key` - `string` - lock name;
1558
2028
  - `:ts` - `numeric`/`epoch` - the time when lock was obtained;
1559
2029
  - `:acq_time` - `float`/`milliseconds` - time spent on lock acquiring;
@@ -1567,6 +2037,7 @@ Detalized event semantics and payload structure:
1567
2037
  - `:hold_time` - `float`/`milliseconds` - lock hold time;
1568
2038
  - `:ttl` - `integer`/`milliseconds` - last lock ttl by reentrant locking;
1569
2039
  - `:acq_id` - `string` - lock acquier identifier;
2040
+ - `:hst_id` - `string` - lock's host identifier;
1570
2041
  - `:ts` - `numeric`/`epoch` - the time when the lock was obtaiend as reentrant lock;
1571
2042
  - `:lock_key` - `string` - lock name;
1572
2043
  - `:acq_time` - `float`/`milliseconds` - time spent on lock acquiring;
@@ -1596,6 +2067,12 @@ Detalized event semantics and payload structure:
1596
2067
  <sup>\[[back to top](#table-of-contents)\]</sup>
1597
2068
 
1598
2069
  - **Major**:
2070
+ - Swarm:
2071
+ - circuit-breaker for long-living failures of your infrastructure inside the swarm elements and supervisor:
2072
+ the supervisor will stop (for some period of time or while the some factor will return `true`)
2073
+ trying to ressurect unexpectedly terminated swarm elements, and will notify about this;
2074
+ - swarm logs (thread/ractor has some limitations so the initial implementation does not include swarm logging);
2075
+ - swarm instrumentation (thread/ractor has some limitations so the initial implementation does not include swarm instrumentation);
1599
2076
  - lock request prioritization;
1600
2077
  - **strict redlock algorithm support** (support for many `RedisClient` instances);
1601
2078
  - `#lock_series` - acquire a series of locks: