redis_queued_locks 1.19.0 → 1.21.0

data/CHANGELOG.md

@@ -1,5 +1,199 @@
 ## [Unreleased]
 
+## [1.21.0] - 2026-10-13
+### Added
+- Logging and instrumentation of all operations that change Redis keys (the swarm is not covered yet):
+  - the lock acquirement: the new `redis_queued_locks.lock_obtaining_failed` instrumentation event and log
+    (`#lock`/`#lock!` and each lock of `#lock_series`/`#lock_series!`: the lock is not obtained, the failure `result`
+    (`:timeout_reached`, `:retry_limit_reached`, `:fail_fast_no_try`, `:fail_fast_after_try`, `:conflict_dead_lock`,
+    `:conflict_lock_upgrade`) is in the payload; with `raise_errors: true` it is reported before the error is raised);
+  - each lock mutation operation is reported with one log line and one instrumentation event with the time of the whole
+    operation (the same name and data; the payload ends with `instrument`): the new `redis_queued_locks.extend_lock_ttl`
+    (`#extend_lock_ttl`), `redis_queued_locks.clear_dead_requests` (`#clear_dead_requests`), `redis_queued_locks.reindex_locks`
+    (`#reindex_locks`) events, and the new log lines of the existing `explicit_lock_release` (`#unlock`),
+    `explicit_read_lock_release` (`#unlock_read`), `explicit_all_locks_release` (`#clear_locks`), `release_locks_of`
+    (`#clear_locks_of`/`#clear_current_locks`) events (the `logger:` option of these methods was not used);
+  - `#reindex_locks`: `logger:`, `instrumenter:`, `instrument:` and the log/instrumentation sampling options;
+### Changed
+- stricter validation (the details are in "Fixed"): invalid values raise `RedisQueuedLocks::ArgumentError`
+  (`RedisQueuedLocks::SwarmArgumentError` for the swarm methods, `RedisQueuedLocks::ConfigValidationError` for the config)
+  instead of the previous silent misbehavior:
+  - `ttl` of `#lock`/`#lock!`/`#lock_series`/`#lock_series!` should be a positive `Integer` in both modes (`ttl: 0` was
+    accepted in the write mode);
+  - `#extend_lock_ttl`: `milliseconds` should be a positive `Integer` (`extend_lock_ttl(name, 0)` was a successful no-op);
+  - the per-call numeric options should be `Integer`s in the ranges of their config keys: `Float` values (e.g.
+    `timeout: 0.5`, `retry_delay: 10.0`) are rejected now;
+  - the config: positive TTLs, periods and batch/scan sizes;
+  - `:meta`: reserved lock data keys are rejected as Symbol keys too;
+- the internal timeout errors of `timed: true` and `timeout` inherit from `RedisQueuedLocks::Exception < ::Exception`
+  (were `Timeout::Error`s): a `rescue Timeout::Error` (or `rescue => e`) inside the block does not catch the lock timeout
+  anymore; the public errors/results are the same;
+- `#locks_info`: the info of write locks has the `"lock_key"` field (like `#lock_info`);
+- instrumentation payloads: `explicit_lock_release`, `explicit_all_locks_release` and `release_locks_of` have `instrument`;
+### Fixed
+- the `instrument:` option of `#unlock`, `#clear_locks`, `#clear_locks_of`/`#clear_current_locks` (README: "passed to
+  the instrumenter's payload") was not passed to the payload (`#unlock` did not even forward it); the payloads of
+  `explicit_lock_release`, `explicit_all_locks_release` and `release_locks_of` have `instrument` now; `#extend_lock_ttl` and
+  `#clear_dead_requests` had no instrumentation events at all (see "Added");
+- `#lock`/`#lock!` with a non-positive `ttl` in the write mode: the lock was reported as obtained, but `PEXPIRE` with
+  a non-positive ttl deleted the lock key in the obtaining transaction, so other acquirers could obtain the same lock
+  at the same time (no mutual exclusion); `ttl` should be a positive `Integer` in both modes now
+  (`RedisQueuedLocks::ArgumentError`, previously validated for read locks only); `ttl: nil` was never supported
+  (the YARD docs described it as "without timeout");
+- `#lock_series`/`#lock_series!` (PoC) with a non-positive `ttl` (both modes): the locks of the series were obtained with
+  the ttl made of the retry delay and jitter only; the series `ttl` should be a positive `Integer` now
+  (`RedisQueuedLocks::ArgumentError`);
+- `config['default_lock_ttl']` should be a positive `Integer` (`RedisQueuedLocks::ConfigValidationError`);
+- the config validation accepted zero and negative TTLs, periods and batch/scan sizes: `config['default_queue_ttl'] = 0`
+  dropped the own lock request in the queueing pipeline (endless `:dead_score_reached`), zero batch/scan sizes failed
+  `SCAN`/`ZSCAN` (`COUNT 0`: `ERR syntax error`) and `each_slice(0)`, zero/negative swarm periods killed the swarm main loops
+  and the supervisor (`sleep` with a negative value) or made them busy loops, a non-positive `zombie_ttl` made all hosts
+  zombies; these keys should be positive `Integer`s now (`RedisQueuedLocks::ConfigValidationError`): `default_queue_ttl`,
+  `lock_release_batch_size`, `clear_locks_of__lock_scan_size`, `clear_locks_of__queue_scan_size`, `key_extraction_batch_size`,
+  `swarm.supervisor.liveness_probing_period`, `swarm.probe_hosts.probe_period`, `swarm.flush_zombies.zombie_ttl`,
+  `swarm.flush_zombies.zombie_flush_period`, `swarm.flush_zombies.zombie_lock_scan_size`,
+  `swarm.flush_zombies.zombie_queue_scan_size`;
+- `#extend_lock_ttl` with a non-positive `milliseconds` decreased the lock TTL (or deleted the lock) and reported a successful
+  extension: `milliseconds` should be a positive `Integer` now (`RedisQueuedLocks::ArgumentError`);
+- `#lock`/`#lock!`/`#lock_series`/`#lock_series!` (both modes): reserved lock data keys were rejected in `:meta` only as
+  String keys, while Symbol keys (`meta: { acq_id: '...' }`) passed the validation and overwrote the lock data fields
+  (`acq_id`, `hst_id`, `ts`, `ini_ttl`, `spc_*`, ...: reentrant locks, owner checks of lock releases and lock info were
+  broken while the lock lived); Symbol keys are validated too now (`RedisQueuedLocks::ArgumentError`);
+- `timed: true` and `timeout`: the internal timeout errors inherited from `Timeout::Error` (a `StandardError`), so `rescue => e`,
+  `rescue StandardError`, `rescue Timeout::Error` and `rescue nil` of the interrupted code swallowed them: a `timed` block with
+  such a `rescue` continued after the lock TTL (another acquirer could obtain the expired lock, and the write lock release
+  after the block deleted the lock of that acquirer), and the acquirement timeout that interrupted a logger/instrumenter
+  call (`rescue nil` of the log/instrumentation events) was lost (with `retry_count: nil` the acquirement waited without
+  the timeout); the internal timeouts inherit from the new private base `RedisQueuedLocks::Exception < ::Exception` now
+  (like `Timeout::ExitException`): only `rescue Exception` catches them, `ensure` still runs, the public errors/results
+  are the same (`RedisQueuedLocks::TimedLockTimeoutError`, `RedisQueuedLocks::LockAcquirementTimeoutError`,
+  `:timeout_reached`); a connection interrupted inside a `WATCH` block is closed (redis-client sends `UNWATCH` only on
+  `StandardError`s: the next transaction on the connection would be aborted);
+- `#lock_series`/`#lock_series!` (PoC): the log events and the `redis_queued_locks.lock_series_hold_and_release`
+  instrumentation event raised the errors of the logger/instrumenter: a failing logger broke the series call (after
+  obtaining all the locks of the series the locks were not released till their ttl), a failing instrumenter replaced
+  the block result with its error; the logger/instrumenter errors are swallowed now (like in all other events);
+- `#locks_info` (and `#locks(with_info: true)`): the info of write locks had no `"lock_key"` field (the info of read locks and
+  `#lock_info` have it), so the info shape depended on the lock mode; the write lock info has `"lock_key"` now (the same
+  fields as `#lock_info`);
+- `RedisQueuedLocks::Logging::Sampler` and `RedisQueuedLocks::Instrument::Sampler` (the default log/instrumentation samplers)
+  sampled `sampling_percent` of 101 values (`SecureRandom.rand(0..100)`): `0%` sampled ~1% of the cases (instead of none),
+  `15%` sampled ~15.8%; they sample exactly `sampling_percent` of 100 values now (`0%` - never, `100%` - always);
+  the private `SAMPLING_PERCENT_RANGE` constants are removed;
+- swarm: the Redis client builder of the swarm elements mutated the `swarm.*.redis_config.config`/`pool_config` hashes of
+  the client config (`transform_keys!`): with frozen hashes each start of the host probing and zombie flushing main loops
+  failed with `FrozenError` (the hosts were not probed, zombie locks were not flushed); the hashes are not mutated now;
+- RBS: the `RedisQueuedLocks::Client#current_acq_id` alias was declared as `ccurrent_acq_id` (a typo: the public method was
+  missing in the signatures); `RedisQueuedLocks::Client#queues_info` returned the `queues` union (`queueList | queuesInfo`)
+  instead of `queuesInfo`;
+- `RedisQueuedLocks::Resource.lock_key_from_queue` (a private debug helper for manual checks in a console, not used by
+  the library) returned a wrong key for the read lock request queues (`rql:lock:ueue:<name>`); it supports the write and
+  read lock request queues now and returns `nil` for other keys;
+- a connection interrupted inside a `WATCH` block by a `Timeout.timeout` of your code without an exception class
+  (`Timeout::ExitException`, e.g. `Timeout.timeout(5) { rql.lock("x") { ... } }`) returned to the connection pool with
+  the active `WATCH` (redis-client sends `UNWATCH` only on `StandardError`s): the next transaction on the connection could
+  be aborted (e.g. a lock release); the connection is closed now (like with the internal timeouts);
+- the per-call numeric options were not validated (zero/negative values broke the operations like the same config values
+  before the config validation fix): they are validated with the same ranges as their config keys now
+  (`RedisQueuedLocks::ArgumentError`; `RedisQueuedLocks::SwarmArgumentError` for the swarm methods), before any Redis
+  command; the values should be `Integer`s (like the config values: `Float` values such as `timeout: 0.5` or
+  `retry_delay: 10.0` are rejected now):
+  - `#lock`/`#lock!`/`#lock_series`/`#lock_series!`: `queue_ttl` > 0, `timeout` and `retry_count` - `nil` or >= 0,
+    `retry_delay`, `retry_jitter` and `lock_index_grace_period` >= 0;
+  - `#locks`, `#locks_info`, `#queues`, `#queues_info`, `#keys`, `#clear_dead_requests`, `#reindex_locks`: `scan_size` > 0;
+    `#clear_locks`: `batch_size` > 0; `#clear_locks_of`/`#clear_current_locks`: `lock_scan_size`, `queue_scan_size` > 0;
+    `#clear_dead_requests`: `dead_ttl` > 0; `#reindex_locks`: `queue_ttl` > 0, `lock_index_grace_period` >= 0;
+  - `#swarm_info`, `#flush_zombies`, `#zombie_locks`, `#zombie_acquirers`, `#zombie_hosts`, `#zombies_info`: `zombie_ttl`
+    and the scan sizes > 0;
+- README/YARD documentation that did not match the code:
+  - config keys: `config['instr_sampling_enabled']` in the configuration example (was `nstr_sampling_enabled`),
+    `config['log_sampling_enabled']` (was `log_samplign_enabled`), `*_sampling_enabled` (was `*_sampling_enalbed`);
+  - `#lock` results: `:retry_limit_reached` (was `:retry_count_reached`), `:conflict_lock_upgrade` is listed, `:unknown`
+    is not returned, `:dead_locking` is not a `:process` value (it fails with `:conflict_dead_lock`), the result has `hst_id`;
+  - `#lock_info`: `"ini_ttl"` (was `"init_ttl"`), `"spc_cnt"` (was `"spc_count"`), host ids `rql:hst:...` in the examples;
+  - `#unlock`: `rel_time` is in milliseconds (was "in seconds");
+  - `#clear_dead_requests` returns `{ ok: true, result: { processed_queues: Set<String> } }` (write and read request queues);
+  - `#extend_lock_ttl`: the write lock TTL is extended atomically by one Lua script, without the lock owner check
+    (was "non-atomic, two commands");
+  - the log lists of the `### Configuration` section: `rw_mode`, the key order, the read/write and lock series logs
+    (the same as in the `## Logging` section);
+  - the `#lock!` link of the Usage list and the `#lock` link of `#lock_info`, the Usage list and the table of contents: `unlock_read`, `clear_locks_of`,
+    `clear_current_locks`, `reindex_locks`;
+  - YARD: the `#clear_current_locks` options (`lock_scan_size`, `queue_scan_size` instead of the missing `batch_size`), the tag
+    typos (`@sine`, `@since ?.?.?`, `@since 19.0.0`, `@api ppublic`/`pulic`/`pub;ic`); the inert `# steep:ingore` comments
+    are removed;
+  - YARD: the `@version` tags of the classes/modules (versions >= 1.17.0) follow the interface changes only (method
+    signatures, added/removed methods, `extend`/`include`/`prepend`, the superclass), not the behavior changes inside
+    the methods;
+
+## [1.20.0] - 2026-10-12
+### Added
+- **Lock index** (opt-in: `config['lock_index_enabled']`, `false` by default: lock-iterating operations scan the keyspace as before):
+  `rql:index:locks` (`ZSET`: lock name => the upper bound of the lifetime of all lock keys in client time, ms); with the enabled
+  lock index all lock-iterating operations find locks without the full keyspace `SCAN` (see the "Lock Index" README section):
+  - the client-wide mode (not a per-call option): the lock acquirement maintains the lock index and lock-iterating operations
+    read it only when it is enabled; use the same mode on all processes that share the Redis database (enable it everywhere,
+    then invoke `#reindex_locks` once); the swarm reads the mode when it is started;
+  - maintained by the lock acquirement itself without new Redis round trips: each lock request adds the lock name (`ZADD GT`
+    with the queue ttl), sets the request queue TTL and drops the expired lock names (`ZREMRANGEBYSCORE`) in the queueing pipeline
+    (4 more commands per attempt), each lock obtaining (write and read) adds the lock name with the lock ttl in the same
+    transaction (1 more command), each extension (`#extend_lock_ttl`, `:extendable_work_through` reentrant locks) increases
+    the score by the extension (`ZADD XX INCR`; `#extend_lock_ttl` of write locks sends `EVAL` and the index update in one
+    transaction);
+  - `#unlock` and `#clear_locks` drop the lock names from the index in the same transaction with the lock keys;
+  - lock request queues have TTL: the queue lives as long as its latest lock request (`PEXPIRE NX` + `GT` with the queue ttl on
+    each lock request), queues of dead acquirers expire by themselves;
+  - lock-iterating operations find locks and lock queues via one `ZSCAN` pass over the lock names (lock keys are derived from
+    the lock names): the cost depends on the number of locks only, keys of other applications are never visited; locks without
+    the lock index entry (older RQL versions, the disabled lock index) are not found until `#reindex_locks`:
+    - swarm (`#flush_zombies`, the zombie flusher, `#zombie_locks`/`#zombie_acquirers`/`#zombies_info`): `zombie_lock_scan_size` is
+      the `ZSCAN` count and the batch size of locks, `zombie_queue_scan_size` - the batch size of lock queues (both processed during
+      the same pass); `#flush_zombies` drops the expired lock names from the index;
+    - `#locks`/`#locks_info`, `#queues`/`#queues_info`, `#keys`: `scan_size` is the `ZSCAN` count and the batch size of locks;
+      `#keys` returns the existing keys of the lock key families of the indexed locks (other `rql:lock*` keys are not listed);
+    - `#clear_locks`: the keys of each batch of locks are deleted with one `DEL` in one transaction with the removal of their lock
+      names from the lock index; `batch_size` is the number of locks;
+    - `#clear_locks_of`/`#clear_current_locks`: `clear_locks_of__lock_scan_size` is the `ZSCAN` count and the batch size of locks,
+      `clear_locks_of__queue_scan_size` - the number of lock queues processed in one round trip;
+    - `#clear_dead_requests`: `scan_size` is the `ZSCAN` count and the number of locks (both lock queues of each lock) processed in
+      one round trip; the expired lock names are dropped from the index;
+  - `config['lock_index_grace_period']` (`60_000` ms by default) and `lock_index_grace_period:` option of `#lock`/`#lock!`/
+    `#lock_series`/`#lock_series!`: the lock name stays in the index for this period after the expiration of all its keys
+    (the client clock skew and network latency tolerance);
+- `#reindex_locks`: adds the lock keys without the lock index entry (keys of older RQL versions and keys created with the disabled
+  lock index) to the lock index (one `SCAN` pass, lock queues without TTL get the queue ttl); **invoke it once after the lock index
+  is enabled on all your processes**;
+### Changed
+- `#flush_zombies` (and the zombie flusher of the swarm): zombie lock requests are dropped with one `ZREM` per lock queue
+  (previously one `ZREM` per zombie acquirer and lock queue);
+- `#queue_info` / `#queues_info`: lock queues are read without `EXISTS` checks (Redis does not keep empty sorted sets);
+- Lock acquirement attempt: read lock requests do not fetch the longest read lock (it is used by write lock requests only),
+  write lock requests do not build the read lock data key, disabled try-lock logs get a shared frozen hash instead of a new one;
+- Lock acquirement attempt: the lock request queueing (`ZADD NX`, expired requests pruning, queue heads) is one pipelined
+  round trip instead of three; with `:wait_for_lock`/`:dead_locking` conflict strategies (without `fail_fast`) it is sent
+  in the lock state round trip: `#lock` with a block takes 4 Redis round trips instead of 7, a failed attempt - 3 instead of 6;
+- SCAN-based operations (`#clear_locks`, `#clear_locks_of`/`#clear_current_locks`, `#clear_dead_requests`, `#locks`/`#locks_info`,
+  `#queues`/`#queues_info`, `#flush_zombies`, `#zombie_locks`/`#zombie_acquirers`/`#zombies_info`) make one `SCAN` pass instead
+  of 2-5 passes (one per key family) and send per-key commands in pipelined batches instead of one round trip per key:
+  - `#clear_locks` deletes the found keys during the scan with one `DEL` per batch (previously all keys were expired at the end,
+    lock queues first);
+  - `#clear_locks_of`/`#clear_current_locks` (`clear_locks_of__queue_scan_size`) and `#flush_zombies` (`zombie_queue_scan_size`):
+    the queue scan size is the number of lock queues processed in one round trip (queues are found by the same pass as locks);
+  - `#zombie_locks`/`#zombie_acquirers`/`#zombies_info` do not scan the keyspace when there are no zombie hosts;
+- Read lock data (`rql:lock_reader:<lock_name>:<acquirer_id>`) does not store `acq_id`/`hst_id` (both are derived from the acquirer id):
+  ~40% less memory per read lock (`MEMORY USAGE` 280 -> 168 bytes without `:meta`); `#lock_info`/`#locks_info` report them as before
+  (the first fields of each reader; read lock data of older versions is supported);
+- `#unlock`: one Redis round trip when there are no read locks (the readers registry is read in the release transaction;
+  the read lock data is dropped with one more `DEL` only when read locks existed);
+- `#lock_series` (PoC): write locks of the series are released (compare-and-delete) in one transaction for the whole series
+  (three round trips per series instead of three round trips per lock);
+- `#flush_zombies` (and the zombie flusher of the swarm): requests of zombie hosts are dropped from the lock queues by the host
+  of the request (derived from its acquirer id): requests of dead hosts that hold no locks are dropped too (previously they stayed
+  until the queue ttl) and their acquirers are reported in `deleted_zombie_acquirers`;
+- Lock acquirement: retries and `fail_fast` attempts pre-check the lock state without `WATCH` (one round trip): a retry that can not
+  obtain the lock takes 1 Redis round trip instead of 3 and a `fail_fast` attempt on an obtained lock - 1 instead of 3 (an attempt
+  that can obtain the lock continues with `WATCH`/`MULTI`: one more round trip); mutual exclusion still relies on `WATCH`/`MULTI` only;
+
 ## [1.19.0] - 2026-10-11
 ### Added
 - `#unlock_read` (aka `#release_read_lock`): an ability to unlock the read lock of any acquirer

data/CLAUDE.md

@@ -14,6 +14,8 @@ Full details (config keys, key layout, events, errors, stack, CI): `.claude/proj
 - `acquirer/acquire_lock.rb`: composed via `extend` mixins in `acquire_lock/` (try_to_lock, with_acq_timeout, delay_execution, dequeue_from_lock_queue, yield_expire) + `LogVisitor`/`InstrVisitor` event hooks.
 - Locking: `ZADD NX` into the request queue, then WATCH/MULTI; Lua only for write TTL extension/decrease.
 - Read/Write locks (`read_write_mode: :write|:read`): write lock `rql:lock:<name>` + queue `rql:lock_queue:<name>`; readers registry `rql:lock_readers:<name>` (ZSET acq_id => Redis-time expiry), read queue `rql:lock_read_queue:<name>`, per-reader data `rql:lock_reader:<name>:<acq_id>`. Readers WATCH the write key only, writers WATCH the write key + registry; queues give ordering only. Details: overview "Read/Write locks".
+- Lock lookup modes: `config['lock_index_enabled']` (client-wide, `false` by default) is read by `Client`/`Swarm` and passed to every lock-iterating module (swarm, `locks`, `queues`, `keys`, `clear_*`) and to the hot path; each module keeps the keyspace lookup (one `SCAN` pass, the pre-index code) and the lock index lookup (one `ZSCAN` pass) side by side with the shared processing in one copy.
+- Lock index `rql:index:locks` (enabled mode only; ZSET lock name => upper bound of the lifetime of all lock keys, client ms + ttl + `lock_index_grace_period`): every key-creating command is followed by `ZADD GT` (queueing pipeline, obtaining `MULTI`), every extension by `ZADD XX INCR`, names are removed only with all their keys (`unlock`, `clear_locks`), queues have TTL, expired names are pruned on queueing; lock index commands are conditional tails of their pipelines/transactions (none in the disabled mode); `reindex_locks` (one keyspace `SCAN`, both modes) adopts keys without entries after enabling. Details: overview "Lock index".
 - Strategies: `access_strategy` `:queued`|`:random`; `conflict_strategy` `:wait_for_lock`|`:work_through`|`:extendable_work_through`|`:dead_locking`.
 - `resource.rb`: all key names and acquirer/host IDs. `config.rb`: `setting`/`validate` DSL, read as `config['a.b']`.
 - `swarm/`: zombie-lock cleanup. `Supervisor` thread revives `ProbeHosts` (`SwarmElement::Threaded`) and `FlushZombies` (`SwarmElement::Isolated`, Ractor talking via `Ractor::Port`s); each element runs a main-loop thread with its own Redis connection.
@@ -24,13 +26,14 @@ Full details (config keys, key layout, events, errors, stack, CI): `.claude/proj
 Detailed, path-scoped rules in `.claude/rules/`: `logic.md` (lib, general), `acquirer.md` (acquirer modules, Redis access), `visitors.md` (log/instrumentation visitors), `arguments.md` (long keyword lists), `swarm.md` (swarm supervisor/elements, Ractor/Thread rules), `tests.md` (spec), `type-checking.md` (sig/Steep), `git.md` (commit messages and commit flow, always loaded).
 - Long explicit keyword/parameter lists are intentional (minimal allocations, signature-as-DSL): never introduce parameter objects, option structs or `**opts`; add new options as explicit keywords to every `Client` variant and forward them by name.
 - Update the matching `sig/*.rbs` for every `lib/` change; keep Steep green.
-- Keep `# frozen_string_literal: true` and YARD `@api`/`@since`/`@version` tags.
+- Keep `# frozen_string_literal: true` and YARD `@api`/`@since`/`@version` tags (class/module `@version` changes only with the interface: signatures, added/removed methods, `extend`/`include`/`prepend`, the superclass; `logic.md#2`).
 - `{ ok:, result: }` is for the public API only (`Client`, `Acquirer::*`, `Swarm` facade actions). Internal swarm element APIs (commands, replies, helpers) use bare scalars/primitives (see `swarm.md`).
 - New operation: `Acquirer::*` module, thin `Client` method (+ `!` variant), RBS, spec.
 - Operation modules behind `Client` methods (all `Acquirer::*` except the `AcquireLock` acquisition core and PoC modules such as `LockSeriesPoC`, which may reuse other modules) are independent: they never call each other or reference each other's RBS types, share logic only via `Resource`/`Utilities`, and duplicate similar logic on purpose (independence over DRY; e.g. `Locks` duplicates the `LockInfo` formatting). Details: `acquirer.md`.
 - New option: `setting` (+ `validate`) in `config.rb`.
 - Dev gems go in `Gemfile`, not the gemspec.
-- Commits (`git.md`): subject only `[<scope>] <summary>` (several features: 1-2 words each, comma-separated), no body; when a step is complete (requirement fully implemented, specs/rubocop/steep green) ask before committing, with the proposed subject and a recap of the changes; an explicit "commit" request needs no extra question; never on `master`, never push.
+- Commits (`git.md`): subject only `[<scope>] <summary>` (several features: 1-2 words each, comma-separated), no body; when a step is complete (requirement fully implemented, specs/rubocop/steep green), and only then, run the `rql-code-review` skill on the uncommitted changes, then ask before committing, with the proposed subject, a recap of the changes and the review verdict; an explicit "commit" request needs no extra question and no review (a commit authorized in advance for a task still gets the review); never on `master`, never push.
 - Before implementing, read the relevant sections of `.claude/project-overview.md` (key layout, lock flow, "Read/Write locks", events) and keep their invariants. In the same change, update the overview and the affected `.claude/rules/*.md` for every new/changed key, option, result shape, event, algorithm step or limitation (and README/CHANGELOG `[Unreleased]` for user-visible behavior). New or changed log keys / instrumentation payload keys (visitors or inline `notify`) also go to README `## Logging` / `### Instrumentation Events` (see `visitors.md`).
-- Lock-touching code is read/write-aware: any operation over locks, queues, zombies or lock info must cover the read family (`rql:lock_readers:*`, `rql:lock_read_queue:*`, `rql:lock_reader:*`) next to `rql:lock:*`/`rql:lock_queue:*`. Mutual exclusion comes only from WATCH/MULTI on lock state (no Lua in the RW acquisition path unless asked); queues never guarantee safety.
+- Observability (`logic.md#16`): logic that creates/changes/deletes keys is logged and instrumented: the lock acquirement per meaningful outcome (visitors), each mutation operation with one log line + one event of the whole root operation called by `Client` (inline, `instrument` last in the payload); lock info queries are neither; the swarm elements are known debt.
+- Lock-touching code is read/write-aware and index-aware: any operation over locks, queues, zombies or lock info must cover the read family (`rql:lock_readers:*`, `rql:lock_read_queue:*`, `rql:lock_reader:*`) next to `rql:lock:*`/`rql:lock_queue:*`, and any new key-creating/extending command must keep the lock index a superset of the existing locks (enabled mode); a new lock-iterating operation implements both lookups (keyspace and lock index, `lock_index_enabled`). Mutual exclusion comes only from WATCH/MULTI on lock state (no Lua in the RW acquisition path unless asked); queues never guarantee safety.
 - Known debt: rspec-retry, disabled coverage minimum, runtime type-check CI uses `--failure-exit-code=0`.