familia 2.6.0 → 2.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5577963e36f6c5ce2ce1fc977458c092b6883430e1c6c3e4c72742ddb4cae795
-  data.tar.gz: bc958eef278b236f894652b4954e7235a917c858f26e8c63427b1fb7bb62713a
+  metadata.gz: c6cb1ccd59d4290c1d75b70e114945180fc5dbb03aaeb405d841c250cd504179
+  data.tar.gz: c2bd7946e7e024c8322d0bf48bea39c22cd61c7690589007f9ab32e6f12e614d
 SHA512:
-  metadata.gz: 2820cc6134dd2a707a8e7c91855c42b622807748cc428984bb725f71faada7489ed5211ac3dedbe887132e13523cd4e8c565abeaf57c7d9df411566fb60766ed
-  data.tar.gz: 1bc50c330c90ac7edcb460c7f8688d8e47d8329e2770a36b8963e24ea00cb258ee017f2c7ec905a2f19b38949ea609f381a9d65d0460711cc725c26b9f664053
+  metadata.gz: 54d3c3020c53fe01de9baf6af78fbcc1ffbea72a671b6e6269c08c9c5eaab786bb267514aa64212ff8ec854e0363cfb1c1de9b9c66674e57cb067ed419a8944b
+  data.tar.gz: d8b8ecd85ed1273c64ae75cab7559dee0ddb92fe5d552b60690c3a66ee9df93b3829aa32deef5438677ad9fb8d9acebb0bdbf071f3bccdb63161c4a772b1c021

data/.pre-commit-config.yaml

@@ -62,8 +62,6 @@ repos:
     rev: v1.32.2
     hooks:
       - id: talisman-commit
-        env:
-          TALISMAN_SCAN_MODE: CI
 
   # Commit message issue tracking integration
   - repo: https://github.com/delano/add-msg-issue-prefix-hook

data/CHANGELOG.rst

@@ -7,6 +7,215 @@ The format is based on `Keep a Changelog <https://keepachangelog.com/en/1.1.0/>`
 
    <!--scriv-insert-here-->
 
+.. _changelog-2.8.0:
+
+2.8.0 — 2026-05-15
+==================
+
+Added
+-----
+
+- Expanded Redis 7 command coverage across all DataType classes:
+
+  - **StringKey**: ``incrbyfloat``, ``getex``, ``getdel``, ``setex``, ``psetex``,
+    ``bitcount``, ``bitpos``, ``bitfield``, plus class methods ``mget``, ``mset``,
+    ``msetnx``, ``bitop`` for multi-key and bitwise operations.
+
+  - **List**: ``trim``/``ltrim``, ``set``/``lset``, ``insert``/``linsert``,
+    ``move``/``lmove``, ``pushx``/``rpushx``, ``unshiftx``/``lpushx``.
+    Updated ``pop`` and ``shift`` to support optional count parameter for batch operations.
+
+  - **UnsortedSet**: ``intersection``/``inter``, ``union``, ``difference``/``diff``,
+    ``member_any?``/``members?``, ``scan``, ``intercard``/``intersection_cardinality``,
+    ``interstore``/``intersection_store``, ``unionstore``/``union_store``,
+    ``diffstore``/``difference_store``.
+
+  - **SortedSet**: ``popmin``, ``popmax``, ``score_count``/``zcount``, ``mscore``,
+    ``union``, ``inter``, ``rangebylex``, ``revrangebylex``, ``remrangebylex``,
+    ``lexcount``, ``randmember``, ``scan``, ``unionstore``, ``interstore``,
+    ``diff``, ``diffstore``.
+
+  - **HashKey**: ``scan``/``hscan``, ``incrbyfloat``/``incrfloat``,
+    ``strlen``/``hstrlen``, ``randfield``/``hrandfield``, plus field-level TTL
+    commands (Redis 7.4+): ``expire_fields``, ``pexpire_fields``, ``expireat_fields``,
+    ``pexpireat_fields``, ``ttl_fields``, ``pttl_fields``, ``persist_fields``,
+    ``expiretime_fields``, ``pexpiretime_fields``.
+
+- Added 158 new tests across 5 test files covering all new methods.
+
+- Instance-scoped ``audit_multi_indexes`` is now fully implemented.
+  Discovers per-scope bucket keys via SCAN, partitions them by scope
+  instance, and reports stale members, orphaned buckets, and missing
+  entries in the same shape as the class-level audit. Orphan entries
+  carry a ``:reason`` (``:scope_missing`` or ``:field_value_unheld``)
+  and a ``:scope_id``. Missing entries are detected via the indexed
+  class's ``participates_in`` relationship to the scope class; when
+  absent, the result carries ``missing_status: :not_audited``.
+  Resolves the ``:not_implemented`` follow-up from #217.
+
+- ``repair_multi_indexes!`` class method that invokes the existing
+  ``rebuild_<index_name>`` methods for both class-level (one call on
+  the indexed class) and instance-scoped (one call per scope
+  instance) multi-indexes. Indexes whose audit status is ``:ok`` are
+  skipped; rebuild methods that don't exist or scope classes
+  without an ``instances`` collection are recorded in ``:skipped``
+  with a reason.
+
+- ``housekeeping`` feature gains a class-level bulk runner,
+  ``Klass.run_chores!(chore_name:, limit:, batch_size:)``. It iterates
+  the class's ``instances`` collection in pipelined batches via
+  ``load_multi``, runs all registered chores (or one named chore)
+  against each record, and returns a stats hash:
+  ``{ model:, scanned:, chores: { name => { modified:, errors: } } }``.
+  Truthy chore returns increment ``modified``; raised exceptions are
+  isolated per-record, logged via ``Familia.warn``, and counted as
+  ``errors`` so a single failure doesn't halt the run. Lifted from the
+  shape proven out in OneTime Secret's ``HousekeepingJob``.
+
+- Trace events for connection-mode conflicts. ``Familia.trace`` now
+  emits ``CONFLICTING_CONTEXT`` when pipeline and transaction
+  contexts collide (in both ``FiberPipelineHandler``/
+  ``FiberTransactionHandler`` and the
+  ``execute_transaction``/``execute_pipeline`` entry points), and
+  ``FAST_WRITER_BLOCKED`` when a fast writer (``field!``) is called
+  inside a transaction or pipeline. These fire just before the
+  corresponding ``ConflictingContextError`` /
+  ``OperationModeError`` is raised, so operators can pinpoint where
+  blocked operations originate when ``FAMILIA_TRACE=1``.
+
+Changed
+-------
+
+- ``repair_all!`` now runs each repair stage inside its own rescue
+  boundary; a failure in one dimension no longer prevents the others
+  from running. The return hash gains ``:status`` (``:ok`` or
+  ``:partial_failure``), ``:errors`` (per-stage exception details
+  when raised), and ``:multi_indexes`` (results from the new
+  ``repair_multi_indexes!``). An opt-in ``verify: true`` kwarg
+  re-runs ``health_check`` after repair and exposes the result as
+  ``:post_audit`` / ``:verified`` so callers can confirm the run
+  actually drove the model back to a healthy state.
+
+- ``AuditReport#complete?`` is no longer false-positive due to
+  ``:not_implemented`` stubs in ``multi_indexes`` -- instance-scoped
+  indexes return ``:ok`` or ``:issues_found`` like class-level ones.
+
+- ``housekeeping`` feature: split the dual-purpose ``tidy!`` into two
+  explicit instance methods. ``do_chore!(name)`` runs a single named
+  chore and returns the block's raw return value (no longer wrapped
+  in a ``{name => result}`` hash). ``do_chores!`` runs every
+  registered chore and returns the ``{name => result}`` hash.
+  ``tidy!`` is preserved as an alias of ``do_chores!`` for backwards
+  compatibility with the 2.7.0 no-arg call site; the single-arg form
+  ``tidy!(:name)`` now raises ``ArgumentError``.
+
+- The connection handler hierarchy has been refactored from class
+  inheritance (``BaseConnectionHandler``) to module composition.
+  Handlers now ``include Familia::Connection::Handler`` and declare
+  their operation-mode capabilities with a small DSL:
+  ``supports transaction: true, pipelined: false``. The
+  ``BaseConnectionHandler`` constant is gone. This is only relevant if
+  you have custom handlers in application code — the public
+  ``allows_transaction`` / ``allows_pipelined`` class methods continue
+  to work, and the singleton ``.instance`` accessors on
+  ``FiberPipelineHandler`` / ``FiberTransactionHandler`` are
+  unchanged. The previous default of "allow all operations" when
+  capability flags were not set has been removed; every handler is now
+  expected to declare its capabilities explicitly via ``supports``.
+- ``Familia.dbclient`` and ``Familia::DataType#dbclient`` now route through ``FiberPipelineHandler`` before ``FiberTransactionHandler``, matching ``Horreum#dbclient``. With both handlers in the chain, attempting to mix pipeline and transaction contexts raises ``Familia::ConflictingContextError`` uniformly from every call site.
+
+Removed
+-------
+
+- ``Familia::DataType#direct_access`` has been removed. The method
+  was a legacy escape hatch for issuing raw Redis commands from
+  inside a DataType wrapper; it predates the chain-based routing of
+  ``Fiber[:familia_transaction]`` and ``Fiber[:familia_pipeline]``.
+  All in-tree call sites now go through the wrapper's own mutating
+  methods (which auto-route through the active transaction or
+  pipeline) or through the wrapper's ``transaction`` / ``pipelined``
+  blocks. If you were calling ``direct_access do |conn, key| ... end``,
+  replace it with either the DataType's own mutator or the
+  corresponding block API.
+
+Fixed
+-----
+
+- ``SortedSet#popmin`` and ``SortedSet#popmax`` now normalize an explicitly
+  passed ``nil`` count to the default of ``1``. Previously, calling
+  ``zset.popmin(nil)`` or ``zset.popmax(nil)`` would bypass the ``count == 1``
+  branch of the structural dispatch added in the prior commit, causing
+  redis-rb's flat ``[member, score]`` return shape to be iterated as if it
+  were a nested result — yielding a malformed pair. Omitting the argument
+  was and remains unaffected.
+
+- Restored ``require 'set'`` in ``lib/familia/horreum/management/audit.rb``. ``Set`` is autoloaded as a core class only on Ruby 3.4+; on Ruby 3.2/3.3 (the gem's supported floor) the require is mandatory for the five ``Set.new`` usages in that file.
+
+AI Assistance
+-------------
+
+- Claude Opus 4.5 analyzed Redis 7 command documentation and compared coverage
+  against existing Familia DataType implementations using parallel Explore agents.
+- Implementation performed by 5 parallel backend-dev agents, one per DataType.
+- Test coverage written by 5 parallel qa-automation-engineer agents focusing on
+  Familia-specific behavior (serialization, deserialization, aliases) rather than
+  re-testing redis-rb gem functionality.
+
+- Edge case identified by the Claude Code Review GitHub Action
+  (``.github/workflows/claude-code-review.yml``) when reviewing the
+  structural-dispatch change in commit ``010d5be``. Fix drafted and verified
+  by Claude Opus 4.7 under supervision.
+
+- Instance-scoped multi-index audit algorithm (bucket discovery,
+  scope existence batching, participation-driven missing detection),
+  ``repair_multi_indexes!``, the ``repair_all!`` robustness
+  refactor, and the accompanying tryouts coverage were authored
+  with Claude Code assistance against the #217 review branch.
+
+- Method split, alias wiring, bulk runner port from OTS, doc updates,
+  and expanded tryouts coverage (25 → 48 testcases) authored with
+  Claude Code.
+
+- Added the trace instrumentation in response to PR #263 review
+  feedback (Claude Code review bot) recommending tracing for
+  conflict detection events.
+
+- The handler refactor, ``direct_access`` removal, and changelog drafting were performed with Claude Code assistance while resolving review feedback on PR #263.
+
+.. _changelog-2.7.0:
+
+2.7.0 — 2026-05-13
+==================
+
+Added
+-----
+
+- New ``housekeeping`` feature for ``Familia::Horreum``: a declarative DSL
+  (``chore :name do |obj| ... end``) for registering named cleanup blocks on
+  a model class, plus an instance method ``tidy!`` that runs all (or one)
+  registered chore against a single object. The feature owns registration
+  and per-instance execution only -- iteration, batching, scheduling and
+  error aggregation are the consumer application's responsibility, keeping
+  it distinct from ``Familia::Migration`` (which is for versioned, one-shot
+  transformations). Resolves #258.
+
+Documentation
+-------------
+
+- Added ``docs/guides/feature-housekeeping.md`` covering the API, the
+  ``housekeeping`` vs ``migration`` vs defensive-setter trade-off,
+  generated method reference, design constraints, and common patterns
+  (multiple chores, sequential steps in one chore, tracking modified
+  records, error aggregation).
+
+AI Assistance
+-------------
+
+- Drafted the housekeeping feature module, the tryouts test suite, and the
+  guide using Claude Code, working from the API proposal in issue #258 and
+  the existing ``feature-relationships.md`` and ``safe_dump.rb`` as style
+  templates.
+
 .. _changelog-2.6.0:
 
 2.6.0 — 2026-04-17

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    familia (2.6.0)
+    familia (2.8.0)
       concurrent-ruby (~> 1.3)
       connection_pool (>= 2.4, < 4.0)
       csv (~> 3.3)

data/docs/guides/feature-housekeeping.md

@@ -0,0 +1,247 @@
+# Housekeeping Feature Guide
+
+The Housekeeping feature provides a declarative DSL for registering named cleanup chores on Horreum models. It is designed for short-lived, repeated tidying against fields whose values have drifted over time -- not for versioned, one-shot migrations.
+
+> [!TIP]
+> Enable with `feature :housekeeping` and register cleanup blocks with `chore :name do |obj| ... end`. Run all of them with `obj.do_chores!` (aliased `tidy!`), or one with `obj.do_chore!(:name)`. Iteration and persistence are the caller's responsibility.
+
+## Quick Start
+
+```ruby
+class Organization < Familia::Horreum
+  feature :housekeeping
+
+  field :planid
+
+  chore :standardize_planid do |org|
+    canonical = case org.planid
+                when "pro", "Pro", "professional_v1" then "professional"
+                when "free", "Free", "basic"         then "free"
+                end
+    if canonical && canonical != org.planid
+      org.planid = canonical
+      org.save
+      true
+    end
+  end
+end
+
+org = Organization.from_identifier("acme-corp")
+org.do_chores!
+# => { standardize_planid: true }
+
+# Or run a single chore by name (returns the block's raw value):
+org.do_chore!(:standardize_planid)
+# => true
+```
+
+## When to Use
+
+| Tool | Use When |
+|------|----------|
+| `Familia::Migration::Base` | Versioned, one-shot transformation tracked across releases |
+| `feature :housekeeping` | Short-lived chore run nightly until data is clean, then removed |
+| Defensive code in setters | Permanent invariant enforced on every write |
+
+Housekeeping fills the gap between migrations (heavy, tracked) and inline coercion (permanent). Register a chore, run it on a schedule for a few days, verify clean data, then delete the chore and the defensive code that handled the messy values.
+
+## Core Capabilities
+
+### Registration -- Class-Level DSL
+
+Each chore is a named block bound to the model class:
+
+```ruby
+class User < Familia::Horreum
+  feature :housekeeping
+
+  field :email, :timezone
+
+  chore :downcase_email do |user|
+    next unless user.email && user.email != user.email.downcase
+    user.email = user.email.downcase
+    user.save
+    true
+  end
+
+  chore :default_timezone do |user|
+    next if user.timezone
+    user.timezone = "UTC"
+    user.save
+    true
+  end
+end
+
+User.chores.keys
+# => [:downcase_email, :default_timezone]
+```
+
+### Execution -- Single Instance
+
+Run all registered chores, or one by name:
+
+```ruby
+user = User.from_identifier("alice@example.com")
+
+user.do_chores!
+# => { downcase_email: true, default_timezone: nil }
+
+user.tidy! # alias for do_chores!
+# => { downcase_email: nil, default_timezone: nil }
+
+user.do_chore!(:downcase_email)
+# => true
+```
+
+`do_chores!` returns a hash mapping chore name to the block's return value. `do_chore!` returns the block's raw return value (not wrapped in a hash). A truthy result signals "modified"; `nil` or `false` signals "no-op". The feature does not interpret these values -- they are passed through for the caller's stats collection.
+
+### Iteration -- Bulk Runner
+
+For running chores across every record, the feature ships a class-level `run_chores!` that iterates the `instances` collection in pipelined batches (via `load_multi`), executes each chore per record with error isolation, and returns a stats hash:
+
+```ruby
+Organization.run_chores!
+# => {
+#      model: "Organization",
+#      scanned: 4200,
+#      chores: {
+#        standardize_planid: { modified: 37, errors: 0 },
+#        uppercase_country:  { modified: 102, errors: 1 },
+#      },
+#    }
+
+Organization.run_chores!(chore_name: :standardize_planid, limit: 500)
+# Filter to one chore and cap records scanned.
+
+Organization.run_chores!(batch_size: 50)
+# Tune the load_multi pipeline batch size (default: 100).
+```
+
+A truthy chore return increments `modified`; a raised exception increments `errors` (logged via `Familia.warn`) and iteration continues. The runner requires the class to expose `instances` (Horreum's default class-level sorted set) and `load_multi`.
+
+For scheduling, cross-model orchestration, custom logging, or non-default iteration (e.g. a configured allowlist of model classes), wrap `run_chores!` in your own job. The feature deliberately stays out of cron, multi-model discovery, and project-specific logging layers.
+
+## Generated Method Reference
+
+### When a class declares `feature :housekeeping`
+
+| Class | Method | Purpose |
+|-------|--------|---------|
+| **Class** | `chore(name, &block)` | Register a chore |
+| | `chores` | Hash of registered chores |
+| | `run_chores!(chore_name:, limit:, batch_size:)` | Bulk runner across `instances`; returns stats hash |
+| **Instance** | `do_chore!(name)` | Run a single chore by name; returns the block's raw value |
+| | `do_chores!` | Run every registered chore; returns Hash |
+| | `tidy!` | Alias for `do_chores!` |
+
+## Design Constraints
+
+1. **No implicit saves.** The block must call `save` (or `commit_fields`) itself. The feature does not auto-persist.
+2. **Bulk via `run_chores!` only.** The feature operates on a single instance (`do_chore!`/`do_chores!`) plus one bulk runner (`run_chores!`) that iterates `instances`. Scheduling, multi-model orchestration, and custom logging stay in the consumer app.
+3. **No ordering.** Chores run in registration order, but should not depend on each other. If order matters, write one chore with sequential steps.
+4. **Idempotent by convention.** Use the conditional pattern (`if canonical && canonical != org.planid`) so a second run is a no-op.
+5. **Errors isolate in `run_chores!`, propagate in `do_chore!`/`do_chores!`.** Single-instance methods let exceptions propagate; the bulk runner rescues per-record and increments the chore's `errors` counter so one failure doesn't halt the run.
+
+## Common Patterns
+
+### Multiple Independent Chores
+
+```ruby
+class Customer < Familia::Horreum
+  feature :housekeeping
+
+  chore :trim_whitespace do |c|
+    next unless c.name && c.name != c.name.strip
+    c.name = c.name.strip
+    c.save
+    true
+  end
+
+  chore :uppercase_country do |c|
+    next unless c.country && c.country != c.country.upcase
+    c.country = c.country.upcase
+    c.save
+    true
+  end
+end
+
+customer.do_chores!
+# => { trim_whitespace: true, uppercase_country: nil }
+```
+
+### Sequential Steps in One Chore
+
+When step B depends on step A's result, keep them in one block:
+
+```ruby
+chore :reconcile_billing do |account|
+  changed = false
+  if account.plan_id == "legacy"
+    account.plan_id = "standard"
+    changed = true
+  end
+  if account.plan_id == "standard" && account.billing_cycle.nil?
+    account.billing_cycle = "monthly"
+    changed = true
+  end
+  if changed
+    account.save
+    true
+  end
+end
+```
+
+### Tracking Modified Records (Bulk)
+
+`run_chores!` already aggregates `modified` and `errors` counts per chore. Use it directly:
+
+```ruby
+report = Organization.run_chores!
+report[:chores].each do |name, counts|
+  puts "#{name}: #{counts[:modified]} modified, #{counts[:errors]} errors"
+end
+```
+
+### Custom Iteration (e.g. SCAN-Based)
+
+If `instances`-driven iteration isn't suitable (sharded data, custom scoping), drop down to `do_chore!`/`do_chores!`:
+
+```ruby
+modified = []
+Organization.instances.each do |id|
+  org = Organization.find_by_identifier(id) or next
+  results = org.do_chores!
+  modified << id if results.values.any?
+end
+puts "Modified #{modified.size} records: #{modified.inspect}"
+```
+
+### Wrapping `run_chores!` for a Job Framework
+
+```ruby
+class HousekeepingJob
+  def self.perform_for(klass)
+    report = klass.run_chores!(batch_size: 50)
+    StatsD.gauge("housekeeping.#{klass.name}.scanned", report[:scanned])
+    report[:chores].each do |chore, counts|
+      StatsD.increment("housekeeping.#{klass.name}.#{chore}.modified", counts[:modified])
+      StatsD.increment("housekeeping.#{klass.name}.#{chore}.errors",   counts[:errors])
+    end
+    report
+  end
+end
+```
+
+## Best Practices
+
+1. **Keep chores short-lived.** Delete the registration once data is clean.
+2. **Use `||=` and conditional checks** so a second run is a no-op.
+3. **Save inside the block** -- the feature does not persist for you.
+4. **Return truthy on modification, nil on no-op** so callers can collect stats.
+5. **Prefer migrations for one-shot, versioned transformations.** Use housekeeping for ongoing tidying that can be run repeatedly.
+
+## See Also
+
+- [**Writing Migrations**](writing-migrations.md) - Versioned, one-shot data transformations
+- [**Field System**](field-system.md) - How field values are stored and serialized
+- [**Feature System**](feature-system.md) - How features are mixed into Horreum classes

data/docs/guides/index.md

@@ -37,9 +37,13 @@ Welcome to the comprehensive documentation for Familia v2.0. This guide collecti
 13. **[Quantization](feature-quantization.md)** - Time-based data bucketing for analytics
 14. **[Time Literals](time-literals.md)** - Time manipulation and formatting utilities
 
+### 🧹 Data Maintenance
+
+15. **[Housekeeping](feature-housekeeping.md)** - Declarative cleanup chores for drifted field values
+
 ### 🛠️ Implementation & Usage
 
-15. **[Optimized Loading](optimized-loading.md)** - Reduce Redis commands by 50-96% for bulk object loading _(new!)_
+16. **[Optimized Loading](optimized-loading.md)** - Reduce Redis commands by 50-96% for bulk object loading _(new!)_
 
 
 ## 🚀 Quick Start Examples