forest_admin_agent 1.44.4 → 1.45.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 82246fa1590f7e734cedc309c456c3dca128ea82b325c3842ad6fa291a7051cf
-  data.tar.gz: 7943c129e8985805327acb63b8f49448ef6fa0a323c501cb420c5030eed514be
+  metadata.gz: b82038f1c254bc91295ded423735b4f481a3c0b997d4c246d606396ad476acec
+  data.tar.gz: 8bae29e2cbb6d1af5b9ebf4228a918eaace4b5489e26aa1ba7a098856da3bb38
 SHA512:
-  metadata.gz: fa96c5af1acbbaca154093885dcc4336c7e1b699de8854200655e9f10ce62564e09232bdffc71d468f08a61d7169dcb1ad08b2389ae518f32cf1432e716ecfee
-  data.tar.gz: 83bfa0a61d14812a660dafdfe2fc8ae8b07856dd7fb58caa6da56215d36e458a69efbc7eaf707bf1fc559416f5decb649b72301673bfbdabea00b4f878a806e7
+  metadata.gz: 336ea0d4c27df5a65ed30f6d5e64fd2a4645ae229a9fb874ea21b2c5501cef641c3a18bd91736cb52ed3b0b085dffbe9aa2f17f59471efcf62ab0fcd3ca3dfb4
+  data.tar.gz: 637d76398dc40a32ec459bb3ae3bdd783c82665a272988185efa1229823639f2cc64b564dbc7be6f1248f31a39f23af241d29bf636d1f48c4b0a7e0306826a55

data/AUDIT_TRAIL.md

@@ -139,6 +139,16 @@ It deliberately does **not** match `operation`, `correlationKey`, `recordId`, `c
 A field masked by `redact` never matches — neither by its `[redacted]` mask nor by the value it hid, which
 was never recorded. A search must not confirm a value the trail refused to keep.
 
+Nor one it withholds (see below). On a record gone for good under a caller's scope, `search` and `fields`
+are matched against the values as served, never as captured — in SQL, which rows come back, `meta.count`
+and `availableUsers` would each say whether a withheld value holds the term. The rows are read without those
+two filters, in batches of 500 that each continue past the last row read, never at an offset, and bounded at
+the instant the scan starts. They are matched and paged as they go, keeping only the page asked for. Only a
+gone record's history pays that scan, and it is one record's history — the same rows the SQL search
+would have scanned without an index. That holds too for a record deleted while the request was in flight: the second read
+of the record decides the withholding, so the SQL-matched answer is discarded and the history scanned the
+same way.
+
 `fields` matches whole keys, never paths, so a name holding a dot (`address.city`) is quoted before it
 reaches SQL. Both sides of the diff are searched, since a field the change added exists in `newValues`
 only and one it removed in `previousValues` only. The JSON test is per adapter (Postgres, SQLite, MySQL /
@@ -217,15 +227,37 @@ are logged, and the row keeps its operation, author and timestamp. This is also
 what makes an update's partial diff safe to test: a diff that never carried the scoped column answers for
 neither side, so both are withheld.
 
-The record is read twice: once before the rows are fetched, to refuse a record that exists outside the
-caller's scope without touching the audit database, and once after, so the answer that decides the
-withholding is never older than the rows it applies to — a record deleted in between would otherwise have
-answered "present and in scope" for rows that already carry its delete. The second read is skipped when no
-scope is in effect.
-
-**This covers the history route only.** `/state` reconstructs a gone record from the same rows and serves
-it unfiltered, and the correlation routes check the record but not the values, so a caller the withholding
-above protects against can still read those values one request away. Closing that is tracked separately.
+A captured `nil` is tested the way the database tests a `NULL`: a negated comparison (`!=`, not in, does not
+contain) never matches it. In memory `status != 'private'` holds for a nil status, while the scoped read that
+guarded the live record left that record out, so without this a record the caller could never read alive
+would become readable once deleted. A scope that asks for the nil itself (`status` is missing) still matches
+it. On a datasource whose own `!=` keeps a NULL (Mongo's `$ne`), this is stricter than the live read, never
+looser.
+
+The record is read twice, and **the values are withheld if either read found it gone**. The first read
+refuses a record that exists outside the caller's scope without touching the audit database, and it is the
+only one that saw the record as it was while the rows were being chosen. The second, once the rows are in
+hand, is the only one that can see a record deleted since — which would otherwise have answered "present and
+in scope" for rows that already carry its delete.
+
+Neither answer cancels the other, because an id outlives the record that held it. A record that took a freed
+id between the two reads answers for itself and not for the life whose rows these are: if this caller cannot
+read it, that is a **404**, the same as for a request starting a moment later; if they can, the earlier
+life's values stay withheld all the same. The second read is skipped when no scope is in effect, and on the
+correlation routes when the answer holds no rows.
+
+Where the delete and the recreation both happened *before* the request, both reads see the live record and
+its history is served whole, earlier lives included — the documented delete/recreate behaviour, and a
+separate question from the one above.
+
+**Every route that serves captured values applies this**, since a rule only one of them applies is one
+lookup away from being no rule at all. The two correlation routes withhold row by row exactly as the history
+route does. `/state` is nothing but those values reassembled, so the reconstruction is tested as a whole and
+`data` is `null` when it fails — including when the scope asks about something the reconstruction cannot
+answer. One difference there: a reconstruction can sit on the far side of a primary-key move the route cannot
+see, so the requested id does not fill in a key the trail redacted, only one that was never captured at all
+(read-only, so it cannot have moved). All of them take the second read of the record too, and skip it when
+there is nothing to withhold — no scope in effect, or no rows in the answer.
 
 ### State route
 
@@ -239,7 +271,8 @@ after** the timestamp — an entry stamped exactly at it counts as part of that
 
 `timestamp` accepts an ISO-8601 instant, or the same wall-clock forms as the filters above read in the
 request `timezone`; it is required (**400** otherwise). `data` is `null` when the record did not exist at
-that instant — either created later, or deleted and never recreated.
+that instant — either created later, or deleted and never recreated — and when the reconstruction of a gone
+record falls outside the caller's scope.
 
 Walking back stops being able to help where the trail stops: only audited (writable) columns are
 reconstructed, and a `create` means the record did not exist before it, while a `delete` restores the whole

data/lib/forest_admin_agent/audit_trail/store.rb

@@ -74,13 +74,15 @@ module ForestAdminAgent
       end
 
       def list_by_record(collection:, record_id:, skip: 0, limit: nil, user_ids: nil, start_timestamp: nil,
-                         end_timestamp: nil, fields: nil, search: nil, order: 'asc')
+                         end_timestamp: nil, fields: nil, search: nil, order: 'asc', after: nil)
+        descending = order.to_s == 'desc'
         # `id` (insertion order) breaks ties on equal timestamps in both directions, keeping pages
         # deterministic and stable.
         relation = scope(collection, record_id, user_ids: user_ids, start_timestamp: start_timestamp,
                                                 end_timestamp: end_timestamp, fields: fields, search: search)
-                   .order(timestamp: order.to_s == 'desc' ? :desc : :asc, id: :asc)
+                   .order(timestamp: descending ? :desc : :asc, id: :asc)
                    .offset(skip || 0)
+        relation = relation.where(*past_row_condition(after, descending)) if after
         relation = relation.limit(limit) unless limit.nil?
 
         relation.map { |row| from_row(row) }
@@ -167,6 +169,15 @@ module ForestAdminAgent
         relation
       end
 
+      # Rows strictly past `row` in the history's own order, so a scan keyed on the last row it read neither
+      # repeats nor skips one when entries are written between its reads, as an offset would.
+      def past_row_condition(row, descending)
+        at = as_time(row[:timestamp])
+        past = descending ? '<' : '>'
+
+        ["timestamp #{past} ? OR (timestamp = ? AND id > ?)", at, at, row[:id]]
+      end
+
       def as_time(value)
         value.is_a?(::Time) ? value : ::Time.iso8601(value.to_s)
       end

data/lib/forest_admin_agent/routes/resources/audit_trail.rb

@@ -13,6 +13,7 @@ module ForestAdminAgent
 
         DEFAULT_PAGE_SIZE = 20
         MAX_PAGE_SIZE = 100
+        SCAN_BATCH_SIZE = 500
         DATE_ONLY = /\A\d{4}-\d{2}-\d{2}\z/
         # Wall-clock datetime, `T` or space separator, seconds optional: `YYYY-MM-DD[T ]HH:mm[:ss]`.
         DATE_TIME = /\A(\d{4}-\d{2}-\d{2})[T ](\d{2}):(\d{2})(?::(\d{2}))?\z/
@@ -39,9 +40,10 @@ module ForestAdminAgent
         def handle_request(args = {})
           context = build(args)
           context.permissions.can?(:read, context.collection)
-          withholding_scope = assert_record_in_scope(context, context.collection, args[:params]['id'])
+          # Before the audit database is touched, for the 404 it raises — and for what it saw of the record
+          # while the rows below were still being chosen, which the read after them can no longer see.
+          gone_at_check = assert_record_in_scope(context, context.collection, args[:params]['id'])
 
-          skip, limit = parse_pagination(args)
           filters = {
             collection: context.collection.name,
             # args[:params]['id'] is already Forest's packed id, the form the audit store keys on — plus any id
@@ -50,18 +52,15 @@ module ForestAdminAgent
             **parse_filters(args)
           }
 
-          history = store.list_by_record(**filters, skip: skip, limit: limit, order: parse_sort(args))
-          # `count` reflects the active filters (not the absolute total) and is independent of the page.
-          count = store.count_by_record(**filters)
-
-          # Asked again now: the check above ran before these rows were read, so a record deleted in between
-          # answered "present and in scope" for rows that already carry its delete.
-          withholding_scope ||= scope_if_gone_since(context, args)
-          data = withhold_out_of_scope_values(history, withholding_scope, context)
+          data, count, authors = if gone_at_check && filters.slice(:search, :fields).any?
+                                   history_matched_after_withholding(context, args, filters, gone_at_check)
+                                 else
+                                   history_matched_in_store(context, args, filters, gone_at_check)
+                                 end
 
           {
             name: args[:params]['collection_name'],
-            content: { data: data.map { |record| serialize_record(record) }, meta: meta(args, filters, count) }
+            content: { data: data.map { |record| serialize_record(record) }, meta: meta(args, count, authors) }
           }
         end
 
@@ -75,6 +74,9 @@ module ForestAdminAgent
           current = scoped_record(
             context, context.collection, args[:params]['id'], audited_projection(context.collection)
           )
+          # Nothing left to evaluate a scope against, so the reconstruction is tested in its own right below
+          # — whatever the read after the rows finds under this id by then.
+          gone_at_check = current.nil? ? context.permissions.get_scope(context.collection) : nil
 
           timestamp = parse_state_timestamp(args)
           entries = store.list_since(
@@ -84,136 +86,151 @@ module ForestAdminAgent
           )
           # Fully qualified: inside this class, `AuditTrail` is the route itself.
           state = ::ForestAdminAgent::AuditTrail::RecordState.at(current, entries)
-
-          { name: args[:params]['collection_name'], content: { data: state } }
+          # Asked again now, for the same reason the history route asks, and the same way: the record read
+          # above can be deleted — or an id that was gone then be taken by somebody else's record — while the
+          # audit read is in flight.
+          withholding_scope = withholding_scope_for(context, context.collection, args[:params]['id'],
+                                                    gone_at_check)
+
+          { name: args[:params]['collection_name'],
+            content: { data: answerable_state(state, withholding_scope, context, args[:params]['id']) } }
         end
 
         private
 
-        # Second read of the record, once the rows are in hand, so the answer that decides the withholding is
-        # never older than what it decides on. Skipped without a scope in effect — there is nothing to withhold
-        # then, and nothing to ask. A record moved out of scope rather than deleted raises the 404 it would
-        # raise for a request starting a moment later.
-        def scope_if_gone_since(context, args)
-          return nil if context.permissions.get_scope(context.collection).nil?
+        # The history route withholds a gone record's captured values from a caller whose scope they fail, and
+        # this route is nothing but those values reassembled: without the same test they come back one request
+        # away. A reconstruction the scope cannot answer withholds too — absent is not the same as passing.
+        def answerable_state(state, scope, context, packed_id)
+          return state if state.nil? || scope.nil?
 
-          assert_record_in_scope(context, context.collection, args[:params]['id'])
+          withholding = Withholding.new(context.collection, scope, context.caller.timezone)
+          # The reconstruction can sit on the far side of a primary-key move this route cannot see, so the
+          # requested id does not answer for a key the trail redacted — only for one never captured at all.
+          in_scope?(state, packed_id, withholding, id_answers_for_keys: false) ? state : nil
         end
 
-        # A record that is gone for good bypasses the scope check — there is nothing left to check it against
-        # — but its rows still carry the column values captured while it existed. When those values would
-        # themselves have failed the caller's scope, withhold them; the row itself stays visible either way,
-        # so that it happened, by whom and when still reads.
-        def withhold_out_of_scope_values(entries, scope, context)
-          return entries if scope.nil?
+        # `availableUsers` rides along on the first fetch only — the front keeps the list it saw — and lists the
+        # distinct authors of the entries the current filters match, whatever page was asked for. The identity
+        # comes from the rows, so someone since renamed or removed still reads as they were when they acted.
+        def meta(args, count, authors)
+          return { count: count } unless first_fetch?(args)
 
-          entries.map { |entry| withhold(entry, scope, context) }
+          { count: count, availableUsers: authors.call.map { |author| available_user(author) } }
         end
 
-        def withhold(entry, scope, context)
-          case entry.operation
-          # `delete`'s previous_values and `create`'s new_values both capture every writable column.
-          when 'delete'
-            in_scope?(entry.previous_values, entry.record_id, scope, context) ? entry : blank(entry, :previous_values)
-          when 'create'
-            in_scope?(entry.new_values, entry.record_id, scope, context) ? entry : blank(entry, :new_values)
-          when 'update'
-            withhold_each_side(entry, scope, context)
-          # `action`/`action_failed` rows hold a submitted form and a result summary, not column values, so
-          # the scope doesn't apply to them.
-          else
-            entry
-          end
-        end
+        def first_fetch?(args)
+          page = args.dig(:params, 'page')
 
-        # An update's two sides are a partial diff, so each is tested against its own values: a diff that never
-        # carried the scoped column answers for neither and is withheld by `in_scope?` anyway. Gating the sides
-        # separately releases the ones that can be proven in scope — "it used to be X" can't escape through a
-        # row whose new value is out of scope, since that side is tested on its own.
-        def withhold_each_side(entry, scope, context)
-          # An update that moved a writable primary key files its row under the id the record ended up with,
-          # and keeps the one it had on `previous_record_id`. Each side is tested against the id it was true
-          # of, or the new state's id would decide whether the old state is in scope.
-          before = entry.previous_record_id || entry.record_id
-          kept = in_scope?(entry.previous_values, before, scope, context) ? entry : blank(entry, :previous_values)
-
-          in_scope?(kept.new_values, after_id(kept), scope, context) ? kept : blank(kept, :new_values)
+          (page.is_a?(Hash) ? page['number'].to_i : 0) <= 1
         end
 
-        # A pending row is filed under the id the record had *before* the write, since the write may not have
-        # landed: it says nothing about the state the update was moving to, so the new side gets no id to fill
-        # from and falls back on what it captured itself.
-        def after_id(entry)
-          entry.status == ::ForestAdminAgent::AuditTrail::Recording::PENDING ? nil : entry.record_id
+        def available_user(author)
+          { id: author[:user_id], firstName: author[:user_first_name],
+            lastName: author[:user_last_name], email: author[:user_email] }
         end
 
-        # Only a snapshot that answers every field the scope asks about, with what was really stored, is worth
-        # matching. The capture keeps the writable columns, so a scope on anything else — a read-only column, a
-        # relation — reads as nil there and would answer for a value the row never held: `status != 'private'`
-        # would match, and an ordered operator would raise on the nil. A redacted value answers no better.
-        def in_scope?(values, packed_id, scope, context)
-          snapshot = answerable_snapshot(values, packed_id, context.collection)
-          return false unless scope.projection.all? { |field| snapshot.key?(field) }
-
-          scope.match(snapshot, context.collection, context.caller.timezone)
-        rescue StandardError => e
-          # Key presence is not answerability: a column captured as nil has its key, and an ordered operator
-          # raises on it. Uncaught that would fail the whole page, and only for the callers a scope applies
-          # to. One withheld row is the smaller loss, and the same answer the field would have got had it
-          # been missing outright.
-          Facades::Container.logger&.log('Warn', "[ForestAdmin] Audit row not scope-checkable: #{e.message}")
-
-          false
+        def history_matched_in_store(context, args, filters, gone_at_check)
+          skip, limit = parse_pagination(args)
+          history = store.list_by_record(**filters, skip: skip, limit: limit, order: parse_sort(args))
+          # `count` reflects the active filters (not the absolute total) and is independent of the page.
+          count = store.count_by_record(**filters)
+          withholding = withholding_for(context, args, gone_at_check)
+
+          # Gone between the check and the read: this answer withholds, so the rows, count and authors matched
+          # in SQL above must not decide what is served either.
+          if withholding.scope && filters.slice(:search, :fields).any?
+            return history_matched_after_withholding(context, args, filters, gone_at_check, withholding)
+          end
+
+          [withhold_out_of_scope_values(history, withholding), count, -> { store.authors_by_record(**filters) }]
         end
 
-        # What this side of the row can answer about. A redacted value answers nothing, so it is dropped rather
-        # than matched against the placeholder — leaving the field unanswered, which withholds. The packed id
-        # then fills in the primary keys: a read-only one never lands in the snapshot at all, and a writable one
-        # the trail redacts was just dropped, while the id the row was filed under proves what the key was.
-        # It only fills what the snapshot cannot answer: on the side of a row that captured the key itself,
-        # that value is the one that was true there.
-        def answerable_snapshot(values, packed_id, collection)
-          answered = (values || {}).reject { |_, value| value == ::ForestAdminAgent::AuditTrail::Recording::REDACTED }
-          return answered if packed_id.nil?
+        # Matched in SQL, `search` and `fields` would test the values as captured, so which rows come back, the
+        # count and the authors would still say what the withholding hides — one probe per character. They are
+        # matched against what is served instead, which means scanning the whole history here, in batches, so
+        # only the page asked for is kept. Only for a gone record: one in scope was the caller's to read whole.
+        def history_matched_after_withholding(context, args, filters, gone_at_check, withholding = nil)
+          skip, limit = parse_pagination(args)
+          value_filters = filters.slice(:search, :fields)
+          page = []
+          count = 0
+          latest = {}
 
-          begin
-            Utils::Id.unpack_id(collection, packed_id, with_key: true).merge(answered)
-          rescue StandardError => e
-            # An id written under a primary key of another shape costs the keys it would have filled, and
-            # nothing else: the snapshot still answers for the columns it captured, so a scope that never
-            # asks about the id is unaffected.
-            Facades::Container.logger&.log('Warn', "[ForestAdmin] Audit row id not decodable: #{e.message}")
-
-            answered
+          each_withheld_batch(context, args, filters.except(*value_filters.keys), gone_at_check,
+                              withholding) do |entry|
+            next unless matches_value_filters?(entry, **value_filters)
+
+            page << entry if count >= skip && page.size < limit
+            count += 1
+            next if entry.user_id.nil?
+
+            latest[entry.user_id] = entry if newer?(entry, latest[entry.user_id])
           end
+
+          [page, count, -> { latest.values.map { |entry| author_of(entry) } }]
         end
 
-        def blank(entry, *fields)
-          entry.dup.tap { |copy| fields.each { |field| copy[field] = {} } }
+        # Each batch continues past the last row read rather than at an offset, which entries written between
+        # batches would shift. Bounded at the instant the scan starts too, so an id taken by another record
+        # since cannot keep an oldest-first scan chasing its new rows.
+        def each_withheld_batch(context, args, filters, gone_at_check, withholding, &block)
+          order = parse_sort(args)
+          snapshot = filters.merge(end_timestamp: [filters[:end_timestamp], Time.now.utc.iso8601(3)].compact.min)
+          cursor = nil
+
+          loop do
+            rows = store.list_by_record(**snapshot, limit: SCAN_BATCH_SIZE, order: order, after: cursor)
+            withholding ||= withholding_for(context, args, gone_at_check)
+            withhold_out_of_scope_values(rows, withholding).each(&block)
+            break if rows.size < SCAN_BATCH_SIZE
+
+            cursor = rows.last.to_h.slice(:timestamp, :id)
+          end
         end
 
-        # `availableUsers` rides along on the first fetch only — the front keeps the list it saw — and lists the
-        # distinct authors of the entries the current filters match, whatever page was asked for. The identity
-        # comes from the rows, so someone since renamed or removed still reads as they were when they acted.
-        def meta(args, filters, count)
-          return { count: count } unless first_fetch?(args)
+        # Asked again now, because the check above ran before these rows were read: a record deleted in
+        # between answered "present and in scope" for rows that already carry its delete.
+        def withholding_for(context, args, gone_at_check)
+          withholding_scope = withholding_scope_for(context, context.collection, args[:params]['id'],
+                                                    gone_at_check)
 
-          { count: count, availableUsers: available_users(filters) }
+          Withholding.new(context.collection, withholding_scope, context.caller.timezone)
         end
 
-        def first_fetch?(args)
-          page = args.dig(:params, 'page')
+        def matches_value_filters?(entry, search: nil, fields: nil)
+          (search.nil? || search_matches?(entry, search)) && (fields.nil? || touches_field?(entry, fields))
+        end
 
-          (page.is_a?(Hash) ? page['number'].to_i : 0) <= 1
+        # `Sql::TextSearch`'s test, on the served values: the term JSON-escaped the way the values serialize,
+        # and a redacted mask removed before matching.
+        def search_matches?(entry, term)
+          text = term.downcase
+          escaped = text.to_json[1..-2]
+
+          ::ForestAdminAgent::AuditTrail::Sql::TextSearch::TEXT_COLUMNS.any? do |column|
+            entry[column].to_s.downcase.include?(text)
+          end || [entry.previous_values, entry.new_values].compact.any? do |values|
+            values.to_json.gsub(::ForestAdminAgent::AuditTrail::Recording::REDACTED, '').downcase.include?(escaped)
+          end
         end
 
-        def available_users(filters)
-          store.authors_by_record(**filters).map do |author|
-            { id: author[:user_id], firstName: author[:user_first_name],
-              lastName: author[:user_last_name], email: author[:user_email] }
+        def touches_field?(entry, fields)
+          [entry.previous_values, entry.new_values].compact.any? do |values|
+            fields.any? { |field| values.key?(field) }
           end
         end
 
+        # An author reads as their latest identity. Not simply the first or last row read: rows sharing a
+        # timestamp come id-ascending whichever way the history is sorted.
+        def newer?(entry, kept)
+          kept.nil? || ([entry.timestamp, entry.id] <=> [kept.timestamp, kept.id]).positive?
+        end
+
+        def author_of(entry)
+          ::ForestAdminAgent::AuditTrail::Store::AUTHOR_COLUMNS.to_h { |column| [column, entry[column]] }
+        end
+
         # An ISO-8601 instant, or the same wall-clock forms the history filters accept, read in the request
         # timezone.
         def parse_state_timestamp(args)

data/lib/forest_admin_agent/routes/resources/audit_trail_correlation.rb

@@ -35,19 +35,20 @@ module ForestAdminAgent
         end
 
         def handle_history(args = {})
-          collection, record_id = assert_scope(args)
+          context, collection, record_id, gone_at_check = assert_scope(args)
 
           history = store.list_by_correlation(
             collection: collection.name,
             record_id: record_id,
             correlation_key: args[:params]['correlation_key']
           )
+          data = withhold_gone_record(history, context, collection, record_id, gone_at_check)
 
-          { name: collection.name, content: { data: history.map { |record| serialize_record(record) } } }
+          { name: collection.name, content: { data: data.map { |record| serialize_record(record) } } }
         end
 
         def handle_batch(args = {})
-          collection, record_id = assert_scope(args)
+          context, collection, record_id, gone_at_check = assert_scope(args)
           correlation_keys = parse_correlation_keys(args)
 
           history = if correlation_keys.empty?
@@ -59,12 +60,25 @@ module ForestAdminAgent
                         correlation_keys: correlation_keys
                       )
                     end
+          data = withhold_gone_record(history, context, collection, record_id, gone_at_check)
 
-          { name: collection.name, content: { data: history.map { |record| serialize_record(record) } } }
+          { name: collection.name, content: { data: data.map { |record| serialize_record(record) } } }
         end
 
         private
 
+        # These routes serve the same rows as the per-record history route, so a gone record's captured values
+        # are tested against the caller's scope here too — otherwise what that route withholds comes back
+        # through a correlation lookup.
+        def withhold_gone_record(history, context, collection, record_id, gone_at_check)
+          # An empty answer has nothing to withhold, so it does not earn a second read of the record.
+          return history if history.empty?
+
+          scope = withholding_scope_for(context, collection, record_id, gone_at_check)
+
+          withhold_out_of_scope_values(history, Withholding.new(collection, scope, context.caller.timezone))
+        end
+
         def assert_scope(args)
           context = build(args)
           name = args.dig(:params, 'collection').to_s
@@ -75,9 +89,9 @@ module ForestAdminAgent
 
           collection = get_collection(context, name)
           context.permissions.can?(:read, collection)
-          assert_record_in_scope(context, collection, record_id)
-
-          [collection, record_id]
+          # Before the audit database is touched, for the 404 it raises — and for what it saw of the record
+          # while the rows were still being chosen, which the read after them can no longer see.
+          [context, collection, record_id, assert_record_in_scope(context, collection, record_id)]
         end
 
         def get_collection(context, name)

data/lib/forest_admin_agent/routes/resources/audit_trail_route.rb

@@ -8,6 +8,7 @@ module ForestAdminAgent
       # and they all serialize audit records the same way.
       module AuditTrailRoute
         include ForestAdminDatasourceToolkit::Components::Query
+        include AuditTrailWithholding
 
         # The caller's scope when the record is gone for good, nil otherwise — no scope in effect, or a record
         # still there and in scope. A scope can't be evaluated against a record that no longer exists, so its
@@ -43,6 +44,29 @@ module ForestAdminAgent
           raise Http::Exceptions::NotFoundError, 'Record does not exists'
         end
 
+        # The scope to withhold with: the caller's own, if the record was gone at either look at it.
+        #
+        # Both looks count, and neither can clear the other. The first is the only one that saw the record as
+        # it was when the rows were chosen, so a replacement the second finds — an id freed by a delete and
+        # taken by another record since — answers for itself, never for the life whose rows these are. The
+        # second is the only one that can see a record deleted since, and it is where the 404 comes from when
+        # that replacement is one this caller cannot read.
+        def withholding_scope_for(context, collection, packed_id, gone_at_check)
+          gone_since = scope_if_gone_since(context, collection, packed_id)
+
+          gone_at_check || gone_since
+        end
+
+        # Second read of the record, once the rows are in hand, so the answer that decides the withholding is
+        # never older than what it decides on. Skipped without a scope in effect — there is nothing to withhold
+        # then, and nothing to ask. A record moved out of scope rather than deleted raises the 404 it would
+        # raise for a request starting a moment later.
+        def scope_if_gone_since(context, collection, packed_id)
+          return nil if context.permissions.get_scope(collection).nil?
+
+          assert_record_in_scope(context, collection, packed_id)
+        end
+
         # Camelize only the top-level keys — the row `id` included, which the front uses as the tiebreaker when
         # merging pages ordered by (timestamp, id). Value hashes keep the keys they were stored with: a record's
         # own column names, or an action answer's camelCase Forest names.

data/lib/forest_admin_agent/routes/resources/audit_trail_withholding.rb

@@ -0,0 +1,142 @@
+module ForestAdminAgent
+  module Routes
+    module Resources
+      # Blanks the captured values a caller's record-level scope does not cover, keeping the rows themselves:
+      # that something happened, by whom and when, stays visible either way. Shared by every route that serves
+      # captured values — the per-record history, the state reconstruction and the two correlation routes —
+      # since a rule only one of them applies is one lookup away from being no rule at all.
+      module AuditTrailWithholding
+        # Everything a withholding decision needs, and nothing a request context would drag along: the
+        # collection the rows belong to, the scope to test them against, and the timezone its date
+        # comparisons read in.
+        Withholding = Struct.new(:collection, :scope, :timezone)
+
+        NULL_EXCLUDING_OPERATORS = [
+          ForestAdminDatasourceToolkit::Components::Query::ConditionTree::Operators::NOT_EQUAL,
+          ForestAdminDatasourceToolkit::Components::Query::ConditionTree::Operators::NOT_IN,
+          ForestAdminDatasourceToolkit::Components::Query::ConditionTree::Operators::NOT_CONTAINS
+        ].freeze
+
+        # A record that is gone for good bypasses the scope check — there is nothing left to check it against
+        # — but its rows still carry the column values captured while it existed. When those values would
+        # themselves have failed the caller's scope, withhold them; the row itself stays visible either way,
+        # so that it happened, by whom and when still reads.
+        def withhold_out_of_scope_values(entries, withholding)
+          return entries if withholding.scope.nil?
+
+          entries.map { |entry| withhold(entry, withholding) }
+        end
+
+        def withhold(entry, withholding)
+          case entry.operation
+          # `delete`'s previous_values and `create`'s new_values both capture every writable column.
+          when 'delete'
+            in_scope?(entry.previous_values, entry.record_id, withholding) ? entry : blank(entry, :previous_values)
+          when 'create'
+            in_scope?(entry.new_values, entry.record_id, withholding) ? entry : blank(entry, :new_values)
+          when 'update'
+            withhold_each_side(entry, withholding)
+          # `action`/`action_failed` rows hold a submitted form and a result summary, not column values, so
+          # the scope doesn't apply to them.
+          else
+            entry
+          end
+        end
+
+        # An update's two sides are a partial diff, so each is tested against its own values: a diff that never
+        # carried the scoped column answers for neither and is withheld by `in_scope?` anyway. Gating the sides
+        # separately releases the ones that can be proven in scope — "it used to be X" can't escape through a
+        # row whose new value is out of scope, since that side is tested on its own.
+        def withhold_each_side(entry, withholding)
+          # An update that moved a writable primary key files its row under the id the record ended up with,
+          # and keeps the one it had on `previous_record_id`. Each side is tested against the id it was true
+          # of, or the new state's id would decide whether the old state is in scope.
+          before = entry.previous_record_id || entry.record_id
+          kept = in_scope?(entry.previous_values, before, withholding) ? entry : blank(entry, :previous_values)
+
+          in_scope?(kept.new_values, after_id(kept), withholding) ? kept : blank(kept, :new_values)
+        end
+
+        # A pending row is filed under the id the record had *before* the write, since the write may not have
+        # landed: it says nothing about the state the update was moving to, so the new side gets no id to fill
+        # from and falls back on what it captured itself.
+        def after_id(entry)
+          entry.status == ::ForestAdminAgent::AuditTrail::Recording::PENDING ? nil : entry.record_id
+        end
+
+        # Only a snapshot that answers every field the scope asks about, with what was really stored, is worth
+        # matching. The capture keeps the writable columns, so a scope on anything else — a read-only column, a
+        # relation — reads as nil there and would answer for a value the row never held: `status != 'private'`
+        # would match, and an ordered operator would raise on the nil. A redacted value answers no better.
+        def in_scope?(values, packed_id, withholding, id_answers_for_keys: true)
+          snapshot = answerable_snapshot(values, packed_id, withholding.collection,
+                                         id_answers_for_keys: id_answers_for_keys)
+          return false unless withholding.scope.projection.all? { |field| snapshot.key?(field) }
+
+          matches_as_stored?(withholding.scope, snapshot, withholding)
+        rescue StandardError => e
+          # Key presence is not answerability: a column captured as nil has its key, and an ordered operator
+          # raises on it. Uncaught that would fail the whole page, and only for the callers a scope applies
+          # to. One withheld row is the smaller loss, and the same answer the field would have got had it
+          # been missing outright.
+          Facades::Container.logger&.log('Warn', "[ForestAdmin] Audit row not scope-checkable: #{e.message}")
+
+          false
+        end
+
+        # `scope.match`, except that a negated comparison never matches a NULL. In memory `status != 'private'`
+        # holds for a nil status, while the database leaves a NULL out of it: a record the caller could never
+        # read alive would become readable once deleted. Stricter than a datasource that matches NULL there
+        # (Mongo's `$ne`), never looser.
+        def matches_as_stored?(tree, snapshot, withholding)
+          if tree.is_a?(ForestAdminDatasourceToolkit::Components::Query::ConditionTree::Nodes::ConditionTreeBranch)
+            evaluate = ->(condition) { matches_as_stored?(condition, snapshot, withholding) }
+
+            return tree.aggregator == 'And' ? tree.conditions.all?(&evaluate) : tree.conditions.any?(&evaluate)
+          end
+          return false if snapshot[tree.field].nil? && NULL_EXCLUDING_OPERATORS.include?(tree.operator)
+
+          tree.match(snapshot, withholding.collection, withholding.timezone)
+        end
+
+        # What this side of the row can answer about. A redacted value answers nothing, so it is dropped rather
+        # than matched against the placeholder — leaving the field unanswered, which withholds. The packed id
+        # then fills in the primary keys: a read-only one never lands in the snapshot at all, and a writable one
+        # the trail redacts was just dropped, while the id the row was filed under proves what the key was.
+        # It only fills what the snapshot cannot answer: on the side of a row that captured the key itself,
+        # that value is the one that was true there.
+        #
+        # `id_answers_for_keys: false` keeps the id off the keys the trail redacted, for a caller holding an id
+        # that may not be the one its values were true under — a state reconstruction, which can sit on the far
+        # side of a primary-key move it cannot see. A key never captured at all is still filled: read-only, so
+        # it cannot have moved.
+        def answerable_snapshot(values, packed_id, collection, id_answers_for_keys: true)
+          captured = values || {}
+          redacted = captured.select { |_, value| value == ::ForestAdminAgent::AuditTrail::Recording::REDACTED }.keys
+          answered = captured.except(*redacted)
+          return answered if packed_id.nil?
+
+          decoded = decoded_keys(collection, packed_id)
+          decoded = decoded.except(*redacted) unless id_answers_for_keys
+
+          decoded.merge(answered)
+        end
+
+        def decoded_keys(collection, packed_id)
+          Utils::Id.unpack_id(collection, packed_id, with_key: true)
+        rescue StandardError => e
+          # An id written under a primary key of another shape costs the keys it would have filled, and
+          # nothing else: the snapshot still answers for the columns it captured, so a scope that never
+          # asks about the id is unaffected.
+          Facades::Container.logger&.log('Warn', "[ForestAdmin] Audit row id not decodable: #{e.message}")
+
+          {}
+        end
+
+        def blank(entry, *fields)
+          entry.dup.tap { |copy| fields.each { |field| copy[field] = {} } }
+        end
+      end
+    end
+  end
+end

data/lib/forest_admin_agent/utils/schema/generator_field.rb

@@ -68,6 +68,12 @@ module ForestAdminAgent
             }
           end
 
+          def relation_filterable?(relation, foreign_collection)
+            return false if relation.respond_to?(:is_filterable) && relation.is_filterable == false
+
+            foreign_collection_filterable?(foreign_collection)
+          end
+
           def foreign_collection_filterable?(foreign_collection)
             foreign_collection.schema[:fields].values.any? do |field|
               field.type == 'Column' && FrontendFilterable.filterable?(field.filter_operators)
@@ -124,7 +130,7 @@ module ForestAdminAgent
               {
                 type: key_field.column_type,
                 defaultValue: nil,
-                isFilterable: foreign_collection_filterable?(foreign_collection),
+                isFilterable: relation_filterable?(relation, foreign_collection),
                 isPrimaryKey: false,
                 isRequired: false,
                 isReadOnly: relation.is_read_only || key_field.is_read_only,
@@ -142,7 +148,7 @@ module ForestAdminAgent
               {
                 type: key_field.column_type,
                 defaultValue: key_field.default_value,
-                isFilterable: foreign_collection_filterable?(foreign_collection),
+                isFilterable: relation_filterable?(relation, foreign_collection),
                 isPrimaryKey: false,
                 isRequired: key_field.validation.any? { |v| v[:operator] == 'Present' },
                 isReadOnly: key_field.is_read_only,

data/lib/forest_admin_agent/utils/schema/schema_emitter.rb

@@ -6,7 +6,7 @@ module ForestAdminAgent
     module Schema
       class SchemaEmitter
         LIANA_NAME = "agent-ruby"
-        LIANA_VERSION = "1.44.4"
+        LIANA_VERSION = "1.45.0"
 
         def self.generate(datasource)
           datasource.collections

data/lib/forest_admin_agent/version.rb

@@ -1,3 +1,3 @@
 module ForestAdminAgent
-  VERSION = "1.44.4"
+  VERSION = "1.45.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: forest_admin_agent
 version: !ruby/object:Gem::Version
-  version: 1.44.4
+  version: 1.45.0
 platform: ruby
 authors:
 - Matthieu
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2026-10-01 00:00:00.000000000 Z
+date: 2026-10-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -378,6 +378,7 @@ files:
 - lib/forest_admin_agent/routes/resources/audit_trail.rb
 - lib/forest_admin_agent/routes/resources/audit_trail_correlation.rb
 - lib/forest_admin_agent/routes/resources/audit_trail_route.rb
+- lib/forest_admin_agent/routes/resources/audit_trail_withholding.rb
 - lib/forest_admin_agent/routes/resources/count.rb
 - lib/forest_admin_agent/routes/resources/csv.rb
 - lib/forest_admin_agent/routes/resources/delete.rb