beskar 0.0.2 → 0.2.0

data/docs/audits/repair-status.md

@@ -0,0 +1,216 @@
+# Review remediation
+
+Tracks [Original project review](project-review.md), which remains the historical baseline.
+Ten implementation batches are complete; this is not a claim that all findings are fixed.
+The runtime is the project's mise default Ruby **4.0.6**.
+
+Batch ten addresses the renewed audit findings 2–6. See
+[Security hardening and rollout](../operations/security-hardening.md) for the entry-point coverage matrix,
+revocation semantics, separate administrative permissions, required export/config
+history, changed availability defaults, and the host/database validation gates
+that remain open. Account deletion still retains events unchanged.
+
+## Implemented and locally verified
+
+| Finding | Change and evidence |
+| --- | --- |
+| F01 | Devise database-password/HTTP Basic admission now reserves and enforces IP/account and opt-in global limits before password verification. Failed known targets count against their account; unknown identities use normalized hashed keys. Native controllers have explicit admission/session guards. Outcome callbacks do not double-count, and credential-free page visits create no attempts. Standard custom Warden strategies now have IP pre-verification admission and account binding after identity resolution; set_user is guarded even when callbacks are disabled. API/Cable adapters and remaining host integration boundaries are documented in docs/operations/security-hardening.md. |
+| F02 | Warden sign-out uses the current request's actual lock result, not recent audit rows, and affects only the current scope. Confirmed locks now always reject admission, rotate a durable account generation, and invalidate all prior Devise sessions/remember cookies. Unlock never restores them; unrelated accounts remain signed in. |
+| F03 | Beskar automatic account locks, sign-outs, and emergency resets honor monitor/whitelist policy. Whitelisted authentication observations cannot consume enforced global capacity. Risk-enabled audits now record would-lock eligibility and adapter availability without claiming an actual lock. Host Devise/Rails policies remain independent. |
+| F04 | Implemented persistent native locks, expiry/manual unlock, all-session revocation, and a transactional session-creation guard. Existing-session readers honor lock state. Separate-connection races and failed/aborted destruction callbacks are covered; failed cleanup requires successful cleanup before manual unlock. Host integration and same-writer-pool requirements are explicit in docs/guides/authentication.md. |
+| F05 | Geographic history pairs normalized JSON locations with their own timestamps/IDs, uses actual elapsed seconds and explicit chronology, and rejects unusable coordinates/times. Real Devise/native requests exercise persisted travel evidence through actual locks. Mock locations never create travel/country risk. Bounded history and geolocation limitations are explicit in docs/guides/risk-scoring.md; production accuracy remains unverified. |
+| F06 | Middleware, authentication audit, rate limiting, and WAF attribution use the Rails-resolved `remote_ip`. Trusted-proxy regression coverage added. Hosts must configure trusted proxies correctly. |
+| F07 | Ban enforcement reads the database, never cached booleans. Regressions cover rollback, IP edits, expiry edits, stale positives, and stale negatives. |
+| F08 | Permanent flag is authoritative; permanent bans are excluded from expiry cleanup and normalize expiry to nil on save. Temporary bans require expiry. Legacy-data review remains an upgrade prerequisite. |
+| F09 | Database-backed state, unique keys, transactional updates, optimistic conflict detection, and bounded retries replace cache read/modify/write. Separate-connection SQLite regressions exercise counters, admissions, WAF history, and repeated bans. Other database concurrency/load runs remain unverified. |
+| F13 | Minitest 5 pinned for Rails 8.0; lint dependencies updated for Ruby 4. CI now includes Ruby 4.0.6 and 3.4; duplicate automatic workflow made manual. Local verification below; hosted CI has not been run here. |
+| F14 | Removed the disconnected `ip_auth_failures` cache representation. Request-wide IP-quota blocking and its fixed-window denial/auto-ban counter are now opt-in; unrelated shared-NAT traffic remains accessible by default. |
+| F15 | Read-only previews, configured counting windows, bounded attempt storage, enforced backoff deadlines, longest retry selection, global backoff separation, and explicit global/IP-denial reset coverage. Global capacity is now opt-in, removing the default shared lock hot spot and distributed-denial budget. |
+| F16 | Added bounded path canonicalization, root/segment-aware signatures, exact-format query matching, and method/path/category exclusions. Ordinary Rails exceptions no longer score without independent scanner evidence by default; broad scoring requires opt-in. Middleware records at most one charge for path plus exception. Benign/attack corpora cover boundary false positives, encoded traversal, query poisoning, malformed input, and exclusions. Production route coverage and false-positive rates remain unverified; this remains a cumulative scanner heuristic, not first-request exploit prevention. |
+| F19 | Monitor WAF/rate observations use separate state and never create automatic active IP bans. WAF audit `would_be_blocked` respects whitelist and auto-block policy. Automatic account actions now honor the same monitor/whitelist policy. |
+
+## Partially addressed
+
+| Finding | Completed so far | Still open |
+| --- | --- | --- |
+| F10 | Dashboard callbacks that render/redirect are not rendered twice; missing-config logging no longer interpolates secrets; credential recipes reject blank/missing values and use secure comparison; recursive installer example removed. | Broader host-adapter authorization coverage and review of custom callbacks. |
+| F11 | Authentication context drops session IDs/forwarded headers and cleans referrers. WAF sinks retain bounded rule evidence, not raw URLs/headers/exception messages. Audit models bound/filter metadata and text using built-in plus host parameter filters, including on legacy model reads. Rescued-exception logs use classes rather than messages. CSV exporters quote cells and visibly prefix formula-like text; JSON projects known fields. Associated Devise/native email presentation now shares filtering across views and exports. Export authorization and no-store behavior are tested. | Arbitrary secrets in allowed free text/paths, historical rows/logs/backups, raw SQL/bulk API bypasses, retention/access policy, and actual spreadsheet-client import/save/reopen QA. Stored-value search can reveal legacy record membership despite read-time redaction. Audit filtering does not anonymize enforcement fields such as ban IPs. |
+| F12 | Logger fallback no longer recurses. Cache availability is not required for authoritative state. Required authentication-state, lock-persistence, and enforced risk-assessment failures reject with 503; optional authentication and WAF audit writes are isolated with savepoints. A failed optional WAF audit does not prevent state updates or bans. Optional host analysis queues only after enclosing commits; rollback cancels it and queue failures do not undo admission/committed work or log raw exception messages. | Unified dependency-failure policy for the other middleware/WAF paths and database-specific fault tests. The optional analysis hook has no outbox/durable-delivery guarantee. |
+| F17 | One assessment provides explicit factors whose points/caps sum to the score and flow into lock evidence. Corrected bot/reason flags, browser captures, and midnight scoring. Removed implicit IP/unlock trust discounts and geographic bypasses; only explicitly admitted, unlocked-success history supplies travel observations. The geographic pattern helper is implemented. Observation history does not raise enforced risk. | Production calibration/false-positive evaluation and any future verified-device/recovery trust mechanism. User-Agent and IP geography remain spoofable/approximate heuristics, not identity proof. |
+| F18 | Native unlock deadlines/manual locks work; Devise's own unlock policy is explicitly separate. Emergency thresholds count confirmed locks with true flags, not duplicated successes or false-flag strings. Password invalidation, session revocation, and mandatory recovery audit are transactional. Opt-in Action Mailer lock/reset/team notices now use independent post-commit jobs with five-attempt retries, validated sender/recipients/recovery page, and isolated failures. Native recovery is exercised through the host reset email/token redemption and authorized manual unlock; resetting the native password alone cannot bypass the lock. A real Devise reset-email/token handoff now verifies its own email-unlock policy, replay rejection and continued revocation of old sessions. | Real SMTP/production worker and broader host recovery verification; durable outbox/reconciliation, duplicate/bounce handling, delivery-status history, and broader host recovery/identity-verification policies. |
+| F20 | Correct migration source/destination paths, destination-aware mount detection, migration numbering, idempotence with engine-copied migrations, monitor-first initializer, truthful immediate installation instructions. Copied migrations run successfully against a fresh SQLite database. | Full fresh host-app boot/authentication exercise and remaining older documentation claims. |
+| F21 | README and gem metadata describe scanner-path and User-Agent heuristics without promising SQLi/XSS filtering, JavaScript challenges, or honeypots. Removed versioned API routes that pointed to missing controllers; authenticated resource exports remain. Devise's invalid callback-registration path was removed. Automatic background analysis now defaults off and requires an explicit host Active Job with a documented post-commit contract. Both authentication paths invoke it only after admission. Removed custom-lock/IP2Location no-op placeholders; unsupported settings fail explicitly. Notification defaults now opt out and their enabled paths perform real Action Mailer delivery, not log-only intent. Current operational-contract documents ship in the gem. | Remaining historical capability/configuration claims. No built-in analyzer, production worker validation, custom lock adapter API, IP2Location integration, or standalone recovery/token endpoint is supplied. |
+| F22 | Added a composite user/event/time index; risk-history projections are ordered/bounded and one device/geographic snapshot supplies scoring/evidence. Exports are capped at 1,000 rows with cursors. Centralized PostgreSQL/SQLite/MySQL JSON search expressions, exact legacy email fallback, bound text values, and literal LIKE escaping. Overview counts reuse grouped queries; related-event tables preload users. Query-count regressions pass. | Live PostgreSQL/MySQL execution, adapter/collation/query-plan/load testing, further query reduction, and retention. JSON-text substring search remains expensive; paging and grouped counts are not transactional snapshots. |
+| F23 | Account deletion retains linked security events unchanged, including original polymorphic IDs, as explicitly chosen. Raw-row comparisons and dashboard/export regressions cover both adapters. Dashboard ban changes require a trusted actor and reason, with bounded before/after snapshots, server operation UUID, and request correlation in a separate journal that survives target/actor deletion. Changes and history share a transaction and ban coordination; bulk selections are bounded, deduplicated, and all-or-nothing. Required-history failures and aborted destruction never report success. SecurityEvent and administrative history reject normal instance CRUD rewrites. Exports require separate permission, actor/reason and a mandatory preparation journal; runtime configuration has its own grant and before/after journal; manual extensions no longer fabricate violations. | No automatic retention period/purge, historical backfill, universal auditing of direct model/automatic changes, failed-attempt journal, database tamper resistance, or production adapter/load verification. Low-level SQL/bulk/counter APIs and privileged host code can bypass model protections. Boot/deployment changes need host source-control/deployment auditing. |
+| F24 | Whitelist parsing follows replaced/mutated configuration; default getters and Devise scope mappings are corrected. Geolocation caches are isolated by provider/database identity, and readers follow database generations. Startup/configure validation checks section schemas, booleans, numeric bounds, whitelist/exclusions, supported providers/strategies, readable MaxMind paths, and enabled host jobs. Partial section assignments merge library defaults; invalid configure blocks do not publish partial changes. Boot/job-autoload and direct unsupported-strategy authentication regressions pass. | Host model/adapter readiness checks, real MaxMind hot-reload/load testing and distributed configuration. Configuration is sealed at boot; authorized runtime changes serialize local publication with required history. This is process-local, not a distributed or crash-atomic publication protocol. |
+| F25 | Lowercase Rack 3 response headers and actual retry deadlines; verified using Rack::Lint. | Content negotiation for API clients. |
+| F26 | Reporting bands, full-history statistics, native-user identity/filtering, and stable event ordering are unified. Ban expiry fields now explicitly use UTC with strict server parsing, browser-compatible milliseconds, and unchanged-value microsecond preservation. Presets use server time; invalid inputs are rejected, custom reasons survive edits, and validation retries preserve duration selections. One nonce-bearing script replaces inline handlers and handwritten method-link submission. Native navigation opts out of host Turbo; real forms retain CSRF and work without JavaScript. Page-size changes preserve filters without duplicate/stale fields; validation feedback stays visible. Chromium regressions cover timezone/DST, skewed clocks, precision, toggles, bulk confirmations, Turbo/back navigation, and no-script forms. | Full strict-style CSP (inline style attributes remain), other browsers, mobile/accessibility and broader host-layout/auth QA, Turbo Frames/Streams, stale-form protection, and remaining dashboard semantics. |
+
+## Next implementation batch
+
+First run the new PostgreSQL/MySQL CI jobs and host multi-worker load/failure
+tests; Docker daemon access was denied locally, including outside the sandbox.
+Verify host API/Cable/native adapters and deployment configuration audit coverage.
+Then address F25 content negotiation and the remaining F26 strict-style CSP/host UI work.
+Continue the remaining privacy and
+real-client validation work in F11.
+Complete the remaining portions of F10/F12/F18/F20/F21/F22/F24/F25 alongside those changes.
+
+## Verification
+
+- Batch ten regular suite: **839 tests, 4,369 assertions, 0 failures/errors,
+  3 existing MaxMind-data skips**, Ruby 4.0.6, seeds `20260915` and `20260916`.
+- Batch ten Chromium suite: **10 tests, 97 assertions, 0 failures/errors/skips**,
+  Chromium/ChromeDriver **153.0.8010.36**, seeds `20260915` and `20260916`. Includes the new
+  reason-bearing export form, required history, and existing form regressions.
+- Batch ten lint: **180 files, no offenses**. Zeitwerk and CI YAML parsing pass.
+  The existing dummy mailer-preview eager-load warning remains. New test-only
+  adapters are pg 1.6.3 and mysql2 0.5.7; their servers/CI jobs were not run locally.
+- New authentication regressions cover custom Warden failure admission, disabled
+  callbacks, same-scope identity substitution, whitelist isolation, multiple
+  browser/remember-cookie revocation, lock rollback, revocation during verification,
+  generic token issuance, stale native sessions, deleted cached Cable users,
+  inbound/outbound Cable denial, and Devise recovery-token replay rejection.
+- New administration/availability regressions cover separate permissions,
+  required export actor/reason/history, append-only events, sealed/runtime config
+  publication, failed journal/transaction rejection, distributed login capacity,
+  shared-NAT page access, eliminated default quota queries and database failures.
+
+- Before implementation, restored baseline: **595 tests, 2,342 assertions,
+  0 failures, 0 errors, 4 skips**.
+- First batch: **626 tests, 2,465 assertions, 0 failures, 0 errors, 4 skips**.
+- Second batch: **655 tests, 2,606 assertions, 0 failures, 0 errors, 4 skips**.
+- Third batch: **679 tests, 2,753 assertions, 0 failures, 0 errors, 4 skips**.
+- Fourth batch: **700 tests, 2,951 assertions, 0 failures, 0 errors, 4 skips**.
+- Fifth batch: **718 tests, 3,177 assertions, 0 failures, 0 errors, 4 skips**.
+- Sixth batch: **747 tests, 3,388 assertions, 0 failures, 0 errors, 3 skips**.
+- Seventh batch: **768 tests, 3,623 assertions, 0 failures, 0 errors, 3 skips**.
+- Eighth batch: **791 tests, 3,932 assertions, 0 failures, 0 errors, 3 skips**.
+- Ninth-batch regular suite: **798 tests, 4,073 assertions, 0 failures, 0 errors,
+  3 existing MaxMind database skips**, Ruby 4.0.6, seeds `20260923` and `20260924`.
+- Ninth-batch Chromium system suite: **9 tests, 91 assertions, 0 failures, 0 errors,
+  0 skips**, Chromium/ChromeDriver 152.0.7977.82, seeds `20260923` and `20260924`.
+  The formerly assertion-free ban-deletion test now verifies a 404 response.
+- Standard Ruby lint passes (164 files). Zeitwerk eager-load verification passes;
+  the dummy application's mailer-preview directory is outside eager-load paths.
+- New regressions include separate database connections (not transactional fixture
+  connection-sharing), null/unavailable caches, retry/rollback, counter isolation,
+  long windows, ban permanence, monitor isolation, trusted proxies, Rack responses,
+  dashboard challenges, and fresh migration execution.
+- Authentication regressions additionally exercise real Devise/Warden requests,
+  pre-password denial, HTTP Basic, independent scopes, disabled/unavailable audits,
+  state/risk failures, native lock/session races, cleanup callbacks, and transactional
+  password-reset rollback.
+- Risk regressions cover persisted travel through both adapters, matching score/
+  lock evidence, history order and bounds, removal of implicit trust, observation
+  isolation, malformed geography/times, midnight and browser scoring, cache/provider
+  generation isolation, and unavailable optional caches. Old discount assertions
+  were replaced with regressions that reject unsupported trust assumptions.
+- Audit/WAF regressions cover nested/host redaction, metadata bounds, legacy model
+  reads and state projection, partial model queries, CSV formulas/column boundaries,
+  filtered-email admission accounting, both export cursors/authorization, canonical
+  matching/exclusions, benign exception policy, single charges, and optional audit
+  failures. Existing tests that required raw emails or exception messages now assert
+  redaction; attack-accounting checks use user associations and hashed counters.
+- Dashboard/search regressions cover reporting boundaries across model/UI/export,
+  invalid/fractional scores, selected-time statistics, full-history ban totals,
+  native/Devise display privacy, stable equal-time ordering, absent phantom API
+  routes, literal search text, legacy JSON email fallback, adapter-specific SQL
+  generation, composition across pages/exports, and aggregate/preload query counts.
+- Configuration/analysis regressions cover real application startup and host job
+  autoloading, host configuration callbacks, invalid-block publication, nested
+  default merging, unsupported providers/strategies, both authentication paths,
+  native final-session denial, real enclosing commits/savepoint rollbacks, and
+  queue exceptions/aborted enqueues. The formerly skipped background-analysis
+  test now exercises the supported opt-in contract.
+- Notification regressions exercise actual Devise/native lock requests, real
+  commits/savepoint rollbacks, reset-token invalidation, independent user/team
+  hooks, enqueue aborts/failures, bounded delivery retries and failed retry enqueue,
+  malformed recipients, deleted users, recipient-list changes, delivery switches,
+  sanitized transport/job logging, and the full dummy native recovery handoff.
+  Only the Rails test delivery backend was used; no real emails were sent.
+- Lifecycle/admin regressions compare every stored event field after native/Devise
+  account deletion, including legacy evidence, and exercise missing-user pages and
+  exports without rewriting retained rows. Administrative tests cover trusted actor
+  resolution, real CSRF rejection, required-history rollback, later-item bulk
+  failure, deletion callbacks, strict selections/durations, no-ops and subsecond
+  changes, ordinary history CRUD rejection, legacy read filtering, bounded history
+  paging, escaped HTML, and confirmation-form verbs. Separate SQLite connections
+  exercise concurrent manual extensions and mixed manual/automatic updates.
+- Form/browser regressions additionally cover non-UTC application-zone parsing,
+  UTC and offset instants, invalid calendar dates/types/overflow/precision, strict
+  creation presets/defaults, permanent/temporary validation, nonce-bearing markup,
+  native navigation, and filter-preserving page sizes. Actual Chromium tests use
+  multiple browser zones, DST-boundary dates, a skewed browser clock, real CSRF,
+  host Turbo loaded from its package, JavaScript disabled, and browser console
+  assertions. They verify persistent validation feedback, precise expiry retention,
+  and no duplicate administrative operations. A separate browser CI job is added.
+- PostgreSQL/MySQL concurrency, production throughput, hosted CI, and broader
+  cross-browser/host UI testing remain unverified. The tested CSP forbids inline
+  script handlers but explicitly allows the existing inline style attributes.
+
+## Rollout contract
+
+See [State storage](../operations/state-storage.md): apply the migration before starting new
+workers, drain old cache-based workers, share one authoritative writer database,
+review legacy bans, and schedule `beskar:cleanup_security_state`. Cache counters
+are not imported. The test database has been migrated locally; no host production
+database was changed.
+
+Rails-native hosts must also adopt the login and existing-session guards in
+[Authentication](../guides/authentication.md). No additional migration beyond the
+shared-state table is required for batches two through seven. Custom lock strategies
+and cross-database authentication models remain unsupported. Batch ten's
+API/Cable/Warden adapters are documented in docs/operations/security-hardening.md.
+
+Risk weights were not calibrated against production traffic. Review the corrected
+factors and removal of implicit trust discounts in [Risk scoring](../guides/risk-scoring.md)
+before enabling enforcement or emergency resets. Legacy records without explicit
+admission evidence are not automatically promoted into travel history.
+
+Review [Audit data and WAF](../guides/audit-and-waf.md) for changed capture/export fields,
+host email-filter behavior, the 1,000-row export cursor contract, visible CSV text
+prefixes, and the narrower default exception policy. No historical personal data
+or existing bans were bulk-rewritten or deleted by this batch.
+
+Review [Dashboard and search](../guides/dashboard-and-search.md) for the corrected reporting
+bands, search behavior/privacy limits, native-user display, and removal of unused
+versioned API routes/helpers. Authentication risk weights and lock thresholds were
+not changed by the dashboard repair.
+
+Review [Configuration](../guides/configuration.md) before deploying the sixth batch:
+invalid/obsolete settings now stop startup, partial section assignments overlay
+library defaults, and `:custom`/IP2Location placeholders are rejected. Automatic
+analysis defaults off; opt in with a host job and the new keyword contract. The
+analysis hook supplies neither a built-in analyzer nor recovery delivery.
+
+Review [Notifications and recovery](../guides/notifications-and-recovery.md) for batch seven.
+The three notification flags now default to false; enabling them requires explicit
+sender/recovery/team settings and a worker consuming `beskar_notifications`.
+Notifications use their own post-commit delivery jobs. They do not issue recovery
+tokens, unlock accounts, guarantee delivery, or supply an outbox. Production
+transports and host recovery/support procedures must be configured and verified.
+
+Review [Audit lifecycle](../guides/audit-lifecycle.md) for batch eight. Apply the new
+administrative-action migration and configure a trusted `audit_actor` before
+dashboard writes; forms/scripted callers must supply `audit_reason`. Drain old
+workers, whose code can still delete events with accounts or make unaudited ban
+changes. Events now survive account deletion unchanged, not anonymized. No audit
+retention period, purge, historical reconstruction, or production data rewrite
+was performed. Journal protections are not a database tamper-resistance guarantee.
+
+Review the form/browser sections of [Dashboard and search](../guides/dashboard-and-search.md)
+for batch nine. No additional migration is needed. Expiry inputs are now UTC;
+scripted callers must use valid UTC/offset timestamps and valid bounded duration
+seconds. The host must allow its nonce for the behavior script; the engine does
+not loosen CSP. Inline style attributes still require separate cleanup for hosts
+with strict style policies. Native navigation is deliberate, including with host
+Turbo loaded; it is not support for arbitrary Turbo Frames/Streams integrations.
+
+For batch ten, apply `ExpandAdministrativeActionTargets`, configure explicit
+dashboard grants and actor resolution, and update exports to supply a reason.
+Adopt the API/Cable/native adapters, bind credential generations, and plan the
+one-time Devise session invalidation. Configuration is now sealed after boot.
+Global login capacity and request-wide quota blocking are explicit opt-ins.
+Read [Security hardening and rollout](../operations/security-hardening.md) before rolling out; host
+integration, deployment auditing and PostgreSQL/MySQL/load gates remain open.

data/docs/guides/audit-and-waf.md

@@ -0,0 +1,175 @@
+# Audit data, exports, and WAF matching
+
+This contract was introduced in the fourth remediation batch. The fifth batch
+adds [consistent dashboard reporting and search](dashboard-and-search.md).
+The eighth batch adds [account-deletion retention and administrative history](audit-lifecycle.md).
+Earlier examples
+that retain full URLs/exception messages or score every Rails exception are
+historical. See [Repair status](../audits/repair-status.md) for remaining findings.
+
+## Audit capture and disclosure
+
+WAF matching uses request data transiently. Persisted WAF evidence contains rule
+IDs, static descriptions/categories, severity, rules version, decoding-pass count,
+an allowlisted HTTP method/exception class, timestamps, and scores. The event
+retains the Rails-resolved IP for attribution, subject to host audit filters.
+Raw/canonical paths, query strings, exception messages, and User-Agent headers are
+not copied into WAF events, ban metadata, state, or WAF messages. Public
+`record_violation` calls also project old-style analysis onto this bounded schema.
+The state API strips legacy raw-path/description fields on reads and on the next
+normal write; it does not bulk-rewrite historical database contents.
+
+Authentication audits still retain bounded paths, cleaned HTTP(S) referrers,
+device/geographic evidence, IPs, and bounded User-Agent text. Referrers exclude
+credentials, queries, and fragments; session IDs and raw forwarded headers are
+not captured. These records remain sensitive and require access controls.
+
+`Services::AuditData` bounds metadata and applies both built-in sensitive-key
+filters and `Rails.application.config.filter_parameters`. Built-ins cover
+passwords, secrets, tokens, authorization, cookies, session IDs, CSRF,
+`exception_message`, `fullpath`, and `matched_path`. Nested keys are filtered.
+Host email filters now redact attempted emails and associated-user presentation
+in views and exports; account associations and hashed admission counters remain intact.
+Plaintext email searches cannot find newly redacted email values.
+
+Model validation and model loading sanitize:
+
+| Model | Filtered audit fields |
+| --- | --- |
+| SecurityEvent | Metadata, event type (100 characters), IP text (64), User-Agent (500), attempted email (320) |
+| BannedIp | Metadata, reason (100 characters), details (2,048) |
+| AdministrativeAction | Actor/request ID (200), reason (1,000), bounded before/after state projections |
+| Native lock state | Metadata supplied when locking; lock authority is separate |
+
+Ban IP addresses, permanence, expiry, IDs, and other enforcement fields are
+deliberately not passed through audit filters. They remain visible to authorized
+dashboard/export readers. Redacting a ban IP must never disable enforcement.
+Host filters targeting event types, geolocation, or authentication evidence can
+reduce audit reporting and history-based risk enrichment; review that policy
+explicitly. Admission state does not depend on a successful optional audit write.
+
+Metadata admits JSON-compatible values, at most 64 entries per hash, 50 per
+array, depth 10, and a 512-node traversal budget. Keys are at most 128 bytes;
+string values are at most 2,048 characters, with controls replaced and invalid
+UTF-8 repaired. Non-finite numbers become null. Excessively large serialized
+metadata becomes `{"_truncated":true}` (64-KiB limit); depth/budget exhaustion uses
+`[TRUNCATED]`. Do not treat truncated audit data as complete evidence.
+
+Read-time filtering changes loaded objects, not stored legacy rows. Raw SQL,
+`pluck`/`pick`, bulk inserts, and writes that skip validation bypass model
+sanitization. No historic rows, logs, backups, or exports were purged by this
+repair. Arbitrary secrets embedded in allowed free text, path segments, or
+User-Agent text cannot be reliably identified by key-based filters. Hosts must
+filter/drop those fields if their application places secrets there.
+
+Beskar's rescued-exception log messages now use the exception class instead of
+its message. This does not sanitize the host application's own request logs,
+exception reporters, custom callbacks, or externally supplied free-text log
+arguments. Account deletion now retains events unchanged, and dashboard ban changes
+require transactional administrative history; see [Audit lifecycle](audit-lifecycle.md).
+An operational retention period, historical cleanup, and tamper-resistant archival
+remain separate work.
+
+## Export contract
+
+Both event and ban exports require dashboard authentication, an explicit `:export`
+permission, trusted actor, and a nonblank reason (`audit_reason` or
+`X-Beskar-Audit-Reason`). Each page requires a persisted administrative export
+record before data is sent; see [Audit lifecycle](audit-lifecycle.md).
+CSV and JSON return at most 1,000 records in descending ID order. They set
+`Cache-Control: private, no-store`, `X-Content-Type-Options: nosniff`, and:
+
+- `X-Beskar-Export-Limit: 1000`.
+- `X-Beskar-Export-Truncated: true|false`.
+- `X-Beskar-Next-Cursor` when more rows remain.
+
+Pass that cursor as `before_id` with the same filters and format for the next
+page. Malformed/nonpositive/out-of-range IDs return 422. Exports no longer load
+the entire relation or silently override ordering with `find_each`.
+Pagination is not a transactional snapshot: concurrent edits/deletions and
+relative-time filters can change membership. Newly inserted higher IDs require
+starting a new export. The 1,000-row cap bounds rows, not database query cost.
+
+CSV fields are quoted/escaped, bounded, and dangerous textual prefixes receive a
+visible `text: ` marker. Detection covers leading `=`, `+`, `-`, and `@`,
+including preceding controls/whitespace/formatting characters and Unicode
+compatibility variants. Numeric database fields stay numeric. This intentionally
+changes exported text; JSON preserves the filtered text without a spreadsheet
+marker. Neither format exports an associated user object's custom serialization:
+only its ID and filtered email are included.
+
+Quoting alone is not a formula defense, and spreadsheet import/save/reopen
+behavior varies. The visible prefix avoids relying solely on a removable
+apostrophe. See [OWASP's CSV injection guidance](https://owasp.org/www-community/attacks/CSV_Injection).
+Automated tests cover emitted CSV cells and parsed column boundaries, not actual
+Excel/LibreOffice/Sheets clients or their save/reopen behavior. Test the supported
+client workflow before claiming spreadsheet-client safety.
+
+## WAF matching contract
+
+The WAF is a scanner-path heuristic, not a general SQL injection/XSS engine.
+It does not parse request bodies or implement JavaScript challenges/honeypots.
+
+- Match at most 8,192 path bytes, with at most two percent-decoding passes.
+- Convert backslashes to slashes, preserve literal plus signs, and retain dot
+  segments so traversal is detectable rather than erased by normalization.
+- Flag oversized paths, malformed escapes, invalid UTF-8/control bytes, and
+  excessive encoding as medium-severity malformed-path evidence.
+- Use root/segment boundaries to avoid matches inside ordinary words.
+  Ordinary `.well-known` routes are not scanner signatures.
+- Ignore arbitrary query text. Only a flat, exact `format` query key with an
+  allowlisted executable value is checked; queries over 8,192 bytes are skipped.
+- Retain matched rule IDs with `rules_version: 1`, not the matching input.
+
+Use narrow exclusions for legitimate host routes:
+
+```ruby
+Beskar.configure do |config|
+  config.waf[:exception_detection] = :suspicious
+  config.waf[:request_exclusions] = [
+    {path: %r{\A/wp-content/}, methods: ["GET", "HEAD"],
+     categories: [:wordpress_static]},
+    {path: %r{\A/reports/}, methods: ["GET"], categories: [:unknown_format]}
+  ]
+end
+```
+
+Exclusions use the decoded path. Omitted methods/categories mean all methods/
+categories for that path. A static-file exclusion does not exclude traversal or
+configuration-file rules. Existing `record_not_found_exclusions` still apply
+to RecordNotFound exception analysis, not independent path signatures.
+
+Exception policies:
+
+- `:suspicious` (default): ordinary known Rails exceptions need independent
+  path/format evidence. A missing record or unsupported format alone is not abuse.
+- `:all`: opt into broad scoring of the four known exception classes; legitimate
+  errors can accumulate enough points to ban an IP.
+- `:none`: disable exception scoring, without disabling request-path matching.
+
+The known classes are UnknownFormat, InvalidType, RecordNotFound, and
+IpSpoofAttackError (matched by exact class name). Middleware only attributes
+IP-spoof exceptions when an IP was safely resolved; it never substitutes an
+attacker-supplied forwarded header. Application exceptions still propagate.
+
+One middleware pass records at most one violation. Matching several path rules
+uses the highest path severity; a subsequent application exception neither
+adds another charge nor upgrades that earlier charge. Scores remain cumulative:
+one critical signature adds 95 points, below the default threshold of 150.
+This is not a promise to block the first exploit request. Monitor/whitelist
+observations remain isolated from enforcement state.
+
+## Rollout and remaining verification
+
+The fourth-batch changes require only the first batch's shared-state migration;
+batch eight also requires the administrative-action table described in
+[Audit lifecycle](audit-lifecycle.md). Review changed audit fields, host filters, cursor-based export
+consumers, and default exception policy before deployment. Existing enforced bans
+are not automatically forgiven by the narrower matching policy.
+
+Run monitor-first against real host routes and mounted paths. Local benign/attack
+corpora cover percent/double encoding, backslashes, query poisoning, boundaries,
+exclusions, ordinary Rails exceptions, legacy state, and single-charge behavior.
+These tests do not establish production false-positive rates or complete scanner
+coverage. Production database fault/load tests, spreadsheet-client QA, historical
+data cleanup, and operational retention policies remain unverified or unimplemented.

data/docs/guides/audit-lifecycle.md

@@ -0,0 +1,172 @@
+# Audit lifecycle and administrative changes
+
+This is the eighth remediation batch's contract. Earlier documentation describing
+account-deletion cascades or unaudited dashboard ban changes is historical.
+
+## Account deletion: retain events unchanged
+
+Deleting a native or Devise account no longer destroys its linked security events.
+Beskar leaves every stored event field unchanged, including `user_type`, `user_id`,
+timestamps, attempted email, IP address, and metadata. The association resolves to
+nil once the account is gone; dashboard pages and exports tolerate that absence.
+JSON exports retain the original polymorphic identifiers without embedding a
+nonexistent user. Existing read-time privacy filters still apply without rewriting
+stored rows.
+
+This is **retention, not anonymization**. Identifiers and evidence can still identify
+people. Beskar does not nullify the linkage, impose an expiry, or supply an automatic
+event purge. Unidentified failed-login events also remain independent of account
+deletion. Previously cascade-deleted events are not recovered, and this change
+does not rewrite historical data or backups.
+
+Hosts must choose and operate any retention period, access controls, backup policy,
+or explicitly authorized erasure workflow separately. Avoid reusing account IDs;
+the retained polymorphic reference can resolve to a replacement row with the same
+type/ID. Model renames/removals also require a host migration strategy. Host-defined
+callbacks, bulk SQL, and database rules can override this lifecycle. Normal
+SecurityEvent save/update/update_columns/destroy/delete operations are now
+rejected after creation. This is append-only model behavior, not tamper-proof
+database storage or a restriction on privileged host code.
+
+## Trusted administrative actor
+
+Dashboard writes require both the existing authorization callback and a separate
+server-side `audit_actor` resolver. For a host with a Devise `User#admin?` role:
+
+```ruby
+Beskar.configure do |config|
+  config.authenticate_admin = ->(request) do
+    request.env["warden"]&.authenticate(scope: :user)&.admin?
+  end
+  config.audit_actor = ->(request) do
+    user = request.env["warden"]&.user(scope: :user)
+    "User:#{user.id}" if user&.admin?
+  end
+  config.authorize_admin = ->(request, permission) do
+    user = request.env["warden"]&.user(scope: :user)
+    user&.admin? && user.beskar_permissions.include?(permission.to_s)
+  end
+end
+```
+
+Adapt the scope, role, and `beskar_permissions` lookup to the host. Permission
+names are `read`, `manage_bans`, `export`, and `read_audit`; none is implied by
+another or by successful authentication. Missing grants deny access (403).
+These callbacks run in controller
+context; the actor callback runs only after dashboard authorization and CSRF
+verification, once per mutation request. Native hosts should derive the identifier
+from their already authenticated administrative session. A shared Basic/token
+credential may use an explicit stable service identifier, but that records a
+shared identity, not which individual used it.
+
+The resolver must return an opaque string of 1–200 ASCII characters, starting with
+a letter/digit and otherwise containing letters, digits, `:`, `_`, `.`, `/`, or `-`.
+Use a stable type/ID, never an email, password, token, request parameter, or
+unverified header. Beskar cannot verify the truth of a host callback's identity.
+Authorization's boolean result is not inferred to be an actor.
+
+`audit_actor` defaults to nil. Separately authorized reads still work, but mutations and exports return
+503 when the resolver is missing, invalid, or raises. Startup validates Proc-or-nil
+without executing the callback. Every mutation also requires `audit_reason`, a
+nonblank string of at most 1,000 characters; invalid input returns 422. Prefer a
+case reference and short explanation. Key-based filtering cannot identify arbitrary
+secrets embedded in free text.
+
+## Journal and transactional behavior
+
+Apply `ExpandAdministrativeActionTargets` as well as the original journal migration.
+Entries now distinguish `BannedIp`, `SecurityEvent`, and `Configuration` targets;
+collection/configuration entries have no target ID. Export preparation records
+the actor, required reason, request, format, filtered query, count, ID bounds,
+and truncation before sending data. A journal failure returns 503 without an
+export body. Supply `audit_reason` or `X-Beskar-Audit-Reason` on every cursor page.
+The record is not proof of successful client download. Runtime configuration
+publication has a separate permission and journal; see [Configuration](configuration.md).
+
+`beskar_administrative_actions` records dashboard ban creation, updates, unbans,
+extensions, and conversion to permanent bans. Each changed target receives:
+
+- A server-resolved actor, required reason, action name, ban ID, and creation time.
+- A server-generated operation UUID shared by all targets of one bulk operation.
+- The Rails request ID, bounded to 200 characters, for correlation only. It may
+  originate in a client `X-Request-ID` header and is not proof of identity.
+- Filtered, bounded before/after projections of ban ID/IP, reason, details,
+  permanence, ban/expiry times, violation count, and metadata. Creation has an empty
+  before-state; unban has an empty after-state.
+
+The projections use [AuditData filtering and bounds](audit-and-waf.md), not a raw
+database copy. Timestamp precision, redaction, and truncation can make projections
+look identical despite a real stored change; such changes still receive an entry.
+The journal is not a full-fidelity restore backup.
+
+Ban changes and required journal inserts share one writer-database transaction and
+the same coordination keys used by automatic escalation. Bulk requests accept
+1–100 positive ban IDs, deduplicate and sort them, and require every target to
+exist. Missing targets return 404; malformed inputs and unsupported operations or
+extension durations return 422. Supported extensions are `1h`, `6h`, `24h`, `7d`,
+and `30d`; making a ban permanent is a separate operation. Existing permanent bans
+cannot be extended. Manual extensions do not increment violation counts; automatic
+`BannedIp.extend_ban!` behavior is unchanged. Dashboard edits cannot change a ban's
+IP identity.
+
+There is no partial bulk success. A failed target callback or required history
+insert rolls back the transaction, including earlier items. Success messages follow
+successful completion; Active Record errors and rejected persistence callbacks
+return 503 rather than claiming success (other unexpected exceptions propagate).
+If a connection fails during commit, the client may not know whether the
+transaction committed: reload state/history before retrying. Operation UUIDs are
+correlation, not a client retry/idempotency protocol. Exact no-op updates create no
+history, and the bulk response reports the count actually changed.
+
+Authenticated history pages at `/beskar/administrative_actions` and `/:id` are
+read-only, paginated (at most 100 rows/page), HTML-escaped, and marked `no-store`.
+Filter the index with `target_id`. There are no history edit/delete/export routes.
+History has no actor or ban foreign key and survives their deletion. Ordinary
+instance save/update, `update_columns`, destroy, and delete attempts are rejected.
+This is **not tamper-proof storage**: low-level counter/bulk APIs, raw SQL, and
+privileged database access can bypass model protections. Database access policy,
+external archival, and tamper detection remain separate work.
+
+Unban and row-extension links now open review pages with real forms and required
+reasons; their GET requests do not mutate state. New/edit/inline-extension/bulk
+forms also require reasons. Batch nine consolidates behavior in a nonce-bearing
+script, removes inline event handlers/method-link synthesis, and adopts native
+navigation. See [Dashboard and search](dashboard-and-search.md) for UTC inputs,
+JavaScript-disabled behavior, browser verification, and the remaining style-CSP limit.
+
+## Coverage and deployment boundaries
+
+The journal is prospective and covers dashboard exports, audited runtime
+configuration publication, and dashboard/explicit `AdministrativeBans` changes.
+It does **not** automatically journal every
+direct model/manager/console write, WAF transition, expiry cleanup, or failed action
+attempt. Existing administrative history is not reconstructed. For host-owned
+manual workflows, authorize the operator first, then create one service instance
+per operation with trusted `actor:`, required `reason:`, and a nonblank `request_id:`:
+
+```ruby
+Beskar::Services::AdministrativeBans.new(
+  actor: "Operator:42", reason: "Case 123: false positive", request_id: SecureRandom.uuid
+).change!([ban.id], action: "unban")
+```
+
+The service itself is not an authorization boundary. As with coordinated security
+state, bans, journal, and state rows must use the same authoritative writer pool.
+Any `Rails.cache` backend remains supported.
+
+Before rollout, copy and apply `CreateBeskarAdministrativeActions` and
+`ExpandAdministrativeActionTargets` along with any
+missing earlier migrations (`bin/rails beskar:install:migrations`, then
+`bin/rails db:migrate`). Configure `authorize_admin` and `audit_actor`, update scripted dashboard callers
+to supply `audit_reason`, and drain/restart old workers: older code can still delete
+events on account deletion or mutate bans without the journal. The new table was
+applied only to the local test database; no production database was changed.
+
+No cleanup job is added for either audit table. Per-entry bounds do not bound total
+storage; hosts must monitor growth and choose retention deliberately. Local tests
+cover both account adapters, unchanged raw rows after deletion/read/export, real
+CSRF rejection, required-history and later-bulk-item rollback, escaped HTML/forms,
+fresh migration execution, and simultaneous manual/automatic updates using separate
+SQLite connections. Batch nine adds Chromium form/CSRF/navigation checks. Live
+PostgreSQL/MySQL, production load, broader host browser flows, and tamper-resistant
+external archival remain unverified.