moderate 1.0.0.beta2 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.rubocop.yml +96 -1
- data/.simplecov +2 -10
- data/AGENTS.md +1 -1
- data/Appraisals +10 -8
- data/CHANGELOG.md +654 -52
- data/CLAUDE.md +1 -1
- data/README.md +147 -7
- data/Rakefile +2 -0
- data/app/controllers/concerns/moderate/moderation.rb +23 -11
- data/app/controllers/moderate/appeals_controller.rb +9 -5
- data/app/controllers/moderate/application_controller.rb +3 -3
- data/app/controllers/moderate/notices_controller.rb +3 -6
- data/app/helpers/moderate/engine_helper.rb +32 -11
- data/app/views/moderate/appeals/new.html.erb +6 -0
- data/app/views/moderate/transparency_reports/_summary_card.html.erb +2 -2
- data/app/views/moderate/transparency_reports/show.html.erb +5 -5
- data/config/locales/en.yml +200 -0
- data/config/locales/es.yml +177 -0
- data/db/migrate/20261018200000_add_pending_key_to_moderate_flags.rb +139 -0
- data/db/migrate/20261018210000_add_retention_and_classifications_to_moderate_tables.rb +103 -0
- data/docs/chats.md +140 -0
- data/docs/compliance.md +92 -3
- data/docs/configuration.md +30 -2
- data/docs/dsa-notice-form.md +1 -1
- data/docs/madmin.md +44 -14
- data/docs/notifications.md +17 -4
- data/examples/aws_rekognition_adapter.rb +6 -6
- data/examples/openai_moderation_adapter.rb +1 -1
- data/examples/regexp_pattern_adapter.rb +68 -0
- data/gemfiles/{rails_7.1.gemfile → rails_8.0.gemfile} +8 -6
- data/gemfiles/rails_8.1.gemfile +6 -5
- data/lib/generators/moderate/install_generator.rb +25 -0
- data/lib/generators/moderate/templates/create_moderate_tables.rb.erb +17 -2
- data/lib/generators/moderate/templates/initializer.rb +9 -15
- data/lib/generators/moderate/views_generator.rb +3 -3
- data/lib/moderate/attachments.rb +137 -0
- data/lib/moderate/configuration.rb +87 -12
- data/lib/moderate/engine.rb +86 -25
- data/lib/moderate/event.rb +21 -0
- data/lib/moderate/filters/base.rb +1 -0
- data/lib/moderate/filters/wordlist.rb +5 -2
- data/lib/moderate/jobs/classify_job.rb +96 -16
- data/lib/moderate/label.rb +2 -2
- data/lib/moderate/macros.rb +8 -3
- data/lib/moderate/models/appeal.rb +10 -1
- data/lib/moderate/models/block.rb +23 -19
- data/lib/moderate/models/classification.rb +24 -0
- data/lib/moderate/models/concerns/actor.rb +17 -2
- data/lib/moderate/models/concerns/content_filterable.rb +181 -55
- data/lib/moderate/models/concerns/deferred_notices.rb +90 -0
- data/lib/moderate/models/concerns/reportable.rb +84 -4
- data/lib/moderate/models/flag.rb +182 -19
- data/lib/moderate/models/report.rb +180 -16
- data/lib/moderate/result.rb +3 -10
- data/lib/moderate/retention.rb +241 -0
- data/lib/moderate/services/intake_notice.rb +10 -10
- data/lib/moderate/services/intake_report.rb +42 -13
- data/lib/moderate/services/resolve_appeal.rb +8 -2
- data/lib/moderate/services/resolve_flag.rb +183 -35
- data/lib/moderate/services/resolve_report.rb +288 -35
- data/lib/moderate/text_validator.rb +15 -14
- data/lib/moderate/version.rb +1 -1
- data/lib/moderate.rb +337 -17
- data/lib/tasks/moderate.rake +23 -0
- metadata +22 -14
- data/gemfiles/rails_7.2.gemfile +0 -36
- data/lib/moderate/text.rb +0 -38
- data/lib/moderate/word_list.rb +0 -90
data/docs/chats.md
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# Trust & Safety for [`chats`](https://github.com/rameerez/chats): report, block and filter messages
|
|
2
|
+
|
|
3
|
+
Messages between users are user-generated content. App Store Guideline 1.2, Google Play's UGC policy and the EU DSA all expect people to be able to **report** a message, **block** its sender, and the app to **filter** and **moderate** what's said. `chats` and `moderate` were built for each other, with no hard dependency either way. `chats` already speaks moderate's reportable contract, so the integration is a handful of lines in your app.
|
|
4
|
+
|
|
5
|
+
This is the moderate side of the recipe. It is the shape two production apps (CarHey and playscbw) converged on. chats' own README covers its half: the [Trust & Safety section](https://github.com/rameerez/chats#%EF%B8%8F-trust--safety-snaps-onto-the-moderate-gem).
|
|
6
|
+
|
|
7
|
+
## What chats already answers
|
|
8
|
+
|
|
9
|
+
`Chats::Message` (and `Chats::Conversation`) implement moderate's duck-typed reportable contract themselves:
|
|
10
|
+
|
|
11
|
+
| moderate asks | chats answers |
|
|
12
|
+
|---|---|
|
|
13
|
+
| `reported_owner` | `author \|\| sender`: the person who wrote it. For an unsigned support-desk reply that's the desk, which moderate treats as "nobody to reach" (`Moderate.responsible_user`): the report still files and the content can still be removed. There's nothing to override. |
|
|
14
|
+
| `moderation_snapshot(:body)` | the body, frozen into the report as evidence |
|
|
15
|
+
| `remove_reported_field!(:body)` | the soft-delete tombstone ("Message deleted"), which also works in a locked conversation |
|
|
16
|
+
| `report_visible_to?(viewer)` | participants only, never the author. moderate enforces it when the report is filed, not only when the link renders. |
|
|
17
|
+
| `moderation_content_type` | `"message"` (`"group"` for a conversation), stored on the report |
|
|
18
|
+
|
|
19
|
+
## 1. Make messages reportable and filtered
|
|
20
|
+
|
|
21
|
+
In an after-boot hook, so a code reload re-applies the macros to the engine's model:
|
|
22
|
+
|
|
23
|
+
```ruby
|
|
24
|
+
# config/initializers/chats_moderation.rb
|
|
25
|
+
Rails.application.config.to_prepare do
|
|
26
|
+
Chats::Message.has_reportable_content :body, labels: { body: "Message" }
|
|
27
|
+
Chats::Message.moderates :body, mode: :flag
|
|
28
|
+
# with attachments on:
|
|
29
|
+
# Chats::Message.has_reportable_content :body, :files, labels: { body: "Message", files: "Photo" }
|
|
30
|
+
# Chats::Message.moderates :files, mode: :flag, with: :your_image_adapter
|
|
31
|
+
end
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
- **`:flag`, never `:block`.** A wordlist false positive must not gag someone mid-conversation. The message sends, and a pending `Moderate::Flag` lands in the queue.
|
|
35
|
+
- **Declare each filter once.** `moderates :body, mode: :flag` and `config.filter "Chats::Message", :body, mode: :flag` write the same policy. Keep one of them; the macro next to `has_reportable_content` reads best.
|
|
36
|
+
- **`labels:`** is what the queue, the statement of reasons and the appeal page call the field. Without it they print "Text" (the gem's generic label for `body`).
|
|
37
|
+
|
|
38
|
+
## 2. Blocks are moderate's, everywhere chats decides something
|
|
39
|
+
|
|
40
|
+
```ruby
|
|
41
|
+
# config/initializers/chats.rb
|
|
42
|
+
config.blocked_messager_ids = ->(messager) { messager.is_a?(User) ? Moderate.blocked_ids_for(messager) : [] }
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
A blocked pair can't open a conversation or send into one, and their direct thread leaves both inboxes. The thread is hidden, never deleted: unblocking brings it back. Group conversations are exempt from pair blocks. The `is_a?(User)` guard keeps non-person messagers (a support desk) out of the block table.
|
|
46
|
+
|
|
47
|
+
`Moderate.blocked_ids_for` always returns a relation of user ids, which chats turns into a subquery. If you compose it with ids of your own, call `.ids` first: `relation | array` gives you User records, not ids.
|
|
48
|
+
|
|
49
|
+
## 3. Report on a bubble
|
|
50
|
+
|
|
51
|
+
chats renders each bubble **once per broadcast** and shares it with every viewer, so the bubble can't ask who is looking. That rules out `moderate_report_link`, whose per-viewer check would vanish from live bubbles. Use the viewer-agnostic path in chats' `message_meta` slot, and hide the link on your own bubbles client-side:
|
|
52
|
+
|
|
53
|
+
```erb
|
|
54
|
+
<%# app/views/chats/slots/_message_meta.html.erb %>
|
|
55
|
+
<% if message.sender && !message.deleted? %>
|
|
56
|
+
<%= link_to t("messages.report"), moderate_report_target_path(message, field: :body), class: "chats-report" %>
|
|
57
|
+
<% end %>
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
```css
|
|
61
|
+
.chats-message--own .chats-report { display: none; } /* cosmetic: the server refuses it anyway */
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
`moderate_report_target_path` builds your report form's path (`new_report_path`, or `new_moderate_report_path`, found through `main_app` from inside chats' views) with a signed, non-expiring target (`message.to_moderation_sgid`), so the URL is identical on every render. If your route has another name, build it yourself: `main_app.new_abuse_report_path(target: message.to_moderation_sgid, field: "body")`. Hiding the link is cosmetic. When the report is filed, moderate refuses anyone `report_visible_to?` rejects (a non-participant, the author) with an error on `:reportable`.
|
|
65
|
+
|
|
66
|
+
Your `ReportsController` stays yours (moderate is UI-agnostic):
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
def create
|
|
70
|
+
reportable = Moderate::Report.locate_signed_reportable(params[:target]) or return head(:not_found)
|
|
71
|
+
current_user.report!(reportable, category: params.dig(:report, :category), details: params.dig(:report, :message),
|
|
72
|
+
field: params[:field], block: params.dig(:report, :block) == "1")
|
|
73
|
+
redirect_back fallback_location: root_path, notice: t(".filed")
|
|
74
|
+
rescue ActiveRecord::RecordInvalid => error
|
|
75
|
+
# …render the form with error.record.errors
|
|
76
|
+
end
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
- **The category select:** `f.select :category, Moderate::Report.category_options`. It honors `config.report_categories` and prints labels in the reader's language (English and Spanish ship with the gem).
|
|
80
|
+
- **"Report and block":** `block: true` blocks the message's author once the report is filed.
|
|
81
|
+
|
|
82
|
+
## 4. Block from the thread menu
|
|
83
|
+
|
|
84
|
+
Put a Block action in chats' `conversation_header_actions` slot, posting to a small controller of yours. Blocking hides the very thread the user is standing in, so land on the inbox, not back:
|
|
85
|
+
|
|
86
|
+
```ruby
|
|
87
|
+
class BlocksController < ApplicationController
|
|
88
|
+
before_action :authenticate_user!
|
|
89
|
+
|
|
90
|
+
def create
|
|
91
|
+
current_user.block!(User.find(params[:user_id]))
|
|
92
|
+
redirect_to chats.root_path, status: :see_other
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
def destroy
|
|
96
|
+
other = User.find(params[:user_id])
|
|
97
|
+
current_user.unblock!(other)
|
|
98
|
+
redirect_to profile_path(other), status: :see_other # there's no thread to go back to while blocked
|
|
99
|
+
end
|
|
100
|
+
end
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
If your inbox badges or notification bell should catch up on both players' screens when a block is placed or lifted, do it from moderate's `user_blocked` / `user_unblocked` events rather than from the controller. They fire after the block commits, and also for blocks placed elsewhere (`report!(…, block: true)`, a console):
|
|
104
|
+
|
|
105
|
+
```ruby
|
|
106
|
+
# config/initializers/moderate.rb
|
|
107
|
+
config.notify = lambda do |event|
|
|
108
|
+
case event.name
|
|
109
|
+
when :user_blocked, :user_unblocked
|
|
110
|
+
InboxSettler.call(event.subject.blocker, event.subject.blocked) # your code, refreshing both sides
|
|
111
|
+
end
|
|
112
|
+
# …
|
|
113
|
+
end
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
## 5. The admin side
|
|
117
|
+
|
|
118
|
+
Reports and flags about messages land in moderate's queues (`Moderate::Report.pending`, `Moderate::Flag.pending`) with no chat-specific branches:
|
|
119
|
+
|
|
120
|
+
- **Open the message in context.** Define `moderation_admin_path(routes)` on `Chats::Message`, in the same `to_prepare`, pointing at your admin transcript, anchored at the line.
|
|
121
|
+
- **Decide.** `include Moderate::Moderation` in your admin's reports and flags controllers (see [the madmin guide](madmin.md)). Resolving with `remove_content` removes the message through chats' tombstone. On a flag, `remove_content` / `ban_user` escalate to a report, so the author gets the statement of reasons and the appeal window. A ban needs `config.ban_handler`.
|
|
122
|
+
- **"Flag for review" from the transcript:**
|
|
123
|
+
|
|
124
|
+
```ruby
|
|
125
|
+
flag = Moderate::Flag.flag_for_review!(message, field: :body, by: current_user)
|
|
126
|
+
flag.previously_new_record? # => false when one was already waiting
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
At most one flag is pending per message and field, and the database enforces it, so two moderators pressing it at once file one flag: the second gets the first's. Skip deleted or system messages yourself; there's nothing left to review.
|
|
130
|
+
|
|
131
|
+
## Launch checklist
|
|
132
|
+
|
|
133
|
+
- [ ] `Chats::Message.has_reportable_content :body` (and `:files`), with `labels:`
|
|
134
|
+
- [ ] one `:flag` filter per field (macro *or* `config.filter`, not both)
|
|
135
|
+
- [ ] `config.blocked_messager_ids` → `Moderate.blocked_ids_for`, for people only
|
|
136
|
+
- [ ] Report on every bubble via `moderate_report_target_path`, hidden on your own
|
|
137
|
+
- [ ] Block in the thread menu, landing on the inbox; Unblock where the person still is (their profile)
|
|
138
|
+
- [ ] `moderation_admin_path` on `Chats::Message`
|
|
139
|
+
- [ ] admin decisions wired (`Moderate::Moderation`), and `config.ban_handler` set before anyone can press "ban"
|
|
140
|
+
- [ ] `config.notify` for at least `report_received`, `report_decision`, `affected_user_decision` (see [notifications](notifications.md))
|
data/docs/compliance.md
CHANGED
|
@@ -64,7 +64,7 @@ The statement must include, at minimum: the **restriction imposed** (and its sco
|
|
|
64
64
|
| State the **specific restriction** imposed and its scope (content removed? account suspended?). | Every resolution records its action (`remove_content:`, `ban_user:`) and emits `affected_user_decision` with that action in `event.payload`. | **[gem]** | `test/services/moderate/resolve_report_test.rb` |
|
|
65
65
|
| State the **facts and circumstances** relied on. | The immutable **evidence snapshot** taken at report time travels with the decision; the moderator's mandatory `note:` is the human-readable ground. | **[gem]** | `test/models/moderate/report_test.rb` |
|
|
66
66
|
| State whether **automated means** were used in detection or decision — Art. 17(3)(c). | Reports/flags carry their `source` (`text_filter`, `image_filter`, `external_classifier`, or `manual`). A decision acting on an auto-`Moderate::Flag` is flagged as automated-means in the `affected_user_decision` payload; a human report is not. | **[gem]** | `test/services/moderate/resolve_report_test.rb` |
|
|
67
|
-
| State the **legal or contractual ground**. | For DSA notices, the `legal_reason` (from `Moderate::DSA_LEGAL_REASONS`) is the legal ground; for in-app reports, the community `category` + your `note:` is the contractual (terms-of-service) ground. Both ride in the decision payload. | **[gem + you]** | `test/services/moderate/resolve_report_test.rb` |
|
|
67
|
+
| State the **legal or contractual ground**. | For DSA notices, the `legal_reason` (from `Moderate::Report::DSA_LEGAL_REASONS`) is the legal ground; for in-app reports, the community `category` + your `note:` is the contractual (terms-of-service) ground. Both ride in the decision payload. | **[gem + you]** | `test/services/moderate/resolve_report_test.rb` |
|
|
68
68
|
| Communicate **redress** options (internal complaint, out-of-court, judicial). | The `affected_user_decision` event carries the appeal entry point; you render the redress text in your decision email (the gem ships the data; the copy is yours, because it names your jurisdiction). | **[gem + you]** | `test/services/moderate/resolve_report_test.rb` |
|
|
69
69
|
| Deliver the statement to the **affected recipient** (the content owner), not just the reporter. | Two distinct events fire: `report_decision` → the **reporter**; `affected_user_decision` → the **content owner** (resolved via the reportable's `reported_owner`). You wire both. | **[gem + you]** | `test/services/moderate/resolve_report_test.rb` |
|
|
70
70
|
|
|
@@ -79,7 +79,7 @@ The statement must include, at minimum: the **restriction imposed** (and its sco
|
|
|
79
79
|
| --- | --- | --- | --- |
|
|
80
80
|
| An **internal** appeal mechanism against moderation decisions. | `Moderate::Appeal` — a complaint filed against a resolved report/notice; the queue is `Moderate::Appeal.pending`. | **[gem]** | `test/models/moderate/appeal_test.rb` |
|
|
81
81
|
| **Free of charge.** | There is no charge anywhere in the appeal path — it's just a record + a queue. (You simply don't bill for it.) | **[gem]** | n/a (no payment code exists in the path) |
|
|
82
|
-
| Open for **at least six months**
|
|
82
|
+
| Open for **at least six months**, counted from the day the person **is informed** of the decision (Art. 20(2)). | Each report stores its **appeal window** (`appeal_deadline_at`): six months from the decision (a minimum: the notice says "at least until"), moved to six months from delivery when your notifier confirms the notice actually went out (`Moderate.delivered!`, which stamps `decision_notified_at` / `affected_user_notified_at` once), never earlier. A queued mail doesn't count. `Moderate::Appeal` refuses a complaint filed after it; one filed in time is decided whenever the review ends. | **[gem]** | `test/models/moderate/appeal_window_test.rb`, `test/models/moderate/appeal_test.rb` |
|
|
83
83
|
| Decisions **reversible** — uphold the complaint and reverse the action. | `appeal.uphold!(by:, note:)` overturns the original decision (and runs the reverse enforcement); `appeal.reject!(by:, note:)` confirms it. | **[gem]** | `test/services/moderate/resolve_appeal_test.rb` |
|
|
84
84
|
| **Not solely automated** — a human decides the complaint. | `uphold!`/`reject!` **require** a `by:` moderator and a `note:`; there is no path to auto-decide an appeal. | **[gem]** | `test/services/moderate/resolve_appeal_test.rb` |
|
|
85
85
|
| Inform the complainant of the **appeal decision** and remaining redress (out-of-court / judicial). | Resolving an appeal emits `appeal_decision` to the complainant; you render the out-of-court / judicial redress copy. | **[gem + you]** | `test/services/moderate/resolve_appeal_test.rb` |
|
|
@@ -98,7 +98,7 @@ The statement must include, at minimum: the **restriction imposed** (and its sco
|
|
|
98
98
|
| **Publish** the report (at least annually). | The gem produces the numbers; **you** publish them (a `/transparency` page, a PDF, whatever) — only you know your reporting period and format. | **[you]** | manual: render `Moderate.transparency(from:, to:)` |
|
|
99
99
|
|
|
100
100
|
> [!TIP]
|
|
101
|
-
> `Moderate.transparency(from: 1.year.ago, to: Time.current)` returns a plain hash you can drop straight into a view, a JSON endpoint, or a rake task that emails it to you each January. The counters are the regulator-aligned ones — same taxonomy as the notice form (`Moderate::DSA_LEGAL_REASONS`) — so the published numbers line up with the intake.
|
|
101
|
+
> `Moderate.transparency(from: 1.year.ago, to: Time.current)` returns a plain hash you can drop straight into a view, a JSON endpoint, or a rake task that emails it to you each January. The counters are the regulator-aligned ones — same taxonomy as the notice form (`Moderate::Report::DSA_LEGAL_REASONS`) — so the published numbers line up with the intake.
|
|
102
102
|
|
|
103
103
|
---
|
|
104
104
|
|
|
@@ -162,6 +162,7 @@ If you read nothing else, this is the table that says "we did the thing."
|
|
|
162
162
|
| **Play UGC** | Report + block, **users and content** | `report!`/`block!` on users; reportable + `blocked_ids_for` on content | ✅ gem |
|
|
163
163
|
| **Play UGC** | Ongoing moderation surface | `Flag.pending` + `:flag`-mode filtering | ✅ gem (you review) |
|
|
164
164
|
| **Play UGC** | Accept terms before UGC | your onboarding gate | ⬜ you (categories align) |
|
|
165
|
+
| **GDPR Art. 5(1)(e)** | Don't keep personal data longer than needed | `Moderate.purge` (opt-in, keeps decisions; see [Retention](#5-retention-what-moderate-keeps-and-for-how-long)) | ✅ gem (you schedule it) |
|
|
165
166
|
|
|
166
167
|
Legend: **✅ gem** — the mechanism ships and is tested. **⬜ you** — your operational/policy step that the gem makes straightforward.
|
|
167
168
|
|
|
@@ -170,6 +171,94 @@ Legend: **✅ gem** — the mechanism ships and is tested. **⬜ you** — your
|
|
|
170
171
|
|
|
171
172
|
---
|
|
172
173
|
|
|
174
|
+
## 5. Retention: what `moderate` keeps, and for how long
|
|
175
|
+
|
|
176
|
+
**Out of the box, nothing is purged.** The gem never deletes or redacts a moderation record on its own: every report, notice, flag, appeal and block stays exactly as it was written until you remove it, or until you schedule the opt-in purge below. That is a safe default for evidence and an unsafe one for privacy, so decide on a retention period and wire the purge before you launch in the EU.
|
|
177
|
+
|
|
178
|
+
### What the gem stores
|
|
179
|
+
|
|
180
|
+
| Table | What a row holds | Personal data in it |
|
|
181
|
+
| --- | --- | --- |
|
|
182
|
+
| `moderate_reports` | An in-app report or a DSA notice: category, legal ground, member state, content type; the reporter's message; an **evidence snapshot** (a copy of the reported text, the reported URLs, the reported account identifier, taken at intake so it survives edits); the **decision** (status, basis, actions, moderator note, decision visibility, automated-means disclosure, appeal deadline, notification timestamps); which earlier decision removed its content while it was pending (`covered_by_report_id`). | reporter, reported user, notifier name and email, message, snapshot, URLs |
|
|
183
|
+
| `moderate_flags` | A filter, classifier or manual flag: source, mode, categories, scores; an **excerpt** of the content (up to 500 characters); **context** (the classifier's raw response, a reviewer's note); the review. | owner, excerpt, context |
|
|
184
|
+
| `moderate_appeals` | An Art. 20 complaint: the reason, the appellant's name and email, a snapshot of the decision appealed, the outcome and its note. | appellant, name, email, reason |
|
|
185
|
+
| `moderate_blocks` | One blocker → blocked edge and when it was made. | both people |
|
|
186
|
+
| `moderate_classifications` | Bookkeeping for attachment classification: which record and field, a digest of the attached blob ids, who is classifying it (a claim) and when it was classified, the adapter and whether it flagged. | none beyond the record's id |
|
|
187
|
+
|
|
188
|
+
**Notices** (receipts, statements of reasons, appeal decisions) are not stored by the gem: it emits events and your mailer sends them. It only stamps when your notifier confirms one went out (`Moderate.delivered!`: `decision_notified_at`, `affected_user_notified_at`). Whatever your mailer or `config.audit` keeps is yours to retain or purge.
|
|
189
|
+
|
|
190
|
+
Rows go away by themselves only through ordinary associations: deleting a user destroys their blocks and nullifies them as reporter or reported user on reports (`Moderate::Actor`); deleting reported content nullifies the report's pointer to it (the evidence snapshot stays).
|
|
191
|
+
|
|
192
|
+
### What the law expects, in plain words
|
|
193
|
+
|
|
194
|
+
Not legal advice, and your national law and your own policies may add to it:
|
|
195
|
+
|
|
196
|
+
- **Keep the decision and its evidence while it can be challenged.** A decision must stay appealable for at least six months (DSA Art. 20), an appeal filed in time is decided whenever the review ends, and afterwards the person may still take it to an out-of-court dispute settlement body (Art. 21) or a court.
|
|
197
|
+
- **Keep the decision itself for longer.** It is the statement of reasons (Art. 17); it is what a repeat-misuse policy (Art. 23: suspending people who frequently post manifestly illegal content) looks back on; and the annual transparency report (Art. 15, 24) counts notices, actions, automated means, complaints and handling times for a whole reporting period, published after it ends.
|
|
198
|
+
- **Don't keep the personal data longer than that purpose needs** (GDPR Art. 5(1)(c) and (e): data minimisation, storage limitation). The copy of the reported content, the notifier's contact details and the complaint text have done their job once the decision can no longer be challenged. A person may ask for erasure (GDPR Art. 17); the exceptions for legal obligations and legal claims (Art. 17(3)(b) and (e)) are what let you keep a decision that is still contestable.
|
|
199
|
+
|
|
200
|
+
So: content copies and contact details get a short life, decisions and counts a long one. That split is what the purge implements.
|
|
201
|
+
|
|
202
|
+
### `Moderate.purge`: opt in
|
|
203
|
+
|
|
204
|
+
```ruby
|
|
205
|
+
Moderate.purge(older_than: 1.year)
|
|
206
|
+
# => { reports: 12, flags: 40, appeals: 1, classifications: 300, claims_kept: 0,
|
|
207
|
+
# held: { reports: 1, flags: 2, appeals: 0 } }
|
|
208
|
+
Moderate.purge(older_than: 18.months, dry_run: true) # the same counts, changes nothing
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
**`1.year` is a default, not a rule.** GDPR storage limitation asks for a period set by *your* purposes (how long a decision can be challenged where you operate, your repeat-misuse policy, your national rules), so set `older_than` to the period your privacy policy states; the gem only refuses one shorter than the six-month appeal window.
|
|
212
|
+
|
|
213
|
+
```bash
|
|
214
|
+
bin/rails moderate:purge # OLDER_THAN_DAYS=365 by default
|
|
215
|
+
bin/rails moderate:purge OLDER_THAN_DAYS=540 DRY_RUN=1
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
Schedule it like any recurring job (cron, `config/recurring.yml` for Solid Queue, `sidekiq-cron`): `Moderate.purge` is safe to run every night. The rake task prints what it redacted and what it kept for a legal hold.
|
|
219
|
+
|
|
220
|
+
**Legal holds.** When a decision is under external redress (an out-of-court dispute settlement body, DSA Art. 21; a court case; an authority's request), hold that report and the nightly purge leaves it, its appeals and the flags on its content alone until the date you give:
|
|
221
|
+
|
|
222
|
+
```ruby
|
|
223
|
+
Moderate.hold(report, until: 1.year.from_now, by: admin, reason: "ODS case 2026-114")
|
|
224
|
+
report.held? # => true
|
|
225
|
+
Moderate.release_hold(report, by: admin) # the dispute ended; the next purge treats it normally
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
Holding again moves the date; a hold that expires lifts itself. Both are audited (`:legal_hold_placed`, `:legal_hold_released`), and `dry_run` counts held rows under `:held` so you can see what a hold is keeping.
|
|
229
|
+
|
|
230
|
+
| It redacts | Only when | It keeps |
|
|
231
|
+
| --- | --- | --- |
|
|
232
|
+
| **Reports and notices**: the message (→ `"[redacted]"`), the evidence snapshot, the reported URLs and account identifier, the notifier's name and email, the reporter. | The report is closed, was decided more than `older_than` ago, its appeal window has closed, no appeal on it is open, and it isn't held. | Every row; category, legal ground, member state, content type, status, timestamps; the whole decision (basis, actions, note, visibility, automated-means disclosure); the reported user. |
|
|
233
|
+
| **Flags**: the excerpt (→ `"[redacted]"`) and the context. | The flag was decided more than `older_than` ago, and no held report is about its content and field. | Every row; source, mode, categories, scores, status, the owner, the review. |
|
|
234
|
+
| **Appeals**: the reason (→ `"[redacted]"`) and the appellant's name and email. | The appeal was decided more than `older_than` ago, and its report isn't held. | Every row; status, outcome note, timestamps, the appellant's account. |
|
|
235
|
+
| **Classification rows** (`moderate_classifications`): deleted, not redacted (nothing in them is evidence). | Last touched more than `older_than` ago, and not under a live claim (a worker classifying those files right now; counted under `:claims_kept`). | — |
|
|
236
|
+
|
|
237
|
+
Guarantees, each with a test in `test/models/moderate/retention_test.rb`:
|
|
238
|
+
|
|
239
|
+
- **Never** a pending report, flag or appeal, a report still inside its appeal window, a report with an open appeal, or anything a legal hold keeps, whatever their age. `older_than` below the six-month appeal window raises.
|
|
240
|
+
- A redacted row is stamped `redacted_at`, and the purge selects on that column, never on the content: a message that literally reads `"[redacted]"` is purged like any other.
|
|
241
|
+
- `Moderate.transparency` returns the same numbers before and after a purge, so a published report stays reproducible.
|
|
242
|
+
- **Blocks are never touched**: they are live safety state, not a record of the past.
|
|
243
|
+
- Idempotent (a second run finds nothing to do), `dry_run: true` only counts, and a run that changed something is audited as `:retention_purged` with the counts and the cutoff.
|
|
244
|
+
- One `UPDATE` per batch of 500, with no callbacks or validations: a redacted DSA notice no longer has the notifier details its validations require, and a final record should not be re-saved.
|
|
245
|
+
|
|
246
|
+
What it does **not** do, on purpose:
|
|
247
|
+
|
|
248
|
+
- **Delete rows.** The default `1.year` keeps every decision forever, as pseudonymous rows that still name the reported account. If your policy says decisions go after N years, delete them yourself once the transparency report that counts them is published: `Moderate::Report.where.not(status: "open").where(resolved_at: ...N.years.ago).destroy_all` (appeals go with their report).
|
|
249
|
+
- **Answer an erasure request.** Whether a given person's records must go now, or can be kept for a legal claim, is a decision for you and your counsel; the records are plain Active Record (`Moderate::Report.where(reporter: user)`, `Moderate::Appeal.where(appellant_email: email)`, `Moderate::Flag.where(owner: user)`).
|
|
250
|
+
- **Know about disputes by itself.** The gem can't see an out-of-court or court case; when one starts, `Moderate.hold` the report.
|
|
251
|
+
|
|
252
|
+
| Requirement | How `moderate` satisfies it | Who | Proof |
|
|
253
|
+
| --- | --- | --- | --- |
|
|
254
|
+
| Keep decisions contestable for ≥ 6 months (Art. 20) and while an appeal is open. | `Moderate.purge` never touches a record inside its appeal window or with an open appeal, and refuses an `older_than` below it. | **[gem]** | `test/models/moderate/retention_test.rb` |
|
|
255
|
+
| Keep what the annual transparency report counts (Art. 15, 24). | Redaction keeps every row and every counted column; `Moderate.transparency` is unchanged by a purge. | **[gem]** | `test/models/moderate/retention_test.rb` |
|
|
256
|
+
| Storage limitation for personal data (GDPR Art. 5(1)(e)). | `Moderate.purge` / `moderate:purge` redacts content copies and contact details from final records. | **[gem + you]** schedule it, choose `older_than` | `test/models/moderate/retention_test.rb` |
|
|
257
|
+
| Keep evidence during external redress (Art. 21, courts, authorities). | `Moderate.hold(report, until:)` keeps the report, its appeals and the flags on its content out of the purge. | **[gem + you]** place the hold | `test/models/moderate/retention_test.rb` |
|
|
258
|
+
| A written retention period, erasure requests. | Out of scope for a gem: your privacy policy and your counsel decide (`older_than` is yours to set). | **[you]** | manual |
|
|
259
|
+
|
|
260
|
+
---
|
|
261
|
+
|
|
173
262
|
## See also
|
|
174
263
|
|
|
175
264
|
- [`docs/dsa-notice-form.md`](dsa-notice-form.md) — the public Art. 16 notice form, in depth (mount, fields, Turnstile gate, the `moderate:views` eject)
|
data/docs/configuration.md
CHANGED
|
@@ -249,7 +249,7 @@ end
|
|
|
249
249
|
Called for **every notifiable event**. One hook drives them all — `goodmail` for user emails, `telegrama` for admin alerts, `noticed` for in-app feed + push — because every event shares the same envelope. The full vocabulary:
|
|
250
250
|
|
|
251
251
|
```
|
|
252
|
-
report_received report_decision affected_user_decision
|
|
252
|
+
report_received notice_received report_decision affected_user_decision
|
|
253
253
|
appeal_received appeal_decision
|
|
254
254
|
user_blocked user_unblocked user_banned
|
|
255
255
|
content_flagged content_removed
|
|
@@ -272,7 +272,17 @@ Optional teardown when one user blocks another — cancel a pending invite, leav
|
|
|
272
272
|
config.ban_handler = ->(user:, by:, reason:) { user.suspend!(reason: reason) }
|
|
273
273
|
```
|
|
274
274
|
|
|
275
|
-
`moderate` doesn't own your user lifecycle, so it **never bans a user itself**. When a moderator resolves a report with `ban_user: true` (see [the madmin queue](madmin.md#step-3--the-controller-call-the-gems-decision-methods)), this proc decides what "banned" means in your domain — `suspend!`, soft-delete, flip a flag, revoke sessions, whatever. Signature is **keyword args** (`user:`, `by:`, `reason:`).
|
|
275
|
+
`moderate` doesn't own your user lifecycle, so it **never bans a user itself**. When a moderator resolves a report with `ban_user: true` (see [the madmin queue](madmin.md#step-3--the-controller-call-the-gems-decision-methods)), this proc decides what "banned" means in your domain — `suspend!`, soft-delete, flip a flag, revoke sessions, whatever. Signature is **keyword args** (`user:`, `by:`, `reason:`).
|
|
276
|
+
|
|
277
|
+
Leave it unset and everything works except a ban: `ban_user: true` (on a report or a flag) and `Moderate.apply_ban` raise `Moderate::ConfigurationError` before anything changes. A ban nothing applies would still audit `user_banned` and send the person a statement of reasons saying their account was suspended, which is a false statement, so the gem refuses it instead. Assigning `nil` puts the default back.
|
|
278
|
+
|
|
279
|
+
### `banned` — is this person banned right now?
|
|
280
|
+
|
|
281
|
+
```ruby
|
|
282
|
+
config.banned = ->(user) { user.suspended? } # optional; set it if your app can unban
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
One ban per person per episode: a second ban decision on someone a decision in the same episode already banned is recorded as covered by that one (`resolution_actions["ban_already_applied_by"]`), and the `ban_handler` isn't called twice. That only holds while the account is still banned. If your app can lift a ban (an admin unbans, a suspension expires), set `banned`: when it returns false, the earlier ban no longer covers the new decision and the `ban_handler` runs again, so the "account suspended" statement of reasons is true. Unset, the earlier ban counts, whatever happened since.
|
|
276
286
|
|
|
277
287
|
---
|
|
278
288
|
|
|
@@ -286,6 +296,24 @@ config.locale = :en # default: your app's I18n.default_locale
|
|
|
286
296
|
|
|
287
297
|
The locale for copy `moderate` generates on its own — filter validation messages, the DSA statement-of-reasons taxonomy labels, the notice-form strings. Leave it unset to follow `I18n.default_locale`.
|
|
288
298
|
|
|
299
|
+
### `migration_copies_check`
|
|
300
|
+
|
|
301
|
+
```ruby
|
|
302
|
+
config.migration_copies_check = :raise # default; or :log, or :off
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
An app that holds copies of the gem's migrations (`*.moderate.rb`, from `rails generate moderate:install` or `bin/rails moderate:install:migrations`) runs only those copies, so a release whose new migration you didn't copy would never run it. At boot `moderate` compares your copies with its migrations by name: if one is missing it raises `Moderate::ConfigurationError` naming it and the command that copies it (`bin/rails moderate:install:migrations`, then `bin/rails db:migrate`). Under a rake task it logs instead, so that command can run. `:log` always logs (to `Rails.logger` and stderr); `:off` skips the check. An app without copies is never checked: it gets the gem's migrations through the engine.
|
|
306
|
+
|
|
307
|
+
### `classification_claim_ttl`
|
|
308
|
+
|
|
309
|
+
```ruby
|
|
310
|
+
config.classification_claim_ttl = 10.minutes # default (600 seconds)
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
How long one worker's claim on a set of attached files lasts. An attachment is classified by whoever claims it first (the inline path, the immediate `ClassifyJob` or the after-upload guard); the others see the live claim and stop, and the adapter runs outside any lock. Make it longer than your slowest adapter call: a claim that expires mid-call lets another worker classify the same files, and the first result is then discarded. When a worker dies mid-call, the files wait this long before a job takes them over. It must be more than 0 and at most a day; anything else raises at boot.
|
|
314
|
+
|
|
315
|
+
An adapter that **raises** leaves its claim in place until it expires, and the failure doesn't schedule a retry by itself. A later attempt takes the files over: the after-upload guard job, a job's single recheck at the claim's expiry, the next save of that field, or your job backend retrying the same job (it keeps its job id, so the claim is still its own). `Moderate.purge` never deletes a row whose claim is live, whatever its age (`:claims_kept` in its counts).
|
|
316
|
+
|
|
289
317
|
---
|
|
290
318
|
|
|
291
319
|
## How the macros relate to config
|
data/docs/dsa-notice-form.md
CHANGED
|
@@ -250,7 +250,7 @@ off the public form's bot gate.
|
|
|
250
250
|
|
|
251
251
|
#### The rate-limit hook
|
|
252
252
|
|
|
253
|
-
|
|
253
|
+
You could use Rails' built-in `rate_limit` API; `moderate` instead implements a tiny per-IP, cache-backed counter (`Rails.cache`) as a `before_action`, so it honors your **runtime** `config.notice_rate_limit` (the class-level macro is evaluated at class load, before your initializer has run). Configure it once:
|
|
254
254
|
|
|
255
255
|
```ruby
|
|
256
256
|
config.notice_rate_limit = { max: 5, within: 1.hour } # default
|
data/docs/madmin.md
CHANGED
|
@@ -100,15 +100,17 @@ class Moderate::ReportResource < Madmin::Resource
|
|
|
100
100
|
# Everything is read-only (form: false): decisions go through the custom
|
|
101
101
|
# member actions below, never through madmin's stock edit form.
|
|
102
102
|
attribute :id, index: true, form: false
|
|
103
|
-
attribute :status, index: true, form: false #
|
|
104
|
-
attribute :
|
|
103
|
+
attribute :status, index: true, form: false # open / actioned / dismissed
|
|
104
|
+
attribute :intake_kind, index: true, form: false # "community" (in-app) or "dsa" (a public notice)
|
|
105
105
|
attribute :category, index: true, form: false # community category OR DSA legal_reason
|
|
106
106
|
attribute :reportable, :polymorphic, index: true, form: false, label: "Target"
|
|
107
|
-
|
|
107
|
+
# The field's human label ("Chat message"), not the raw column ("body") — see
|
|
108
|
+
# `labels:` on has_reportable_content. A virtual attribute, so it needs a type.
|
|
109
|
+
attribute :reported_field_label, :string, index: true, form: false, label: "Field"
|
|
108
110
|
attribute :reported_user, index: true, form: false
|
|
109
111
|
attribute :reporter, index: true, form: false
|
|
110
112
|
attribute :notifier_email, index: true, form: false, label: "Notifier" # DSA notices
|
|
111
|
-
attribute :
|
|
113
|
+
attribute :message, index: false, form: false, label: "Reason" # what the reporter wrote
|
|
112
114
|
attribute :snapshot, index: false, form: false # the immutable evidence snapshot
|
|
113
115
|
attribute :resolution_note, index: false, form: false
|
|
114
116
|
attribute :created_at, index: true, form: false, label: "Received"
|
|
@@ -116,7 +118,7 @@ class Moderate::ReportResource < Madmin::Resource
|
|
|
116
118
|
|
|
117
119
|
# Queue scopes — these are the gem's own scopes, surfaced as madmin filters.
|
|
118
120
|
scope :pending
|
|
119
|
-
scope :
|
|
121
|
+
scope :actioned
|
|
120
122
|
scope :dismissed
|
|
121
123
|
|
|
122
124
|
menu label: "Reports", parent: "Trust & Safety"
|
|
@@ -135,10 +137,10 @@ class Moderate::FlagResource < Madmin::Resource
|
|
|
135
137
|
model Moderate::Flag
|
|
136
138
|
|
|
137
139
|
attribute :id, index: true, form: false
|
|
138
|
-
attribute :status, index: true, form: false # pending /
|
|
140
|
+
attribute :status, index: true, form: false # pending / actioned / dismissed
|
|
139
141
|
attribute :source, index: true, form: false # wordlist / image / <your adapter> / manual
|
|
140
142
|
attribute :flaggable, :polymorphic, index: true, form: false, label: "Target"
|
|
141
|
-
attribute :
|
|
143
|
+
attribute :field_label, :string, index: true, form: false, label: "Field" # "Chat message", not "body"
|
|
142
144
|
attribute :owner, index: true, form: false
|
|
143
145
|
attribute :categories, index: false, form: false # e.g. [:hate, :threats]
|
|
144
146
|
attribute :scores, index: false, form: false # { hate: 1.0 } (0..1 for ML adapters)
|
|
@@ -147,7 +149,7 @@ class Moderate::FlagResource < Madmin::Resource
|
|
|
147
149
|
attribute :reviewed_at, index: true, form: false
|
|
148
150
|
|
|
149
151
|
scope :pending
|
|
150
|
-
scope :
|
|
152
|
+
scope :actioned
|
|
151
153
|
scope :dismissed
|
|
152
154
|
|
|
153
155
|
menu label: "Flags", parent: "Trust & Safety"
|
|
@@ -296,8 +298,14 @@ end
|
|
|
296
298
|
# app/controllers/madmin/flags_controller.rb
|
|
297
299
|
module Madmin
|
|
298
300
|
class FlagsController < Madmin::ResourceController
|
|
301
|
+
# Without the checkboxes this only triages the flag. With either, the flag
|
|
302
|
+
# escalates to a report (the open one on the same content, or a system one) and
|
|
303
|
+
# resolves it: the content goes through your remove_reported_field!, the ban
|
|
304
|
+
# through your ban_handler, and the author gets the statement of reasons and the
|
|
305
|
+
# appeal window a restriction requires. `@record.report` is that report after.
|
|
299
306
|
def resolve
|
|
300
|
-
@record.resolve!(by: current_user, note: params[:note]
|
|
307
|
+
@record.resolve!(by: current_user, note: params[:note],
|
|
308
|
+
remove_content: params[:remove_content], ban_user: params[:ban_user])
|
|
301
309
|
redirect_to main_app.madmin_flag_path(@record), notice: "Flag actioned.", status: :see_other
|
|
302
310
|
rescue => error
|
|
303
311
|
redirect_to main_app.madmin_flag_path(@record), alert: "Could not action flag: #{error.message}", status: :see_other
|
|
@@ -379,7 +387,10 @@ Hand-rolling (above) is the right call when your redirects/flashes need to match
|
|
|
379
387
|
<section>
|
|
380
388
|
<h2>Resolve</h2>
|
|
381
389
|
<%= form_with url: resolve_madmin_report_path(report), method: :post do %>
|
|
382
|
-
|
|
390
|
+
<% if report.reportable.respond_to?(:removable_reported_field?) && report.reportable.removable_reported_field?(report.reported_field) %>
|
|
391
|
+
<%# Only where removal can happen: a removal that removes nothing refuses the decision. %>
|
|
392
|
+
<label><%= check_box_tag :remove_content, "1" %> Remove reported content</label>
|
|
393
|
+
<% end %>
|
|
383
394
|
<label><%= check_box_tag :ban_user, "1" %> Ban reported user</label>
|
|
384
395
|
<%= text_area_tag :note, nil, placeholder: "Decision note", required: true %>
|
|
385
396
|
<%= submit_tag "Resolve", data: { turbo_confirm: "Resolve and notify the parties?" } %>
|
|
@@ -414,11 +425,29 @@ The appeal show view is the same shape with `uphold`/`reject`:
|
|
|
414
425
|
|
|
415
426
|
Two details worth copying:
|
|
416
427
|
|
|
428
|
+
- **Resolve needs an action.** A report's `resolve!` with neither `remove_content` nor `ban_user` ticked is refused (the flash says why): it would tell the author a restriction was imposed when none was. Offer "Dismiss" for a decision that restricts nothing.
|
|
417
429
|
- **Gate on `pending?`.** Show the decision forms only while the record is open; render "closed" once it isn't. The decision methods are also guarded server-side (calling `resolve!` on an already-resolved report raises), but hiding the buttons is the better UX.
|
|
418
430
|
- **Require the note.** `required: true` on the textarea, and the gem requires it too — every decision must carry a rationale (that's what feeds the DSA Art. 17 statement of reasons). Belt and suspenders.
|
|
419
431
|
|
|
420
432
|
The evidence snapshot (`report.snapshot`) is plain JSON on the record — render it in a `<pre>` so the moderator sees exactly what was reported, even if the original content was since edited or deleted. That immutability is the whole point of the snapshot.
|
|
421
433
|
|
|
434
|
+
The flag show view is the report's, pointed at the flag routes: the same two checkboxes on `resolve` (each escalates the flag to a report, see the controller above), and `dismiss`. Print labels, not stored values: `Moderate.taxonomy_label(:flag_status, flag.status)`, `Moderate.taxonomy_label(:flag_source, flag.source)`, `flag.field_label`, and for reports `Moderate.taxonomy_label(:category, report.category)` and `report.reported_field_label`. The gem ships English and Spanish for every value it defines.
|
|
435
|
+
|
|
436
|
+
### "Flag for review" from a browse screen
|
|
437
|
+
|
|
438
|
+
Moderators often find bad content while browsing something else: a conversation's transcript, a user's page. Don't grow enforcement buttons there; file the content into the queue, where the decision tools already are:
|
|
439
|
+
|
|
440
|
+
```ruby
|
|
441
|
+
# app/controllers/madmin/messages_controller.rb
|
|
442
|
+
def flag
|
|
443
|
+
flag = Moderate::Flag.flag_for_review!(@record, field: :body, by: current_user, note: params[:note])
|
|
444
|
+
notice = flag.previously_new_record? ? "Filed for review." : "It's already waiting for review."
|
|
445
|
+
redirect_to main_app.madmin_flag_path(flag), notice: notice, status: :see_other
|
|
446
|
+
end
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
`flag_for_review!` files a `source: "manual"` flag owned by the content's person, or returns the one already waiting on that field. At most one flag is pending per content and field, and the database enforces it (a unique index), so two moderators pressing it at once file one flag: the second gets the first's.
|
|
450
|
+
|
|
422
451
|
---
|
|
423
452
|
|
|
424
453
|
## The moderation-queue pattern (the index)
|
|
@@ -456,14 +485,15 @@ The headline trick of the model design: **a human admin and an ML/automation con
|
|
|
456
485
|
class AutoModerationJob < ApplicationJob
|
|
457
486
|
def perform
|
|
458
487
|
Moderate::Flag.pending.find_each do |flag|
|
|
459
|
-
next unless flag.scores[
|
|
460
|
-
|
|
488
|
+
next unless flag.scores["sexual/minors"].to_f >= 0.99 # only the unambiguous ones
|
|
489
|
+
# `by: nil`: an automated decision is never attributed to a person who didn't look.
|
|
490
|
+
flag.resolve!(by: nil, note: "Removed automatically: high-confidence sexual/minors", remove_content: true)
|
|
461
491
|
end
|
|
462
492
|
end
|
|
463
493
|
end
|
|
464
494
|
```
|
|
465
495
|
|
|
466
|
-
Because the job calls `resolve!` (not a raw `update`), the auto-decision is identical to a human one — atomic enforcement, `notify`/`audit` hooks, statement-of-reasons, appeal window. Whatever the job doesn't touch stays in `pending` for a human in madmin. One queue, two consumers, zero divergence.
|
|
496
|
+
Because the job calls `resolve!` (not a raw `update`), the auto-decision is identical to a human one — atomic enforcement, `notify`/`audit` hooks, statement-of-reasons, appeal window. (Declare the automation honestly: a decision taken by a job is "automated means" under DSA Art. 17(3)(c).) Whatever the job doesn't touch stays in `pending` for a human in madmin. One queue, two consumers, zero divergence.
|
|
467
497
|
|
|
468
498
|
---
|
|
469
499
|
|
|
@@ -474,7 +504,7 @@ To be explicit about the boundary this guide sits on:
|
|
|
474
504
|
| `moderate` ships (the primitives) | You build (the UI chrome) |
|
|
475
505
|
| --- | --- |
|
|
476
506
|
| The models (`Report`/`Flag`/`Appeal`/`Block`) | The madmin resources (columns, labels, fields) |
|
|
477
|
-
| The queue scopes (`.pending`, `.
|
|
507
|
+
| The queue scopes (`.pending`, `.actioned`, `.dismissed`, `.recent_first`) | The index/show screens & nav grouping |
|
|
478
508
|
| The decision methods (`resolve!`/`dismiss!`/`uphold!`/`reject!`) — atomic enforcement + notify + audit + appeal window | The buttons/forms that call them |
|
|
479
509
|
| The optional `Moderate::Moderation` controller concern | Auth (`current_user`, admin gate) |
|
|
480
510
|
| Helpers + the evidence snapshot on each record | Your branding, layout, extra columns |
|
data/docs/notifications.md
CHANGED
|
@@ -80,7 +80,7 @@ event.to_h # the whole envelope as a Hash (handy for logging, tests, and
|
|
|
80
80
|
Two fields do most of the work:
|
|
81
81
|
|
|
82
82
|
- **`event.recipients`** — already resolved to *the right people* for this event. For `report_received` it's the reporter (their receipt). For `report_decision` it's the reporter (the outcome). For `affected_user_decision` it's the person whose content/account was acted on (their statement of reasons). You don't compute audiences — `moderate` does, and hands you the array. Iterate it.
|
|
83
|
-
- **`event.payload[:summary]`** — a short, human, already-redaction-safe one-liner describing what happened (e.g. `"New harassment report on Comment #4213 by joh…e@example.com"`). It exists so your **admin Telegram ping is literally one line** — no string-building, no leaking. Everything richer (the model, the category, the moderator's note) is also in `payload` if you want to compose your own copy.
|
|
83
|
+
- **`event.payload[:summary]`** — a short, human, already-redaction-safe one-liner describing what happened (e.g. `"New harassment report on Comment #4213 by joh…e@example.com"`). It exists so your **admin Telegram ping is literally one line** — no string-building, no leaking. Everything richer (the model, the category, the moderator's note) is also in `payload` if you want to compose your own copy. Events about a specific field carry both the raw name and its human label (`reported_field` + `reported_field_label` on reports, `field` + `field_label` on flags) — print the label, keep the raw name for code (see [Reportable content](../README.md#-reportable-content)).
|
|
84
84
|
|
|
85
85
|
A recipient is usually one of your `User` records (it responds to `email`, `id`, etc.), **except** for DSA-notice events (`notice_received`, and the `affected_user_decision` for a public notice) where the notifier may be an anonymous person — there `moderate` gives you a lightweight recipient that responds to `email`/`name` but is **not** a `User`. The `next unless user.respond_to?(:email)` / `user.is_a?(User)` guards in the recipes above handle that cleanly.
|
|
86
86
|
|
|
@@ -137,6 +137,16 @@ end
|
|
|
137
137
|
class ModerationMailer < ApplicationMailer
|
|
138
138
|
# `event` and `recipient` arrive via `.with(...)` and are available as `params[:event]` / `params[:recipient]`.
|
|
139
139
|
|
|
140
|
+
# DSA Art. 20(2): the appeal window runs from when the person was informed. Confirm a
|
|
141
|
+
# decision notice when it actually leaves, not when deliver_later queued it.
|
|
142
|
+
after_deliver do
|
|
143
|
+
event = params[:event]
|
|
144
|
+
case event&.name
|
|
145
|
+
when :affected_user_decision then Moderate.delivered!(event.subject, :affected_user)
|
|
146
|
+
when :report_decision then Moderate.delivered!(event.subject, :decision)
|
|
147
|
+
end
|
|
148
|
+
end
|
|
149
|
+
|
|
140
150
|
def report_received
|
|
141
151
|
event = params[:event]
|
|
142
152
|
goodmail_mail(to: params[:recipient].email, from: "trust@myapp.com",
|
|
@@ -156,11 +166,13 @@ class ModerationMailer < ApplicationMailer
|
|
|
156
166
|
subject: "An update about your content") do
|
|
157
167
|
h1 "We took action on your content"
|
|
158
168
|
text event.payload[:reason] # the human-readable ground
|
|
169
|
+
info_row "Content", event.payload[:reported_field_label] # e.g. "Chat message" (nil for a whole-record report)
|
|
159
170
|
info_row "Action", event.payload[:action] # e.g. "Content removed"
|
|
160
171
|
info_row "Automated means", event.payload[:automated] ? "Yes" : "No"
|
|
161
172
|
space
|
|
162
173
|
text "If you believe this was a mistake, you can appeal — it's free and reviewed by a person."
|
|
163
|
-
button "Appeal this decision",
|
|
174
|
+
button "Appeal this decision", # the engine's appeal form, behind a signed link
|
|
175
|
+
moderate.new_appeal_url(token: event.subject.signed_appeal_gid, source: "affected_user")
|
|
164
176
|
sign
|
|
165
177
|
end
|
|
166
178
|
end
|
|
@@ -183,7 +195,8 @@ Why this shape:
|
|
|
183
195
|
- **`.with(event:, recipient:)`** is the standard Action Mailer params path, so the same arguments survive `deliver_later` serialization (the event's records are GlobalID-serializable AR objects).
|
|
184
196
|
- **`public_send(event.name)`** lets the *event name* select the mailer action — `report_received` → `#report_received` — so adding a new email is "add a method," not "add a branch."
|
|
185
197
|
- **`goodmail_mail`** is goodmail's mailer helper: it renders the DSL block, wires `List-Unsubscribe`, and hands a normal multipart message to Action Mailer. Use its DSL (`h1`, `text`, `info_row`, `button`, `sign`, …) instead of ERB templates. `info_row`/`price_row` are perfect for the reference/date/action rows a decision email needs.
|
|
186
|
-
- **The appeal link** in a decision email
|
|
198
|
+
- **The appeal link** in a decision email points at the engine's appeal form with the report's signed token: `moderate.new_appeal_url(token: event.subject.signed_appeal_gid, source: "affected_user")` (`source: "notifier"` in the reporter's `report_decision` email). The token names only that report and can't be replayed to find anything else; `event.payload[:appeal_deadline_at]` is a minimum: say "you can appeal at least until …".
|
|
199
|
+
- **Confirm delivery with `Moderate.delivered!`** (the `after_deliver` above). A decision notice your hook queued is not one the person has received: the gem stamps `decision_notified_at` / `affected_user_notified_at` only on that confirmation, once, and moves the appeal deadline to six months after it (never earlier). Without it, the deadline stays decision + six months. A notifier that isn't Action Mailer calls it wherever it learns the message went out (a provider webhook, a push receipt).
|
|
187
200
|
|
|
188
201
|
> [!TIP]
|
|
189
202
|
> Per-product or per-tenant branding? `goodmail_mail` accepts `config: { company_name:, brand_color:, logo_url: }` for a scoped, thread-local override — handy if your app is white-labeled. See goodmail's "Per-message branding."
|
|
@@ -351,7 +364,7 @@ Moderate.configure do |config|
|
|
|
351
364
|
end
|
|
352
365
|
```
|
|
353
366
|
|
|
354
|
-
`audit` fires for
|
|
367
|
+
`audit` fires for every intake, decision, block, ban and removal (including the silent ones like `user_blocked` that you usually don't notify on), plus the gem's own housekeeping: `report_received`, `appeal_received`, `report_decision`, `flag_decision`, `appeal_decision`, `flag_filed`, `user_blocked`, `user_unblocked`, `user_banned`, `content_removed`, `notify_failed`, `classification_skipped`, `retention_purged`, `legal_hold_placed`, `legal_hold_released` (`Moderate::Event::AUDIT_NAMES`). It's append-only by intent, and it's the trail behind the DSA **Art. 24 transparency counters** (notices received, actions taken, appeal outcomes), which `Moderate.transparency` computes from the moderation tables themselves. Wire it once next to `notify` and you have both halves: humans get told, history gets kept.
|
|
355
368
|
|
|
356
369
|
> [!NOTE]
|
|
357
370
|
> Same envelope (`name` / `subject` / `actor` / `recipients` / `payload` / `to_h`), two destinations. Keep both hooks fast and side-effect-light; push real work (emails, HTTP, heavy writes) to background jobs.
|
|
@@ -53,13 +53,13 @@ class AwsRekognitionAdapter
|
|
|
53
53
|
# change that default if a different fallback fits your app.
|
|
54
54
|
CATEGORY_MAP = {
|
|
55
55
|
"Explicit Nudity" => [:sexual, nil],
|
|
56
|
-
"Sexual"
|
|
56
|
+
"Sexual" => [:sexual, nil],
|
|
57
57
|
"Non-Explicit Nudity of Intimate parts and Kissing" => [:sexual, nil],
|
|
58
|
-
"Violence"
|
|
59
|
-
"Visually Disturbing" => [
|
|
60
|
-
"Hate Symbols"
|
|
58
|
+
"Violence" => [:violence, nil],
|
|
59
|
+
"Visually Disturbing" => %i[violence graphic],
|
|
60
|
+
"Hate Symbols" => [:hate, nil],
|
|
61
61
|
"Drugs & Tobacco" => [:illicit, nil],
|
|
62
|
-
"Gambling"
|
|
62
|
+
"Gambling" => [:illicit, nil]
|
|
63
63
|
}.freeze
|
|
64
64
|
|
|
65
65
|
def initialize(client: nil)
|
|
@@ -89,7 +89,7 @@ class AwsRekognitionAdapter
|
|
|
89
89
|
source: "image_filter",
|
|
90
90
|
raw: response.to_h
|
|
91
91
|
)
|
|
92
|
-
rescue => error
|
|
92
|
+
rescue StandardError => error
|
|
93
93
|
# FAIL OPEN on everything, same rationale as any network classifier: a moderation
|
|
94
94
|
# API is defense-in-depth, not a gatekeeper that should block uploads on an AWS
|
|
95
95
|
# blip. The image simply isn't auto-flagged this time; users can still report it.
|
|
@@ -67,7 +67,7 @@ class OpenAIModerationAdapter
|
|
|
67
67
|
source: "external_classifier",
|
|
68
68
|
raw: result.results
|
|
69
69
|
)
|
|
70
|
-
rescue => error
|
|
70
|
+
rescue StandardError => error
|
|
71
71
|
# FAIL OPEN on EVERYTHING (network error, timeout, auth failure, malformed
|
|
72
72
|
# response, …). A moderation API is best-effort defense-in-depth, not a
|
|
73
73
|
# gatekeeper that should take down user posting when OpenAI has a hiccup — and a
|