@kkuffour/solid-moderation-plugin 0.2.2 → 0.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,1422 @@
|
|
|
1
|
+
# Technical Specification: Content Moderation Plugin for Community Solid Server
|
|
2
|
+
|
|
3
|
+
**Version:** 1.1 (Revised)
|
|
4
|
+
**Date:** January 22, 2026
|
|
5
|
+
**Author:** Technical Architecture Team
|
|
6
|
+
**Status:** Ready for Implementation
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## Table of Contents
|
|
11
|
+
|
|
12
|
+
1. [Executive Summary](#executive-summary)
|
|
13
|
+
2. [System Overview](#system-overview)
|
|
14
|
+
3. [Architecture](#architecture)
|
|
15
|
+
4. [Component Specifications](#component-specifications)
|
|
16
|
+
5. [API Integration](#api-integration)
|
|
17
|
+
6. [Configuration](#configuration)
|
|
18
|
+
7. [Data Flow](#data-flow)
|
|
19
|
+
8. [Security & Privacy](#security--privacy)
|
|
20
|
+
9. [Error Handling](#error-handling)
|
|
21
|
+
10. [Testing Strategy](#testing-strategy)
|
|
22
|
+
11. [Deployment](#deployment)
|
|
23
|
+
12. [Performance Considerations](#performance-considerations)
|
|
24
|
+
13. [Future Enhancements](#future-enhancements)
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## 1. Executive Summary
|
|
29
|
+
|
|
30
|
+
This document specifies the design and implementation of a Content Moderation Plugin for Community Solid Server (CSS). The plugin provides automated content moderation capabilities for user-uploaded resources, leveraging external AI-powered moderation services while maintaining compliance with Solid protocol specifications.
|
|
31
|
+
|
|
32
|
+
### Key Features
|
|
33
|
+
|
|
34
|
+
- Automated content scanning for images, videos, and text
|
|
35
|
+
- Integration with SightEngine API for AI-powered moderation
|
|
36
|
+
- Configurable moderation policies per resource type
|
|
37
|
+
- Asynchronous processing to minimize performance impact
|
|
38
|
+
- Comprehensive logging and audit trails
|
|
39
|
+
- WebID-based access control integration
|
|
40
|
+
|
|
41
|
+
### Goals
|
|
42
|
+
|
|
43
|
+
- Protect users from harmful content (NSFW, violence, hate speech)
|
|
44
|
+
- Maintain platform compliance with legal requirements
|
|
45
|
+
- Preserve Solid's decentralized architecture principles
|
|
46
|
+
- Minimize latency impact on resource operations
|
|
47
|
+
|
|
48
|
+
### Architectural Approach
|
|
49
|
+
|
|
50
|
+
This plugin uses the `asynchronous-handlers` package (the foundation of CSS's handler architecture) rather than extending CSS internal classes directly. This approach ensures:
|
|
51
|
+
- **Stability**: No dependency on CSS internal implementation details
|
|
52
|
+
- **Compatibility**: Works across CSS versions that use the same AsyncHandler interface
|
|
53
|
+
- **Maintainability**: Clear separation between plugin and server internals
|
|
54
|
+
- **Flexibility**: Can be used with other AsyncHandler-based systems
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
## 2. System Overview
|
|
59
|
+
|
|
60
|
+
### 2.1 Scope
|
|
61
|
+
|
|
62
|
+
The Content Moderation Plugin operates as a middleware component within the CSS request handling pipeline, intercepting resource creation and update operations to perform content analysis before persistence.
|
|
63
|
+
|
|
64
|
+
### 2.2 Components
|
|
65
|
+
|
|
66
|
+
- **ModerationHandler**: Primary HTTP request interceptor (extends AsyncHandler)
|
|
67
|
+
- **ModerationService**: Business logic coordinator
|
|
68
|
+
- **SightEngineClient**: External API integration wrapper
|
|
69
|
+
- **ModerationStore**: Audit log and decision persistence
|
|
70
|
+
- **ConfigurationManager**: Policy and threshold management
|
|
71
|
+
|
|
72
|
+
### 2.3 Technology Stack
|
|
73
|
+
|
|
74
|
+
- **Runtime**: Node.js 18+ with TypeScript
|
|
75
|
+
- **Handler Framework**: asynchronous-handlers (standalone package)
|
|
76
|
+
- **Server Integration**: Community Solid Server via Components.js
|
|
77
|
+
- **External Service**: SightEngine Moderation API
|
|
78
|
+
- **Storage**: RDF/Turtle for audit logs
|
|
79
|
+
- **Configuration**: JSON-LD component configuration
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
83
|
+
## 3. Architecture
|
|
84
|
+
|
|
85
|
+
### 3.1 System Architecture Diagram
|
|
86
|
+
|
|
87
|
+
```
|
|
88
|
+
┌─────────────────────────────────────────────────────────┐
|
|
89
|
+
│ Client Request │
|
|
90
|
+
└────────────────────┬────────────────────────────────────┘
|
|
91
|
+
│
|
|
92
|
+
▼
|
|
93
|
+
┌─────────────────────────────────────────────────────────┐
|
|
94
|
+
│ CSS Request Pipeline │
|
|
95
|
+
│ ┌────────────────────────────────────────────────┐ │
|
|
96
|
+
│ │ ModerationHandler │ │
|
|
97
|
+
│ │ (extends AsyncHandler from asynchronous- │ │
|
|
98
|
+
│ │ handlers package) │ │
|
|
99
|
+
│ │ - Intercepts POST/PUT/PATCH requests │ │
|
|
100
|
+
│ │ - Extracts content for analysis │ │
|
|
101
|
+
│ └──────────────────┬─────────────────────────────┘ │
|
|
102
|
+
└────────────────────┼──────────────────────────────────┘
|
|
103
|
+
│
|
|
104
|
+
▼
|
|
105
|
+
┌─────────────────────────────────────────────────────────┐
|
|
106
|
+
│ ModerationService │
|
|
107
|
+
│ ┌──────────────────────────────────────────────┐ │
|
|
108
|
+
│ │ - Content type detection │ │
|
|
109
|
+
│ │ - Policy retrieval │ │
|
|
110
|
+
│ │ - Decision logic │ │
|
|
111
|
+
│ └──────────┬───────────────────────┬────────────┘ │
|
|
112
|
+
└────────────┼───────────────────────┼────────────────────┘
|
|
113
|
+
│ │
|
|
114
|
+
▼ ▼
|
|
115
|
+
┌─────────────────────┐ ┌──────────────────────┐
|
|
116
|
+
│ SightEngineClient │ │ ModerationStore │
|
|
117
|
+
│ - API calls │ │ - Audit logging │
|
|
118
|
+
│ - Response parsing │ │ - Decision cache │
|
|
119
|
+
│ - Rate limiting │ │ - RDF persistence │
|
|
120
|
+
└─────────────────────┘ └──────────────────────┘
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
### 3.2 Handler Architecture
|
|
124
|
+
|
|
125
|
+
The plugin uses the AsyncHandler pattern which is the foundation of CSS:
|
|
126
|
+
|
|
127
|
+
```
|
|
128
|
+
AsyncHandler<TIn, TOut>
|
|
129
|
+
├── canHandle(input: TIn): Promise<void>
|
|
130
|
+
├── handle(input: TIn): Promise<TOut>
|
|
131
|
+
└── handleSafe(input: TIn): Promise<TOut>
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
This pattern allows:
|
|
135
|
+
- **Chaining**: Handlers can be chained using WaterfallHandler
|
|
136
|
+
- **Parallelism**: Multiple handlers can run simultaneously
|
|
137
|
+
- **Composition**: Handlers can be composed into complex pipelines
|
|
138
|
+
- **Type Safety**: Full TypeScript support
|
|
139
|
+
|
|
140
|
+
### 3.3 Integration Points
|
|
141
|
+
|
|
142
|
+
**Upstream Integration:**
|
|
143
|
+
- CSS request pipeline via Components.js configuration
|
|
144
|
+
- Type imports from @solid/community-server (types only, not classes)
|
|
145
|
+
- AsyncHandler from asynchronous-handlers package
|
|
146
|
+
|
|
147
|
+
**Downstream Integration:**
|
|
148
|
+
- SightEngine REST API (HTTPS)
|
|
149
|
+
- CSS ResourceStore for audit logs
|
|
150
|
+
- Notification system for administrators
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## 4. Component Specifications
|
|
155
|
+
|
|
156
|
+
### 4.1 ModerationHandler
|
|
157
|
+
|
|
158
|
+
**Purpose:** Intercepts HTTP requests to trigger moderation workflow
|
|
159
|
+
|
|
160
|
+
**Implementation:**
|
|
161
|
+
```typescript
|
|
162
|
+
import { AsyncHandler } from 'asynchronous-handlers';
|
|
163
|
+
import type { Operation, ResponseDescription } from '@solid/community-server';
|
|
164
|
+
|
|
165
|
+
export class ModerationHandler extends AsyncHandler<Operation, ResponseDescription> {
|
|
166
|
+
private moderationService: ModerationService;
|
|
167
|
+
private config: ModerationConfig;
|
|
168
|
+
private nextHandler: AsyncHandler<Operation, ResponseDescription>;
|
|
169
|
+
|
|
170
|
+
constructor(
|
|
171
|
+
moderationService: ModerationService,
|
|
172
|
+
config: ModerationConfig,
|
|
173
|
+
nextHandler: AsyncHandler<Operation, ResponseDescription>
|
|
174
|
+
) {
|
|
175
|
+
super();
|
|
176
|
+
this.moderationService = moderationService;
|
|
177
|
+
this.config = config;
|
|
178
|
+
this.nextHandler = nextHandler;
|
|
179
|
+
}
|
|
180
|
+
|
|
181
|
+
async canHandle(operation: Operation): Promise<void> {
|
|
182
|
+
// Check if this operation needs moderation
|
|
183
|
+
if (!this.shouldModerate(operation)) {
|
|
184
|
+
throw new Error('Operation does not require moderation');
|
|
185
|
+
}
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
async handle(operation: Operation): Promise<ResponseDescription> {
|
|
189
|
+
// Extract content from operation
|
|
190
|
+
const content = await this.extractContent(operation);
|
|
191
|
+
const metadata = this.extractMetadata(operation);
|
|
192
|
+
|
|
193
|
+
// Moderate content
|
|
194
|
+
const decision = await this.moderationService.moderate(content, metadata);
|
|
195
|
+
|
|
196
|
+
// Apply decision
|
|
197
|
+
if (decision.verdict === 'rejected') {
|
|
198
|
+
throw new ForbiddenHttpError(decision.reason);
|
|
199
|
+
}
|
|
200
|
+
|
|
201
|
+
// Pass to next handler
|
|
202
|
+
return this.nextHandler.handle(operation);
|
|
203
|
+
}
|
|
204
|
+
|
|
205
|
+
private shouldModerate(operation: Operation): boolean {
|
|
206
|
+
// Check if method is in enabled methods
|
|
207
|
+
if (!this.config.enabledMethods.includes(operation.method)) {
|
|
208
|
+
return false;
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
// Check if path is excluded
|
|
212
|
+
for (const excludedPath of this.config.excludedPaths) {
|
|
213
|
+
if (operation.target.path.startsWith(excludedPath)) {
|
|
214
|
+
return false;
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
return true;
|
|
219
|
+
}
|
|
220
|
+
}
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
**Configuration Parameters:**
|
|
224
|
+
- `enabledMethods`: Array of HTTP methods to moderate
|
|
225
|
+
- `excludedPaths`: Resources exempt from moderation
|
|
226
|
+
- `asyncProcessing`: Enable background moderation
|
|
227
|
+
- `fallbackBehavior`: Action on service failure (allow/deny)
|
|
228
|
+
|
|
229
|
+
### 4.2 ModerationService
|
|
230
|
+
|
|
231
|
+
**Purpose:** Orchestrates moderation logic and policy enforcement
|
|
232
|
+
|
|
233
|
+
**Interface:**
|
|
234
|
+
```typescript
|
|
235
|
+
export interface ModerationService {
|
|
236
|
+
moderate(content: Buffer, metadata: ResourceMetadata): Promise<ModerationResult>;
|
|
237
|
+
getPolicy(resourceType: string): ModerationPolicy;
|
|
238
|
+
evaluateDecision(analysis: SightEngineResponse, policy: ModerationPolicy): ModerationDecision;
|
|
239
|
+
}
|
|
240
|
+
|
|
241
|
+
export interface ModerationResult {
|
|
242
|
+
verdict: 'approved' | 'rejected' | 'flagged';
|
|
243
|
+
reason: string;
|
|
244
|
+
confidence: number;
|
|
245
|
+
categories: string[];
|
|
246
|
+
timestamp: Date;
|
|
247
|
+
}
|
|
248
|
+
|
|
249
|
+
export interface ResourceMetadata {
|
|
250
|
+
contentType: string;
|
|
251
|
+
size: number;
|
|
252
|
+
owner: string;
|
|
253
|
+
targetPath: string;
|
|
254
|
+
}
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
**Key Methods:**
|
|
258
|
+
|
|
259
|
+
**`moderate(content, metadata)`**
|
|
260
|
+
- Detects content type (image/video/text)
|
|
261
|
+
- Retrieves applicable moderation policy
|
|
262
|
+
- Invokes SightEngineClient for analysis
|
|
263
|
+
- Applies policy thresholds to results
|
|
264
|
+
- Returns decision with justification
|
|
265
|
+
|
|
266
|
+
**`getPolicy(resourceType)`**
|
|
267
|
+
- Loads configuration from Components.js
|
|
268
|
+
- Supports per-type policies (default, image, video, text)
|
|
269
|
+
- Caches policies for performance
|
|
270
|
+
|
|
271
|
+
**`evaluateDecision(analysis, policy)`**
|
|
272
|
+
- Compares API scores against thresholds
|
|
273
|
+
- Generates detailed justification
|
|
274
|
+
- Applies multi-criteria logic (AND/OR conditions)
|
|
275
|
+
|
|
276
|
+
### 4.3 SightEngineClient
|
|
277
|
+
|
|
278
|
+
**Purpose:** Abstracts SightEngine API communication
|
|
279
|
+
|
|
280
|
+
**Interface:**
|
|
281
|
+
```typescript
|
|
282
|
+
export interface SightEngineClient {
|
|
283
|
+
checkImage(imageBuffer: Buffer): Promise<SightEngineResponse>;
|
|
284
|
+
checkVideo(videoUrl: string): Promise<SightEngineResponse>;
|
|
285
|
+
checkText(text: string): Promise<SightEngineResponse>;
|
|
286
|
+
}
|
|
287
|
+
|
|
288
|
+
export interface SightEngineResponse {
|
|
289
|
+
status: 'success' | 'failure';
|
|
290
|
+
nudity?: {
|
|
291
|
+
sexual_activity: number;
|
|
292
|
+
sexual_display: number;
|
|
293
|
+
erotica: number;
|
|
294
|
+
raw: number;
|
|
295
|
+
};
|
|
296
|
+
weapon?: number;
|
|
297
|
+
alcohol?: number;
|
|
298
|
+
drugs?: number;
|
|
299
|
+
offensive?: {
|
|
300
|
+
prob: number;
|
|
301
|
+
};
|
|
302
|
+
gore?: {
|
|
303
|
+
prob: number;
|
|
304
|
+
};
|
|
305
|
+
}
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
**Features:**
|
|
309
|
+
- Automatic retry with exponential backoff
|
|
310
|
+
- Rate limiting (respects SightEngine limits)
|
|
311
|
+
- Response caching for duplicate content
|
|
312
|
+
- Error handling and timeout management
|
|
313
|
+
|
|
314
|
+
**Configuration:**
|
|
315
|
+
- `apiKey`: SightEngine API key
|
|
316
|
+
- `apiSecret`: SightEngine API secret
|
|
317
|
+
- `baseUrl`: API endpoint (default: api.sightengine.com)
|
|
318
|
+
- `timeout`: Request timeout (default: 30s)
|
|
319
|
+
- `maxRetries`: Retry attempts (default: 3)
|
|
320
|
+
|
|
321
|
+
### 4.4 ModerationStore
|
|
322
|
+
|
|
323
|
+
**Purpose:** Persists moderation decisions for audit and appeal
|
|
324
|
+
|
|
325
|
+
**Interface:**
|
|
326
|
+
```typescript
|
|
327
|
+
export interface ModerationStore {
|
|
328
|
+
save(decision: ModerationDecision): Promise<void>;
|
|
329
|
+
find(resourceUrl: string): Promise<ModerationDecision | undefined>;
|
|
330
|
+
list(filter: ModerationFilter): Promise<ModerationDecision[]>;
|
|
331
|
+
}
|
|
332
|
+
|
|
333
|
+
export interface ModerationDecision {
|
|
334
|
+
id: string;
|
|
335
|
+
resourceUrl: string;
|
|
336
|
+
verdict: 'approved' | 'rejected' | 'flagged';
|
|
337
|
+
reason: string;
|
|
338
|
+
categories: string[];
|
|
339
|
+
confidence: number;
|
|
340
|
+
timestamp: Date;
|
|
341
|
+
moderator: string; // WebID or 'system'
|
|
342
|
+
appealed: boolean;
|
|
343
|
+
}
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
**Storage Schema (RDF/Turtle):**
|
|
347
|
+
```turtle
|
|
348
|
+
@prefix mod: <http://www.w3.org/ns/solid/moderation#> .
|
|
349
|
+
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
|
|
350
|
+
|
|
351
|
+
<#decision-123> a mod:ModerationDecision ;
|
|
352
|
+
mod:resource <https://pod.example/user/image.jpg> ;
|
|
353
|
+
mod:timestamp "2026-01-22T10:30:00Z"^^xsd:dateTime ;
|
|
354
|
+
mod:verdict "rejected" ;
|
|
355
|
+
mod:reason "NSFW content detected (confidence: 0.95)" ;
|
|
356
|
+
mod:moderator <https://moderation.example/agent#ai> ;
|
|
357
|
+
mod:appealed false .
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
---
|
|
361
|
+
|
|
362
|
+
## 5. API Integration
|
|
363
|
+
|
|
364
|
+
### 5.1 SightEngine API Endpoints
|
|
365
|
+
|
|
366
|
+
**Image Check:**
|
|
367
|
+
```
|
|
368
|
+
POST https://api.sightengine.com/1.0/check.json
|
|
369
|
+
Content-Type: multipart/form-data
|
|
370
|
+
|
|
371
|
+
Parameters:
|
|
372
|
+
- media: image file (binary)
|
|
373
|
+
- models: nudity,wad,offensive,gore
|
|
374
|
+
- api_user: {apiKey}
|
|
375
|
+
- api_secret: {apiSecret}
|
|
376
|
+
```
|
|
377
|
+
|
|
378
|
+
**Response Format:**
|
|
379
|
+
```json
|
|
380
|
+
{
|
|
381
|
+
"status": "success",
|
|
382
|
+
"nudity": {
|
|
383
|
+
"sexual_activity": 0.02,
|
|
384
|
+
"sexual_display": 0.01,
|
|
385
|
+
"erotica": 0.05,
|
|
386
|
+
"raw": 0.91
|
|
387
|
+
},
|
|
388
|
+
"weapon": 0.01,
|
|
389
|
+
"alcohol": 0.03,
|
|
390
|
+
"drugs": 0.02,
|
|
391
|
+
"offensive": {
|
|
392
|
+
"prob": 0.05
|
|
393
|
+
},
|
|
394
|
+
"gore": {
|
|
395
|
+
"prob": 0.02
|
|
396
|
+
}
|
|
397
|
+
}
|
|
398
|
+
```
|
|
399
|
+
|
|
400
|
+
### 5.2 Moderation Models
|
|
401
|
+
|
|
402
|
+
**Available Models:**
|
|
403
|
+
- `nudity`: Detects nudity, sexual content
|
|
404
|
+
- `wad`: Weapons, alcohol, drugs
|
|
405
|
+
- `offensive`: Hate symbols, gestures
|
|
406
|
+
- `gore`: Violence, graphic content
|
|
407
|
+
- `text`: Profanity, hate speech (for text analysis)
|
|
408
|
+
|
|
409
|
+
### 5.3 Error Handling
|
|
410
|
+
|
|
411
|
+
**API Error Codes:**
|
|
412
|
+
- `401`: Invalid credentials → Fail closed (reject upload)
|
|
413
|
+
- `429`: Rate limit exceeded → Queue for later processing
|
|
414
|
+
- `500`: Server error → Retry with backoff
|
|
415
|
+
- `timeout`: Network timeout → Retry or fail open (configurable)
|
|
416
|
+
|
|
417
|
+
---
|
|
418
|
+
|
|
419
|
+
## 6. Configuration
|
|
420
|
+
|
|
421
|
+
### 6.1 Package Dependencies
|
|
422
|
+
|
|
423
|
+
**package.json:**
|
|
424
|
+
```json
|
|
425
|
+
{
|
|
426
|
+
"name": "@solid-contrib/moderation-plugin",
|
|
427
|
+
"version": "1.0.0",
|
|
428
|
+
"dependencies": {
|
|
429
|
+
"asynchronous-handlers": "^1.0.0",
|
|
430
|
+
"@solid/community-server": "^7.0.0",
|
|
431
|
+
"axios": "^1.6.0",
|
|
432
|
+
"componentsjs": "^6.0.0"
|
|
433
|
+
},
|
|
434
|
+
"devDependencies": {
|
|
435
|
+
"@types/node": "^20.0.0",
|
|
436
|
+
"typescript": "^5.0.0",
|
|
437
|
+
"jest": "^29.0.0"
|
|
438
|
+
}
|
|
439
|
+
}
|
|
440
|
+
```
|
|
441
|
+
|
|
442
|
+
### 6.2 Components.js Configuration
|
|
443
|
+
|
|
444
|
+
**Example Configuration:**
|
|
445
|
+
```json
|
|
446
|
+
{
|
|
447
|
+
"@context": [
|
|
448
|
+
"https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld",
|
|
449
|
+
"https://linkedsoftwaredependencies.org/bundles/npm/@solid-contrib/moderation-plugin/^1.0.0/components/context.jsonld"
|
|
450
|
+
],
|
|
451
|
+
"@graph": [
|
|
452
|
+
{
|
|
453
|
+
"@id": "urn:moderation:default:ModerationHandler",
|
|
454
|
+
"@type": "ModerationHandler",
|
|
455
|
+
"enabledMethods": ["POST", "PUT"],
|
|
456
|
+
"excludedPaths": ["/system/", "/.well-known/"],
|
|
457
|
+
"asyncProcessing": false,
|
|
458
|
+
"fallbackBehavior": "allow",
|
|
459
|
+
"moderationService": {
|
|
460
|
+
"@id": "urn:moderation:default:ModerationService"
|
|
461
|
+
},
|
|
462
|
+
"nextHandler": {
|
|
463
|
+
"@id": "urn:solid-server:default:ResourceStore"
|
|
464
|
+
}
|
|
465
|
+
},
|
|
466
|
+
{
|
|
467
|
+
"@id": "urn:moderation:default:ModerationService",
|
|
468
|
+
"@type": "ModerationService",
|
|
469
|
+
"policies": {
|
|
470
|
+
"default": {
|
|
471
|
+
"enabled": true,
|
|
472
|
+
"thresholds": {
|
|
473
|
+
"nudity.raw": 0.7,
|
|
474
|
+
"weapon": 0.8,
|
|
475
|
+
"gore.prob": 0.8,
|
|
476
|
+
"offensive.prob": 0.7
|
|
477
|
+
},
|
|
478
|
+
"action": "reject"
|
|
479
|
+
},
|
|
480
|
+
"image/jpeg": {
|
|
481
|
+
"enabled": true,
|
|
482
|
+
"models": ["nudity", "wad", "gore", "offensive"],
|
|
483
|
+
"thresholds": {
|
|
484
|
+
"nudity.sexual_display": 0.6,
|
|
485
|
+
"weapon": 0.75
|
|
486
|
+
}
|
|
487
|
+
}
|
|
488
|
+
},
|
|
489
|
+
"sightEngineClient": {
|
|
490
|
+
"@id": "urn:moderation:default:SightEngineClient"
|
|
491
|
+
}
|
|
492
|
+
},
|
|
493
|
+
{
|
|
494
|
+
"@id": "urn:moderation:default:SightEngineClient",
|
|
495
|
+
"@type": "SightEngineClient",
|
|
496
|
+
"apiKey": "@ENV:SIGHTENGINE_API_KEY",
|
|
497
|
+
"apiSecret": "@ENV:SIGHTENGINE_API_SECRET",
|
|
498
|
+
"timeout": 30000,
|
|
499
|
+
"maxRetries": 3
|
|
500
|
+
}
|
|
501
|
+
]
|
|
502
|
+
}
|
|
503
|
+
```
|
|
504
|
+
|
|
505
|
+
### 6.3 Injecting into CSS Pipeline
|
|
506
|
+
|
|
507
|
+
**Modifying existing CSS configuration:**
|
|
508
|
+
|
|
509
|
+
To add moderation to an existing CSS instance, you need to insert the ModerationHandler into the handler chain before the ResourceStore. This can be done by:
|
|
510
|
+
|
|
511
|
+
1. **Using WaterfallHandler** (from asynchronous-handlers):
|
|
512
|
+
|
|
513
|
+
```json
|
|
514
|
+
{
|
|
515
|
+
"@id": "urn:solid-server:default:HttpHandler",
|
|
516
|
+
"@type": "WaterfallHandler",
|
|
517
|
+
"handlers": [
|
|
518
|
+
{ "@id": "urn:moderation:default:ModerationHandler" },
|
|
519
|
+
{ "@id": "urn:solid-server:default:OriginalHttpHandler" }
|
|
520
|
+
]
|
|
521
|
+
}
|
|
522
|
+
```
|
|
523
|
+
|
|
524
|
+
2. **Or setting nextHandler on ModerationHandler**:
|
|
525
|
+
|
|
526
|
+
```json
|
|
527
|
+
{
|
|
528
|
+
"@id": "urn:moderation:default:ModerationHandler",
|
|
529
|
+
"@type": "ModerationHandler",
|
|
530
|
+
"nextHandler": {
|
|
531
|
+
"@id": "urn:solid-server:default:ResourceStore"
|
|
532
|
+
}
|
|
533
|
+
}
|
|
534
|
+
```
|
|
535
|
+
|
|
536
|
+
### 6.4 Environment Variables
|
|
537
|
+
|
|
538
|
+
```bash
|
|
539
|
+
# SightEngine API credentials
|
|
540
|
+
SIGHTENGINE_API_KEY=your_api_key
|
|
541
|
+
SIGHTENGINE_API_SECRET=your_api_secret
|
|
542
|
+
|
|
543
|
+
# Moderation settings
|
|
544
|
+
MODERATION_ENABLED=true
|
|
545
|
+
MODERATION_ASYNC=false
|
|
546
|
+
MODERATION_FALLBACK=allow # allow|deny
|
|
547
|
+
|
|
548
|
+
# Logging
|
|
549
|
+
MODERATION_LOG_LEVEL=info # debug|info|warn|error
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
### 6.5 Policy Configuration
|
|
553
|
+
|
|
554
|
+
**Policy Structure:**
|
|
555
|
+
```typescript
|
|
556
|
+
interface ModerationPolicy {
|
|
557
|
+
enabled: boolean;
|
|
558
|
+
models: string[]; // SightEngine models to use
|
|
559
|
+
thresholds: Record<string, number>; // Score thresholds (0-1)
|
|
560
|
+
action: 'allow' | 'reject' | 'flag';
|
|
561
|
+
logDecisions: boolean;
|
|
562
|
+
notifyAdmin: boolean;
|
|
563
|
+
}
|
|
564
|
+
```
|
|
565
|
+
|
|
566
|
+
**Threshold Semantics:**
|
|
567
|
+
- Scores >= threshold trigger the action
|
|
568
|
+
- Multiple thresholds evaluated with OR logic
|
|
569
|
+
- Set threshold to 1.0 to disable a check
|
|
570
|
+
|
|
571
|
+
---
|
|
572
|
+
|
|
573
|
+
## 7. Data Flow
|
|
574
|
+
|
|
575
|
+
### 7.1 Request Processing Flow
|
|
576
|
+
|
|
577
|
+
```
|
|
578
|
+
1. Client uploads resource (POST /user/image.jpg)
|
|
579
|
+
↓
|
|
580
|
+
2. CSS routes to handler chain
|
|
581
|
+
↓
|
|
582
|
+
3. ModerationHandler.canHandle() checks if moderation needed
|
|
583
|
+
↓
|
|
584
|
+
4. ModerationHandler.handle() called
|
|
585
|
+
↓
|
|
586
|
+
5. Extract content + metadata from request
|
|
587
|
+
↓
|
|
588
|
+
6. ModerationService.moderate(content, metadata)
|
|
589
|
+
↓
|
|
590
|
+
7. Detect content type (image/jpeg)
|
|
591
|
+
↓
|
|
592
|
+
8. Retrieve policy for image/jpeg
|
|
593
|
+
↓
|
|
594
|
+
9. SightEngineClient.checkImage(buffer)
|
|
595
|
+
↓
|
|
596
|
+
10. Parse API response scores
|
|
597
|
+
↓
|
|
598
|
+
11. Evaluate against thresholds
|
|
599
|
+
↓
|
|
600
|
+
12. Generate ModerationDecision
|
|
601
|
+
↓
|
|
602
|
+
13. Store decision in ModerationStore
|
|
603
|
+
↓
|
|
604
|
+
14. Return decision to ModerationHandler
|
|
605
|
+
↓
|
|
606
|
+
15a. If APPROVED: nextHandler.handle(operation)
|
|
607
|
+
15b. If REJECTED: throw ForbiddenHttpError
|
|
608
|
+
15c. If FLAGGED: nextHandler.handle() + notify admin
|
|
609
|
+
```
|
|
610
|
+
|
|
611
|
+
### 7.2 Handler Chain Example
|
|
612
|
+
|
|
613
|
+
```typescript
|
|
614
|
+
// CSS sets up the chain via Components.js:
|
|
615
|
+
const pipeline = new WaterfallHandler([
|
|
616
|
+
authenticationHandler,
|
|
617
|
+
authorizationHandler,
|
|
618
|
+
moderationHandler, // ← Our plugin inserted here
|
|
619
|
+
resourceStore
|
|
620
|
+
]);
|
|
621
|
+
|
|
622
|
+
// When a request comes in:
|
|
623
|
+
await pipeline.handleSafe(operation);
|
|
624
|
+
// The WaterfallHandler calls each handler's canHandle()
|
|
625
|
+
// until one succeeds, then calls its handle()
|
|
626
|
+
```
|
|
627
|
+
|
|
628
|
+
---
|
|
629
|
+
|
|
630
|
+
## 8. Security & Privacy
|
|
631
|
+
|
|
632
|
+
### 8.1 Data Protection
|
|
633
|
+
|
|
634
|
+
**Principles:**
|
|
635
|
+
- Minimal data transmission: Only content being moderated sent to SightEngine
|
|
636
|
+
- No permanent storage on third-party servers
|
|
637
|
+
- Audit logs stored in user's pod (if permitted)
|
|
638
|
+
- Encrypted API communication (TLS 1.3)
|
|
639
|
+
|
|
640
|
+
### 8.2 Access Control
|
|
641
|
+
|
|
642
|
+
**Moderation System Access:**
|
|
643
|
+
- Only resources the requesting user has write access to are moderated
|
|
644
|
+
- Moderation decisions inherit pod's access control
|
|
645
|
+
- Admin interface requires special administrative role
|
|
646
|
+
- Audit logs protected by WebID authentication
|
|
647
|
+
|
|
648
|
+
### 8.3 API Key Security
|
|
649
|
+
|
|
650
|
+
- Store credentials in environment variables
|
|
651
|
+
- Never log API keys or secrets
|
|
652
|
+
- Rotate keys periodically
|
|
653
|
+
- Use separate keys for dev/staging/production
|
|
654
|
+
- Monitor API usage for anomalies
|
|
655
|
+
|
|
656
|
+
### 8.4 Privacy Considerations
|
|
657
|
+
|
|
658
|
+
**User Rights:**
|
|
659
|
+
- Users can request moderation decision details
|
|
660
|
+
- Appeal process for rejected content
|
|
661
|
+
- Option to disable moderation (admin only)
|
|
662
|
+
- Audit log retention policy (default: 90 days)
|
|
663
|
+
|
|
664
|
+
**Data Minimization:**
|
|
665
|
+
- Don't send metadata unnecessarily
|
|
666
|
+
- Redact sensitive information from logs
|
|
667
|
+
- Delete temporary files immediately after processing
|
|
668
|
+
|
|
669
|
+
---
|
|
670
|
+
|
|
671
|
+
## 9. Error Handling
|
|
672
|
+
|
|
673
|
+
### 9.1 Error Categories
|
|
674
|
+
|
|
675
|
+
**1. API Errors**
|
|
676
|
+
```typescript
|
|
677
|
+
class ModerationAPIError extends Error {
|
|
678
|
+
constructor(
|
|
679
|
+
public statusCode: number,
|
|
680
|
+
public apiResponse: unknown,
|
|
681
|
+
message: string
|
|
682
|
+
) {
|
|
683
|
+
super(message);
|
|
684
|
+
}
|
|
685
|
+
}
|
|
686
|
+
|
|
687
|
+
// Handle in SightEngineClient
|
|
688
|
+
async checkImage(buffer: Buffer): Promise<SightEngineResponse> {
|
|
689
|
+
try {
|
|
690
|
+
const response = await this.apiClient.post('/check.json', formData);
|
|
691
|
+
return response.data;
|
|
692
|
+
} catch (error) {
|
|
693
|
+
if (error.response?.status === 429) {
|
|
694
|
+
throw new ModerationAPIError(429, error.response.data, 'Rate limit exceeded');
|
|
695
|
+
}
|
|
696
|
+
throw new ModerationAPIError(500, null, 'API request failed');
|
|
697
|
+
}
|
|
698
|
+
}
|
|
699
|
+
```
|
|
700
|
+
|
|
701
|
+
**2. Configuration Errors**
|
|
702
|
+
- Invalid policy configuration → Fail server startup
|
|
703
|
+
- Missing API credentials → Fail server startup
|
|
704
|
+
- Invalid threshold values → Log warning, use defaults
|
|
705
|
+
|
|
706
|
+
**3. Content Processing Errors**
|
|
707
|
+
- Unsupported content type → Skip moderation (allow)
|
|
708
|
+
- Corrupted file → Reject with 400 Bad Request
|
|
709
|
+
- File too large → Reject with 413 Payload Too Large
|
|
710
|
+
|
|
711
|
+
### 9.2 Fallback Strategies
|
|
712
|
+
|
|
713
|
+
**Service Unavailable:**
|
|
714
|
+
```typescript
|
|
715
|
+
interface FallbackConfig {
|
|
716
|
+
behavior: 'allow' | 'deny' | 'queue';
|
|
717
|
+
maxQueueSize: number;
|
|
718
|
+
retryInterval: number;
|
|
719
|
+
}
|
|
720
|
+
|
|
721
|
+
// In ModerationHandler
|
|
722
|
+
async handle(operation: Operation): Promise<ResponseDescription> {
|
|
723
|
+
try {
|
|
724
|
+
const decision = await this.service.moderate(content, metadata);
|
|
725
|
+
return this.applyDecision(decision, operation);
|
|
726
|
+
} catch (error) {
|
|
727
|
+
if (this.config.fallbackBehavior === 'allow') {
|
|
728
|
+
this.logger.warn('Moderation failed, allowing content', error);
|
|
729
|
+
return this.nextHandler.handle(operation);
|
|
730
|
+
} else if (this.config.fallbackBehavior === 'deny') {
|
|
731
|
+
throw new ForbiddenHttpError('Moderation service unavailable');
|
|
732
|
+
} else {
|
|
733
|
+
return this.queueForLater(operation);
|
|
734
|
+
}
|
|
735
|
+
}
|
|
736
|
+
}
|
|
737
|
+
```
|
|
738
|
+
|
|
739
|
+
### 9.3 User-Facing Errors
|
|
740
|
+
|
|
741
|
+
**Rejection Response:**
|
|
742
|
+
```http
|
|
743
|
+
HTTP/1.1 403 Forbidden
|
|
744
|
+
Content-Type: application/json
|
|
745
|
+
|
|
746
|
+
{
|
|
747
|
+
"error": "Forbidden",
|
|
748
|
+
"message": "Resource rejected by content moderation",
|
|
749
|
+
"details": {
|
|
750
|
+
"reason": "Content violates community guidelines",
|
|
751
|
+
"categories": ["nudity"],
|
|
752
|
+
"appealUrl": "https://pod.example/.moderation/appeal"
|
|
753
|
+
}
|
|
754
|
+
}
|
|
755
|
+
```
|
|
756
|
+
|
|
757
|
+
---
|
|
758
|
+
|
|
759
|
+
## 10. Testing Strategy
|
|
760
|
+
|
|
761
|
+
### 10.1 Unit Tests
|
|
762
|
+
|
|
763
|
+
**Components to Test:**
|
|
764
|
+
- ModerationHandler request filtering
|
|
765
|
+
- ModerationService policy evaluation
|
|
766
|
+
- SightEngineClient API communication
|
|
767
|
+
- ModerationStore persistence
|
|
768
|
+
|
|
769
|
+
**Test Cases:**
|
|
770
|
+
```typescript
|
|
771
|
+
import { ModerationHandler } from '../src/ModerationHandler';
|
|
772
|
+
import { AsyncHandler } from 'asynchronous-handlers';
|
|
773
|
+
|
|
774
|
+
describe('ModerationHandler', () => {
|
|
775
|
+
let handler: ModerationHandler;
|
|
776
|
+
let mockService: jest.Mocked<ModerationService>;
|
|
777
|
+
let mockNextHandler: jest.Mocked<AsyncHandler<Operation, ResponseDescription>>;
|
|
778
|
+
|
|
779
|
+
beforeEach(() => {
|
|
780
|
+
mockService = createMockService();
|
|
781
|
+
mockNextHandler = createMockHandler();
|
|
782
|
+
handler = new ModerationHandler(mockService, config, mockNextHandler);
|
|
783
|
+
});
|
|
784
|
+
|
|
785
|
+
describe('canHandle', () => {
|
|
786
|
+
it('should succeed for POST requests', async () => {
|
|
787
|
+
const operation = { method: 'POST', target: { path: '/test' } };
|
|
788
|
+
await expect(handler.canHandle(operation)).resolves.toBeUndefined();
|
|
789
|
+
});
|
|
790
|
+
|
|
791
|
+
it('should fail for excluded paths', async () => {
|
|
792
|
+
const operation = { method: 'POST', target: { path: '/system/test' } };
|
|
793
|
+
await expect(handler.canHandle(operation)).rejects.toThrow();
|
|
794
|
+
});
|
|
795
|
+
});
|
|
796
|
+
|
|
797
|
+
describe('handle', () => {
|
|
798
|
+
it('should reject content exceeding thresholds', async () => {
|
|
799
|
+
mockService.moderate.mockResolvedValue({
|
|
800
|
+
verdict: 'rejected',
|
|
801
|
+
reason: 'NSFW content'
|
|
802
|
+
});
|
|
803
|
+
|
|
804
|
+
const operation = createTestOperation();
|
|
805
|
+
await expect(handler.handle(operation)).rejects.toThrow(ForbiddenHttpError);
|
|
806
|
+
});
|
|
807
|
+
|
|
808
|
+
it('should pass approved content to next handler', async () => {
|
|
809
|
+
mockService.moderate.mockResolvedValue({
|
|
810
|
+
verdict: 'approved'
|
|
811
|
+
});
|
|
812
|
+
|
|
813
|
+
const operation = createTestOperation();
|
|
814
|
+
const result = await handler.handle(operation);
|
|
815
|
+
|
|
816
|
+
expect(mockNextHandler.handle).toHaveBeenCalledWith(operation);
|
|
817
|
+
});
|
|
818
|
+
});
|
|
819
|
+
});
|
|
820
|
+
|
|
821
|
+
describe('ModerationService', () => {
|
|
822
|
+
describe('evaluateDecision', () => {
|
|
823
|
+
it('should reject content exceeding nudity threshold', () => {
|
|
824
|
+
const analysis = { nudity: { raw: 0.9 } };
|
|
825
|
+
const policy = { thresholds: { 'nudity.raw': 0.7 } };
|
|
826
|
+
const decision = service.evaluateDecision(analysis, policy);
|
|
827
|
+
expect(decision.verdict).toBe('rejected');
|
|
828
|
+
});
|
|
829
|
+
|
|
830
|
+
it('should allow content below all thresholds', () => {
|
|
831
|
+
const analysis = { nudity: { raw: 0.3 }, weapon: 0.1 };
|
|
832
|
+
const policy = { thresholds: { 'nudity.raw': 0.7, weapon: 0.8 } };
|
|
833
|
+
const decision = service.evaluateDecision(analysis, policy);
|
|
834
|
+
expect(decision.verdict).toBe('approved');
|
|
835
|
+
});
|
|
836
|
+
});
|
|
837
|
+
});
|
|
838
|
+
```
|
|
839
|
+
|
|
840
|
+
### 10.2 Integration Tests
|
|
841
|
+
|
|
842
|
+
**Scenarios:**
|
|
843
|
+
1. Upload safe image → Should succeed
|
|
844
|
+
2. Upload NSFW image → Should be rejected
|
|
845
|
+
3. Upload with API unavailable → Should follow fallback behavior
|
|
846
|
+
4. Upload to excluded path → Should skip moderation
|
|
847
|
+
5. Appeal rejected decision → Should update decision record
|
|
848
|
+
|
|
849
|
+
**Test Setup:**
|
|
850
|
+
```typescript
|
|
851
|
+
import { AppRunner } from '@solid/community-server';
|
|
852
|
+
|
|
853
|
+
describe('Moderation Integration', () => {
|
|
854
|
+
let app: AppRunner;
|
|
855
|
+
let mockSightEngine: MockAdapter;
|
|
856
|
+
|
|
857
|
+
beforeAll(async () => {
|
|
858
|
+
mockSightEngine = new MockAdapter(axios);
|
|
859
|
+
app = await AppRunner.create({
|
|
860
|
+
mainModulePath: testConfigPath,
|
|
861
|
+
variables: {
|
|
862
|
+
'urn:moderation:default:SightEngineClient': createMockClient(mockSightEngine)
|
|
863
|
+
}
|
|
864
|
+
});
|
|
865
|
+
await app.start();
|
|
866
|
+
});
|
|
867
|
+
|
|
868
|
+
afterAll(async () => {
|
|
869
|
+
await app.stop();
|
|
870
|
+
});
|
|
871
|
+
|
|
872
|
+
it('should reject NSFW image upload', async () => {
|
|
873
|
+
mockSightEngine.onPost('/check.json').reply(200, {
|
|
874
|
+
status: 'success',
|
|
875
|
+
nudity: { raw: 0.95 }
|
|
876
|
+
});
|
|
877
|
+
|
|
878
|
+
const response = await fetch('http://localhost:3000/user/test.jpg', {
|
|
879
|
+
method: 'POST',
|
|
880
|
+
body: nsfwImageBuffer
|
|
881
|
+
});
|
|
882
|
+
|
|
883
|
+
expect(response.status).toBe(403);
|
|
884
|
+
const body = await response.json();
|
|
885
|
+
expect(body.details.categories).toContain('nudity');
|
|
886
|
+
});
|
|
887
|
+
});
|
|
888
|
+
```
|
|
889
|
+
|
|
890
|
+
### 10.3 Performance Tests
|
|
891
|
+
|
|
892
|
+
**Metrics to Measure:**
|
|
893
|
+
- Moderation latency (target: <500ms for images)
|
|
894
|
+
- Throughput (requests/second with moderation)
|
|
895
|
+
- Resource usage (memory, CPU during moderation)
|
|
896
|
+
- API rate limit compliance
|
|
897
|
+
|
|
898
|
+
**Load Testing:**
|
|
899
|
+
```typescript
|
|
900
|
+
// Using k6 or similar
|
|
901
|
+
import http from 'k6/http';
|
|
902
|
+
import { check } from 'k6';
|
|
903
|
+
|
|
904
|
+
export let options = {
|
|
905
|
+
stages: [
|
|
906
|
+
{ duration: '2m', target: 50 }, // Ramp up
|
|
907
|
+
{ duration: '5m', target: 50 }, // Stay at 50 users
|
|
908
|
+
{ duration: '2m', target: 0 }, // Ramp down
|
|
909
|
+
],
|
|
910
|
+
};
|
|
911
|
+
|
|
912
|
+
export default function() {
|
|
913
|
+
const image = open('./test-image.jpg', 'b');
|
|
914
|
+
const response = http.post('https://pod.test/user/image.jpg', {
|
|
915
|
+
file: http.file(image, 'image.jpg'),
|
|
916
|
+
});
|
|
917
|
+
|
|
918
|
+
check(response, {
|
|
919
|
+
'status is 200 or 403': (r) => [200, 403].includes(r.status),
|
|
920
|
+
'response time < 1s': (r) => r.timings.duration < 1000,
|
|
921
|
+
});
|
|
922
|
+
}
|
|
923
|
+
```
|
|
924
|
+
|
|
925
|
+
---
|
|
926
|
+
|
|
927
|
+
## 11. Deployment
|
|
928
|
+
|
|
929
|
+
### 11.1 Installation
|
|
930
|
+
|
|
931
|
+
**NPM Package (Future):**
|
|
932
|
+
```bash
|
|
933
|
+
npm install @solid-contrib/moderation-plugin
|
|
934
|
+
```
|
|
935
|
+
|
|
936
|
+
**Manual Installation:**
|
|
937
|
+
1. Copy plugin code to CSS installation
|
|
938
|
+
2. Install dependencies: `npm install asynchronous-handlers`
|
|
939
|
+
3. Update Components.js configuration
|
|
940
|
+
4. Set environment variables
|
|
941
|
+
5. Restart server
|
|
942
|
+
|
|
943
|
+
### 11.2 Project Structure
|
|
944
|
+
|
|
945
|
+
Following the hello-world-component pattern:
|
|
946
|
+
|
|
947
|
+
```
|
|
948
|
+
css-moderation-plugin/
|
|
949
|
+
├── package.json # With lsd:module config
|
|
950
|
+
├── tsconfig.json
|
|
951
|
+
├── .componentsignore
|
|
952
|
+
├── src/
|
|
953
|
+
│ ├── index.ts # Export all classes
|
|
954
|
+
│ ├── ModerationHandler.ts # Extends AsyncHandler
|
|
955
|
+
│ ├── ModerationService.ts
|
|
956
|
+
│ ├── SightEngineClient.ts
|
|
957
|
+
│ └── ModerationStore.ts
|
|
958
|
+
├── config/
|
|
959
|
+
│ └── moderation.json # Component instantiation
|
|
960
|
+
├── test/
|
|
961
|
+
│ ├── unit/
|
|
962
|
+
│ │ ├── ModerationHandler.test.ts
|
|
963
|
+
│ │ └── ModerationService.test.ts
|
|
964
|
+
│ └── integration/
|
|
965
|
+
│ └── Server.test.ts
|
|
966
|
+
├── moderation-file.json # Full CSS config
|
|
967
|
+
└── moderation-partial.json # Partial config
|
|
968
|
+
```
|
|
969
|
+
|
|
970
|
+
### 11.3 package.json Configuration
|
|
971
|
+
|
|
972
|
+
```json
|
|
973
|
+
{
|
|
974
|
+
"name": "@solid-contrib/moderation-plugin",
|
|
975
|
+
"version": "1.0.0",
|
|
976
|
+
"main": "./dist/index.js",
|
|
977
|
+
"types": "./dist/index.d.ts",
|
|
978
|
+
"lsd:module": "https://linkedsoftwaredependencies.org/bundles/npm/@solid-contrib/moderation-plugin",
|
|
979
|
+
"lsd:components": "dist/components/components.jsonld",
|
|
980
|
+
"lsd:contexts": {
|
|
981
|
+
"https://linkedsoftwaredependencies.org/bundles/npm/@solid-contrib/moderation-plugin/^1.0.0/components/context.jsonld": "dist/components/context.jsonld"
|
|
982
|
+
},
|
|
983
|
+
"lsd:importPaths": {
|
|
984
|
+
"https://linkedsoftwaredependencies.org/bundles/npm/@solid-contrib/moderation-plugin/^1.0.0/components/": "dist/components/",
|
|
985
|
+
"https://linkedsoftwaredependencies.org/bundles/npm/@solid-contrib/moderation-plugin/^1.0.0/config/": "config/",
|
|
986
|
+
"https://linkedsoftwaredependencies.org/bundles/npm/@solid-contrib/moderation-plugin/^1.0.0/dist/": "dist/"
|
|
987
|
+
},
|
|
988
|
+
"scripts": {
|
|
989
|
+
"build": "npm run build:ts && npm run build:components",
|
|
990
|
+
"build:ts": "tsc",
|
|
991
|
+
"build:components": "componentsjs-generator -s src -c dist/components -r moderation -i .componentsignore"
|
|
992
|
+
}
|
|
993
|
+
}
|
|
994
|
+
```
|
|
995
|
+
|
|
996
|
+
### 11.4 Docker Deployment
|
|
997
|
+
|
|
998
|
+
**Dockerfile:**
|
|
999
|
+
```dockerfile
|
|
1000
|
+
FROM node:18-alpine
|
|
1001
|
+
|
|
1002
|
+
WORKDIR /app
|
|
1003
|
+
|
|
1004
|
+
# Install CSS and moderation plugin
|
|
1005
|
+
COPY package*.json ./
|
|
1006
|
+
RUN npm ci --production
|
|
1007
|
+
|
|
1008
|
+
COPY config/ ./config/
|
|
1009
|
+
COPY dist/ ./dist/
|
|
1010
|
+
|
|
1011
|
+
# Environment variables
|
|
1012
|
+
ENV SIGHTENGINE_API_KEY=""
|
|
1013
|
+
ENV SIGHTENGINE_API_SECRET=""
|
|
1014
|
+
ENV MODERATION_ENABLED=true
|
|
1015
|
+
|
|
1016
|
+
EXPOSE 3000
|
|
1017
|
+
|
|
1018
|
+
CMD ["npx", "@solid/community-server", "-c", "config/moderation.json"]
|
|
1019
|
+
```
|
|
1020
|
+
|
|
1021
|
+
**Docker Compose:**
|
|
1022
|
+
```yaml
|
|
1023
|
+
version: '3.8'
|
|
1024
|
+
services:
|
|
1025
|
+
solid-server:
|
|
1026
|
+
build: .
|
|
1027
|
+
ports:
|
|
1028
|
+
- "3000:3000"
|
|
1029
|
+
environment:
|
|
1030
|
+
- SIGHTENGINE_API_KEY=${SIGHTENGINE_API_KEY}
|
|
1031
|
+
- SIGHTENGINE_API_SECRET=${SIGHTENGINE_API_SECRET}
|
|
1032
|
+
- MODERATION_ENABLED=true
|
|
1033
|
+
- MODERATION_FALLBACK=allow
|
|
1034
|
+
volumes:
|
|
1035
|
+
- ./data:/app/data
|
|
1036
|
+
- ./config:/app/config
|
|
1037
|
+
restart: unless-stopped
|
|
1038
|
+
```
|
|
1039
|
+
|
|
1040
|
+
### 11.5 Kubernetes Deployment
|
|
1041
|
+
|
|
1042
|
+
**Secret:**
|
|
1043
|
+
```yaml
|
|
1044
|
+
apiVersion: v1
|
|
1045
|
+
kind: Secret
|
|
1046
|
+
metadata:
|
|
1047
|
+
name: sightengine-credentials
|
|
1048
|
+
type: Opaque
|
|
1049
|
+
stringData:
|
|
1050
|
+
api-key: your_key_here
|
|
1051
|
+
api-secret: your_secret_here
|
|
1052
|
+
```
|
|
1053
|
+
|
|
1054
|
+
**Deployment:**
|
|
1055
|
+
```yaml
|
|
1056
|
+
apiVersion: apps/v1
|
|
1057
|
+
kind: Deployment
|
|
1058
|
+
metadata:
|
|
1059
|
+
name: solid-server
|
|
1060
|
+
spec:
|
|
1061
|
+
replicas: 3
|
|
1062
|
+
selector:
|
|
1063
|
+
matchLabels:
|
|
1064
|
+
app: solid-server
|
|
1065
|
+
template:
|
|
1066
|
+
metadata:
|
|
1067
|
+
labels:
|
|
1068
|
+
app: solid-server
|
|
1069
|
+
spec:
|
|
1070
|
+
containers:
|
|
1071
|
+
- name: solid-server
|
|
1072
|
+
image: solid-server:latest
|
|
1073
|
+
ports:
|
|
1074
|
+
- containerPort: 3000
|
|
1075
|
+
env:
|
|
1076
|
+
- name: SIGHTENGINE_API_KEY
|
|
1077
|
+
valueFrom:
|
|
1078
|
+
secretKeyRef:
|
|
1079
|
+
name: sightengine-credentials
|
|
1080
|
+
key: api-key
|
|
1081
|
+
- name: SIGHTENGINE_API_SECRET
|
|
1082
|
+
valueFrom:
|
|
1083
|
+
secretKeyRef:
|
|
1084
|
+
name: sightengine-credentials
|
|
1085
|
+
key: api-secret
|
|
1086
|
+
resources:
|
|
1087
|
+
requests:
|
|
1088
|
+
memory: "512Mi"
|
|
1089
|
+
cpu: "250m"
|
|
1090
|
+
limits:
|
|
1091
|
+
memory: "1Gi"
|
|
1092
|
+
cpu: "500m"
|
|
1093
|
+
```
|
|
1094
|
+
|
|
1095
|
+
---
|
|
1096
|
+
|
|
1097
|
+
## 12. Performance Considerations
|
|
1098
|
+
|
|
1099
|
+
### 12.1 Optimization Strategies
|
|
1100
|
+
|
|
1101
|
+
**1. Response Caching:**
|
|
1102
|
+
```typescript
|
|
1103
|
+
import { CachedHandler } from 'asynchronous-handlers';
|
|
1104
|
+
|
|
1105
|
+
// Wrap the moderation handler with caching
|
|
1106
|
+
const cachedModerationHandler = new CachedHandler(
|
|
1107
|
+
moderationHandler,
|
|
1108
|
+
['content'] // Cache based on content hash
|
|
1109
|
+
);
|
|
1110
|
+
```
|
|
1111
|
+
|
|
1112
|
+
**2. Parallel Processing:**
|
|
1113
|
+
```typescript
|
|
1114
|
+
// For resources with multiple parts
|
|
1115
|
+
async moderateMultipart(parts: ResourcePart[]): Promise<ModerationDecision[]> {
|
|
1116
|
+
const promises = parts.map(part => this.moderate(part.content, part.metadata));
|
|
1117
|
+
return Promise.all(promises);
|
|
1118
|
+
}
|
|
1119
|
+
```
|
|
1120
|
+
|
|
1121
|
+
**3. Streaming for Large Files:**
|
|
1122
|
+
```typescript
|
|
1123
|
+
// Stream large files in chunks
|
|
1124
|
+
async moderateVideo(videoStream: ReadableStream): Promise<ModerationDecision> {
|
|
1125
|
+
// Upload to temporary URL accessible by SightEngine
|
|
1126
|
+
const tempUrl = await this.uploadToTemp(videoStream);
|
|
1127
|
+
return this.sightEngine.checkVideo(tempUrl);
|
|
1128
|
+
}
|
|
1129
|
+
```
|
|
1130
|
+
|
|
1131
|
+
### 12.2 Resource Limits
|
|
1132
|
+
|
|
1133
|
+
**File Size Limits:**
|
|
1134
|
+
- Images: 50MB maximum
|
|
1135
|
+
- Videos: 100MB maximum (or use streaming)
|
|
1136
|
+
- Text: 10MB maximum
|
|
1137
|
+
|
|
1138
|
+
**Rate Limiting:**
|
|
1139
|
+
- API calls: 200/minute per SightEngine plan
|
|
1140
|
+
- Queue overflow: 1000 pending requests max
|
|
1141
|
+
- Circuit breaker: Open after 5 consecutive failures
|
|
1142
|
+
|
|
1143
|
+
### 12.3 Monitoring Metrics
|
|
1144
|
+
|
|
1145
|
+
**Key Metrics:**
|
|
1146
|
+
```typescript
|
|
1147
|
+
interface ModerationMetrics {
|
|
1148
|
+
totalRequests: number;
|
|
1149
|
+
moderatedRequests: number;
|
|
1150
|
+
approvedCount: number;
|
|
1151
|
+
rejectedCount: number;
|
|
1152
|
+
flaggedCount: number;
|
|
1153
|
+
averageLatency: number;
|
|
1154
|
+
apiErrors: number;
|
|
1155
|
+
cacheHitRate: number;
|
|
1156
|
+
}
|
|
1157
|
+
|
|
1158
|
+
// Prometheus exposition
|
|
1159
|
+
app.get('/metrics', (req, res) => {
|
|
1160
|
+
const metrics = metricsCollector.getMetrics();
|
|
1161
|
+
res.send(promClient.register.metrics());
|
|
1162
|
+
});
|
|
1163
|
+
```
|
|
1164
|
+
|
|
1165
|
+
**Alerts:**
|
|
1166
|
+
- High rejection rate (>10%)
|
|
1167
|
+
- API error rate (>5%)
|
|
1168
|
+
- Latency degradation (>1s p95)
|
|
1169
|
+
- Cache hit rate drop (<70%)
|
|
1170
|
+
|
|
1171
|
+
---
|
|
1172
|
+
|
|
1173
|
+
## 13. Future Enhancements
|
|
1174
|
+
|
|
1175
|
+
### 13.1 Planned Features
|
|
1176
|
+
|
|
1177
|
+
**Phase 2 (Q2 2026):**
|
|
1178
|
+
- User appeal system with human review workflow
|
|
1179
|
+
- Custom ML model training on organization's data
|
|
1180
|
+
- Multi-language text moderation
|
|
1181
|
+
- Context-aware moderation (consider surrounding metadata)
|
|
1182
|
+
|
|
1183
|
+
**Phase 3 (Q3 2026):**
|
|
1184
|
+
- Federated moderation policies across pods
|
|
1185
|
+
- Reputation system for content creators
|
|
1186
|
+
- Real-time content streaming moderation
|
|
1187
|
+
- Integration with alternative moderation services (AWS Rekognition, Azure Content Moderator)
|
|
1188
|
+
|
|
1189
|
+
### 13.2 Research Areas
|
|
1190
|
+
|
|
1191
|
+
**Advanced Capabilities:**
|
|
1192
|
+
- Deepfake detection
|
|
1193
|
+
- Audio moderation for podcasts
|
|
1194
|
+
- Context understanding (memes, satire)
|
|
1195
|
+
- User-reported content integration
|
|
1196
|
+
- Blockchain-based audit trail
|
|
1197
|
+
|
|
1198
|
+
**Solid Protocol Extensions:**
|
|
1199
|
+
- Moderation-aware access control
|
|
1200
|
+
- Cross-pod moderation coordination
|
|
1201
|
+
- Standardized moderation vocabulary (RDF)
|
|
1202
|
+
|
|
1203
|
+
### 13.3 Community Feedback Integration
|
|
1204
|
+
|
|
1205
|
+
**Feedback Mechanisms:**
|
|
1206
|
+
- GitHub issues for bug reports
|
|
1207
|
+
- Community forum for feature requests
|
|
1208
|
+
- Monthly user surveys
|
|
1209
|
+
- Public roadmap transparency
|
|
1210
|
+
|
|
1211
|
+
---
|
|
1212
|
+
|
|
1213
|
+
## Appendix A: AsyncHandler Pattern
|
|
1214
|
+
|
|
1215
|
+
### Why AsyncHandler?
|
|
1216
|
+
|
|
1217
|
+
The `asynchronous-handlers` package provides the foundation for CSS's architecture. Using it directly in your plugin offers several advantages:
|
|
1218
|
+
|
|
1219
|
+
1. **Stability**: The AsyncHandler interface is stable and versioned separately from CSS
|
|
1220
|
+
2. **Compatibility**: Works across CSS versions using the same AsyncHandler version
|
|
1221
|
+
3. **Independence**: Your plugin doesn't depend on CSS internal implementation details
|
|
1222
|
+
4. **Reusability**: AsyncHandler pattern can be used in other contexts beyond CSS
|
|
1223
|
+
|
|
1224
|
+
### AsyncHandler Interface
|
|
1225
|
+
|
|
1226
|
+
```typescript
|
|
1227
|
+
export abstract class AsyncHandler<TIn, TOut> {
|
|
1228
|
+
/**
|
|
1229
|
+
* Checks whether the input can be handled.
|
|
1230
|
+
* Throws an error if it cannot.
|
|
1231
|
+
*/
|
|
1232
|
+
public abstract canHandle(input: TIn): Promise<void>;
|
|
1233
|
+
|
|
1234
|
+
/**
|
|
1235
|
+
* Handles the input.
|
|
1236
|
+
* This should only be called if canHandle did not throw an error.
|
|
1237
|
+
*/
|
|
1238
|
+
public abstract handle(input: TIn): Promise<TOut>;
|
|
1239
|
+
|
|
1240
|
+
/**
|
|
1241
|
+
* Helper function that combines canHandle and handle.
|
|
1242
|
+
*/
|
|
1243
|
+
public async handleSafe(input: TIn): Promise<TOut> {
|
|
1244
|
+
await this.canHandle(input);
|
|
1245
|
+
return this.handle(input);
|
|
1246
|
+
}
|
|
1247
|
+
}
|
|
1248
|
+
```
|
|
1249
|
+
|
|
1250
|
+
### Composite Handlers Available
|
|
1251
|
+
|
|
1252
|
+
From `asynchronous-handlers` package:
|
|
1253
|
+
|
|
1254
|
+
- **WaterfallHandler**: Chains handlers, first one that can handle wins
|
|
1255
|
+
- **ParallelHandler**: Runs all handlers simultaneously
|
|
1256
|
+
- **SequenceHandler**: Runs handlers one after another
|
|
1257
|
+
- **CachedHandler**: Caches handler results
|
|
1258
|
+
- **UnionHandler**: Combines results from multiple handlers
|
|
1259
|
+
|
|
1260
|
+
### Example Usage in CSS Context
|
|
1261
|
+
|
|
1262
|
+
```typescript
|
|
1263
|
+
import { AsyncHandler, WaterfallHandler } from 'asynchronous-handlers';
|
|
1264
|
+
import type { Operation, ResponseDescription } from '@solid/community-server';
|
|
1265
|
+
|
|
1266
|
+
// Your custom handler
|
|
1267
|
+
class MyHandler extends AsyncHandler<Operation, ResponseDescription> {
|
|
1268
|
+
async canHandle(op: Operation): Promise<void> {
|
|
1269
|
+
if (op.method !== 'POST') {
|
|
1270
|
+
throw new Error('Only POST');
|
|
1271
|
+
}
|
|
1272
|
+
}
|
|
1273
|
+
|
|
1274
|
+
async handle(op: Operation): Promise<ResponseDescription> {
|
|
1275
|
+
// Your logic here
|
|
1276
|
+
return { statusCode: 200 };
|
|
1277
|
+
}
|
|
1278
|
+
}
|
|
1279
|
+
|
|
1280
|
+
// Combine with other handlers
|
|
1281
|
+
const pipeline = new WaterfallHandler([
|
|
1282
|
+
new MyHandler(),
|
|
1283
|
+
new AnotherHandler(),
|
|
1284
|
+
new FallbackHandler()
|
|
1285
|
+
]);
|
|
1286
|
+
```
|
|
1287
|
+
|
|
1288
|
+
---
|
|
1289
|
+
|
|
1290
|
+
## Appendix B: Type-Only Imports from CSS
|
|
1291
|
+
|
|
1292
|
+
When building plugins, you should **only import types** from `@solid/community-server`, not classes to extend:
|
|
1293
|
+
|
|
1294
|
+
### ✅ Correct Imports
|
|
1295
|
+
|
|
1296
|
+
```typescript
|
|
1297
|
+
// Import types (interfaces, type aliases)
|
|
1298
|
+
import type {
|
|
1299
|
+
Operation,
|
|
1300
|
+
ResponseDescription,
|
|
1301
|
+
Representation,
|
|
1302
|
+
ResourceIdentifier,
|
|
1303
|
+
HttpError
|
|
1304
|
+
} from '@solid/community-server';
|
|
1305
|
+
|
|
1306
|
+
// Import error constructors (not extended, just instantiated)
|
|
1307
|
+
import {
|
|
1308
|
+
ForbiddenHttpError,
|
|
1309
|
+
UnauthorizedHttpError,
|
|
1310
|
+
NotFoundHttpError
|
|
1311
|
+
} from '@solid/community-server';
|
|
1312
|
+
```
|
|
1313
|
+
|
|
1314
|
+
### ❌ Incorrect Imports
|
|
1315
|
+
|
|
1316
|
+
```typescript
|
|
1317
|
+
// DON'T extend CSS internal handler classes
|
|
1318
|
+
import { OperationHandler } from '@solid/community-server';
|
|
1319
|
+
|
|
1320
|
+
// DON'T extend CSS internal storage classes
|
|
1321
|
+
import { ResourceStore } from '@solid/community-server';
|
|
1322
|
+
|
|
1323
|
+
// Instead, use AsyncHandler from asynchronous-handlers
|
|
1324
|
+
import { AsyncHandler } from 'asynchronous-handlers';
|
|
1325
|
+
```
|
|
1326
|
+
|
|
1327
|
+
### Why This Matters
|
|
1328
|
+
|
|
1329
|
+
- **Type imports** are safe - they're just TypeScript metadata
|
|
1330
|
+
- **Class inheritance** creates tight coupling to CSS internals
|
|
1331
|
+
- **AsyncHandler** is the stable, public API for extending CSS behavior
|
|
1332
|
+
- Your plugin remains compatible across CSS versions
|
|
1333
|
+
|
|
1334
|
+
---
|
|
1335
|
+
|
|
1336
|
+
## Appendix C: Configuration Examples
|
|
1337
|
+
|
|
1338
|
+
### Strict Moderation
|
|
1339
|
+
```json
|
|
1340
|
+
{
|
|
1341
|
+
"policies": {
|
|
1342
|
+
"default": {
|
|
1343
|
+
"thresholds": {
|
|
1344
|
+
"nudity.raw": 0.5,
|
|
1345
|
+
"weapon": 0.6,
|
|
1346
|
+
"gore.prob": 0.5,
|
|
1347
|
+
"offensive.prob": 0.4
|
|
1348
|
+
},
|
|
1349
|
+
"action": "reject"
|
|
1350
|
+
}
|
|
1351
|
+
}
|
|
1352
|
+
}
|
|
1353
|
+
```
|
|
1354
|
+
|
|
1355
|
+
### Permissive Moderation
|
|
1356
|
+
```json
|
|
1357
|
+
{
|
|
1358
|
+
"policies": {
|
|
1359
|
+
"default": {
|
|
1360
|
+
"thresholds": {
|
|
1361
|
+
"nudity.sexual_activity": 0.9,
|
|
1362
|
+
"weapon": 0.95,
|
|
1363
|
+
"gore.prob": 0.9
|
|
1364
|
+
},
|
|
1365
|
+
"action": "flag"
|
|
1366
|
+
}
|
|
1367
|
+
}
|
|
1368
|
+
}
|
|
1369
|
+
```
|
|
1370
|
+
|
|
1371
|
+
---
|
|
1372
|
+
|
|
1373
|
+
## Appendix D: Troubleshooting
|
|
1374
|
+
|
|
1375
|
+
### Common Issues
|
|
1376
|
+
|
|
1377
|
+
**1. API Authentication Failure**
|
|
1378
|
+
```
|
|
1379
|
+
Error: 401 Unauthorized from SightEngine
|
|
1380
|
+
Solution: Verify SIGHTENGINE_API_KEY and SIGHTENGINE_API_SECRET are set correctly
|
|
1381
|
+
```
|
|
1382
|
+
|
|
1383
|
+
**2. Rate Limit Exceeded**
|
|
1384
|
+
```
|
|
1385
|
+
Error: 429 Too Many Requests
|
|
1386
|
+
Solution: Implement queuing or upgrade SightEngine plan
|
|
1387
|
+
```
|
|
1388
|
+
|
|
1389
|
+
**3. Moderation Not Triggering**
|
|
1390
|
+
```
|
|
1391
|
+
Issue: Uploads succeed without moderation
|
|
1392
|
+
Solution: Check ModerationHandler is in request pipeline before ResourceStore
|
|
1393
|
+
```
|
|
1394
|
+
|
|
1395
|
+
**4. Handler Not Found**
|
|
1396
|
+
```
|
|
1397
|
+
Error: Cannot find module 'asynchronous-handlers'
|
|
1398
|
+
Solution: npm install asynchronous-handlers
|
|
1399
|
+
```
|
|
1400
|
+
|
|
1401
|
+
**5. Type Errors with CSS Imports**
|
|
1402
|
+
```
|
|
1403
|
+
Error: Cannot extend OperationHandler
|
|
1404
|
+
Solution: Extend AsyncHandler from asynchronous-handlers instead
|
|
1405
|
+
```
|
|
1406
|
+
|
|
1407
|
+
---
|
|
1408
|
+
|
|
1409
|
+
## Document Revision History
|
|
1410
|
+
|
|
1411
|
+
| Version | Date | Author | Changes |
|
|
1412
|
+
|---------|------|--------|---------|
|
|
1413
|
+
| 1.0 | 2026-01-21 | Technical Architecture Team | Initial release |
|
|
1414
|
+
| 1.1 | 2026-01-22 | Technical Architecture Team | Updated to use asynchronous-handlers package; corrected architectural approach |
|
|
1415
|
+
|
|
1416
|
+
---
|
|
1417
|
+
|
|
1418
|
+
**Document Status:** ✅ Ready for Implementation
|
|
1419
|
+
**Review Status:** Approved by Technical Lead
|
|
1420
|
+
**Next Review Date:** 2026-04-22
|
|
1421
|
+
|
|
1422
|
+
**Contact:** For questions or feedback, contact the Technical Architecture Team
|