@cubis/foundry 0.3.83 → 0.3.84

package/workflows/skills/sanitize-pii/references/audit-trails.md

@@ -1,309 +0,0 @@
-# Audit Trail Design Reference
-
-## Overview
-
-Audit trails record who accessed, modified, or deleted PII, when, why, and from where. They are required by GDPR (Article 5, accountability principle), HIPAA, PCI DSS, and SOC 2. Beyond compliance, audit trails enable breach impact assessment, insider threat detection, and data governance.
-
-## What to Audit
-
-### PII Access Events
-
-| Event Type | What to Log | Example |
-|-----------|------------|---------|
-| Read | Who accessed which PII fields | "user:admin_42 read customer:1234 fields:[email, phone]" |
-| Write | Who modified which PII fields, old/new values (hashed) | "user:api_service wrote customer:1234 field:email" |
-| Delete | Who deleted which PII records | "user:admin_42 deleted customer:1234 (erasure request)" |
-| Export | Who exported PII data and in what format | "user:admin_42 exported customers CSV, 5000 records" |
-| Search | Who searched for PII values | "user:support_01 searched email=*@example.com" |
-| Bulk access | Who accessed PII at scale | "user:analytics_service read 10000 customer records" |
-| Failed access | Who attempted unauthorized PII access | "user:intern_01 denied access to customer:1234 salary" |
-| Consent change | When consent was granted/withdrawn | "customer:1234 withdrew marketing consent" |
-
-### System Events
-
-| Event | Purpose |
-|-------|---------|
-| Masking pipeline execution | Track when non-production data was refreshed |
-| Retention policy execution | Track automated deletions |
-| Backup creation/restoration | Track when PII was copied |
-| Access policy changes | Track permission modifications |
-| Encryption key rotation | Track key management activities |
-
-## Audit Log Schema
-
-### Structured Audit Event
-
-```json
-{
-  "event_id": "evt_a7b3c9d1e5f2g8h4",
-  "timestamp": "2025-03-14T10:23:45.123Z",
-  "event_type": "pii_access",
-  "action": "read",
-  "actor": {
-    "type": "user",
-    "id": "user_42",
-    "role": "support_agent",
-    "ip_address": "10.0.1.50",
-    "user_agent": "Chrome/120.0",
-    "session_id": "sess_xyz789"
-  },
-  "resource": {
-    "type": "customer",
-    "id": "cust_1234",
-    "fields_accessed": ["email", "phone", "address"]
-  },
-  "context": {
-    "purpose": "support_ticket_resolution",
-    "ticket_id": "TICKET-5678",
-    "legal_basis": "legitimate_interest",
-    "application": "support-portal",
-    "environment": "production"
-  },
-  "result": "success",
-  "data_classification": "pii_standard"
-}
-```
-
-### Database Schema
-
-```sql
-CREATE TABLE pii_audit_log (
-  id BIGSERIAL PRIMARY KEY,
-  event_id VARCHAR(50) UNIQUE NOT NULL,
-  timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-  event_type VARCHAR(50) NOT NULL,
-  action VARCHAR(20) NOT NULL,
-
-  -- Actor
-  actor_type VARCHAR(20) NOT NULL,
-  actor_id VARCHAR(100) NOT NULL,
-  actor_role VARCHAR(50),
-  actor_ip INET,
-  session_id VARCHAR(100),
-
-  -- Resource
-  resource_type VARCHAR(50) NOT NULL,
-  resource_id VARCHAR(100) NOT NULL,
-  fields_accessed TEXT[],
-
-  -- Context
-  purpose VARCHAR(100),
-  legal_basis VARCHAR(50),
-  application VARCHAR(100),
-  environment VARCHAR(20),
-
-  -- Result
-  result VARCHAR(20) NOT NULL,
-  error_message TEXT,
-  data_classification VARCHAR(50),
-
-  -- Metadata
-  metadata JSONB
-);
-
--- Indexes for common queries
-CREATE INDEX idx_audit_timestamp ON pii_audit_log (timestamp);
-CREATE INDEX idx_audit_actor ON pii_audit_log (actor_id, timestamp);
-CREATE INDEX idx_audit_resource ON pii_audit_log (resource_type, resource_id, timestamp);
-CREATE INDEX idx_audit_event_type ON pii_audit_log (event_type, timestamp);
-
--- Partitioning by month for performance
-CREATE TABLE pii_audit_log_2025_03 PARTITION OF pii_audit_log
-  FOR VALUES FROM ('2025-03-01') TO ('2025-04-01');
-```
-
-## Implementation Patterns
-
-### Middleware-Based Audit Logging
-
-```javascript
-// Express.js middleware for automatic PII access logging
-function piiAuditMiddleware(piiFields) {
-  return (req, res, next) => {
-    const originalJson = res.json.bind(res);
-
-    res.json = (body) => {
-      // Detect which PII fields are in the response
-      const accessedFields = detectPIIFields(body, piiFields);
-
-      if (accessedFields.length > 0) {
-        auditLogger.log({
-          event_type: 'pii_access',
-          action: req.method === 'GET' ? 'read' : 'write',
-          actor: {
-            id: req.user?.id,
-            role: req.user?.role,
-            ip: req.ip,
-            session_id: req.session?.id,
-          },
-          resource: {
-            type: extractResourceType(req.path),
-            id: req.params.id,
-            fields_accessed: accessedFields,
-          },
-          context: {
-            purpose: req.headers['x-processing-purpose'],
-            application: 'api-server',
-          },
-          result: 'success',
-        });
-      }
-
-      return originalJson(body);
-    };
-
-    next();
-  };
-}
-
-// Apply to routes that serve PII
-app.use('/api/customers', piiAuditMiddleware([
-  'email', 'phone', 'address', 'date_of_birth', 'ssn'
-]));
-```
-
-### Database-Level Audit Triggers
-
-```sql
--- PostgreSQL trigger for PII table changes
-CREATE OR REPLACE FUNCTION audit_pii_changes()
-RETURNS TRIGGER AS $$
-BEGIN
-  INSERT INTO pii_audit_log (
-    event_id, event_type, action,
-    actor_type, actor_id,
-    resource_type, resource_id,
-    fields_accessed, result, metadata
-  ) VALUES (
-    'evt_' || gen_random_uuid(),
-    'pii_modification',
-    TG_OP,
-    'database_user',
-    current_user,
-    TG_TABLE_NAME,
-    CASE TG_OP
-      WHEN 'DELETE' THEN OLD.id::TEXT
-      ELSE NEW.id::TEXT
-    END,
-    ARRAY(SELECT column_name FROM information_schema.columns
-          WHERE table_name = TG_TABLE_NAME
-          AND column_name IN ('email', 'phone', 'ssn', 'address', 'name')),
-    'success',
-    jsonb_build_object(
-      'operation', TG_OP,
-      'table', TG_TABLE_NAME,
-      'application_name', current_setting('application_name')
-    )
-  );
-  RETURN COALESCE(NEW, OLD);
-END;
-$$ LANGUAGE plpgsql;
-
-CREATE TRIGGER audit_customers_pii
-  AFTER INSERT OR UPDATE OR DELETE ON customers
-  FOR EACH ROW EXECUTE FUNCTION audit_pii_changes();
-```
-
-## Audit Log Protection
-
-### Immutability
-
-| Approach | Implementation | Cost |
-|----------|---------------|------|
-| Append-only table | Revoke DELETE/UPDATE on audit table | Low |
-| Write-once storage | S3 with Object Lock, WORM storage | Medium |
-| Blockchain/hash chain | Each entry hashes the previous | High |
-| Separate audit database | Different credentials, network isolation | Medium |
-
-### Append-Only Implementation
-
-```sql
--- Revoke modification permissions on audit table
-REVOKE UPDATE, DELETE ON pii_audit_log FROM PUBLIC;
-REVOKE UPDATE, DELETE ON pii_audit_log FROM app_user;
-
--- Only the audit_writer role can insert
-GRANT INSERT ON pii_audit_log TO audit_writer;
-
--- Create hash chain for tamper detection
-ALTER TABLE pii_audit_log ADD COLUMN previous_hash VARCHAR(64);
-ALTER TABLE pii_audit_log ADD COLUMN entry_hash VARCHAR(64);
-
-CREATE OR REPLACE FUNCTION compute_audit_hash()
-RETURNS TRIGGER AS $$
-BEGIN
-  NEW.previous_hash := (
-    SELECT entry_hash FROM pii_audit_log
-    ORDER BY id DESC LIMIT 1
-  );
-  NEW.entry_hash := encode(
-    sha256(
-      (NEW.event_id || NEW.timestamp || NEW.actor_id || NEW.resource_id
-       || COALESCE(NEW.previous_hash, 'genesis'))::bytea
-    ), 'hex'
-  );
-  RETURN NEW;
-END;
-$$ LANGUAGE plpgsql;
-```
-
-## Retention and Archival
-
-| Regulation | Minimum Retention | Maximum Retention |
-|-----------|-------------------|-------------------|
-| GDPR | Duration of processing + dispute period | Purpose-dependent (minimize) |
-| HIPAA | 6 years | No maximum specified |
-| PCI DSS | 1 year (immediately available) + archive | No maximum specified |
-| SOC 2 | Per control objectives | Per control objectives |
-
-### Archival Strategy
-
-```
-Active (0-90 days): Hot storage, indexed, queryable
-Warm (90-365 days): Compressed, queryable with delay
-Cold (1-7 years): Archived to S3/Glacier, restore on request
-Deletion (>7 years): Automated purge per retention policy
-```
-
-## Monitoring and Alerting
-
-### Anomaly Detection Rules
-
-| Signal | Threshold | Alert |
-|--------|-----------|-------|
-| Bulk PII read | > 1000 records in 1 hour by single actor | Immediate |
-| Off-hours PII access | Any PII access outside business hours | Review next day |
-| Failed PII access attempts | > 5 failures in 10 minutes | Immediate |
-| PII export | Any export of > 100 records | Immediate |
-| New actor accessing PII | First-time PII access by a user | Informational |
-| Unusual fields accessed | Actor reads fields they have never read before | Review |
-| Cross-tenant access | Actor accesses PII from multiple tenants | Immediate |
-
-### Dashboard Metrics
-
-- Total PII access events per day/week/month
-- PII access by actor (top N actors)
-- PII access by resource type
-- Failed access attempts trend
-- Erasure request completion rate
-- Average DSAR response time
-- Audit log integrity check status
-
-## Compliance Reporting
-
-### DSAR Response Evidence
-
-When responding to a data subject access request, the audit trail provides:
-- Complete list of who has accessed the subject's data
-- When each access occurred
-- What fields were accessed
-- The stated purpose of each access
-
-### Breach Impact Assessment
-
-When a breach occurs, the audit trail enables:
-- Determine exactly which records were accessed
-- Identify the breach window (first unauthorized access to detection)
-- Identify all affected data subjects
-- Determine what data was exposed
-- Generate the required notification content

package/workflows/skills/sanitize-pii/references/compliance.md

@@ -1,294 +0,0 @@
-# GDPR/CCPA Compliance Reference
-
-## Overview
-
-This reference covers the key privacy regulations affecting PII handling: the European Union's General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) / California Privacy Rights Act (CPRA). It maps regulatory requirements to technical implementation.
-
-## GDPR Core Principles (Article 5)
-
-| Principle | Requirement | Technical Implementation |
-|-----------|------------|------------------------|
-| Lawfulness | Valid legal basis for processing | Consent management system, purpose tracking |
-| Purpose limitation | Data used only for stated purposes | Access controls per purpose, audit logging |
-| Data minimization | Collect only what is necessary | Schema review, field-level access control |
-| Accuracy | Data must be kept accurate | Update mechanisms, data quality checks |
-| Storage limitation | Delete when no longer needed | Retention policies, automated deletion |
-| Integrity & confidentiality | Protect against unauthorized access | Encryption, access controls, security measures |
-| Accountability | Demonstrate compliance | Audit trails, DPIAs, documentation |
-
-## GDPR Data Subject Rights
-
-### Right to Access (Article 15)
-
-The data subject can request all personal data held about them.
-
-```python
-class DSARHandler:
-    def handle_access_request(self, data_subject_id):
-        """Export all PII for a data subject across all systems."""
-        report = {
-            'data_subject_id': data_subject_id,
-            'generated_at': datetime.utcnow().isoformat(),
-            'data_sources': []
-        }
-
-        # Collect from all data stores
-        for source in self.registered_data_sources:
-            records = source.find_by_subject(data_subject_id)
-            report['data_sources'].append({
-                'source': source.name,
-                'purpose': source.processing_purpose,
-                'legal_basis': source.legal_basis,
-                'retention_period': source.retention_period,
-                'records': records,
-            })
-
-        return report  # Must be delivered within 30 days
-```
-
-**Implementation requirements:**
-- Respond within 30 days (extendable to 90 for complex requests)
-- Provide data in a commonly used, machine-readable format
-- Include processing purposes, categories, recipients, retention periods
-- Free of charge (unless manifestly excessive)
-
-### Right to Erasure (Article 17)
-
-The data subject can request deletion of all personal data.
-
-```python
-class ErasureHandler:
-    def handle_erasure_request(self, data_subject_id):
-        """Delete or anonymize all PII for a data subject."""
-        results = []
-
-        for source in self.registered_data_sources:
-            # Check if legal obligation requires retention
-            if source.has_legal_retention_requirement(data_subject_id):
-                results.append({
-                    'source': source.name,
-                    'action': 'retained',
-                    'reason': 'Legal obligation (e.g., tax records)',
-                })
-                continue
-
-            # Delete or anonymize
-            deleted = source.delete_subject_data(data_subject_id)
-            results.append({
-                'source': source.name,
-                'action': 'deleted',
-                'records_affected': deleted,
-            })
-
-        # Also notify third parties who received the data
-        for processor in self.data_processors:
-            processor.request_erasure(data_subject_id)
-
-        return results
-```
-
-**Erasure must cover:**
-- Primary database records
-- Backups (within reasonable timeframe)
-- Caches and CDN
-- Analytics and data warehouses
-- Third-party processors
-- Search engine indexes (request de-indexing)
-
-### Right to Portability (Article 20)
-
-```json
-// Export format example (JSON, CSV, or XML)
-{
-  "data_subject": {
-    "email": "sarah@example.com",
-    "name": "Sarah Jones",
-    "account_created": "2023-01-15"
-  },
-  "orders": [
-    {"id": "ORD-001", "date": "2023-02-20", "total": 49.99}
-  ],
-  "preferences": {
-    "newsletter": true,
-    "language": "en"
-  }
-}
-```
-
-## CCPA/CPRA Requirements
-
-### Key Rights (Comparison with GDPR)
-
-| Right | CCPA | GDPR Equivalent |
-|-------|------|-----------------|
-| Right to know | Categories + specific pieces of data | Right to access (Art. 15) |
-| Right to delete | Delete personal information | Right to erasure (Art. 17) |
-| Right to opt out | Opt out of sale/sharing | No direct equivalent (consent-based) |
-| Right to correct | Correct inaccurate data | Right to rectification (Art. 16) |
-| Right to limit use | Limit use of sensitive PI | Purpose limitation (Art. 5) |
-| Non-discrimination | No retaliation for exercising rights | Similar protection (Art. 21) |
-
-### CCPA Technical Requirements
-
-```python
-# "Do Not Sell My Personal Information" implementation
-class CCPAOptOutHandler:
-    def process_opt_out(self, consumer_id):
-        """Stop selling/sharing consumer's personal information."""
-
-        # 1. Set opt-out flag
-        self.set_consumer_flag(consumer_id, 'do_not_sell', True)
-
-        # 2. Notify all data sharing partners
-        for partner in self.data_sharing_partners:
-            partner.cease_data_sharing(consumer_id)
-
-        # 3. Update advertising/analytics integrations
-        self.analytics.exclude_consumer(consumer_id)
-
-        # 4. Must process within 15 business days
-        # 5. Must not ask consumer to opt in again for 12 months
-```
-
-## Consent Management
-
-### Consent Tracking Schema
-
-```sql
-CREATE TABLE consent_records (
-  id SERIAL PRIMARY KEY,
-  data_subject_id VARCHAR(255) NOT NULL,
-  purpose VARCHAR(100) NOT NULL,
-  legal_basis VARCHAR(50) NOT NULL,  -- consent, legitimate_interest, contract, legal_obligation
-  granted_at TIMESTAMP NOT NULL,
-  withdrawn_at TIMESTAMP,
-  consent_version VARCHAR(20),
-  consent_text_hash VARCHAR(64),  -- Hash of the exact consent text shown
-  collection_method VARCHAR(50),  -- web_form, mobile_app, api
-  ip_address VARCHAR(45),  -- IP at time of consent (for evidence)
-  is_active BOOLEAN DEFAULT TRUE,
-
-  INDEX idx_subject_purpose (data_subject_id, purpose)
-);
-```
-
-### Consent Validation
-
-```python
-def is_processing_allowed(data_subject_id, purpose):
-    """Check if processing is allowed for this purpose."""
-    consent = db.query("""
-        SELECT * FROM consent_records
-        WHERE data_subject_id = %s
-        AND purpose = %s
-        AND is_active = TRUE
-        AND withdrawn_at IS NULL
-        ORDER BY granted_at DESC
-        LIMIT 1
-    """, [data_subject_id, purpose])
-
-    if not consent:
-        return False
-
-    # Consent must be for the specific purpose
-    # Consent must not be withdrawn
-    # Legal basis must be valid for the processing type
-    return True
-```
-
-## Data Retention
-
-### Retention Schedule
-
-| Data Category | Retention Period | Legal Basis | Auto-Delete |
-|--------------|-----------------|-------------|-------------|
-| Account data | Until account deletion + 30 days | Contract | Yes |
-| Transaction records | 7 years | Legal obligation (tax) | Yes |
-| Marketing consent | Until withdrawal | Consent | On withdrawal |
-| Log data | 90 days | Legitimate interest | Yes |
-| Support tickets | 2 years after resolution | Legitimate interest | Yes |
-| Analytics (anonymized) | Indefinite | N/A (not personal data) | No |
-
-### Automated Retention Enforcement
-
-```python
-class RetentionEnforcer:
-    RETENTION_RULES = {
-        'logs': timedelta(days=90),
-        'session_data': timedelta(days=30),
-        'deleted_accounts': timedelta(days=30),
-        'support_tickets': timedelta(days=730),
-    }
-
-    def enforce(self):
-        """Run daily to delete data past retention period."""
-        for data_type, retention_period in self.RETENTION_RULES.items():
-            cutoff = datetime.utcnow() - retention_period
-            deleted = self.data_stores[data_type].delete_before(cutoff)
-            self.audit_log.record(
-                action='retention_deletion',
-                data_type=data_type,
-                records_deleted=deleted,
-                cutoff_date=cutoff.isoformat()
-            )
-```
-
-## Data Protection Impact Assessment (DPIA)
-
-Required under GDPR Article 35 for high-risk processing.
-
-### When Required
-
-- Systematic evaluation of personal aspects (profiling)
-- Processing special category data at scale
-- Systematic monitoring of public areas
-- New technologies processing personal data
-- Cross-referencing or combining datasets
-
-### DPIA Template Sections
-
-1. **Processing description** — What data, what purpose, what systems
-2. **Necessity assessment** — Is this processing necessary and proportionate?
-3. **Risk assessment** — What could go wrong? (data breach, unauthorized access, discrimination)
-4. **Risk mitigation** — Technical and organizational measures in place
-5. **Residual risk** — Risk remaining after mitigation
-6. **DPO consultation** — Data Protection Officer review and sign-off
-
-## Breach Notification
-
-### GDPR (Article 33-34)
-
-| Action | Timeline | Recipient |
-|--------|----------|-----------|
-| Internal assessment | Immediately on discovery | Incident response team |
-| Supervisory authority notification | Within 72 hours | Data protection authority |
-| Data subject notification | Without undue delay | Affected individuals (if high risk) |
-
-### Notification Content
-
-```markdown
-## Breach Notification
-
-- Nature of the breach
-- Categories and approximate number of data subjects affected
-- Categories and approximate number of records affected
-- Name and contact details of DPO
-- Likely consequences of the breach
-- Measures taken or proposed to address the breach
-- Measures to mitigate adverse effects
-```
-
-## Compliance Checklist
-
-- [ ] Legal basis documented for each processing activity
-- [ ] Privacy policy published and current
-- [ ] Consent mechanism with granular purpose selection
-- [ ] Data subject request handling (access, erasure, portability)
-- [ ] Data retention schedule defined and automated
-- [ ] DPIAs completed for high-risk processing
-- [ ] Data processing agreements with all third-party processors
-- [ ] Breach notification procedure documented and tested
-- [ ] PII encrypted at rest and in transit
-- [ ] Audit trails for all PII access
-- [ ] Staff trained on data protection responsibilities
-- [ ] DPO appointed (if required)