@cubis/foundry 0.3.83 → 0.3.84

package/workflows/workflows/agent-environment-setup/platforms/codex/skills/sanitize-pii/references/compliance.md

@@ -1,294 +0,0 @@
-# GDPR/CCPA Compliance Reference
-
-## Overview
-
-This reference covers the key privacy regulations affecting PII handling: the European Union's General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) / California Privacy Rights Act (CPRA). It maps regulatory requirements to technical implementation.
-
-## GDPR Core Principles (Article 5)
-
-| Principle | Requirement | Technical Implementation |
-|-----------|------------|------------------------|
-| Lawfulness | Valid legal basis for processing | Consent management system, purpose tracking |
-| Purpose limitation | Data used only for stated purposes | Access controls per purpose, audit logging |
-| Data minimization | Collect only what is necessary | Schema review, field-level access control |
-| Accuracy | Data must be kept accurate | Update mechanisms, data quality checks |
-| Storage limitation | Delete when no longer needed | Retention policies, automated deletion |
-| Integrity & confidentiality | Protect against unauthorized access | Encryption, access controls, security measures |
-| Accountability | Demonstrate compliance | Audit trails, DPIAs, documentation |
-
-## GDPR Data Subject Rights
-
-### Right to Access (Article 15)
-
-The data subject can request all personal data held about them.
-
-```python
-class DSARHandler:
-    def handle_access_request(self, data_subject_id):
-        """Export all PII for a data subject across all systems."""
-        report = {
-            'data_subject_id': data_subject_id,
-            'generated_at': datetime.utcnow().isoformat(),
-            'data_sources': []
-        }
-
-        # Collect from all data stores
-        for source in self.registered_data_sources:
-            records = source.find_by_subject(data_subject_id)
-            report['data_sources'].append({
-                'source': source.name,
-                'purpose': source.processing_purpose,
-                'legal_basis': source.legal_basis,
-                'retention_period': source.retention_period,
-                'records': records,
-            })
-
-        return report  # Must be delivered within 30 days
-```
-
-**Implementation requirements:**
-- Respond within 30 days (extendable to 90 for complex requests)
-- Provide data in a commonly used, machine-readable format
-- Include processing purposes, categories, recipients, retention periods
-- Free of charge (unless manifestly excessive)
-
-### Right to Erasure (Article 17)
-
-The data subject can request deletion of all personal data.
-
-```python
-class ErasureHandler:
-    def handle_erasure_request(self, data_subject_id):
-        """Delete or anonymize all PII for a data subject."""
-        results = []
-
-        for source in self.registered_data_sources:
-            # Check if legal obligation requires retention
-            if source.has_legal_retention_requirement(data_subject_id):
-                results.append({
-                    'source': source.name,
-                    'action': 'retained',
-                    'reason': 'Legal obligation (e.g., tax records)',
-                })
-                continue
-
-            # Delete or anonymize
-            deleted = source.delete_subject_data(data_subject_id)
-            results.append({
-                'source': source.name,
-                'action': 'deleted',
-                'records_affected': deleted,
-            })
-
-        # Also notify third parties who received the data
-        for processor in self.data_processors:
-            processor.request_erasure(data_subject_id)
-
-        return results
-```
-
-**Erasure must cover:**
-- Primary database records
-- Backups (within reasonable timeframe)
-- Caches and CDN
-- Analytics and data warehouses
-- Third-party processors
-- Search engine indexes (request de-indexing)
-
-### Right to Portability (Article 20)
-
-```json
-// Export format example (JSON, CSV, or XML)
-{
-  "data_subject": {
-    "email": "sarah@example.com",
-    "name": "Sarah Jones",
-    "account_created": "2023-01-15"
-  },
-  "orders": [
-    {"id": "ORD-001", "date": "2023-02-20", "total": 49.99}
-  ],
-  "preferences": {
-    "newsletter": true,
-    "language": "en"
-  }
-}
-```
-
-## CCPA/CPRA Requirements
-
-### Key Rights (Comparison with GDPR)
-
-| Right | CCPA | GDPR Equivalent |
-|-------|------|-----------------|
-| Right to know | Categories + specific pieces of data | Right to access (Art. 15) |
-| Right to delete | Delete personal information | Right to erasure (Art. 17) |
-| Right to opt out | Opt out of sale/sharing | No direct equivalent (consent-based) |
-| Right to correct | Correct inaccurate data | Right to rectification (Art. 16) |
-| Right to limit use | Limit use of sensitive PI | Purpose limitation (Art. 5) |
-| Non-discrimination | No retaliation for exercising rights | Similar protection (Art. 21) |
-
-### CCPA Technical Requirements
-
-```python
-# "Do Not Sell My Personal Information" implementation
-class CCPAOptOutHandler:
-    def process_opt_out(self, consumer_id):
-        """Stop selling/sharing consumer's personal information."""
-
-        # 1. Set opt-out flag
-        self.set_consumer_flag(consumer_id, 'do_not_sell', True)
-
-        # 2. Notify all data sharing partners
-        for partner in self.data_sharing_partners:
-            partner.cease_data_sharing(consumer_id)
-
-        # 3. Update advertising/analytics integrations
-        self.analytics.exclude_consumer(consumer_id)
-
-        # 4. Must process within 15 business days
-        # 5. Must not ask consumer to opt in again for 12 months
-```
-
-## Consent Management
-
-### Consent Tracking Schema
-
-```sql
-CREATE TABLE consent_records (
-  id SERIAL PRIMARY KEY,
-  data_subject_id VARCHAR(255) NOT NULL,
-  purpose VARCHAR(100) NOT NULL,
-  legal_basis VARCHAR(50) NOT NULL,  -- consent, legitimate_interest, contract, legal_obligation
-  granted_at TIMESTAMP NOT NULL,
-  withdrawn_at TIMESTAMP,
-  consent_version VARCHAR(20),
-  consent_text_hash VARCHAR(64),  -- Hash of the exact consent text shown
-  collection_method VARCHAR(50),  -- web_form, mobile_app, api
-  ip_address VARCHAR(45),  -- IP at time of consent (for evidence)
-  is_active BOOLEAN DEFAULT TRUE,
-
-  INDEX idx_subject_purpose (data_subject_id, purpose)
-);
-```
-
-### Consent Validation
-
-```python
-def is_processing_allowed(data_subject_id, purpose):
-    """Check if processing is allowed for this purpose."""
-    consent = db.query("""
-        SELECT * FROM consent_records
-        WHERE data_subject_id = %s
-        AND purpose = %s
-        AND is_active = TRUE
-        AND withdrawn_at IS NULL
-        ORDER BY granted_at DESC
-        LIMIT 1
-    """, [data_subject_id, purpose])
-
-    if not consent:
-        return False
-
-    # Consent must be for the specific purpose
-    # Consent must not be withdrawn
-    # Legal basis must be valid for the processing type
-    return True
-```
-
-## Data Retention
-
-### Retention Schedule
-
-| Data Category | Retention Period | Legal Basis | Auto-Delete |
-|--------------|-----------------|-------------|-------------|
-| Account data | Until account deletion + 30 days | Contract | Yes |
-| Transaction records | 7 years | Legal obligation (tax) | Yes |
-| Marketing consent | Until withdrawal | Consent | On withdrawal |
-| Log data | 90 days | Legitimate interest | Yes |
-| Support tickets | 2 years after resolution | Legitimate interest | Yes |
-| Analytics (anonymized) | Indefinite | N/A (not personal data) | No |
-
-### Automated Retention Enforcement
-
-```python
-class RetentionEnforcer:
-    RETENTION_RULES = {
-        'logs': timedelta(days=90),
-        'session_data': timedelta(days=30),
-        'deleted_accounts': timedelta(days=30),
-        'support_tickets': timedelta(days=730),
-    }
-
-    def enforce(self):
-        """Run daily to delete data past retention period."""
-        for data_type, retention_period in self.RETENTION_RULES.items():
-            cutoff = datetime.utcnow() - retention_period
-            deleted = self.data_stores[data_type].delete_before(cutoff)
-            self.audit_log.record(
-                action='retention_deletion',
-                data_type=data_type,
-                records_deleted=deleted,
-                cutoff_date=cutoff.isoformat()
-            )
-```
-
-## Data Protection Impact Assessment (DPIA)
-
-Required under GDPR Article 35 for high-risk processing.
-
-### When Required
-
-- Systematic evaluation of personal aspects (profiling)
-- Processing special category data at scale
-- Systematic monitoring of public areas
-- New technologies processing personal data
-- Cross-referencing or combining datasets
-
-### DPIA Template Sections
-
-1. **Processing description** — What data, what purpose, what systems
-2. **Necessity assessment** — Is this processing necessary and proportionate?
-3. **Risk assessment** — What could go wrong? (data breach, unauthorized access, discrimination)
-4. **Risk mitigation** — Technical and organizational measures in place
-5. **Residual risk** — Risk remaining after mitigation
-6. **DPO consultation** — Data Protection Officer review and sign-off
-
-## Breach Notification
-
-### GDPR (Article 33-34)
-
-| Action | Timeline | Recipient |
-|--------|----------|-----------|
-| Internal assessment | Immediately on discovery | Incident response team |
-| Supervisory authority notification | Within 72 hours | Data protection authority |
-| Data subject notification | Without undue delay | Affected individuals (if high risk) |
-
-### Notification Content
-
-```markdown
-## Breach Notification
-
-- Nature of the breach
-- Categories and approximate number of data subjects affected
-- Categories and approximate number of records affected
-- Name and contact details of DPO
-- Likely consequences of the breach
-- Measures taken or proposed to address the breach
-- Measures to mitigate adverse effects
-```
-
-## Compliance Checklist
-
-- [ ] Legal basis documented for each processing activity
-- [ ] Privacy policy published and current
-- [ ] Consent mechanism with granular purpose selection
-- [ ] Data subject request handling (access, erasure, portability)
-- [ ] Data retention schedule defined and automated
-- [ ] DPIAs completed for high-risk processing
-- [ ] Data processing agreements with all third-party processors
-- [ ] Breach notification procedure documented and tested
-- [ ] PII encrypted at rest and in transit
-- [ ] Audit trails for all PII access
-- [ ] Staff trained on data protection responsibilities
-- [ ] DPO appointed (if required)

package/workflows/workflows/agent-environment-setup/platforms/codex/skills/sanitize-pii/references/data-masking.md

@@ -1,272 +0,0 @@
-# Data Masking Techniques Reference
-
-## Overview
-
-Data masking creates realistic but non-identifying versions of production data for use in non-production environments (staging, development, testing, analytics). Unlike redaction (which removes data), masking replaces data with realistic alternatives that preserve format, distribution, and referential integrity.
-
-## Masking Techniques
-
-### 1. Substitution
-
-Replace real values with values from a lookup table or generated set.
-
-```python
-from faker import Faker
-
-fake = Faker(['en_US', 'en_GB', 'de_DE'])
-
-SUBSTITUTION_MAP = {
-    'first_name': lambda: fake.first_name(),
-    'last_name': lambda: fake.last_name(),
-    'email': lambda row: f"user_{row['id']}@masked.example.com",
-    'phone': lambda: fake.phone_number(),
-    'address': lambda: fake.address(),
-    'company': lambda: fake.company(),
-    'job_title': lambda: fake.job(),
-}
-
-def mask_row(row, table_config):
-    masked = dict(row)
-    for column, generator in table_config.items():
-        if column in masked:
-            masked[column] = generator(row) if callable(generator) else generator
-    return masked
-```
-
-### 2. Shuffling
-
-Swap values between rows within the same column, preserving the value distribution but breaking the link to individuals.
-
-```sql
--- PostgreSQL: shuffle email column
-UPDATE customers c1
-SET email = c2.email
-FROM (
-  SELECT id, email,
-    ROW_NUMBER() OVER (ORDER BY random()) as rn
-  FROM customers
-) c2
-WHERE c1.id = (
-  SELECT id FROM (
-    SELECT id, ROW_NUMBER() OVER (ORDER BY random()) as rn
-    FROM customers
-  ) c3
-  WHERE c3.rn = c2.rn
-);
-```
-
-**Limitation:** Shuffling preserves the exact set of values, so unique or rare values may still be identifiable. Use substitution for small datasets.
-
-### 3. Format-Preserving Encryption (FPE)
-
-Encrypts data while maintaining the original format (length, character set).
-
-```python
-from ff3 import FF3Cipher
-
-# FF3-1 format-preserving encryption
-key = os.environ['FPE_KEY']  # 128 or 192 or 256 bit key
-tweak = os.environ['FPE_TWEAK']  # 56 bit tweak
-
-cipher = FF3Cipher(key, tweak, radix=10)
-
-# Encrypt SSN (preserves format: 9 digits)
-real_ssn = "123456789"
-masked_ssn = cipher.encrypt(real_ssn)
-# Returns: "947583216" (still 9 digits)
-
-# Deterministic: same input always produces same output
-# Reversible: can decrypt with the key (if needed)
-```
-
-**Use when:**
-- Data must pass format validation
-- Downstream systems expect specific formats
-- Reversibility may be needed (with strict key access controls)
-
-### 4. Variance / Perturbation
-
-Add random noise to numerical values, preserving statistical properties.
-
-```python
-import numpy as np
-
-def perturb_numeric(value, variance_pct=10):
-    """Add random noise within +/- variance_pct percent."""
-    noise = np.random.uniform(-variance_pct/100, variance_pct/100)
-    return round(value * (1 + noise), 2)
-
-def perturb_date(date, days_variance=30):
-    """Shift date by random number of days."""
-    shift = timedelta(days=np.random.randint(-days_variance, days_variance))
-    return date + shift
-
-# Usage
-salary = 85000.00
-masked_salary = perturb_numeric(salary, variance_pct=15)
-# Returns: 78,350.00 to 97,750.00 (random within range)
-
-dob = date(1990, 5, 15)
-masked_dob = perturb_date(dob, days_variance=60)
-# Returns: date within 60 days of original
-```
-
-### 5. Generalization
-
-Replace specific values with broader categories.
-
-```python
-GENERALIZATION_RULES = {
-    'age': lambda age: f"{(age // 10) * 10}-{(age // 10) * 10 + 9}",
-    # 34 → "30-39"
-
-    'postal_code': lambda pc: pc[:3] + "**",
-    # 62704 → "627**"
-
-    'date_of_birth': lambda dob: str(dob.year),
-    # 1990-05-15 → "1990"
-
-    'salary': lambda s: f"${(s // 10000) * 10}k-${(s // 10000 + 1) * 10}k",
-    # 85000 → "$80k-$90k"
-}
-```
-
-### 6. Synthetic Data Generation
-
-Generate entirely new datasets that match the statistical properties of production data.
-
-```python
-# Using SDV (Synthetic Data Vault)
-from sdv.single_table import GaussianCopulaSynthesizer
-from sdv.metadata import SingleTableMetadata
-
-# Learn data distribution
-metadata = SingleTableMetadata()
-metadata.detect_from_dataframe(production_df)
-metadata.update_column('email', sdtype='email')
-metadata.update_column('phone', sdtype='phone_number')
-
-synthesizer = GaussianCopulaSynthesizer(metadata)
-synthesizer.fit(production_df)
-
-# Generate synthetic data
-synthetic_df = synthesizer.sample(num_rows=200000)
-# Statistical properties match production, but no real individuals
-```
-
-## Database Masking Pipeline
-
-### PostgreSQL Masking Script
-
-```sql
--- Step 1: Create masking functions
-CREATE OR REPLACE FUNCTION mask_email(email TEXT)
-RETURNS TEXT AS $$
-BEGIN
-  RETURN 'user_' || md5(email)::TEXT || '@masked.example.com';
-END;
-$$ LANGUAGE plpgsql;
-
-CREATE OR REPLACE FUNCTION mask_name(name TEXT)
-RETURNS TEXT AS $$
-DECLARE
-  names TEXT[] := ARRAY['Alex', 'Jordan', 'Sam', 'Casey', 'Morgan',
-                        'Riley', 'Quinn', 'Avery', 'Blake', 'Drew'];
-BEGIN
-  RETURN names[1 + (hashtext(name) % array_length(names, 1))::INT];
-END;
-$$ LANGUAGE plpgsql;
-
-CREATE OR REPLACE FUNCTION mask_phone(phone TEXT)
-RETURNS TEXT AS $$
-BEGIN
-  RETURN '+1-555-' || lpad((hashtext(phone) % 10000000)::TEXT, 7, '0');
-END;
-$$ LANGUAGE plpgsql;
-
--- Step 2: Apply masking
-UPDATE customers SET
-  email = mask_email(email),
-  first_name = mask_name(first_name),
-  last_name = mask_name(last_name || '_last'),
-  phone = mask_phone(phone),
-  national_id = 'XX' || lpad((hashtext(national_id) % 1000000)::TEXT, 6, '0') || 'X',
-  address = 'Masked Address ' || id,
-  date_of_birth = date_of_birth + (hashtext(email) % 60 - 30);  -- shift +/- 30 days
-
--- Step 3: Mask free-text fields (full replacement)
-UPDATE support_tickets SET
-  body = 'Synthetic ticket content for testing purposes. Ticket ID: ' || id,
-  agent_notes = 'Synthetic agent notes for ticket ' || id;
-
-UPDATE medical_records SET
-  diagnosis = 'Condition-' || (hashtext(diagnosis) % 50 + 1),
-  treatment = 'Treatment-Protocol-' || (hashtext(treatment) % 30 + 1),
-  doctor_notes = 'Synthetic clinical notes for record ' || id;
-```
-
-## Referential Integrity
-
-### Cross-Table Consistency
-
-When masking related tables, the same source value must produce the same masked value across all tables.
-
-```python
-class ConsistentMasker:
-    def __init__(self, salt):
-        self.salt = salt
-        self.cache = {}
-
-    def mask(self, value, pii_type):
-        cache_key = f"{pii_type}:{value}"
-        if cache_key not in self.cache:
-            seed = int(hashlib.sha256(
-                f"{self.salt}:{cache_key}".encode()
-            ).hexdigest()[:8], 16)
-            fake.seed_instance(seed)
-
-            if pii_type == 'email':
-                self.cache[cache_key] = f"user_{seed % 999999}@masked.test"
-            elif pii_type == 'name':
-                self.cache[cache_key] = fake.name()
-
-        return self.cache[cache_key]
-
-# Same email in customers and orders tables → same masked email
-masker = ConsistentMasker(salt="random-salt-per-masking-run")
-```
-
-### Foreign Key Preservation
-
-```
-Production:              Masked:
-customers.id = 42        customers.id = 42  (preserved)
-orders.customer_id = 42  orders.customer_id = 42  (preserved)
-customers.email = real   customers.email = masked
-
-FK relationships remain valid because IDs are preserved.
-PII is masked in the columns that contain it.
-```
-
-## Masking Quality Validation
-
-| Check | Method | Expected Result |
-|-------|--------|-----------------|
-| No real PII | Cross-reference with production hashes | Zero matches |
-| Format preserved | Run application validation against masked data | All records pass |
-| FK integrity | Run `ANALYZE` + FK constraint checks | Zero violations |
-| Data distribution | Compare histograms (age, amounts) | Similar shape |
-| Row counts | Compare counts per table | Exact match |
-| Uniqueness | Check unique constraints | No duplicates where originals were unique |
-| Null handling | Check null patterns | Same null distribution |
-
-## Tools
-
-| Tool | Type | Open Source | Features |
-|------|------|------------|----------|
-| Faker | Library | Yes | Generates fake data by locale |
-| SDV | Library | Yes | ML-based synthetic data |
-| Presidio | Library | Yes | PII detection + anonymization |
-| AWS DMS | Service | No | Database migration with masking |
-| Tonic.ai | SaaS | No | Enterprise data masking |
-| Delphix | Enterprise | No | Dynamic data masking |