@zigrivers/scaffold 3.8.0 → 3.9.1

package/content/knowledge/data-pipeline/data-pipeline-security.md

@@ -0,0 +1,326 @@
+---
+name: data-pipeline-security
+description: PII handling, encryption at rest and in transit, access control, and audit logging for data pipelines
+topics: [data-pipeline, security, pii, encryption, access-control, audit-logging, gdpr, data-masking]
+---
+
+Data pipelines are high-value targets for security incidents because they aggregate sensitive data from multiple systems and often run with broad service account permissions. A pipeline that ingests payment data, user PII, and health records and writes to a central warehouse becomes a single point of compromise. Security must be designed into the pipeline from the start — encryption, least-privilege access, PII handling, and audit logging cannot be retrofitted economically.
+
+## Summary
+
+Encrypt all data at rest and in transit using current standards (TLS 1.2+, AES-256). Apply least-privilege access: each pipeline service account reads only the sources it needs and writes only to its designated destinations. Detect and classify all PII fields at ingestion; mask, tokenize, or drop PII before it reaches analytical layers. Log all data access and transformation events to an immutable audit trail. Test security controls in CI.
+
+## Deep Guidance
+
+### Encryption at Rest
+
+Every data store the pipeline writes to must encrypt data at rest:
+
+**Object storage (S3, GCS, Azure Blob)**
+
+```python
+# S3: server-side encryption with AWS KMS
+s3_client = boto3.client("s3")
+s3_client.put_object(
+    Bucket="raw-data",
+    Key="payments/2024-01-15/transactions.parquet",
+    Body=parquet_bytes,
+    ServerSideEncryption="aws:kms",
+    SSEKMSKeyId="arn:aws:kms:us-east-1:123456789:key/mrk-abc123",
+)
+```
+
+Use customer-managed KMS keys (CMKs), not AWS-managed keys, for data subject to regulatory requirements. CMKs allow key rotation control and key deletion for data erasure compliance (GDPR right to erasure via key deletion).
+
+**Database encryption**
+
+Configure encryption at rest for all databases the pipeline reads from or writes to:
+- PostgreSQL: `pg_tde` extension or filesystem-level encryption (LUKS, dm-crypt)
+- BigQuery: CMEK (customer-managed encryption keys) via `--kms_key` parameter
+- Snowflake: Tri-Secret Secure (Snowflake + customer key + Snowflake-managed key)
+- Kafka: enable log encryption using KMS-backed encryption keys
+
+**Parquet/Avro field-level encryption**
+
+For sensitive fields, apply field-level encryption in addition to file-level encryption. This allows analytic queries on encrypted files while keeping specific columns (PII) unreadable without the column-level key:
+
+```python
+from pyarrow.parquet import ParquetFile
+from pyarrow import compute as pc
+
+# Write with field-level encryption using Apache Parquet encryption
+encryption_config = pq.EncryptionConfiguration(
+    footer_key="footer_key_id",
+    column_keys={
+        "user_pii_key": ["email", "phone", "ssn_hash"],
+        "default_key": None,  # all other columns use footer key
+    },
+)
+pq.write_table(table, "output.parquet", encryption_properties=encryption_config)
+```
+
+### Encryption in Transit
+
+All data in transit must use TLS 1.2 or higher. Disable older protocols and weak cipher suites:
+
+**Kafka TLS configuration**
+
+```properties
+# Kafka broker: enforce TLS
+listeners=SSL://:9093
+ssl.keystore.location=/ssl/kafka.server.keystore.jks
+ssl.keystore.password=${KEYSTORE_PASSWORD}
+ssl.truststore.location=/ssl/kafka.server.truststore.jks
+ssl.truststore.password=${TRUSTSTORE_PASSWORD}
+ssl.client.auth=required
+ssl.enabled.protocols=TLSv1.2,TLSv1.3
+ssl.cipher.suites=TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256
+
+# Producer/consumer: mutual TLS
+security.protocol=SSL
+ssl.keystore.location=/ssl/client.keystore.jks
+ssl.keystore.password=${CLIENT_KEYSTORE_PASSWORD}
+```
+
+**API and database connections**
+
+```python
+# PostgreSQL: require SSL
+conn = psycopg2.connect(
+    host="db.example.com",
+    dbname="source_db",
+    user=os.environ["DB_USER"],
+    password=os.environ["DB_PASSWORD"],
+    sslmode="verify-full",           # reject if cert invalid
+    sslrootcert="/certs/ca.pem",
+)
+
+# HTTP clients: verify certificates, no plain HTTP
+import httpx
+client = httpx.Client(verify=True)  # default; never set verify=False in production
+```
+
+### Least-Privilege Access Control
+
+Each pipeline component should have the minimum permissions necessary to perform its function. Service accounts should never have admin-level permissions.
+
+**AWS IAM policy for pipeline service account**
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["s3:GetObject", "s3:ListBucket"],
+      "Resource": [
+        "arn:aws:s3:::raw-data",
+        "arn:aws:s3:::raw-data/payments/*"
+      ]
+    },
+    {
+      "Effect": "Allow",
+      "Action": ["s3:PutObject"],
+      "Resource": "arn:aws:s3:::silver-data/transactions/*"
+    },
+    {
+      "Effect": "Deny",
+      "Action": ["s3:DeleteObject", "s3:DeleteBucket"],
+      "Resource": "*"
+    }
+  ]
+}
+```
+
+**BigQuery column-level access control**
+
+```python
+# Grant access to non-PII columns only for analytics users
+from google.cloud import bigquery
+
+client = bigquery.Client()
+table_ref = client.get_table("project.silver.transactions")
+
+# Column-level security policy: analytics role cannot read PII columns
+table_ref.schema = [
+    bigquery.SchemaField("transaction_id", "STRING"),
+    bigquery.SchemaField("amount", "FLOAT64"),
+    bigquery.SchemaField("currency", "STRING"),
+    bigquery.SchemaField("email", "STRING",
+        policy_tags=bigquery.PolicyTagList(["projects/project/locations/us/taxonomies/123/policyTags/456"])),
+]
+```
+
+**Secret management**
+
+Never store credentials in pipeline code, DAG files, config files, or environment variables on shared machines:
+
+```python
+# Bad: hardcoded in code
+DB_PASSWORD = "s3cr3t"
+
+# Bad: plaintext in config file
+# config/prod.yaml: db_password: s3cr3t
+
+# Good: fetch from secrets manager at runtime
+import boto3
+
+def get_secret(secret_name: str) -> str:
+    client = boto3.client("secretsmanager", region_name="us-east-1")
+    response = client.get_secret_value(SecretId=secret_name)
+    return response["SecretString"]
+
+DB_PASSWORD = get_secret("prod/pipeline/db-password")
+```
+
+Use short-lived credentials where possible (IAM roles with STS, Workload Identity for GKE) rather than long-lived API keys or passwords.
+
+### PII Handling
+
+**PII detection at ingestion**
+
+Scan incoming records for PII patterns before writing to bronze:
+
+```python
+import re
+from dataclasses import dataclass
+
+PII_PATTERNS = {
+    "email": re.compile(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b"),
+    "ssn": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
+    "credit_card": re.compile(r"\b(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14})\b"),
+    "phone": re.compile(r"\b\+?1?\s?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}\b"),
+}
+
+def detect_pii(record: dict) -> dict[str, list[str]]:
+    """Return mapping of PII type to field names containing that PII."""
+    findings = {}
+    for field_name, value in record.items():
+        if not isinstance(value, str):
+            continue
+        for pii_type, pattern in PII_PATTERNS.items():
+            if pattern.search(value):
+                findings.setdefault(pii_type, []).append(field_name)
+    return findings
+```
+
+**PII masking strategies**
+
+Choose the masking strategy based on the downstream use case:
+
+| Strategy | Description | When to use |
+|----------|-------------|-------------|
+| Tokenization | Replace PII with consistent token (HMAC, UUID) | Join keys across datasets; token is stable |
+| Hashing | One-way hash (SHA-256 + pepper) | Deduplication; cannot reverse |
+| Masking | Replace with masked value (`****@example.com`) | Display in dashboards |
+| Pseudonymization | Replace with synthetic but realistic value | ML training data |
+| Suppression | Remove field entirely | When field has no analytical value |
+
+```python
+import hashlib
+import hmac
+
+PII_PEPPER = os.environ["PII_PEPPER"]  # secret pepper stored in secrets manager
+
+def tokenize_pii(value: str) -> str:
+    """Generate a consistent, irreversible token for a PII value."""
+    return hmac.new(
+        PII_PEPPER.encode(),
+        value.lower().strip().encode(),
+        hashlib.sha256,
+    ).hexdigest()
+
+def mask_email(email: str) -> str:
+    """Mask email address: j***@example.com"""
+    local, domain = email.split("@", 1)
+    return f"{local[0]}***@{domain}"
+```
+
+**PII in logs**
+
+Ensure PII never appears in pipeline logs. Implement a log sanitizer:
+
+```python
+import logging
+
+SENSITIVE_PATTERNS = [
+    (re.compile(r'email=["\']?[\w.@+]+', re.I), 'email=***'),
+    (re.compile(r'password=["\']?[\S]+', re.I), 'password=***'),
+    (re.compile(r'\b\d{3}-\d{2}-\d{4}\b'), '***-**-****'),
+]
+
+class PIIRedactingFilter(logging.Filter):
+    def filter(self, record: logging.LogRecord) -> bool:
+        message = str(record.getMessage())
+        for pattern, replacement in SENSITIVE_PATTERNS:
+            message = pattern.sub(replacement, message)
+        record.msg = message
+        record.args = ()
+        return True
+```
+
+### Audit Logging
+
+Audit logs record who accessed what data, when, and what they did with it. They are required for GDPR, SOC 2, and HIPAA compliance.
+
+**Audit event structure**
+
+```python
+from dataclasses import dataclass
+from datetime import datetime
+
+@dataclass
+class AuditEvent:
+    timestamp: datetime
+    pipeline_id: str
+    pipeline_version: str
+    service_account: str
+    action: str              # "read", "write", "delete", "transform"
+    resource: str            # table name, topic, file path
+    record_count: int
+    pii_fields_accessed: list[str]
+    execution_id: str        # correlation ID for tracing
+    environment: str
+
+def emit_audit_event(event: AuditEvent) -> None:
+    """Write audit event to immutable audit log (append-only, no delete permission)."""
+    audit_client.log(
+        log_name="data-pipeline-audit",
+        payload=dataclasses.asdict(event),
+        severity="INFO",
+    )
+```
+
+**Immutable audit log requirements**:
+- Stored in a separate, append-only log store from pipeline data
+- Service accounts that write to audit logs cannot delete from them
+- Retain audit logs for minimum 7 years (or regulatory requirement)
+- Export to SIEM system for anomaly detection
+
+**GDPR erasure audit trail**
+
+When processing a right-to-erasure request, emit a deletion audit event for every store purged:
+
+```python
+def process_erasure_request(user_id: str, request_id: str) -> ErasureResult:
+    """Process GDPR right-to-erasure request."""
+    stores_purged = []
+    for store in REGISTERED_PII_STORES:
+        count = store.delete_user_records(user_id)
+        if count > 0:
+            emit_audit_event(AuditEvent(
+                action="delete",
+                resource=store.name,
+                record_count=count,
+                pii_fields_accessed=store.pii_fields,
+                execution_id=request_id,
+            ))
+            stores_purged.append(store.name)
+
+    return ErasureResult(
+        user_id=user_id,
+        request_id=request_id,
+        stores_purged=stores_purged,
+        completed_at=datetime.utcnow(),
+    )
+```

package/content/knowledge/data-pipeline/data-pipeline-streaming-patterns.md

@@ -0,0 +1,280 @@
+---
+name: data-pipeline-streaming-patterns
+description: Event time vs processing time, windowing strategies, watermarks, and exactly-once semantics for streaming data pipelines
+topics: [data-pipeline, streaming, event-time, processing-time, windowing, watermarks, exactly-once, flink, kafka-streams]
+---
+
+Streaming pipeline correctness depends on understanding the distinction between when an event occurred and when it was processed, handling late-arriving data with watermarks, and ensuring that each event is processed exactly once even in the face of failures. Getting these fundamentals wrong produces pipelines that appear to work but silently compute incorrect aggregations — bugs that surface only when business users notice wrong numbers weeks later.
+
+## Summary
+
+Use event time (when the event occurred) rather than processing time (when it was received) for all time-based aggregations. Define watermarks to bound how late an event can arrive before being dropped. Choose window types (tumbling, sliding, session) based on the business question. Implement exactly-once semantics at the processing layer and idempotent sinks to guarantee end-to-end exactly-once delivery.
+
+## Deep Guidance
+
+### Event Time vs. Processing Time
+
+Every streaming event has two timestamps:
+
+- **Event time**: When the event actually occurred in the real world (the user clicked, the transaction was authorized, the sensor fired)
+- **Processing time**: When the event arrived at and was processed by the pipeline
+
+These two times are rarely the same. Events are delayed by:
+- Network latency (milliseconds to seconds)
+- Mobile clients syncing offline activity (minutes to hours)
+- Source system batch uploads (hours to days)
+- Retries and error recovery (variable)
+
+**Why event time matters**
+
+Computing aggregations using processing time produces incorrect results when events arrive late. A mobile app that syncs a purchase made at 11:58 PM at 12:05 AM should count that purchase against the 11 PM hour, not the midnight hour — if using processing time, it appears in the wrong time window.
+
+**Always use event time for**:
+- Revenue and transaction aggregations
+- User activity metrics
+- Time-series analytics
+- SLA measurement
+
+**Processing time is appropriate for**:
+- Pipeline throughput monitoring (how many events did the system process per second)
+- System health metrics
+- Deduplication windows within the pipeline itself (not based on business events)
+
+### Watermarks
+
+A watermark is the pipeline's estimate of how far behind the current event time is relative to the real world. It defines the maximum expected delay for events: events more than W minutes late are considered lost and their window closes.
+
+**Watermark definition**
+
+```
+watermark(t) = max(event_time seen so far) - max_delay
+```
+
+If the pipeline has seen events with timestamps up to 14:55:00, and the configured max delay is 5 minutes, the watermark is 14:50:00. All windows ending before 14:50:00 are considered complete and can be emitted.
+
+**Choosing watermark delay**
+
+The watermark delay must be large enough to allow late-arriving events to arrive, but small enough to not indefinitely delay results:
+
+| Event source | Typical delay | Recommended watermark |
+|-------------|--------------|----------------------|
+| Server-side events | < 1 second | 10–30 seconds |
+| Mobile app events | < 5 minutes | 5–15 minutes |
+| IoT devices | < 1 minute | 2–5 minutes |
+| Batch file uploads | < 1 hour | 2–4 hours |
+
+**Late event handling strategies**
+
+When an event arrives after the watermark (too late for its window):
+1. **Drop** (default): The event is discarded; the window has already closed
+2. **Side output / late data stream**: Route to a separate stream for separate processing or DLQ
+3. **Allowed lateness**: Keep the window open for an extra duration, recompute and emit updated results
+
+```python
+# Apache Flink: watermark with late data handling
+stream
+    .assignTimestampsAndWatermarks(
+        WatermarkStrategy
+            .forBoundedOutOfOrderness(Duration.ofMinutes(5))  # 5-minute watermark
+            .withTimestampAssigner(lambda e, _: e['event_timestamp'])
+    )
+    .key_by(lambda e: e['merchant_id'])
+    .window(TumblingEventTimeWindows.of(Time.hours(1)))
+    .allowed_lateness(Duration.ofMinutes(30))  # keep window 30 min past watermark
+    .side_output_late_data(late_output_tag)    # capture late events
+    .aggregate(RevenueAggregator())
+```
+
+### Window Types
+
+Choose the window type based on the business question:
+
+**Tumbling windows (fixed, non-overlapping)**
+
+Each event belongs to exactly one window. Windows are discrete and consecutive.
+
+```
+|--- 10:00 – 11:00 ---|--- 11:00 – 12:00 ---|--- 12:00 – 13:00 ---|
+```
+
+Use for: Hourly/daily aggregations, fixed-period KPIs, batch-style streaming.
+
+```python
+# Transactions per hour
+stream.window(TumblingEventTimeWindows.of(Time.hours(1)))
+      .aggregate(CountAggregator())
+```
+
+**Sliding windows (fixed size, overlapping)**
+
+Windows slide at a fixed interval. Each event may belong to multiple windows.
+
+```
+Window size=60min, slide=15min:
+|--- 10:00 – 11:00 ---|
+         |--- 10:15 – 11:15 ---|
+                  |--- 10:30 – 11:30 ---|
+```
+
+Use for: Moving averages, rolling metrics, anomaly detection over recent history.
+
+```python
+# 1-hour rolling average, updated every 15 minutes
+stream.window(SlidingEventTimeWindows.of(Time.hours(1), Time.minutes(15)))
+      .aggregate(MovingAverageAggregator())
+```
+
+**Session windows (dynamic, gap-based)**
+
+Windows close after a period of inactivity. Window size varies per user.
+
+```
+User A: |--active--| gap |--active--|
+User B:      |----active----|
+```
+
+Use for: User session analysis, visit duration, activity burst detection.
+
+```python
+# Session window with 30-minute inactivity gap
+stream.key_by(lambda e: e['user_id'])
+      .window(EventTimeSessionWindows.with_gap(Time.minutes(30)))
+      .aggregate(SessionAggregator())
+```
+
+**Global windows**
+
+Unbounded single window; requires custom trigger to emit results.
+
+Use for: Stream-table joins, custom trigger-based aggregations.
+
+### Exactly-Once Semantics
+
+Exactly-once means each event is processed and its effects are reflected in the output exactly once, even if the processing system fails and restarts.
+
+**Delivery guarantees**
+
+- **At-most-once**: Events may be lost, never duplicated. Simplest but loses data on failure.
+- **At-least-once**: Events are never lost but may be duplicated. Requires idempotent consumers.
+- **Exactly-once**: Events are neither lost nor duplicated. Most complex.
+
+Most production streaming systems combine at-least-once delivery with idempotent sinks to achieve end-to-end exactly-once behavior.
+
+**Flink exactly-once checkpointing**
+
+Apache Flink implements exactly-once via distributed checkpointing:
+
+```python
+env = StreamExecutionEnvironment.get_execution_environment()
+env.enable_checkpointing(60000)  # checkpoint every 60 seconds
+
+checkpoint_config = env.get_checkpoint_config()
+checkpoint_config.set_checkpointing_mode(CheckpointingMode.EXACTLY_ONCE)
+checkpoint_config.set_min_pause_between_checkpoints(30000)  # 30s between checkpoints
+checkpoint_config.set_checkpoint_timeout(120000)            # fail if checkpoint takes >2min
+checkpoint_config.enable_externalized_checkpoints(
+    ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION
+)
+```
+
+Flink checkpoints snapshot the full operator state to durable storage (S3, HDFS). On failure, the job restarts from the last successful checkpoint and replays events from Kafka from the committed offset.
+
+**Kafka Streams exactly-once**
+
+```python
+props = {
+    "processing.guarantee": "exactly_once_v2",  # EOS v2 (Kafka 2.5+)
+    "bootstrap.servers": "localhost:9092",
+    "application.id": "payments-processor",
+}
+```
+
+Kafka Streams EOS uses transactions: reads from input topics, writes to output topics, and commits consumer offsets atomically in a single Kafka transaction. Either all three happen or none do.
+
+**Idempotent sinks for exactly-once delivery**
+
+Even with exactly-once processing semantics, the sink (database, warehouse) must be idempotent to achieve end-to-end exactly-once:
+
+```python
+# PostgreSQL: use INSERT ON CONFLICT for idempotent writes
+def write_to_postgres(records: list[dict], conn) -> None:
+    with conn.cursor() as cur:
+        for record in records:
+            cur.execute("""
+                INSERT INTO transactions (id, amount, currency, processed_at)
+                VALUES (%(id)s, %(amount)s, %(currency)s, %(processed_at)s)
+                ON CONFLICT (id) DO UPDATE SET
+                    amount = EXCLUDED.amount,
+                    currency = EXCLUDED.currency,
+                    processed_at = EXCLUDED.processed_at
+            """, record)
+
+# BigQuery: use MERGE for idempotent loads
+MERGE_SQL = """
+MERGE silver_transactions T
+USING (SELECT * FROM UNNEST(@records)) S
+ON T.transaction_id = S.transaction_id
+WHEN MATCHED THEN UPDATE SET
+    amount = S.amount, currency = S.currency
+WHEN NOT MATCHED THEN INSERT ROW
+"""
+```
+
+### State Management in Streaming
+
+Stateful streaming operations (aggregations, joins, deduplication) require managing state across events:
+
+**State backend selection**
+
+| Backend | Use case | Tradeoffs |
+|---------|----------|-----------|
+| In-memory (HashMap) | Dev/test, small state | Lost on failure, no persistence |
+| RocksDB (embedded) | Production, large state | Persisted locally, spills to disk |
+| Remote (Redis, DynamoDB) | Cross-instance shared state | Network latency, operational cost |
+
+**State TTL (Time-To-Live)**
+
+Always set TTL on state to prevent unbounded state growth:
+
+```python
+# Flink: expire deduplication state after 24 hours
+state_ttl_config = StateTtlConfig \
+    .new_builder(Time.hours(24)) \
+    .set_update_type(StateTtlConfig.UpdateType.OnReadAndWrite) \
+    .set_state_visibility(StateTtlConfig.StateVisibility.NeverReturnExpired) \
+    .build()
+
+dedup_state = RuntimeContext.get_state(
+    ValueStateDescriptor("seen_events", Types.STRING())
+)
+dedup_state.enable_time_to_live(state_ttl_config)
+```
+
+**Deduplication using state**
+
+```python
+class DeduplicationFunction(KeyedProcessFunction):
+    def open(self, parameters):
+        state_descriptor = ValueStateDescriptor("seen", Types.BOOLEAN())
+        ttl_config = StateTtlConfig.new_builder(Time.hours(24)).build()
+        state_descriptor.enable_time_to_live(ttl_config)
+        self.seen = self.runtime_context.get_state(state_descriptor)
+
+    def process_element(self, event, ctx, out):
+        if self.seen.value() is None:
+            self.seen.update(True)
+            out.collect(event)  # first occurrence: emit
+        # else: duplicate, silently drop
+```
+
+### Streaming Pipeline Operational Metrics
+
+Monitor these metrics on every production streaming pipeline:
+
+- **Consumer lag** (Kafka): events behind the latest offset per partition
+- **Event time lag**: difference between watermark and wall clock
+- **Checkpoint duration and success rate** (Flink)
+- **Late event rate**: percentage of events arriving after watermark
+- **DLQ write rate**: events being routed to dead-letter queue
+- **Throughput** (events/sec, bytes/sec): actual vs. capacity headroom
+- **State size**: total operator state in bytes; watch for unbounded growth