@zigrivers/scaffold 3.8.0 → 3.9.0

package/content/knowledge/browser-extension/browser-extension-testing.md

@@ -0,0 +1,270 @@
+---
+name: browser-extension-testing
+description: Extension testing with Puppeteer and Playwright, unit testing shared logic, and manual cross-browser smoke test procedures
+topics: [browser-extension, testing, puppeteer, playwright, unit-testing, e2e, smoke-tests, cross-browser]
+---
+
+Browser extension testing is harder than web app testing because the extension runs in a privileged browser context that most test frameworks cannot easily access. The strategy is to maximize the code that lives in plain TypeScript (easily unit-tested), minimize the code that requires a real browser to test (expensive), and write targeted end-to-end tests that exercise the extension in a real browser for the scenarios that matter most.
+
+## Summary
+
+Test browser extensions at three levels: unit tests for all shared and context-agnostic logic (Vitest, no browser required), integration tests for message handlers using `jest-chrome` or `sinon-chrome` to mock the `chrome.*` APIs, and end-to-end tests using Playwright or Puppeteer with the extension loaded into a real browser instance. Run a manual cross-browser smoke test checklist before every release. Never consider an extension release done without smoke testing in both Chrome and Firefox.
+
+## Deep Guidance
+
+### Testing Strategy Overview
+
+The extension architecture determines the testing strategy:
+
+- **Shared logic** (`src/shared/`) — Pure TypeScript functions with no browser API dependencies. Unit test with Vitest, Jest, or any standard test runner. No special setup required.
+- **Background handlers** (`src/background/handlers/`) — Functions that call `chrome.*` APIs. Test with chrome API mocks (`jest-chrome` or manual mocks). No real browser required.
+- **Content script logic** (`src/content/`) — DOM manipulation functions. Test with jsdom (Vitest/Jest built-in) for unit tests. Test integration with Playwright for page injection scenarios.
+- **Popup/options UI** (`src/popup/`, `src/options/`) — React/framework components. Test with component testing tools (Vitest + Testing Library). E2E test the full popup flow with Playwright.
+- **Full extension integration** — Test the complete extension loaded in a real browser with Playwright or Puppeteer.
+
+### Unit Tests for Shared Logic
+
+Unit tests are the fastest feedback loop and highest ROI in extension testing:
+
+```typescript
+// tests/unit/shared/url-helpers.test.ts
+import { describe, it, expect } from 'vitest';
+import { matchesPattern, normalizeUrl } from '../../../src/shared/url-helpers';
+
+describe('matchesPattern', () => {
+  it('matches exact URL', () => {
+    expect(matchesPattern('https://example.com/page', 'https://example.com/page')).toBe(true);
+  });
+
+  it('matches wildcard pattern', () => {
+    expect(matchesPattern('https://example.com/anything', 'https://example.com/*')).toBe(true);
+  });
+
+  it('rejects non-matching URL', () => {
+    expect(matchesPattern('https://other.com/', 'https://example.com/*')).toBe(false);
+  });
+});
+```
+
+**Architecture rule:** Any logic that can be written without importing `chrome.*` or `browser.*` should be. Utilities for URL parsing, config validation, data transformation, and business logic belong in `src/shared/` and are easily unit-tested.
+
+### Mocking chrome.* APIs
+
+For testing background message handlers and storage operations without a real browser, use chrome API mocks:
+
+**jest-chrome** (for Jest or Vitest with compatibility layer):
+
+```bash
+npm install -D jest-chrome
+```
+
+```typescript
+// tests/setup.ts (Vitest global setup)
+import chrome from 'jest-chrome';
+Object.assign(global, { chrome });
+
+// Setup default mock implementations
+chrome.storage.sync.get.mockImplementation((keys, callback) => {
+  callback?.({ enabled: true, sitesConfig: [] });
+  return Promise.resolve({ enabled: true, sitesConfig: [] });
+});
+```
+
+```typescript
+// tests/unit/background/message-router.test.ts
+import { describe, it, expect, vi, beforeEach } from 'vitest';
+import chrome from 'jest-chrome';
+import { handleMessage } from '../../../src/background/message-router';
+import { Messages } from '../../../src/shared/messages';
+
+describe('handleMessage', () => {
+  beforeEach(() => {
+    chrome.storage.sync.get.mockClear();
+  });
+
+  it('responds to POPUP_GET_STATUS with current state', async () => {
+    chrome.storage.sync.get.mockResolvedValue({ enabled: true });
+
+    const sendResponse = vi.fn();
+    handleMessage(
+      { type: Messages.POPUP_GET_STATUS },
+      { id: chrome.runtime.id },
+      sendResponse,
+    );
+
+    await vi.waitFor(() => expect(sendResponse).toHaveBeenCalledWith({ enabled: true }));
+  });
+});
+```
+
+**Manual chrome mock** (for projects that prefer explicit control):
+
+```typescript
+// tests/mocks/chrome.ts
+export const chromeMock = {
+  storage: {
+    sync: {
+      get: vi.fn(),
+      set: vi.fn().mockResolvedValue(undefined),
+    },
+    onChanged: {
+      addListener: vi.fn(),
+      removeListener: vi.fn(),
+    },
+  },
+  runtime: {
+    id: 'test-extension-id',
+    sendMessage: vi.fn(),
+    onMessage: {
+      addListener: vi.fn(),
+    },
+  },
+  tabs: {
+    query: vi.fn(),
+    sendMessage: vi.fn(),
+  },
+};
+```
+
+### End-to-End Tests with Playwright
+
+Playwright supports loading Chrome extensions in a persistent browser context:
+
+```bash
+npm install -D @playwright/test
+npx playwright install chromium
+```
+
+```typescript
+// tests/e2e/extension.spec.ts
+import { test, expect, chromium, BrowserContext } from '@playwright/test';
+import path from 'path';
+
+const extensionPath = path.resolve(__dirname, '../../dist/chrome');
+
+let context: BrowserContext;
+
+test.beforeAll(async () => {
+  context = await chromium.launchPersistentContext('', {
+    headless: false, // Extensions require headed mode in Playwright
+    args: [
+      `--disable-extensions-except=${extensionPath}`,
+      `--load-extension=${extensionPath}`,
+    ],
+  });
+});
+
+test.afterAll(async () => {
+  await context.close();
+});
+
+test('extension popup opens and displays status', async () => {
+  // Get the extension ID from the background service worker URL
+  let extensionId: string;
+  const backgroundPages = context.backgroundPages();
+
+  if (backgroundPages.length > 0) {
+    const backgroundPage = backgroundPages[0];
+    extensionId = new URL(backgroundPage.url()).hostname;
+  } else {
+    // Wait for the service worker
+    const serviceWorker = await context.waitForEvent('serviceworker');
+    extensionId = new URL(serviceWorker.url()).hostname;
+  }
+
+  // Open the popup as a regular page (full Playwright API access)
+  const popupPage = await context.newPage();
+  await popupPage.goto(`chrome-extension://${extensionId}/popup/index.html`);
+
+  await expect(popupPage.locator('#status-indicator')).toBeVisible();
+  await expect(popupPage.locator('#toggle-btn')).toBeEnabled();
+});
+
+test('content script injects on target page', async () => {
+  const page = await context.newPage();
+  await page.goto('https://example.com');
+
+  // Wait for content script to inject
+  await page.waitForSelector('#my-ext-overlay', { timeout: 5000 });
+
+  const overlay = page.locator('#my-ext-overlay');
+  await expect(overlay).toBeVisible();
+});
+```
+
+**Playwright limitations with extensions:**
+- `headless: false` is required — Chrome does not load extensions in headless mode (as of Playwright 1.40). Use `headless: 'new'` with Chromium 112+ for limited headless extension support.
+- The extension must be built before tests run. Wire `build:chrome` to run before the e2e test suite in your CI pipeline.
+
+### End-to-End Tests with Puppeteer
+
+Puppeteer supports Chrome extensions and can run in CI:
+
+```typescript
+// tests/e2e/puppeteer-extension.test.ts
+import puppeteer, { Browser } from 'puppeteer';
+import path from 'path';
+
+const extensionPath = path.resolve(__dirname, '../../dist/chrome');
+
+let browser: Browser;
+let extensionId: string;
+
+beforeAll(async () => {
+  browser = await puppeteer.launch({
+    headless: false, // Required for extensions
+    args: [
+      `--disable-extensions-except=${extensionPath}`,
+      `--load-extension=${extensionPath}`,
+    ],
+  });
+
+  // Find the extension ID by checking the background page URL
+  const targets = await browser.targets();
+  const extensionTarget = targets.find(
+    t => t.type() === 'service_worker' && t.url().includes('background')
+  );
+  extensionId = new URL(extensionTarget!.url()).hostname;
+});
+
+afterAll(async () => {
+  await browser.close();
+});
+
+test('extension popup renders correctly', async () => {
+  const page = await browser.newPage();
+  await page.goto(`chrome-extension://${extensionId}/popup/index.html`);
+  const title = await page.$eval('h1', el => el.textContent);
+  expect(title).toBe('My Extension');
+});
+```
+
+### Manual Cross-Browser Smoke Test Checklist
+
+Automated tests cannot catch everything. Run this checklist before every release:
+
+**Chrome smoke test:**
+- [ ] Build `dist/chrome` with `npm run build:chrome`.
+- [ ] Load unpacked from `chrome://extensions`.
+- [ ] Extension icon appears in toolbar with correct icon at 16×16.
+- [ ] Click toolbar icon — popup opens without errors (check DevTools console for the popup page).
+- [ ] All popup controls are interactive and functional.
+- [ ] Toggle enabled/disabled — page content changes as expected.
+- [ ] Navigate to a target URL — content script injects without errors (check page DevTools console).
+- [ ] Open options page (`chrome://extensions` → Details → Extension options) — loads and saves correctly.
+- [ ] Disable the extension — injected content is removed from the page.
+- [ ] Check `chrome://extensions` — no errors shown under the extension card.
+- [ ] Check service worker DevTools (`Inspect views: Service Worker`) — no uncaught errors.
+
+**Firefox smoke test:**
+- [ ] Build `dist/firefox` with `npm run build:firefox`.
+- [ ] Load via `about:debugging` → This Firefox → Load Temporary Add-on (select `manifest.json`).
+- [ ] Repeat all functional checks from Chrome smoke test.
+- [ ] Check Browser Console (`Ctrl+Shift+J`) — no extension errors.
+- [ ] Verify `browser.storage.sync` reads/writes work (Firefox syncs separately from Chrome).
+
+**Regression test after every change to:**
+- `manifest.json` — Reload the extension and verify all declared features still work.
+- Content script matches — Verify the script injects on all intended URLs and not on excluded URLs.
+- `chrome.storage` schema — Verify existing storage data is read correctly after the schema change.
+- Message types — Verify all message senders and receivers still agree on the message format.

package/content/knowledge/data-pipeline/data-pipeline-architecture.md

@@ -0,0 +1,175 @@
+---
+name: data-pipeline-architecture
+description: Lambda vs Kappa architecture tradeoffs, medallion architecture (bronze/silver/gold), and CDC patterns for data pipelines
+topics: [data-pipeline, architecture, lambda, kappa, medallion, bronze-silver-gold, cdc, change-data-capture]
+---
+
+Data pipeline architecture is the set of structural decisions that determine how data flows from sources to consumers, how it is stored at each stage, and how historical data is reprocessed when logic changes. The wrong architecture creates systems that are either operationally complex (Lambda), too rigid for historical reprocessing (pure streaming), or without clear data quality boundaries (no medallion layers). These decisions are expensive to reverse and must be made explicitly.
+
+## Summary
+
+Lambda architecture separates batch and streaming paths into two systems with a serving layer that merges them — powerful but operationally expensive. Kappa architecture uses a single streaming system with replayable logs for both real-time and historical processing. The medallion architecture (bronze/silver/gold) provides clear data quality tiers regardless of which processing model is used. CDC (Change Data Capture) is the standard mechanism for streaming relational database changes into pipelines.
+
+## Deep Guidance
+
+### Lambda Architecture
+
+Lambda architecture processes data through two parallel paths that converge at query time:
+
+**Batch layer (high latency, high accuracy)**
+- Reprocesses the complete historical dataset on a schedule (daily, hourly)
+- Produces accurate, complete views by operating on all data
+- Tolerates high latency — results available hours after data arrives
+- Implemented with Spark, Hadoop MapReduce, BigQuery batch jobs
+
+**Speed layer (low latency, approximate)**
+- Processes only recent data in real-time as events arrive
+- Produces approximate views with low latency (seconds to minutes)
+- Covers only the gap since the last batch run
+- Implemented with Kafka Streams, Flink, Spark Streaming
+
+**Serving layer**
+- Merges batch and speed layer outputs at query time
+- Returns the batch view for historical ranges; supplements with speed layer for recency
+- Implemented with systems like Cassandra, HBase, or a query engine that unions both stores
+
+**Lambda tradeoffs**
+- Benefits: Accurate historical data (batch), low-latency recent data (speed), well-understood pattern
+- Costs: Two codebases implementing the same business logic (divergence risk), complex serving layer merging logic, double infrastructure cost, testing complexity
+- Use when: Latency requirements genuinely differ for historical vs. recent data, e.g., a dashboard that shows real-time last-hour data but accurate historical monthly reports
+
+The most common Lambda failure mode is the batch and speed layers producing different results for the same time window due to logic divergence. This requires continuous reconciliation effort.
+
+### Kappa Architecture
+
+Kappa architecture replaces the Lambda dual-path with a single streaming system:
+
+**Core premise**: Everything is a stream. Historical reprocessing is just replaying old events through the same streaming pipeline.
+
+**Requirements for Kappa**:
+- Event log is durable and replayable (Kafka with long retention, or event store)
+- Events are immutable and ordered within a partition
+- Processing logic is stateless or uses externally managed state (RocksDB, Redis)
+
+**Kappa pipeline flow**:
+1. All events land in the replayable log (Kafka, Kinesis, Pulsar)
+2. Streaming jobs process events in real-time
+3. When logic changes, deploy new job version and replay from the beginning of the log
+4. New job catches up to current time; old job is decommissioned
+5. Serving layer reads from the streaming job's output store
+
+**Kappa tradeoffs**
+- Benefits: Single codebase for all processing, no logic divergence, simpler architecture, streaming-native
+- Costs: Reprocessing large historical datasets takes time and cluster resources, log retention costs scale with history depth, complex state management for aggregations over long windows
+- Use when: Business logic is unified (same calculation for historical and real-time), event log is durable, acceptable reprocessing latency for backfills
+
+**Choosing Lambda vs. Kappa**
+
+| Factor | Prefer Lambda | Prefer Kappa |
+|--------|---------------|--------------|
+| Logic divergence risk | High (two systems) | Low (one system) |
+| Infrastructure cost | High | Lower |
+| Real-time latency | Sub-second achievable | Sub-second achievable |
+| Historical reprocessing | Fast (dedicated batch) | Slower (streaming catchup) |
+| Team capability | Strong batch + streaming | Streaming-focused team |
+| Use today | Rarely — mostly legacy | Default choice for new pipelines |
+
+### Medallion Architecture
+
+The medallion architecture (also called multi-hop) organizes data into three quality tiers regardless of whether Lambda or Kappa is used:
+
+**Bronze layer — Raw ingestion**
+- Stores data exactly as received from source systems, with no transformations
+- Append-only; records are never modified or deleted (except for compliance purges)
+- Schema-on-read: no schema enforcement at write time
+- Retains original field names, formats, and any encoding quirks from the source
+- Adds pipeline metadata: `_ingested_at`, `_source_system`, `_pipeline_version`, `_raw_id`
+- Retention: indefinite (or compliance minimum) — this is the recovery point for all downstream layers
+
+Bronze is the safety net. When a transformation bug is discovered, bronze data allows rebuilding silver and gold from scratch without re-ingesting from source.
+
+**Silver layer — Cleaned and conformed**
+- Applies schema enforcement, type casting, and null handling from bronze
+- Deduplicates records using business keys
+- Normalizes field names to the canonical data model (snake_case, consistent naming)
+- Resolves encoding issues, trims whitespace, standardizes date formats
+- Joins with slowly-changing dimension tables (currency conversion rates, country codes)
+- Enforces data quality rules; routes failing records to DLQ
+- Schema-on-write: strict Avro/Parquet schema applied at write time
+- Retention: 1–3 years (or per regulatory requirement)
+
+Silver is the trusted, clean, integrated dataset. Most analytical consumers should read from silver, not bronze.
+
+**Gold layer — Aggregated and business-ready**
+- Pre-aggregated views optimized for specific business questions
+- Applies business logic: revenue rollups, user segmentation, cohort calculations
+- Denormalized for query performance (no joins required by consumers)
+- Named for the business concept, not the data source: `daily_revenue`, `user_ltv`, `product_performance`
+- Retention: dependent on business reporting requirements (often indefinitely for monthly/yearly aggregates)
+
+Gold is what business users, dashboards, and ML features consume. Schema changes in gold require migration planning and consumer coordination.
+
+**Medallion implementation rules**
+- Bronze → Silver is an automated pipeline; never manually edit bronze data
+- Silver → Gold is also automated; never manually edit silver data
+- Data flows only forward (bronze → silver → gold), never backward
+- Consumer systems should read from the highest-quality layer that satisfies their latency requirements
+- Schema changes in bronze require no consumer coordination; changes in silver/gold do
+
+### CDC (Change Data Capture) Patterns
+
+CDC streams database changes (INSERT, UPDATE, DELETE) as events into the pipeline without polling or application-layer hooks.
+
+**Why CDC**
+- Zero-impact on source database performance (reads WAL, not production tables)
+- Sub-second latency from database commit to pipeline event
+- Captures all changes including bulk updates and direct SQL modifications
+- Works for initial data load and ongoing changes through the same mechanism
+
+**CDC implementation options**
+
+*Log-based CDC (recommended)*: Reads the database Write-Ahead Log (WAL) directly
+- PostgreSQL: logical replication using `pgoutput` plugin (Debezium connector)
+- MySQL: reads binary log (binlog) using Debezium MySQL connector
+- SQL Server: uses Change Data Capture feature built into SQL Server
+
+*Query-based CDC (polling)*: Queries for rows modified after a watermark timestamp
+- Simpler to implement but misses DELETEs, requires `updated_at` column, higher DB load
+- Acceptable for low-volume tables where log-based CDC is not available
+
+*Triggers*: Database triggers write changes to a staging table
+- High database overhead, blocking risk, not recommended for high-volume tables
+
+**Debezium CDC event structure**
+
+Debezium transforms each database change into a structured event:
+
+```json
+{
+  "before": { "id": 123, "status": "pending", "amount": 100.00 },
+  "after":  { "id": 123, "status": "completed", "amount": 100.00 },
+  "op": "u",
+  "ts_ms": 1705329825000,
+  "source": {
+    "db": "payments",
+    "table": "transactions",
+    "lsn": 1847392
+  }
+}
+```
+
+`op` values: `c` (create/insert), `u` (update), `d` (delete), `r` (read, initial snapshot)
+
+**CDC pipeline topology**
+
+```
+Source DB → Debezium → Kafka (per-table topics) → Stream Processor → Bronze → Silver
+```
+
+Each source table gets its own Kafka topic: `{db}.{schema}.{table}`. This allows consumers to subscribe to specific tables independently.
+
+**CDC operational considerations**
+- Schema changes in the source database must be handled gracefully (Avro schema evolution, schema registry)
+- Initial snapshot: Debezium can snapshot existing table data before beginning to tail the log — manage this carefully for large tables (can take hours)
+- Replication slot management: PostgreSQL logical replication slots accumulate WAL if the consumer falls behind — monitor replication slot lag as a critical metric
+- Exactly-once delivery: Kafka + Debezium provides at-least-once delivery; implement idempotent consumers for exactly-once semantics