groove-dev 0.27.70 → 0.27.72

package/MOE_TRAINING_PIPELINE.md

@@ -0,0 +1,720 @@
+# MoE Training Data Pipeline — Daemon Team Integration Guide
+
+Handoff spec from the Network Team to the Daemon Team.
+Last updated: 2026-04-23 | Target: HN launch April 26, 2026
+
+---
+
+## 1. Overview
+
+The MoE training pipeline captures opted-in user workflow data (prompts,
+completions, conversations) and stores it as PII-scrubbed JSONL for future
+MoE expert training. The pipeline has two halves:
+
+**Network team built (DONE):**
+- Consent management (SQLite-backed, GDPR-style)
+- PII scrubber (regex-based, 13 pattern categories)
+- Domain tagger (journalism, code, research, planning, general)
+- Training corpus storage (daily-rotated JSONL)
+- Intake API and CaptureSession — the entry points the daemon calls
+- Corpus statistics and reporting
+
+**Daemon team builds (THIS DOC):**
+- User-facing opt-in toggle in the Groove app
+- Data capture hooks at prompt/completion/session boundaries
+- Background capture queue (non-blocking)
+- Data management UI (view stats, download, delete)
+
+The daemon team does NOT need to implement PII scrubbing, consent
+checking, domain classification, or storage. The network team's API
+handles all of that internally. The daemon team's job is to wire up
+the capture points and build the user-facing opt-in experience.
+
+**Timeline:** Must be ready for HN launch on April 26, 2026. New users
+who opt in should immediately start contributing training data.
+
+---
+
+## 2. What the Network Team Built (Backend API)
+
+All source code lives in `moe-team/src/training/`. The daemon team
+interacts with two main classes: `TrainingDataIntake` (simple per-call
+API) and `CaptureSession` (session-oriented wrapper with lifecycle).
+
+### 2.1 TrainingDataIntake — Single Entry Point
+
+```python
+from moe-team.src.training.intake import TrainingDataIntake
+```
+
+Constructor:
+
+```python
+class TrainingDataIntake:
+    def __init__(
+        self,
+        consent_manager: ConsentManager,
+        corpus: TrainingCorpus,
+        scrubber: PIIScrubber,
+        tagger: DomainTagger,
+    ) -> None
+```
+
+#### submit()
+
+```python
+def submit(
+    self,
+    user_id: str,
+    session_id: str,
+    content: str,
+    content_type: str,
+    metadata: dict[str, Any] | None = None,
+) -> str | None
+```
+
+Returns `record_id` (hex UUID) on success, `None` if user is not opted in.
+
+What happens inside submit():
+1. Checks `consent_manager.is_opted_in(user_id)` — returns None if not
+2. Runs `scrubber.scrub(content)` — strips all PII
+3. Runs `tagger.tag(scrubbed_content)` — classifies domain
+4. Looks up current `consent_version` from consent history
+5. Creates a `TrainingRecord` and writes it to JSONL via `corpus.add_record()`
+
+The daemon team does NOT need to scrub PII or check consent — submit()
+handles both internally.
+
+#### submit_batch()
+
+```python
+def submit_batch(
+    self,
+    records: list[dict[str, Any]],
+) -> dict[str, Any]
+# Returns: {"accepted": int, "rejected": int, "record_ids": list[str]}
+```
+
+Each record in the list must have keys: `user_id`, `session_id`,
+`content`, `content_type`, and optionally `metadata`.
+
+#### delete_user()
+
+```python
+def delete_user(self, user_id: str) -> dict[str, Any]
+# Returns: {"consent_revoked": bool, "records_deleted": int}
+```
+
+Revokes consent AND deletes all stored training data for the user.
+Call this when a user opts out and chooses to delete their data.
+
+### 2.2 CaptureSession — Session-Oriented Wrapper
+
+```python
+from moe-team.src.training import CaptureSession
+```
+
+Designed for per-session capture with start/end lifecycle and built-in
+stats tracking.
+
+```python
+class CaptureSession:
+    def __init__(
+        self,
+        user_id: str,
+        session_id: str,
+        consent_manager: ConsentManager,
+        corpus: TrainingCorpus,
+        scrubber: PIIScrubber,
+        tagger: DomainTagger,
+    ) -> None
+
+    def start(self) -> None
+    # Raises PermissionError if user not opted in
+
+    def record_prompt(self, text: str, metadata: dict[str, Any] | None = None) -> None
+    def record_completion(self, text: str, metadata: dict[str, Any] | None = None) -> None
+    def record_conversation(self, messages: list[dict[str, Any]], metadata: dict[str, Any] | None = None) -> None
+
+    def end(self) -> dict[str, Any]
+    # Returns: {"session_id", "records_captured", "bytes_captured", "domains", "duration_seconds"}
+
+    def is_active(self) -> bool
+    # Re-checks consent on every call — stops if user revokes mid-session
+```
+
+CaptureSession is the recommended API for the daemon team. It handles
+consent re-checking on every capture call, so if a user revokes consent
+mid-session, capture stops immediately.
+
+### 2.3 ConsentManager — Consent Operations
+
+```python
+from moe-team.src.training import ConsentManager
+```
+
+```python
+class ConsentManager:
+    def __init__(self, db_path: str = "~/.groove/consent.db") -> None
+
+    def record_consent(
+        self,
+        user_id: str,
+        opted_in: bool,
+        consent_version: str,
+        metadata: dict[str, Any] | None = None,
+    ) -> None
+
+    def is_opted_in(self, user_id: str) -> bool
+
+    def revoke_consent(self, user_id: str) -> bool
+    # Returns False if user had no consent records
+
+    def request_deletion(self, user_id: str) -> dict[str, Any]
+    # Returns: {"user_id", "consent_records", "status": "pending"}
+
+    def get_opted_in_count(self) -> int
+
+    def get_consent_history(self, user_id: str) -> list[ConsentRecord]
+```
+
+Key behavior of `is_opted_in()`:
+- Returns `False` if no consent record exists (default opt-out)
+- Returns `False` if the latest consent version doesn't match
+  `CURRENT_CONSENT_VERSION` (currently "1.0") — this forces re-consent
+  when terms change
+
+### 2.4 CorpusStats — Monitoring
+
+```python
+from moe-team.src.training import CorpusStats
+
+class CorpusStats:
+    def __init__(self, corpus: TrainingCorpus, consent_manager: ConsentManager) -> None
+
+    def summary(self) -> dict[str, Any]
+    # Returns: {"total_records", "storage_size_mb", "opted_in_users", "domains"}
+
+    def daily_growth(self, days: int = 7) -> list[dict[str, Any]]
+    # Returns: [{"date": "2026-04-26", "records": 142}, ...]
+
+    def domain_breakdown(self) -> dict[str, Any]
+    # Returns: {"journalism": {"count": 50, "percentage": 33.3}, ...}
+
+    def print_report(self) -> None
+    # Prints formatted report to stdout
+```
+
+---
+
+## 3. What the Daemon Team Needs to Build
+
+### 3.1 User-Facing Opt-In
+
+Add a settings toggle in the Groove app/IDE:
+
+**Label:** "Share usage data to improve Groove"
+**Default state:** OFF. No data collected without explicit user action.
+
+**When toggled ON:**
+
+1. Show a clear disclosure dialog before enabling. The disclosure must state:
+   - What data is collected: prompts, completions, workflow metadata
+   - How it's used: training MoE expert models to improve Groove
+   - PII is automatically scrubbed before storage (emails, phone numbers,
+     API keys, file paths, etc. are all replaced with placeholders)
+   - How to opt out: toggle the setting OFF at any time
+   - How to delete data: option available in settings to delete all
+     previously collected data
+
+2. Generate or load a persistent `user_id`:
+   - Store at `~/.groove/user_id`
+   - Generate as a random UUID (e.g., `uuid.uuid4().hex`)
+   - Generated once per install, reused across sessions
+   - NOT tied to identity: no hardware IDs, no email, no name, no IP
+
+3. Record consent:
+   ```python
+   consent_manager.record_consent(
+       user_id=user_id,
+       opted_in=True,
+       consent_version="1.0",
+   )
+   ```
+
+**When toggled OFF:**
+
+1. Revoke consent:
+   ```python
+   consent_manager.revoke_consent(user_id)
+   ```
+
+2. Stop all capture immediately. Any active CaptureSession will
+   self-deactivate on the next `is_active()` check.
+
+3. Offer the option to delete previously collected data:
+   ```python
+   intake.delete_user(user_id)
+   # This revokes consent AND deletes all JSONL records for the user
+   ```
+
+### 3.2 Data Capture Points
+
+Hook capture at these points in the daemon/app:
+
+#### PROMPTS
+
+When user submits a prompt for inference, capture the prompt text.
+
+```python
+session.record_prompt(prompt_text, metadata={
+    "model_name": model_name,        # e.g., "qwen3-moe-a3b"
+    "timestamp": time.time(),
+    "workflow_type": workflow_type,   # e.g., "journalism", "code_editor"
+})
+```
+
+Or with TrainingDataIntake:
+```python
+intake.submit(user_id, session_id, prompt_text, "prompt", metadata={
+    "model_name": model_name,
+    "timestamp": time.time(),
+    "workflow_type": workflow_type,
+})
+```
+
+#### COMPLETIONS
+
+When the model returns a completion, capture the output text.
+
+```python
+session.record_completion(completion_text, metadata={
+    "model_name": model_name,
+    "token_count": token_count,
+    "latency_ms": latency_ms,
+    "finish_reason": finish_reason,  # e.g., "stop", "length", "eos"
+})
+```
+
+Or with TrainingDataIntake:
+```python
+intake.submit(user_id, session_id, completion_text, "completion", metadata={
+    "model_name": model_name,
+    "token_count": token_count,
+    "latency_ms": latency_ms,
+    "finish_reason": finish_reason,
+})
+```
+
+#### CONVERSATIONS
+
+For multi-turn sessions, capture the full conversation at session end.
+
+```python
+messages = [
+    {"role": "user", "content": "..."},
+    {"role": "assistant", "content": "..."},
+    {"role": "user", "content": "..."},
+    {"role": "assistant", "content": "..."},
+]
+session.record_conversation(messages, metadata={
+    "turn_count": len(messages),
+    "total_tokens": total_tokens,
+    "session_duration_s": duration_seconds,
+})
+```
+
+Or with TrainingDataIntake:
+```python
+import json
+intake.submit(user_id, session_id, json.dumps(messages), "conversation", metadata={
+    "turn_count": len(messages),
+    "total_tokens": total_tokens,
+    "session_duration_s": duration_seconds,
+})
+```
+
+#### WORKFLOW METADATA
+
+Capture what type of work the user is doing (journalism, coding,
+research, planning). The domain tagger classifies content automatically
+based on keywords, but explicit `workflow_type` in metadata helps
+validate and improve the tagger.
+
+If the app knows the user is in "journalist mode" or "code editor",
+pass that as metadata on every submit call:
+
+```python
+metadata={"workflow_type": "journalism"}
+```
+
+### 3.3 Capture Requirements
+
+**ZERO OVERHEAD when opted out:**
+Don't import `moe-team.src.training` modules if the user hasn't opted in.
+Check a local flag file or config value first. Recommended pattern:
+
+```python
+import os
+
+def is_capture_enabled() -> bool:
+    """Check local flag before touching any training modules."""
+    user_id_path = os.path.expanduser("~/.groove/user_id")
+    if not os.path.exists(user_id_path):
+        return False
+    # Only import consent manager if user_id exists
+    from moe-team.src.training import ConsentManager
+    consent = ConsentManager()
+    with open(user_id_path) as f:
+        user_id = f.read().strip()
+    return consent.is_opted_in(user_id)
+```
+
+**NON-BLOCKING:**
+Capture calls must not slow down inference. The `intake.submit()` and
+`CaptureSession.record_*()` calls do synchronous I/O (SQLite reads +
+JSONL file writes). Do NOT call them in the inference hot path.
+
+Recommended pattern — background queue:
+
+```python
+import queue
+import threading
+
+capture_queue = queue.Queue(maxsize=10000)
+
+def capture_worker(intake, user_id):
+    """Background thread that drains the capture queue."""
+    while True:
+        item = capture_queue.get()
+        if item is None:
+            break
+        try:
+            intake.submit(
+                user_id=user_id,
+                session_id=item["session_id"],
+                content=item["content"],
+                content_type=item["content_type"],
+                metadata=item.get("metadata"),
+            )
+        except Exception:
+            pass  # fail silent — never crash over capture
+        capture_queue.task_done()
+
+# Start worker thread at app startup (only if opted in)
+worker = threading.Thread(target=capture_worker, args=(intake, user_id), daemon=True)
+worker.start()
+
+# In the inference path — non-blocking enqueue
+def on_prompt(session_id, prompt_text, model_name):
+    try:
+        capture_queue.put_nowait({
+            "session_id": session_id,
+            "content": prompt_text,
+            "content_type": "prompt",
+            "metadata": {"model_name": model_name},
+        })
+    except queue.Full:
+        pass  # drop if queue is full — never block inference
+```
+
+If using asyncio:
+
+```python
+import asyncio
+
+async def capture_submit(intake, user_id, session_id, content, content_type, metadata):
+    loop = asyncio.get_event_loop()
+    await loop.run_in_executor(None, intake.submit, user_id, session_id, content, content_type, metadata)
+```
+
+**FAIL SILENT:**
+If capture fails (disk full, permission error, SQLite locked), log a
+warning and continue. Never crash the app over training data capture.
+Wrap every capture call in a try/except that swallows all exceptions.
+
+### 3.4 User Data Management UI
+
+Provide a way for users to:
+
+**View contribution stats:**
+```python
+from moe-team.src.training import CorpusStats, TrainingCorpus, ConsentManager
+
+corpus = TrainingCorpus()
+consent = ConsentManager()
+stats = CorpusStats(corpus, consent)
+
+summary = stats.summary()
+# {"total_records": 1423, "storage_size_mb": 2.34, "opted_in_users": 1, "domains": {...}}
+
+growth = stats.daily_growth(days=7)
+# [{"date": "2026-04-26", "records": 142}, {"date": "2026-04-27", "records": 203}, ...]
+```
+
+**Download their data:**
+```python
+corpus = TrainingCorpus()
+count = corpus.export_jsonl(
+    output_path="/tmp/my_groove_data.jsonl",
+    domain=None,   # all domains, or filter by "journalism", "code", etc.
+    after=None,     # all time, or filter by timestamp
+)
+# count = number of records exported
+```
+
+Note: `export_jsonl()` exports ALL users' data. To filter by user_id,
+the daemon team should read the JSONL and filter client-side, or the
+network team can add a user_id filter (file a request).
+
+**Delete their data:**
+```python
+intake.delete_user(user_id)
+# Revokes consent AND deletes all JSONL records for the user
+```
+
+This can be a settings panel, CLI command (`groove data --stats`,
+`groove data --export`, `groove data --delete`), or both.
+
+---
+
+## 4. Data Format Reference
+
+### TrainingRecord Schema
+
+Each record stored in the JSONL corpus has these fields:
+
+| Field             | Type              | Description                                              |
+|-------------------|-------------------|----------------------------------------------------------|
+| `record_id`       | `str`             | Hex UUID, unique per record                              |
+| `user_id`         | `str`             | Anonymized install ID (random UUID from ~/.groove/user_id)|
+| `session_id`      | `str`             | Groups records from the same inference session           |
+| `timestamp`       | `float`           | Unix timestamp (time.time())                             |
+| `domain`          | `str`             | Auto-tagged: journalism, code, research, planning, general|
+| `content_type`    | `str`             | prompt, completion, conversation, workflow               |
+| `content`         | `str`             | PII-scrubbed text                                        |
+| `metadata`        | `dict[str, Any]`  | Model name, token count, latency, workflow_type, etc.    |
+| `consent_version` | `str`             | Version of consent terms (currently "1.0")               |
+
+### Storage Location
+
+JSONL files at `~/.groove/training_data/training_YYYY-MM-DD.jsonl`
+
+One JSON object per line, daily rotation. Standard format for LLM
+training pipelines.
+
+### Example JSONL Line
+
+```json
+{"record_id":"a1b2c3d4e5f6","user_id":"f8e7d6c5b4a3","session_id":"sess_001","timestamp":1745625600.0,"domain":"journalism","content_type":"prompt","content":"Write a lead paragraph about the city council vote on [EMAIL] proposed budget","metadata":{"model_name":"qwen3-moe-a3b","workflow_type":"journalism"},"consent_version":"1.0"}
+```
+
+---
+
+## 5. Content Types and What to Capture
+
+| content_type   | When to capture                        | What to include                        |
+|----------------|----------------------------------------|----------------------------------------|
+| `prompt`       | User submits text for inference        | The raw prompt text                    |
+| `completion`   | Model returns generated text           | The generated output                   |
+| `conversation` | Multi-turn session ends                | Full message history as JSON           |
+| `workflow`     | User completes a workflow/task         | Summary of the workflow steps          |
+
+Notes:
+- `prompt` and `completion` are captured in real-time during inference
+- `conversation` is captured once at session end — the full message array
+- `workflow` is optional — capture if the app has a concept of discrete
+  tasks or workflows (e.g., "user finished writing an article")
+
+---
+
+## 6. Integration Checklist
+
+- [ ] Generate persistent `user_id` (random UUID) at `~/.groove/user_id`
+- [ ] Add opt-in toggle to settings UI (default: OFF)
+- [ ] Show data collection disclosure dialog on first opt-in
+- [ ] Call `consent_manager.record_consent(user_id, opted_in=True, consent_version="1.0")` on opt-in
+- [ ] Call `consent_manager.revoke_consent(user_id)` on opt-out
+- [ ] Hook capture at prompt submission point
+- [ ] Hook capture at completion receipt point
+- [ ] Hook capture at session end (conversation type)
+- [ ] Implement background capture queue (non-blocking)
+- [ ] Guard all capture behind opted-in check (zero overhead when off)
+- [ ] Add data management UI (view stats, download, delete)
+- [ ] Test: opt-in -> capture -> opt-out -> capture stops
+- [ ] Test: data deletion removes all user records
+- [ ] Test: no PII leaks through (the intake API scrubs, but verify)
+
+---
+
+## 7. Quick Start Code Example
+
+Minimal end-to-end integration using CaptureSession:
+
+```python
+import os
+import uuid
+
+from moe-team.src.training import (
+    CaptureSession,
+    ConsentManager,
+    TrainingCorpus,
+    PIIScrubber,
+    DomainTagger,
+)
+
+# --- One-time setup (app startup) ---
+
+USER_ID_PATH = os.path.expanduser("~/.groove/user_id")
+
+def get_or_create_user_id() -> str:
+    if os.path.exists(USER_ID_PATH):
+        with open(USER_ID_PATH) as f:
+            return f.read().strip()
+    uid = uuid.uuid4().hex
+    os.makedirs(os.path.dirname(USER_ID_PATH), exist_ok=True)
+    with open(USER_ID_PATH, "w") as f:
+        f.write(uid)
+    return uid
+
+user_id = get_or_create_user_id()
+consent = ConsentManager()           # db at ~/.groove/consent.db
+corpus = TrainingCorpus()             # JSONL at ~/.groove/training_data/
+scrubber = PIIScrubber()
+tagger = DomainTagger()
+
+# --- User opts in (settings toggle) ---
+
+consent.record_consent(user_id, opted_in=True, consent_version="1.0")
+
+# --- Per-session capture ---
+
+session_id = uuid.uuid4().hex
+capture = CaptureSession(user_id, session_id, consent, corpus, scrubber, tagger)
+capture.start()
+
+# User sends a prompt
+capture.record_prompt("Summarize the city council meeting notes", metadata={
+    "model_name": "qwen3-moe-a3b",
+})
+
+# Model returns a completion
+capture.record_completion("The city council voted 7-2 to approve...", metadata={
+    "model_name": "qwen3-moe-a3b",
+    "token_count": 142,
+    "latency_ms": 820,
+})
+
+# Session ends — capture full conversation
+capture.record_conversation([
+    {"role": "user", "content": "Summarize the city council meeting notes"},
+    {"role": "assistant", "content": "The city council voted 7-2 to approve..."},
+], metadata={"turn_count": 2, "total_tokens": 168})
+
+summary = capture.end()
+# {"session_id": "...", "records_captured": 3, "bytes_captured": 312, ...}
+
+# --- User opts out and deletes data ---
+
+from moe-team.src.training.intake import TrainingDataIntake
+
+intake = TrainingDataIntake(consent, corpus, scrubber, tagger)
+result = intake.delete_user(user_id)
+# {"consent_revoked": True, "records_deleted": 3}
+```
+
+Alternative using TrainingDataIntake directly (simpler, no session lifecycle):
+
+```python
+from moe-team.src.training.intake import TrainingDataIntake
+
+intake = TrainingDataIntake(consent, corpus, scrubber, tagger)
+
+# Submit individual records
+intake.submit(user_id, session_id, prompt_text, "prompt", {"model_name": "qwen3-moe-a3b"})
+intake.submit(user_id, session_id, completion_text, "completion", {"token_count": 142})
+
+# User opts out — revokes consent + deletes all data
+intake.delete_user(user_id)
+```
+
+---
+
+## 8. PII Scrubbing — What the Daemon Team Does NOT Need to Do
+
+The intake API and CaptureSession scrub ALL content before storage.
+The daemon team does NOT need to scrub PII. Just pass raw text.
+
+The scrubber (`PIIScrubber` in `moe-team/src/training/scrubber.py`)
+handles these PII categories:
+
+| PII Type           | Replacement      | Example                                    |
+|--------------------|------------------|--------------------------------------------|
+| Email addresses    | `[EMAIL]`        | `user@example.com` -> `[EMAIL]`            |
+| Phone numbers      | `[PHONE]`        | `(555) 123-4567` -> `[PHONE]`              |
+| IPv4 addresses     | `[IP]`           | `192.168.1.1` -> `[IP]`                    |
+| IPv6 addresses     | `[IP]`           | `2001:db8::1` -> `[IP]`                    |
+| SSNs               | `[SSN]`          | `123-45-6789` -> `[SSN]`                   |
+| Credit cards       | `[CREDIT_CARD]`  | `4111-1111-1111-1111` -> `[CREDIT_CARD]`   |
+| AWS access keys    | `[AWS_KEY]`      | `AKIAIOSFODNN7EXAMPLE` -> `[AWS_KEY]`      |
+| Private keys       | `[PRIVATE_KEY]`  | `-----BEGIN RSA PRIVATE KEY-----...`       |
+| Bearer tokens      | `[API_KEY]`      | `Bearer eyJhbGc...` -> `[API_KEY]`         |
+| sk_/pk_ API keys   | `[API_KEY]`      | `sk-abc123...` -> `[API_KEY]`              |
+| Long hex strings   | `[API_KEY]`      | 40+ char hex strings -> `[API_KEY]`        |
+| User file paths    | `[FILE_PATH]`    | `/Users/john/docs/...` -> `[FILE_PATH]`    |
+| URLs with tokens   | `[REDACTED_URL]` | `https://...?token=abc` -> `[REDACTED_URL]`|
+
+Credit card detection includes Luhn checksum validation to reduce false
+positives.
+
+If the daemon team discovers PII types the scrubber misses, report to
+the network team to add patterns to `scrubber.py`.
+
+---
+
+## 9. Privacy Principles
+
+1. **Default opt-out:** No data collection without explicit user action.
+   The toggle defaults to OFF. `is_opted_in()` returns `False` when no
+   consent record exists.
+
+2. **Anonymous user_id:** The `user_id` is a random UUID generated once
+   per install. It is NOT derived from any personal information — no
+   hardware IDs, no email, no name, no IP address.
+
+3. **PII scrubbed before storage:** All content passes through the
+   PIIScrubber before hitting disk. The scrubber replaces 13 categories
+   of PII with placeholder tokens.
+
+4. **User can delete all data at any time:** `intake.delete_user(user_id)`
+   removes all JSONL records and revokes consent. The consent database
+   also supports `request_deletion()` for formal deletion requests.
+
+5. **Consent is versioned:** `is_opted_in()` checks that the user's
+   consent version matches `CURRENT_CONSENT_VERSION` (currently "1.0").
+   If terms change and the version is bumped, all users must re-consent.
+   Old consent is treated as not-opted-in until re-consented.
+
+6. **Training data stays local:** Data is stored on the machine running
+   the Groove service (`~/.groove/training_data/`). No data leaves the
+   machine without explicit export. Future network-level aggregation
+   will be a separate opt-in.
+
+7. **Mid-session revocation:** If a user revokes consent during an active
+   session, `CaptureSession.is_active()` detects this on the next call
+   and stops capture immediately. No buffered data is flushed.
+
+---
+
+## 10. File Reference
+
+| File | Description |
+|------|-------------|
+| `moe-team/src/training/__init__.py` | Package exports: CaptureSession, ConsentManager, ConsentRecord, CorpusStats, DomainTagger, PIIScrubber, TrainingCorpus, TrainingRecord |
+| `moe-team/src/training/intake.py` | TrainingDataIntake — simple submit/batch/delete API |
+| `moe-team/src/training/consent.py` | ConsentManager — SQLite consent storage, versioned consent |
+| `moe-team/src/training/scrubber.py` | PIIScrubber — 13 compiled regex patterns for PII removal |
+| `moe-team/src/training/domain_tagger.py` | DomainTagger — keyword-based domain classification |
+| `moe-team/src/training/corpus.py` | TrainingCorpus — JSONL storage with daily rotation |
+| `moe-team/src/training/capture.py` | CaptureSession — session-oriented capture with lifecycle |
+| `moe-team/src/training/stats.py` | CorpusStats — summary, daily growth, domain breakdown |