crprotocol 2.3.1__tar.gz → 4.1.1__tar.gz

crprotocol-4.1.1/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,34 @@
+## Description
+
+<!-- Briefly describe what this PR changes and why -->
+
+## Type of Change
+
+- [ ] Typo / wording fix
+- [ ] Specification clarification (no semantic change)
+- [ ] Non-breaking addition (new optional field, new example)
+- [ ] Breaking change (requires RFC)
+- [ ] Schema change
+- [ ] Example addition/update
+- [ ] Documentation (README, CONTRIBUTING, etc.)
+
+## Specification Sections Affected
+
+<!-- List affected sections, e.g., §3.8, §6.10.2, §22 -->
+
+## RFC 2119 Impact
+
+<!-- If this changes any MUST/SHOULD/MAY requirements, list them here -->
+
+- [ ] No RFC 2119 changes
+- [ ] Changes MUST/MUST NOT requirements
+- [ ] Changes SHOULD/SHOULD NOT requirements
+- [ ] Changes MAY requirements
+
+## Checklist
+
+- [ ] I have read the [Contributing Guide](../CONTRIBUTING.md)
+- [ ] JSON Schemas validate correctly (if modified)
+- [ ] Cross-references are correct (§ references point to valid sections)
+- [ ] CHANGELOG.md is updated (for user-facing changes)
+- [ ] No broken links in modified documents

crprotocol-4.1.1/.github/workflows/docs.yml

@@ -0,0 +1,41 @@
+# Copyright © 2025 Constantinos Vidiniotis. All rights reserved.
+# Licensed under Elastic License 2.0 - see LICENSE.md for details.
+
+name: Deploy Docs
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'site-docs/**'
+      - 'mkdocs.yml'
+      - '.github/workflows/docs.yml'
+      - 'pyproject.toml'
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    env:
+      DISABLE_MKDOCS_2_WARNING: true
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.13'
+
+      - name: Install docs dependencies
+        run: pip install -e ".[docs]"
+
+      - name: Regenerate API reference
+        run: python scripts/generate_api_reference.py
+
+      - name: Build and deploy
+        run: mkdocs gh-deploy --force --strict

crprotocol-4.1.1/.gitignore

@@ -0,0 +1,81 @@
+# OS files
+.DS_Store
+Thumbs.db
+desktop.ini
+
+# Editor files
+*.swp
+*.swo
+*~
+.vscode/
+.idea/
+*.sublime-*
+
+# Node modules (for schema validation)
+node_modules/
+
+# Build outputs
+dist/
+build/
+*.egg-info/
+site/
+.cache/
+
+# Python
+__pycache__/
+*.py[cod]
+*.so
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+*.egg
+pip-wheel-metadata/
+
+# Temporary files
+*.tmp
+*.bak
+
+# Runtime session data
+crp_sessions/
+.hypothesis/
+
+# Private keys / secrets — NEVER commit
+*.pem
+*.key
+*.p8
+encrypted-*.pem
+*.env
+*_railway.env
+.env.local
+.env.production
+.env.staging
+.env.development
+secrets/
+
+# Dev scratch files (root-level only)
+/_*.py
+/_*.txt
+/_*.json
+/_*.jsonl
+/_full_*
+/_long_gen*
+/_live_*
+/_check_*
+/_dep_*
+/_pytest_count*
+/_test_*
+/_tr.*
+/_batch*
+/0.80)
+/_long_gen_results/
+/type_confidence
+/[0-9][0-9][0-9]*
+
+# Numeric test output directories
+/[0-9]*/
+
+# Logs
+*.log
+
+.coverage
+.env.hosted

crprotocol-4.1.1/.kimi/skills/crp-auth-expert/SKILL.md

@@ -0,0 +1,450 @@
+# CRP Auth & User Management Expert
+
+**Scope:** Project  
+**Purpose:** Deep expertise on CRP's unified authentication, identity, entitlement, and PostgreSQL persistence layer. Use this whenever working on auth, billing, user state, Clerk integration, GitHub connectivity, or database schema across CRP Gateway and CRP Comply.
+
+---
+
+## Canonical Documents
+
+Always read these first before modifying auth-related code:
+
+```
+SPECS_5_06_2026_CRP_v4/CRP-AUTH-POSTGRESQL-MIGRATION-PLAN.md    # This plan
+SPECS_5_06_2026_CRP_v4/CRP-AUTH-USER-MANAGEMENT-SOLUTION.md      # Auth spec
+SPECS_5_06_2026_CRP_v4/CRP-GITHUB-CONNECTIVITY-SOLUTION.md       # GitHub spec
+SPECS_5_06_2026_CRP_v4/specs/CRP-SPEC-047-monetisation-payments.md  # Billing spec
+SPECS_5_06_2026_CRP_v4/specs/CRP-SPEC-048-comply-lowcode-github.md  # GitHub App spec
+AGENTS.md                                                        # Project conventions
+```
+
+---
+
+## Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  CLERK (clerk.crprotocol.io) — Identity + Entitlement        │
+│  - Users, orgs, memberships                                  │
+│  - public_metadata: {comply_plan, gateway_plan, scan_plan,   │
+│                     stripeCustomerId, creditBalanceCents}    │
+│  - Stripe webhooks write here ONLY                           │
+└────────────────────────┬────────────────────────────────────┘
+                         │ verify JWT (JWKS)
+           ┌─────────────┼───────────────┐
+           ▼             ▼               ▼
+   ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
+   │   Gateway    │ │   Comply     │ │    Scan      │
+   │  FastAPI     │ │  FastAPI     │ │  (in Comply) │
+   └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
+          │ read entitlement│ read entitlement│ read entitlement
+          │ (Clerk metadata)│ (Clerk metadata)│ (Clerk metadata)
+          ▼                 ▼                 ▼
+   ┌─────────────────────────────────────────────────────────────┐
+   │  POSTGRESQL (Railway managed) — Application State            │
+   │  - tenants, api_keys, github_installations, github_repos,    │
+   │    gateway_systems, gateway_usage, scan_results, etc.        │
+   └─────────────────────────────────────────────────────────────┘
+```
+
+**Golden rule:** Clerk stores WHO you are and WHAT you can do. PostgreSQL stores WHAT you have done.
+
+---
+
+## The Identity Model
+
+### Identity Dataclass
+
+```python
+from dataclasses import dataclass
+
+@dataclass(frozen=True)
+class Identity:
+    user_id: str       # Clerk "sub" claim
+    org_id: str | None # Clerk "org_id" claim (null = personal account)
+    org_role: str | None # Clerk "org_role" claim
+```
+
+### resolve_account()
+
+```python
+def resolve_account(identity: Identity) -> tuple[str, str]:
+    """Return (account_type, account_id)."""
+    if identity.org_id:
+        return ("org", identity.org_id)
+    return ("user", identity.user_id)
+```
+
+**Usage pattern:**
+```python
+identity = await current_clerk_identity(request)
+account_type, account_id = resolve_account(identity)
+tenant_id = account_id  # tenant_id = org_id for teams, user_id for personal
+```
+
+### Personal vs Organization Accounts
+
+| Context | `org_id` | `resolve_account()` returns | `tenant_id` |
+|---------|----------|----------------------------|-------------|
+| Personal (solo developer) | `None` | `("user", "user_abc")` | `"user_abc"` |
+| Organization (team/company) | `"org_xyz"` | `("org", "org_xyz")` | `"org_xyz"` |
+
+**Critical:** Always use `resolve_account()` — never branch on `org_id` directly in route handlers.
+
+---
+
+## Entitlement: Clerk Metadata
+
+### The Five Rules
+
+1. **Stripe webhook is the ONLY writer.** Never write entitlement from app code.
+2. **Per-product keys.** `comply_plan`, `gateway_plan`, `scan_plan` — never `"plan"`.
+3. **Apps read via `get_entitlement()`.** Caches for 60s, reads from Clerk Backend API.
+4. **PostgreSQL never stores entitlement.** No `plan` column on any table.
+5. **Customer ID lives in Clerk metadata.** `stripeCustomerId` is the linkage.
+
+### Clerk Metadata Schema
+
+```json
+{
+  "comply_plan": "free | starter | scale",
+  "gateway_plan": "free | developer | team",
+  "scan_plan": "free | pro | business",
+  "comply_quota": 100,
+  "gateway_quota": 1000,
+  "scan_quota": 50,
+  "stripeCustomerId": "cus_...",
+  "stripeSubscriptionId": "sub_...",
+  "creditBalanceCents": 0,
+  "features": ["advanced_analytics", "multi_org"]
+}
+```
+
+### Reading Entitlement
+
+```python
+from crp_shared.auth import get_entitlement, Identity
+
+async def some_endpoint(identity: Identity = Depends(current_clerk_identity)):
+    entitlement = await get_entitlement(identity, product="comply")
+    # Returns: {"plan": "scale", "quota": 50000, "credits": 2000, "stripe_customer_id": "cus_..."}
+    if entitlement["plan"] == "free":
+        raise HTTPException(403, "This feature requires a paid plan")
+```
+
+### Writing Entitlement (Stripe Webhook Only)
+
+```python
+# In Stripe webhook handler — NEVER in app code
+async def handle_checkout_completed(event):
+    price_id = event["data"]["object"]["items"][0]["price"]["id"]
+    product = price_id_to_product(price_id)  # "comply" | "gateway" | "scan"
+    plan = price_id_to_plan(price_id)        # "starter" | "developer" | "pro"
+    
+    meta_update = {
+        f"{product}_plan": plan,
+        f"{product}_quota": PLAN_QUOTAS[plan],
+    }
+    
+    await clerk_backend_api.update_org_metadata(
+        org_id=org_id,
+        public_metadata=meta_update
+    )
+```
+
+---
+
+## PostgreSQL Schema
+
+### Shared Tables (both databases)
+
+```sql
+-- tenants: links Clerk identity to app state
+CREATE TABLE tenants (
+    tenant_id         TEXT PRIMARY KEY,           -- user_id (personal) or tnt_xxx (org)
+    clerk_org_id      TEXT UNIQUE,                -- NULL for personal
+    clerk_user_id     TEXT NOT NULL,
+    account_type      TEXT NOT NULL 
+                      CHECK (account_type IN ('org', 'personal'))
+                      DEFAULT 'org',
+    name              TEXT,
+    created_at        TIMESTAMPTZ DEFAULT NOW(),
+    updated_at        TIMESTAMPTZ DEFAULT NOW()
+);
+
+-- api_keys: hashed, per-tenant
+CREATE TABLE api_keys (
+    key_id            TEXT PRIMARY KEY,
+    tenant_id         TEXT NOT NULL REFERENCES tenants(tenant_id) ON DELETE CASCADE,
+    key_hash          TEXT NOT NULL,              -- SHA-256
+    key_preview       TEXT,                       -- last 8 chars
+    name              TEXT DEFAULT 'default',
+    scopes            TEXT[] DEFAULT '{}',
+    revoked           BOOLEAN DEFAULT FALSE,
+    created_at        TIMESTAMPTZ DEFAULT NOW(),
+    last_used_at      TIMESTAMPTZ
+);
+```
+
+### Gateway-Specific Tables
+
+```sql
+CREATE TABLE gateway_systems (
+    system_id         TEXT PRIMARY KEY,
+    tenant_id         TEXT NOT NULL REFERENCES tenants(tenant_id) ON DELETE CASCADE,
+    name              TEXT NOT NULL,
+    domain            TEXT,
+    intended_purpose  TEXT,
+    eu_ai_act_class   TEXT CHECK (eu_ai_act_class IN ('MINIMAL','LIMITED','HIGH','UNACCEPTABLE')),
+    created_at        TIMESTAMPTZ DEFAULT NOW()
+);
+
+CREATE TABLE gateway_providers (
+    provider_id       TEXT PRIMARY KEY,
+    tenant_id         TEXT NOT NULL REFERENCES tenants(tenant_id) ON DELETE CASCADE,
+    provider_type     TEXT NOT NULL CHECK (provider_type IN ('openai','anthropic','ollama','custom')),
+    api_key_encrypted TEXT NOT NULL,
+    default_model     TEXT,
+    created_at        TIMESTAMPTZ DEFAULT NOW()
+);
+
+CREATE TABLE gateway_usage (
+    usage_id          BIGSERIAL PRIMARY KEY,
+    tenant_id         TEXT NOT NULL REFERENCES tenants(tenant_id) ON DELETE CASCADE,
+    call_type         TEXT CHECK (call_type IN ('chat','completion','embedding')),
+    model             TEXT,
+    tokens_in         INT DEFAULT 0,
+    tokens_out        INT DEFAULT 0,
+    risk_class        TEXT,
+    halted            BOOLEAN DEFAULT FALSE,
+    created_at        TIMESTAMPTZ DEFAULT NOW()
+);
+```
+
+### Comply-Specific Tables
+
+```sql
+CREATE TABLE github_installations (
+    installation_id   BIGINT PRIMARY KEY,
+    tenant_id         TEXT NOT NULL REFERENCES tenants(tenant_id) ON DELETE CASCADE,
+    account_login     TEXT NOT NULL,
+    account_type      TEXT NOT NULL DEFAULT 'Organization' CHECK (account_type IN ('User','Organization')),
+    suspended         BOOLEAN DEFAULT FALSE,
+    created_at        TIMESTAMPTZ DEFAULT NOW()
+);
+
+CREATE TABLE github_repos (
+    repo_internal_id  BIGSERIAL PRIMARY KEY,
+    installation_id   BIGINT NOT NULL REFERENCES github_installations(installation_id) ON DELETE CASCADE,
+    repo_id           BIGINT NOT NULL,
+    full_name         TEXT NOT NULL,
+    private           BOOLEAN NOT NULL DEFAULT TRUE,
+    connected         BOOLEAN DEFAULT FALSE,
+    last_scanned_at   TIMESTAMPTZ,
+    findings_count    INT DEFAULT 0,
+    UNIQUE(installation_id, repo_id)
+);
+
+CREATE TABLE scan_results (
+    scan_id           BIGSERIAL PRIMARY KEY,
+    repo_id           BIGINT NOT NULL REFERENCES github_repos(repo_internal_id) ON DELETE CASCADE,
+    tenant_id         TEXT NOT NULL REFERENCES tenants(tenant_id) ON DELETE CASCADE,
+    commit_sha        TEXT,
+    findings          JSONB DEFAULT '[]',
+    status            TEXT DEFAULT 'pending' CHECK (status IN ('pending','running','completed','failed')),
+    created_at        TIMESTAMPTZ DEFAULT NOW(),
+    completed_at      TIMESTAMPTZ
+);
+```
+
+---
+
+## GitHub App Connectivity
+
+### The Correct Flow
+
+```
+[Frontend] clicks "Connect GitHub"
+   │
+   ▼ POST /api/v1/github/connect-start (Clerk-authed)
+[Backend] creates signed state token → returns install URL
+   │
+   ▼ User installs on GitHub
+[GitHub] redirects to Setup URL
+   │
+   ▼ GET /api/v1/github/setup?installation_id=NNN&state=TOKEN
+[Backend] validates state → recovers identity → fetches repos → persists → redirects to app
+   │
+   ▼ Frontend calls GET /api/v1/github/repos
+[Backend] returns repos from PostgreSQL (no GitHub API call)
+```
+
+### State Token Format
+
+```python
+import hmac, hashlib, base64, json, time, secrets
+
+STATE_SECRET = os.environ["GITHUB_STATE_SECRET"].encode()
+
+def _sign_state(payload: dict) -> str:
+    raw = base64.urlsafe_b64encode(json.dumps(payload).encode()).decode()
+    sig = hmac.new(STATE_SECRET, raw.encode(), hashlib.sha256).hexdigest()
+    return f"{raw}.{sig}"
+
+def _verify_state(state: str) -> dict:
+    raw, sig = state.split(".")
+    expect = hmac.new(STATE_SECRET, raw.encode(), hashlib.sha256).hexdigest()
+    if not hmac.compare_digest(sig, expect):
+        raise ValueError("bad signature")
+    payload = json.loads(base64.urlsafe_b64decode(raw))
+    if payload["exp"] < time.time():
+        raise ValueError("expired")
+    return payload
+
+# Usage in connect-start:
+state = _sign_state({
+    "clerk_user_id": identity.user_id,
+    "clerk_org_id": identity.org_id,
+    "exp": time.time() + 600,
+    "nonce": secrets.token_hex(8),
+})
+```
+
+### Critical GitHub Webhook Events
+
+| Event | Action | SQL Operation |
+|-------|--------|---------------|
+| `installation` + `deleted`/`suspend` | User uninstalled app | `DELETE FROM github_installations WHERE installation_id = $1` |
+| `installation_repositories` + `added` | Repos granted | `INSERT INTO github_repos ... ON CONFLICT DO NOTHING` |
+| `installation_repositories` + `removed` | Repos revoked | `DELETE FROM github_repos WHERE repo_id = $1` |
+| `push` | New commit | Enqueue scan job (async worker) |
+
+---
+
+## API Key Handling
+
+### Creation
+
+```python
+import secrets, hashlib
+
+async def create_api_key(tenant_id: str, name: str = "default") -> str:
+    prefix = "crp_gw_" if is_gateway else "crp_cm_"
+    random_part = secrets.token_urlsafe(24)
+    full_key = f"{prefix}{random_part}"
+    
+    key_hash = hashlib.sha256(full_key.encode()).hexdigest()
+    key_preview = full_key[-8:]
+    
+    await db.execute("""
+        INSERT INTO api_keys (key_id, tenant_id, key_hash, key_preview, name)
+        VALUES ($1, $2, $3, $4, $5)
+    """, f"key_{secrets.token_urlsafe(8)}", tenant_id, key_hash, key_preview, name)
+    
+    return full_key  # Returned ONCE — never stored or logged
+```
+
+### Validation
+
+```python
+async def validate_api_key(full_key: str) -> str | None:
+    key_hash = hashlib.sha256(full_key.encode()).hexdigest()
+    row = await db.fetchrow("""
+        SELECT tenant_id FROM api_keys
+        WHERE key_hash = $1 AND revoked = FALSE
+    """, key_hash)
+    return row["tenant_id"] if row else None
+```
+
+---
+
+## Railway Environment Variables
+
+### Both Apps (must be identical)
+
+```bash
+CLERK_ISSUER=https://clerk.crprotocol.io
+CLERK_SECRET_KEY=sk_live_...
+CLERK_PUBLISHABLE_KEY=pk_live_Y2xlcmsuY3Jwcm90b2NvbC5pbyQ
+DATABASE_URL=postgresql://USER:PASS@HOST:5432/DB_NAME
+```
+
+### Comply Only
+
+```bash
+GITHUB_APP_ID=...
+GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n..."
+GITHUB_APP_WEBHOOK_SECRET=...
+GITHUB_STATE_SECRET=<openssl rand -hex 32>
+STRIPE_SECRET_KEY=sk_live_...
+STRIPE_WEBHOOK_SECRET=whsec_...
+STRIPE_COMPLY_STARTER_PRICE_ID=price_...
+STRIPE_COMPLY_SCALE_PRICE_ID=price_...
+STRIPE_SCAN_PRO_PRICE_ID=price_...
+STRIPE_SCAN_BUSINESS_PRICE_ID=price_...
+STRIPE_GATEWAY_DEVELOPER_PRICE_ID=price_...
+STRIPE_GATEWAY_TEAM_PRICE_ID=price_...
+```
+
+### Gateway Only
+
+```bash
+SECRET_KEY=<openssl rand -hex 32>
+STRIPE_SECRET_KEY=sk_live_...
+STRIPE_WEBHOOK_SECRET=whsec_...
+STRIPE_GATEWAY_DEVELOPER_PRICE_ID=price_...
+STRIPE_GATEWAY_TEAM_PRICE_ID=price_...
+```
+
+---
+
+## Common Bugs & Fixes
+
+| Bug | Cause | Fix |
+|-----|-------|-----|
+| Repos disappear after restart | `_connected_repos` is in-memory dict | Migrate to `github_repos` table |
+| Billing plan overwritten | Shared `"plan"` key in Clerk metadata | Use `comply_plan` / `gateway_plan` / `scan_plan` |
+| GitHub callback forged | Raw `user_id` in state param | Use HMAC-signed state tokens with TTL |
+| Gateway loses all tenants | `_tenants` is module-level dict | Migrate to PostgreSQL `tenants` table |
+| API keys visible in logs | Stored plaintext | Hash with SHA-256, store hash only |
+| Entitlement drift | Stripe webhook writes local JSON | Write to Clerk metadata ONLY |
+| Personal account 404s | `org_id` missing, no fallback | Use `resolve_account()` — falls back to `user_id` |
+| CORS errors on cross-product links | Different Clerk instances | Align both apps to `clerk.crprotocol.io` |
+
+---
+
+## Testing Auth Changes
+
+```bash
+# Test Clerk JWT verification
+curl -H "Authorization: Bearer $(clerk_token)" https://gateway.crprotocol.io/api/v1/systems
+
+# Test API key validation
+curl -H "Authorization: Bearer crp_gw_xxx" https://gateway.crprotocol.io/v1/chat/completions
+
+# Test GitHub connect-start
+curl -X POST -H "Authorization: Bearer $(clerk_token)" \
+  https://comply.crprotocol.io/api/v1/github/connect-start
+
+# Test entitlement read
+python -c "
+import asyncio
+from crp_shared.auth import get_entitlement, Identity
+async def test():
+    e = await get_entitlement(Identity('user_xxx', 'org_yyy', 'admin'), 'comply')
+    print(e)
+asyncio.run(test())
+"
+```
+
+---
+
+## Related Skills
+
+- `crp-comply-expert` — Comply-specific architecture, frontend patterns, dead code
+- `crp-gateway-expert` — Gateway-specific endpoints, provider vault, deployment
+- `crp-v4-specs` — Spec lookup, implementation rounds, ownership
+
+---
+
+*Last updated: 2026-06-05. Keep in sync with CRP-AUTH-POSTGRESQL-MIGRATION-PLAN.md.*