programasweights 0.1.0.dev6__tar.gz → 0.1.0.dev8__tar.gz

programasweights-0.1.0.dev8/.cursor/rules/vllm-deployment.mdc

@@ -0,0 +1,106 @@
+---
+description: vLLM deployment rules and known issues for PAW servers
+alwaysApply: true
+---
+
+# vLLM Deployment Rules
+
+## Server: 31.22.104.226 (4x RTX PRO 6000 Blackwell, 98 GB each)
+
+### Code Location
+
+All server code lives at `/root/paw-website/` (git clone of `programasweights/website`).
+
+**Deployment (the ONLY way to update code):**
+```bash
+# Local: push changes
+git push origin main
+
+# Server: pull and restart
+cd /root/paw-website && git pull
+cd server && bash scripts/start_all.sh
+```
+
+**NEVER use rsync or scp for code.** Model weights use scp between servers.
+
+### GPU Layout
+
+| GPU | Service | Port | VRAM |
+|-----|---------|------|------|
+| 0 | Pseudo-gen (Qwen3-4B) | 8001 | 90% |
+| 1 | Qwen3 compiler pooling (4B) | 8002 | 85% |
+| 2 | GPT-2 compiler pooling (4B) | 8012 | 85% |
+| 3 | Qwen3 inference (0.6B) + GPT-2 inference (124M) | 8003 + 8013 | 50% + 40% |
+
+GPU 3 services must start sequentially: GPT-2 first (smaller, faster), then Qwen3 after GPT-2 claims its VRAM.
+
+### Services
+
+| Service | Port | Working Dir |
+|---------|------|-------------|
+| vLLM pseudo-gen | 8001 | - |
+| vLLM Qwen3 compiler | 8002 | /root/paw-website/server |
+| vLLM GPT-2 compiler | 8012 | /root/paw-website/server |
+| vLLM Qwen3 inference | 8003 | - |
+| vLLM GPT-2 inference | 8013 | - |
+| Qwen3 CompileProvider | 9000 | /root/paw-website/server |
+| GPT-2 CompileProvider | 9001 | /root/paw-website/server |
+| API gateway | 8000 | /root/paw-website/server |
+
+### Startup
+
+```bash
+cd /root/paw-website/server
+bash scripts/start_all.sh    # starts everything, waits for models to load
+```
+
+Or for full redeploy (git pull + patch + start):
+```bash
+cd /root/paw-website
+bash server/scripts/restart.sh
+```
+
+### Key Files
+
+- `.env`: `/root/paw-website/server/.env` (secrets, NOT in git)
+- Provider configs: `server/provider_configs/qwen3-0.6b.json`, `server/provider_configs/gpt2.json`
+- Compiler registry: `server/compiler_registry.json`
+- vLLM launchers: `server/scripts/launch_qwen3_compiler.py`, `server/scripts/launch_gpt2_compiler.py`
+- Patch script: `server/scripts/patch_vllm.sh` (run after every pip install vllm)
+
+## Critical Environment Variables
+
+### VLLM_ALLOW_RUNTIME_LORA_UPDATING=1
+MUST be set for inference servers with --enable-lora.
+
+### VLLM_TORCH_COMPILE_LEVEL=0
+Set on servers without full CUDA dev toolchain.
+
+## vLLM Compiler Pooling (PawCompilerForPooling)
+
+- Must use `--enforce-eager` flag
+- Must use `--runner pooling` flag
+- Must run `scripts/patch_vllm.sh` after every vLLM install
+- Endpoint: `POST /v1/embeddings`
+- Launcher scripts use `if __name__ == '__main__'` guard for multiprocessing
+
+## Post-Install Patch
+
+```bash
+cd /root/paw-website/server && bash scripts/patch_vllm.sh /root/paw-venv
+```
+
+## GPT-2 Specifics
+
+- Interpreter uses NO chat template (use_chat_template: false)
+- Compiler port: 8012 (NOT 8002 which is Qwen3)
+- Inference port: 8013 (NOT 8003 which is Qwen3)
+
+## Testing
+
+Full benchmark: 12 specs, 36 examples, expected 30/36 (83.3%) for Qwen3
+```bash
+cd /root/paw-website/server
+python benchmarks/benchmark_api.py --server https://programasweights.com
+python benchmarks/stress_test.py --server https://programasweights.com --scenario all
+```

programasweights-0.1.0.dev8/.git

@@ -0,0 +1 @@
+gitdir: ../.git/modules/programasweights

programasweights-0.1.0.dev8/.github/workflows/web-sdk.yml

@@ -0,0 +1,51 @@
+name: Web SDK CI
+
+on:
+  push:
+    paths: ['web-sdk/**']
+  pull_request:
+    paths: ['web-sdk/**']
+
+jobs:
+  lint-and-test:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: web-sdk
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: npm
+          cache-dependency-path: web-sdk/package-lock.json
+      - run: npm ci
+      - run: npm run typecheck
+      - run: npm test
+      - run: npm run build
+
+  publish:
+    if: startsWith(github.ref, 'refs/tags/web-sdk-v')
+    needs: [lint-and-test]
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: web-sdk
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          registry-url: https://registry.npmjs.org
+      - run: npm ci
+      - run: npm run build
+      - run: npm publish --access public --provenance
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

programasweights-0.1.0.dev8/.gitmodules

@@ -0,0 +1,3 @@
+[submodule "web-sdk/wllama"]
+	path = web-sdk/wllama
+	url = git@github.com:programasweights/wllama.git

programasweights-0.1.0.dev8/.readthedocs.yaml

@@ -0,0 +1,13 @@
+version: 2
+
+build:
+  os: ubuntu-24.04
+  tools:
+    python: "3.12"
+
+mkdocs:
+  configuration: mkdocs.yml
+
+python:
+  install:
+    - requirements: docs/requirements.txt

programasweights-0.1.0.dev8/AGENTS.md

@@ -0,0 +1,127 @@
+# ProgramAsWeights (PAW)
+
+PAW compiles natural language specifications into tiny neural functions that run locally. Use it when you need fuzzy text processing — classification, extraction, format repair, search, triage — that regex can't handle but a full LLM is overkill for.
+
+Website: https://programasweights.com
+Docs: https://programasweights.readthedocs.io
+
+## Install
+
+```bash
+pip install --pre programasweights
+```
+
+## Usage
+
+```python
+import programasweights as paw
+
+# Use a pre-compiled function (downloads once, runs locally forever)
+fn = paw.function("email-triage")
+fn("Urgent: server is down!")  # "immediate"
+fn("Newsletter: spring picnic")  # "wait"
+
+# Compile your own from a description
+program = paw.compile(
+    "Fix malformed JSON: repair missing quotes and trailing commas",
+    compiler="paw-4b-qwen3-0.6b",  # or "paw-4b-gpt2" for smaller/faster
+    slug="json-fixer"              # optional: creates da03/json-fixer handle
+)
+fn = paw.function(program.slug)    # or paw.function(program.id)
+fn("{name: 'Alice',}")  # '{"name": "Alice"}'
+```
+
+## Two Compilers
+
+| | Standard (Qwen3 0.6B) | Compact (GPT-2 124M) |
+|---|---|---|
+| Compiler name | `paw-4b-qwen3-0.6b` | `paw-4b-gpt2` |
+| Accuracy | Higher | Lower |
+| Base model size | 594 MB | 105 MB |
+| Program size | ~22 MB | ~5 MB |
+| Inference speed | ~90ms (server) | ~50ms (server) |
+| Runs in browser | No | Yes |
+
+Default is Standard (Qwen3 0.6B). Use Compact (GPT-2) when you need smaller files or browser deployment.
+
+## Browser / JavaScript SDK
+
+Programs compiled with GPT-2 also run entirely in the browser via WebAssembly — no server needed. Users' data never leaves their device.
+
+```bash
+npm install @programasweights/web
+```
+
+```javascript
+import paw from '@programasweights/web';
+
+const fn = await paw.function('programasweights/email-triage');
+const result = await fn('Urgent: server is down!');
+// result: "immediate"
+```
+
+Use this when you want to offload all computation to the user's browser: zero server costs, zero API keys, full data privacy. The 105 MB base model downloads once and is cached.
+
+## When to Use PAW
+
+- **Fuzzy search** — typo-tolerant matching, semantic search, near-duplicate detection
+- **Format repair** — fix broken JSON, normalize dates, repair malformed inputs
+- **Classification** — sentiment, urgency, categories defined in your own words
+- **Extraction** — emails, names, dates from messy unstructured text
+- **Log triage** — extract errors from verbose output, filter noise
+- **Intent routing** — map user descriptions to the closest URL, menu item, or setting
+- **Agent preprocessing** — parse tool calls, validate outputs, route tasks
+
+## Authentication
+
+```bash
+# Option 1: environment variable (recommended)
+export PAW_API_KEY=paw_sk_...
+
+# Option 2: CLI login (opens browser to generate key)
+paw login
+```
+
+Generate API keys at https://programasweights.com/settings
+
+The SDK automatically reads `PAW_API_KEY` from the environment. Authenticated users get higher rate limits (60 compiles/hr vs 5 for anonymous).
+
+## Program Naming
+
+Programs have an immutable hash ID (e.g. `a6b454023d41ac9ca845`). Authenticated
+users can optionally assign a slug at compile time to create a human-readable
+handle like `da03/json-fixer`. Both forms work everywhere:
+
+```python
+fn = paw.function("da03/json-fixer")     # by slug
+fn = paw.function("a6b454023d41ac9ca845") # by hash
+```
+
+## CLI
+
+```bash
+paw compile --spec "Classify sentiment" --slug "my-classifier" --json
+paw run --program "da03/my-classifier" --input "I love this!" --json
+paw rename <program> <new-slug>   # rename a program's slug
+paw info <program>                # show program metadata
+paw login                         # save API key
+```
+
+`--json` gives structured output for programmatic use.
+
+## API
+
+```python
+paw.compile(spec, slug="my-fn")   # Compile with optional slug
+paw.function(name_or_id)          # Load by slug or hash
+paw.login()                       # Save API key
+```
+
+## Browse Programs
+
+https://programasweights.com/hub
+
+## Add PAW to Your Project
+
+Copy this file into your project as AGENTS.md:
+https://programasweights.com/agents

programasweights-0.1.0.dev8/PKG-INFO

@@ -0,0 +1,142 @@
+Metadata-Version: 2.4
+Name: programasweights
+Version: 0.1.0.dev8
+Summary: Compile natural language specifications into neural programs that run locally via llama.cpp.
+Project-URL: Homepage, https://programasweights.com
+Project-URL: Repository, https://github.com/programasweights/programasweights-python
+Project-URL: Documentation, https://programasweights.readthedocs.io
+Project-URL: Bug Tracker, https://github.com/programasweights/programasweights-python/issues
+Author-email: ProgramAsWeights <support@programasweights.com>
+License: MIT
+Keywords: inference,llama-cpp,lora,neural-programs,nlp
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Requires-Dist: httpx<1.0,>=0.27.0
+Requires-Dist: llama-cpp-python<1.0,>=0.3.0
+Provides-Extra: test
+Requires-Dist: pytest; extra == 'test'
+Description-Content-Type: text/markdown
+
+# ProgramAsWeights
+
+**Compile natural language specs into tiny neural functions that run locally.**
+
+Define what a function should do in plain English. PAW compiles it into a small neural program that runs on your machine — no API keys at runtime, no internet needed after setup, fully deterministic.
+
+## Install
+
+```bash
+pip install programasweights
+```
+
+## Quick Start
+
+```python
+import programasweights as paw
+
+# Use a pre-compiled function (downloads once, runs locally forever)
+fn = paw.function("email-triage")
+fn("Urgent: the server is down!")        # "immediate"
+fn("Newsletter: spring picnic")          # "wait"
+
+# Compile your own from a description
+program = paw.compile(
+    "Fix malformed JSON: repair missing quotes and trailing commas",
+    compiler="paw-4b-qwen3-0.6b"  # or "paw-4b-gpt2" for smaller/faster
+)
+fn = paw.function(program.id)
+fn("{name: 'Alice',}")  # '{"name": "Alice"}'
+```
+
+## Two Compilers
+
+|                    | Standard (Qwen3 0.6B)   | Compact (GPT-2 124M)   |
+|--------------------|-------------------------|------------------------|
+| Compiler name      | `paw-4b-qwen3-0.6b`    | `paw-4b-gpt2`          |
+| Accuracy           | Higher                  | Lower                  |
+| Base model size    | 594 MB                  | 105 MB                 |
+| Program size       | ~22 MB                  | ~5 MB                  |
+| Inference speed    | ~90ms (server)          | ~50ms (server)         |
+| Runs in browser    | No                      | Yes                    |
+
+Default is Standard (Qwen3 0.6B). Use Compact (GPT-2) when you need smaller files or browser deployment.
+
+## Browser SDK
+
+Programs compiled with GPT-2 also run entirely in the browser via WebAssembly — no server needed, data never leaves the user's device.
+
+```bash
+npm install @programasweights/web
+```
+
+```javascript
+import paw from '@programasweights/web';
+
+const fn = await paw.function('programasweights/email-triage');
+const result = await fn('Urgent: the server is down!');
+// result: "immediate"
+```
+
+See the [browser SDK repo](https://github.com/programasweights/programasweights-js) for full documentation.
+
+## Use with AI Agents
+
+PAW works with Cursor, Claude, Codex, and other AI coding assistants. Paste this into your agent's chat:
+
+> I want to use ProgramAsWeights (PAW) to create fuzzy text functions that run locally. Read the instructions at https://programasweights.com/agents and help me integrate it.
+
+Or save [`AGENTS.md`](https://programasweights.com/agents) to your project root — agents read it automatically.
+
+## When to Use PAW
+
+- **Fuzzy search** — typo-tolerant matching, semantic search, near-duplicate detection
+- **Format repair** — fix broken JSON, normalize dates, repair malformed inputs
+- **Classification** — sentiment, urgency, categories defined in your own words
+- **Extraction** — emails, names, dates from messy unstructured text
+- **Log triage** — extract errors from verbose output, filter noise
+- **Intent routing** — map user descriptions to the closest URL, menu item, or setting
+- **Agent preprocessing** — parse tool calls, validate outputs, route tasks
+
+## Authentication
+
+```bash
+# Option 1: environment variable (recommended)
+export PAW_API_KEY=paw_sk_...
+
+# Option 2: CLI login (opens browser to generate key)
+paw login
+```
+
+Generate API keys at [programasweights.com/settings](https://programasweights.com/settings). Authenticated users get higher rate limits.
+
+## CLI
+
+```bash
+paw compile --spec "Extract error lines from logs" --json
+paw run --program <program_id> --input "[ERROR] timeout" --json
+paw login
+```
+
+`--json` gives structured output for programmatic use.
+
+## Links
+
+- **Website**: [programasweights.com](https://programasweights.com)
+- **Documentation**: [programasweights.readthedocs.io](https://programasweights.readthedocs.io)
+- **Python SDK**: [github.com/programasweights/programasweights-python](https://github.com/programasweights/programasweights-python)
+- **Browser SDK**: [github.com/programasweights/programasweights-js](https://github.com/programasweights/programasweights-js)
+- **Program Hub**: [programasweights.com/hub](https://programasweights.com/hub)
+
+## License
+
+MIT

programasweights-0.1.0.dev8/PYPI_README.md

@@ -0,0 +1,113 @@
+# ProgramAsWeights
+
+**Compile natural language specs into tiny neural functions that run locally.**
+
+Define what a function should do in plain English. PAW compiles it into a small neural program that runs on your machine — no API keys at runtime, no internet needed after setup, fully deterministic.
+
+## Install
+
+```bash
+pip install programasweights
+```
+
+## Quick Start
+
+```python
+import programasweights as paw
+
+# Use a pre-compiled function (downloads once, runs locally forever)
+fn = paw.function("email-triage")
+fn("Urgent: the server is down!")        # "immediate"
+fn("Newsletter: spring picnic")          # "wait"
+
+# Compile your own from a description
+program = paw.compile(
+    "Fix malformed JSON: repair missing quotes and trailing commas",
+    compiler="paw-4b-qwen3-0.6b"  # or "paw-4b-gpt2" for smaller/faster
+)
+fn = paw.function(program.id)
+fn("{name: 'Alice',}")  # '{"name": "Alice"}'
+```
+
+## Two Compilers
+
+|                    | Standard (Qwen3 0.6B)   | Compact (GPT-2 124M)   |
+|--------------------|-------------------------|------------------------|
+| Compiler name      | `paw-4b-qwen3-0.6b`    | `paw-4b-gpt2`          |
+| Accuracy           | Higher                  | Lower                  |
+| Base model size    | 594 MB                  | 105 MB                 |
+| Program size       | ~22 MB                  | ~5 MB                  |
+| Inference speed    | ~90ms (server)          | ~50ms (server)         |
+| Runs in browser    | No                      | Yes                    |
+
+Default is Standard (Qwen3 0.6B). Use Compact (GPT-2) when you need smaller files or browser deployment.
+
+## Browser SDK
+
+Programs compiled with GPT-2 also run entirely in the browser via WebAssembly — no server needed, data never leaves the user's device.
+
+```bash
+npm install @programasweights/web
+```
+
+```javascript
+import paw from '@programasweights/web';
+
+const fn = await paw.function('programasweights/email-triage');
+const result = await fn('Urgent: the server is down!');
+// result: "immediate"
+```
+
+See the [browser SDK repo](https://github.com/programasweights/programasweights-js) for full documentation.
+
+## Use with AI Agents
+
+PAW works with Cursor, Claude, Codex, and other AI coding assistants. Paste this into your agent's chat:
+
+> I want to use ProgramAsWeights (PAW) to create fuzzy text functions that run locally. Read the instructions at https://programasweights.com/agents and help me integrate it.
+
+Or save [`AGENTS.md`](https://programasweights.com/agents) to your project root — agents read it automatically.
+
+## When to Use PAW
+
+- **Fuzzy search** — typo-tolerant matching, semantic search, near-duplicate detection
+- **Format repair** — fix broken JSON, normalize dates, repair malformed inputs
+- **Classification** — sentiment, urgency, categories defined in your own words
+- **Extraction** — emails, names, dates from messy unstructured text
+- **Log triage** — extract errors from verbose output, filter noise
+- **Intent routing** — map user descriptions to the closest URL, menu item, or setting
+- **Agent preprocessing** — parse tool calls, validate outputs, route tasks
+
+## Authentication
+
+```bash
+# Option 1: environment variable (recommended)
+export PAW_API_KEY=paw_sk_...
+
+# Option 2: CLI login (opens browser to generate key)
+paw login
+```
+
+Generate API keys at [programasweights.com/settings](https://programasweights.com/settings). Authenticated users get higher rate limits.
+
+## CLI
+
+```bash
+paw compile --spec "Extract error lines from logs" --json
+paw run --program <program_id> --input "[ERROR] timeout" --json
+paw login
+```
+
+`--json` gives structured output for programmatic use.
+
+## Links
+
+- **Website**: [programasweights.com](https://programasweights.com)
+- **Documentation**: [programasweights.readthedocs.io](https://programasweights.readthedocs.io)
+- **Python SDK**: [github.com/programasweights/programasweights-python](https://github.com/programasweights/programasweights-python)
+- **Browser SDK**: [github.com/programasweights/programasweights-js](https://github.com/programasweights/programasweights-js)
+- **Program Hub**: [programasweights.com/hub](https://programasweights.com/hub)
+
+## License
+
+MIT

programasweights-0.1.0.dev8/docs/advanced/adrs.md

@@ -0,0 +1,66 @@
+# Architecture Decision Records
+
+Concise records of major technical choices. Full ADR files may live elsewhere in the repository; this page is the canonical summary for documentation.
+
+---
+
+## ADR 001: llama.cpp instead of PyTorch for the SDK runtime
+
+**Decision:** Ship local inference via **llama-cpp-python** (llama.cpp), not a PyTorch stack.
+
+**Context:** A PyTorch install typically exceeds **2 GB** of dependencies. llama.cpp bindings add on the order of **80 MB**, which keeps the SDK viable as a lightweight dependency.
+
+**Consequence:** Adapter weights must be converted to **GGUF** and loaded through the llama.cpp path. Training and server-side tooling may still use PyTorch where needed; the **end-user runtime** is GGUF-centric.
+
+---
+
+## ADR 002: Q4_0 adapter and Q6_K base quantization
+
+**Decision:** Use **Q4_0** for adapters and **Q6_K** for the base model in shipped bundles.
+
+**Context:** Empirical evaluation on **4096** held-out examples across quantization settings informed the trade-off.
+
+**Consequence:**
+
+- **Q6_K base** — quality is preserved while the footprint is roughly **60% smaller** than fp16.
+- **Q4_0 adapter** — quality loss is negligible; adapter size drops to about **23 MB** versus **78 MB** for a heavier format at comparable settings in prior experiments.
+
+---
+
+## ADR 003: Single specification field in the API
+
+**Decision:** The compile API accepts **one text field** for the specification. There is **no separate “examples” field** in the contract.
+
+**Context:** The compiler was trained on specs that naturally embed examples inside the same prose block.
+
+**Consequence:** The web UI may offer structured fields for examples or hints, but the client **merges** them into a single string before calling the API.
+
+---
+
+## ADR 004: Two-level compiler naming
+
+**Decision:** Expose **pretty aliases** (for example `paw-4b-qwen3-0.6b`) that resolve to **dated snapshots** (for example `paw-4b-qwen3-0.6b-20260325`).
+
+**Context:** Mirrors patterns such as OpenAI’s `gpt-4o` mapping to dated model IDs like `gpt-4o-2024-11-20`.
+
+**Consequence:** Users get stable marketing names while the platform can roll forward immutable snapshots without breaking references that pin the dated id.
+
+---
+
+## ADR 005: vLLM for GPU services
+
+**Decision:** Run **all** GPU-heavy paths (pseudo-program generation, hidden-state extraction / pooling, multi-LoRA inference) on **vLLM** instead of hand-rolled Hugging Face inference loops in production.
+
+**Context:** vLLM improves **throughput**, **batching**, and **memory efficiency** for serving and batched extraction workloads.
+
+**Consequence:** Operations that might classically be expressed as “run HuggingFace model X” are implemented as vLLM-managed models and schedules in the deployed stack.
+
+---
+
+## ADR 006: GitHub OAuth for authentication
+
+**Decision:** Use **GitHub OAuth** rather than email verification or password accounts for primary sign-in.
+
+**Context:** The target audience already maintains GitHub accounts; OAuth removes friction for naming programs, voting, and submitting cases.
+
+**Consequence:** Identity is tied to GitHub; users without GitHub need an alternative path if one is offered separately.

programasweights-0.1.0.dev8/docs/advanced/architecture.md

@@ -0,0 +1,47 @@
+# Architecture
+
+This page summarizes how the ProgramAsWeights production system is structured end to end.
+
+## Components
+
+| Layer | Technology | Role |
+|--------|------------|------|
+| Frontend | React, Vite | Web UI; static assets served by nginx |
+| API | FastAPI, uvicorn | REST endpoints, orchestration, auth integration |
+| Database | PostgreSQL | Users, programs, aliases, votes, cases, operational logs |
+| GPU services | Three vLLM instances | Pseudo-program generation, compiler (including hidden-state work), inference |
+| Storage | Hugging Face, local disk | `.paw` bundles on Hugging Face; PEFT adapter artifacts on server disk |
+| Auth | GitHub OAuth | Sign-in; session backed by HTTP cookies |
+
+### GPU layout
+
+Typical allocation:
+
+- **GPU 0** — pseudo-program generation
+- **GPU 1** — compiler workload (including pooling / hidden-state extraction used in the pipeline)
+- **GPU 2** — multi-LoRA inference
+
+Exact mapping may vary by deployment; the important split is dedicated vLLM roles per stage.
+
+## Compile pipeline
+
+High-level flow:
+
+1. **Pseudo-generation** (vLLM) — turn the natural-language spec into a pseudo-program representation.
+2. **LoRA extraction** — derive adapter weights from vLLM hidden states / pooling as implemented in the compiler stack.
+3. **Quantization** — convert adapters to **Q4_0 GGUF** for the bundle format used by the runtime.
+4. **Bundle** — assemble the **`.paw`** package with metadata and weights.
+5. **Upload** — publish the `.paw` artifact to Hugging Face for CDN-backed distribution.
+
+## Caching
+
+The system uses **two-level caching**:
+
+1. **Pseudo-generation cache** — avoid recomputing pseudo-programs for identical or equivalent spec inputs where the cache key applies.
+2. **Program-level disk cache** — reuse compiled artifacts and intermediate state on the server when the same content-addressed program is requested again.
+
+Together these reduce redundant GPU work and speed up repeat compiles.
+
+## Downloads and the SDK
+
+The Python SDK **downloads `.paw` files from the Hugging Face CDN** (or equivalent object storage fronted as a CDN). Programs are **not** served as large binary payloads from the ProgramAsWeights API host, which keeps the API focused on metadata, auth, and orchestration.