programasweights 0.2.7__tar.gz → 0.2.8__tar.gz

{programasweights-0.2.7 → programasweights-0.2.8}/AGENTS.md

@@ -78,6 +78,11 @@ Output: delete
 - **Include examples from your actual data**: Examples outperform prose-only descriptions.
 - **Debug failures before sweeping**: Look at specific failing examples and understand WHY before trying many variants.
 
+## Context Window
+
+- Spec + input + output share a ~2048 token context window. Inputs that exceed it will error.
+- `max_tokens` defaults to `None`: generation runs until EOS or the context limit.
+
 ## Chaining Functions
 
 Multiple PAW functions can be composed for multi-step tasks:
@@ -93,11 +98,7 @@ if label != "other":
 
 Chain them with regular Python logic.
 
-## Case Studies
-
-Production examples with iterative spec-tuning walkthroughs: [site navigation](https://programasweights.readthedocs.io/en/latest/case-studies/site-navigation/) (5-program pipeline, Cmd+K helper) and [semantic search](https://programasweights.readthedocs.io/en/latest/case-studies/semantic-search/) (reranking without a vector database).
-
-## Event-Driven Monitoring
+## Log Monitoring
 
 PAW functions can classify log output. Compile once with examples from your specific logs, then reuse the function locally forever:
 
@@ -199,8 +200,7 @@ paw.login()
 - **Cache**: `~/.cache/programasweights/`. Override with `PAW_CACHE_DIR`.
 - **Offline** after first download.
 
-## Limits
+## Case Studies
 
-- Spec + input + output share a ~2048 token context window. Inputs that exceed it will error.
-- `max_tokens` defaults to `None`: generation runs until EOS or the context limit.
+Detailed walkthroughs of building production systems with PAW, including what we tried and what we learned: [log monitoring](https://programasweights.readthedocs.io/en/latest/case-studies/log-monitoring/), [site navigation](https://programasweights.readthedocs.io/en/latest/case-studies/site-navigation/), [semantic search](https://programasweights.readthedocs.io/en/latest/case-studies/semantic-search/), [tool calling](https://programasweights.readthedocs.io/en/latest/case-studies/tool-calling/).
 

{programasweights-0.2.7 → programasweights-0.2.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: programasweights
-Version: 0.2.7
+Version: 0.2.8
 Summary: Compile natural language specifications into neural programs that run locally via llama.cpp.
 Project-URL: Homepage, https://programasweights.com
 Project-URL: Repository, https://github.com/programasweights/programasweights-python

{programasweights-0.2.7 → programasweights-0.2.8}/docs/adr/001-llama-cpp-over-pytorch.md

@@ -17,5 +17,5 @@ Replace the PyTorch runtime with llama-cpp-python (~80MB install). Use GGUF mode
 - Inference uses Metal (Mac), CPU (Linux/Windows) — no GPU required
 - Must pre-render chat templates server-side (no transformers tokenizer on client)
 - .paw format must change from v2 (safetensors) to v3 (GGUF adapter)
-- Base model is downloaded once (~623MB for Q6_K) and shared across all functions
+- Base model is downloaded once (~594 MB for Q6_K) and shared across all functions
 - Per-function adapter download is ~23MB (Q4_0, confirmed lossless at 4096-scale eval)

{programasweights-0.2.7 → programasweights-0.2.8}/docs/api-reference/cli.md

@@ -57,7 +57,7 @@ paw rename <program_id> <new-slug> [--json]
 | Argument | Description |
 |----------|-------------|
 | `program_id` | Program hash ID or current slug. |
-| `new-slug` | New slug name (2-50 chars, lowercase alphanumeric and hyphens). |
+| `new-slug` | New slug name (2-50 chars, lowercase alphanumeric and hyphens). Pass `""` to remove. |
 
 ## `paw info`
 

{programasweights-0.2.7 → programasweights-0.2.8}/docs/api-reference/python-sdk.md

@@ -41,7 +41,7 @@ output: str = fn(input_text, max_tokens=None, temperature=0.0)
 | `max_tokens` | Maximum tokens to generate. `None` (default) = use all remaining context window. |
 | `temperature` | Sampling temperature (default `0.0`). |
 
-**Context limits:** Spec + input + output share a ~2048 token window. If input + output together exceed the window, generation will error. For short inputs, high `max_tokens` is safe (stops at EOS). For long inputs, set `max_tokens` conservatively.
+**Context limits:** Spec + input + output share a ~2048 token window. Inputs that exceed it will error. `max_tokens` defaults to `None`: generation runs until EOS or the context limit.
 
 ## `paw.compile`
 

{programasweights-0.2.7 → programasweights-0.2.8}/docs/api-reference/rest-api.md

@@ -34,7 +34,11 @@ Compile a specification.
 | `name` | string | No | Display title. Auto-generated if omitted. |
 | `tags` | string[] | No | Tags for discovery. |
 
-**Response (JSON):** includes `job_id`, `status`, `program_id`, `slug` (if created), `pseudo_program`, and `timings`.
+**Response:**
+- **202 Accepted** on success — JSON with `job_id`, `status`, `program_id`, `slug` (if created), `pseudo_program`, and `timings`
+- **400** — unknown compiler
+- **422** — spec too short, too long, or token limit exceeded
+- **500** — compilation failed (server error)
 
 ### `POST /infer`
 
@@ -88,7 +92,11 @@ Owner-only fields: `public`, `name`, `tags`. Any authenticated user can set thei
 
 ### `GET /programs/{id_or_slug}/download`
 
-Redirects to the Hugging Face CDN for the `.paw` artifact. Private programs return 404 to non-owners.
+Downloads the `.paw` artifact. Returns one of:
+- **302** redirect to HuggingFace CDN (program already uploaded)
+- **200** file from server (freshly compiled, not yet on HF)
+- **202 Accepted** with `Retry-After` header (program assets still generating, retry after the indicated seconds)
+- **404** if program not found or private and not owned by the requester
 
 ### `GET /programs/resolve/{slug}`
 

{programasweights-0.2.7 → programasweights-0.2.8}/docs/architecture.md

@@ -31,7 +31,7 @@ vLLM multi-LoRA serving on GPU 2 (or GPU 1 in 2-GPU config). Each compiled progr
 
 ## Inference (Client-side SDK)
 
-llama-cpp-python loads the base GGUF model (623MB, downloaded once) and hot-swaps Q4_0 adapters per function call. No PyTorch or transformers required.
+llama-cpp-python loads the base GGUF model (594 MB, downloaded once) and hot-swaps Q4_0 adapters per function call. No PyTorch or transformers required.
 
 ## GPU Assignment
 

programasweights-0.2.8/docs/case-studies/log-monitoring.md

@@ -0,0 +1,132 @@
+# Event-Driven Log Monitoring
+
+Long-running processes -- training runs, deployments, data pipelines -- produce thousands of log lines. You want to know when something important happens (checkpoint saved, error, completion) without watching the terminal. PAW lets you compile a classifier that runs locally and alerts on the lines that matter.
+
+**Full tool:** [examples/paw_monitor.py](https://github.com/programasweights/programasweights-python/blob/main/examples/paw_monitor.py) on GitHub.
+
+## How we built it
+
+### Attempt 1: Keyword matching
+
+The obvious first approach: grep for "error", "fail", "complete".
+
+**Result:** Too noisy. "error" matches "error_count=0" (routine metric). "complete" matches "batch complete" (every 10 seconds). Missing important lines like "Traceback" or "CUDA out of memory" that don't contain the keywords.
+
+**Lesson:** The whole reason to use PAW is that keyword matching can't express "important enough to interrupt me."
+
+### Attempt 2: One-shot description
+
+```
+Classify if this log line is important enough to alert on. Return ALERT or QUIET.
+```
+
+**Result:** Too vague. The model alerted on everything that looked unusual, including routine debug output. "Important" means different things in different contexts.
+
+**Lesson:** Don't rely on abstract instructions. The model needs concrete examples of what ALERT and QUIET look like in YOUR specific logs.
+
+### Attempt 3: Example-based spec
+
+```
+Classify log lines. Return ONLY one word: ALERT or QUIET.
+
+Input: [step 100] loss=0.05 lr=0.0001
+Output: QUIET
+
+Input: [Checkpoint] Saved model at step 1000
+Output: ALERT
+
+Input: Traceback (most recent call last):
+Output: ALERT
+
+Input: Training complete. Final loss: 0.11
+Output: ALERT
+```
+
+**Result:** This worked. The model learned the boundary between routine metrics (QUIET) and significant events (ALERT) from the examples. Adding 3-5 examples from actual logs was the key.
+
+**Lesson:** Include examples from your actual data. 3-5 representative examples consistently outperform prose-only descriptions.
+
+### Refinement: Developer feedback
+
+A developer using PAW for monitoring diffusion model training shared their experience. Key insights:
+
+- **Positive instructions work better than negative ones.** "Output ALERT only if X, Y, or Z" is more reliable than "Don't alert on routine output."
+- **The spec should match the monitoring context.** Different training phases produce different log patterns. Examples should cover the specific domain.
+- **Stall detection needs a separate mechanism.** PAW classifies what it sees -- it can't detect the absence of output. The monitoring tool needs a timer for "no new output in N minutes."
+
+## The solution
+
+### The spec
+
+```python
+program = paw.compile("""
+Classify log lines. Return ONLY one word: ALERT or QUIET.
+
+Input: [step 100] loss=0.05 lr=0.0001
+Output: QUIET
+
+Input: [Checkpoint] Saved model at step 1000
+Output: ALERT
+
+Input: Traceback (most recent call last):
+Output: ALERT
+
+Input: Training complete. Final loss: 0.11
+Output: ALERT
+""")
+
+fn = paw.function(program.id)
+```
+
+Compile once, save the program ID, reuse forever. The function runs locally with no internet after the first download.
+
+### The monitoring loop
+
+```python
+import time
+
+fn = paw.function("your-program-id")
+
+last_size = 0
+while True:
+    size = os.path.getsize(log_file)
+    if size > last_size:
+        with open(log_file) as f:
+            f.seek(last_size)
+            new_text = f.read()
+        last_size = size
+
+        # Truncate to last ~1000 chars to fit context window
+        chunk = new_text[-1000:] if len(new_text) > 1000 else new_text
+        result = fn(chunk)
+        if result.strip() == "ALERT":
+            send_notification(chunk)
+
+    time.sleep(10)
+```
+
+### Full tool
+
+The complete [paw_monitor.py](https://github.com/programasweights/programasweights-python/blob/main/examples/paw_monitor.py) adds:
+
+- **File watching** with `seek()` to only process new content
+- **Input truncation** to fit the ~2048 token context window
+- **Stall detection** -- alerts if no new output for a configurable timeout
+- **`--focus` and `--ignore`** flags to guide what the classifier pays attention to
+- **`--local` flag** to run entirely offline after first compile
+- **`--json` output** for integration with other tools
+
+## Adapting this for your use case
+
+1. **Collect 5-10 example log lines** from your actual process -- a mix of routine output and important events
+2. **Write a spec** with `Input: ... Output: ALERT/QUIET` pairs
+3. **Test with real logs** -- pipe a log file through the classifier and check which lines it flags
+4. **Iterate** -- if it alerts too much, add more QUIET examples. If it misses things, add ALERT examples.
+5. **Save the program ID** -- compile once, monitor forever
+
+## Takeaways
+
+- **Examples beat descriptions** for teaching the model your specific log patterns.
+- **Compile once, run forever.** The compiled function is cached locally and needs no internet.
+- **PAW classifies what it sees** -- combine with a timer for stall detection.
+- **Iterate with real data.** The first spec is rarely perfect. Test, check failures, adjust examples.

programasweights-0.2.8/docs/case-studies/tool-calling.md

@@ -0,0 +1,477 @@
+# Tool Calling with a 10-Function Pipeline
+
+Large language models are increasingly expected to call tools — pick the right API, extract parameters, chain multi-step flows, and know when NOT to act. This normally requires large models (70B+). We used PAW to build a tool-calling system from 10 tiny compiled functions running on a 0.6B interpreter, and tested it on the [ToolCall-15](https://github.com/stevibe/ToolCall-15) benchmark.
+
+The entire pipeline was built in a single Cursor session by an AI coding agent (Claude Opus 4.6) that read PAW's [AGENTS.md](https://programasweights.com/AGENTS.md) and the benchmark source.
+
+**Result:** 93% (28/30 points) on ToolCall-15, with 14 out of 15 scenarios passing.
+
+!!! warning "Benchmark leakage"
+    The AI agent that built this pipeline had access to the benchmark source code, including the 15 scenario definitions and scoring logic. This means the specs and heuristics were likely tailored to these specific test cases. We report 93% as a proof of concept for the architecture, not a generalizable accuracy claim. A principled evaluation would test on held-out scenarios the builder has never seen.
+
+## The benchmark: ToolCall-15
+
+[ToolCall-15](https://github.com/stevibe/ToolCall-15) (by [stevibe](https://x.com/stevibe)) is a visual benchmark for LLM tool use. It runs 15 hand-picked scenarios through an OpenAI-compatible chat completions interface, scores each deterministically, and renders results in a live dashboard.
+
+**12 available tools** (given to every scenario):
+
+`web_search`, `get_weather`, `calculator`, `send_email`, `search_files`, `read_file`, `create_calendar_event`, `get_contacts`, `translate_text`, `get_stock_price`, `set_reminder`, `run_code`
+
+**15 scenarios across 5 categories** (3 each, max 6 points per category):
+
+| Category | Scenarios | What it tests |
+|---|---|---|
+| A: Tool Selection | TC-01 Weather in Berlin, TC-02 AAPL stock price, TC-03 "Let Sarah know..." | Pick the right tool, resist distractors, infer implicit chains |
+| B: Parameter Precision | TC-04 Tokyo in Fahrenheit, TC-05 "Next Monday 9:30am", TC-06 Translate to both Spanish & Japanese | Pass correct arguments, parse dates, handle multi-value |
+| C: Multi-Step Chains | TC-07 Find report → read → email total, TC-08 Check weather → if raining → set reminder, TC-09 Weather AND stock price | Thread data across steps, branch conditionally, run in parallel |
+| D: Restraint & Refusal | TC-10 "When did WWII end?", TC-11 "15% of 200", TC-12 "Delete my emails" | Don't use tools for trivial knowledge, refuse impossible requests |
+| E: Error Recovery | TC-13 Empty search results → retry, TC-14 Stock API rate limited, TC-15 Search population → calculate 2% | Handle errors gracefully, maintain data integrity |
+
+Scoring: pass = 2 points, partial = 1, fail = 0. Final score = average of 5 category percentages.
+
+## The approach: decompose into specialists
+
+The key insight is the same one from our [site navigation](site-navigation.md) case study: **split the task into small, focused functions.** A single monolithic prompt trying to handle tool selection, parameter extraction, multi-step reasoning, and restraint all at once would fail. Instead, each PAW function handles one decision.
+
+The 10 compiled functions divide into three groups:
+
+**Routing (which tool?):**
+
+| Function | Spec (abbreviated) | Output |
+|---|---|---|
+| `tc15-needs-tool` | "Does this request need an external tool, or can it be answered directly?" | YES / NO |
+| `tc15-tool-router` | "Which tool should be called?" with 10 examples covering all 12 tools + NONE | One of 13 labels |
+| `tc15-impossible-check` | "Is this request something no available tool can fulfill?" | IMPOSSIBLE / POSSIBLE |
+| `tc15-second-tool` | "After the first tool, is a second tool needed?" | Tool name or NONE |
+
+**Extraction (what parameters?):**
+
+| Function | Spec (abbreviated) | Output |
+|---|---|---|
+| `tc15-extract-location` | "Extract the city name from a weather request" | City name |
+| `tc15-extract-ticker` | "Extract the stock ticker symbol" | Ticker (e.g. AAPL) |
+| `tc15-extract-units` | "Extract temperature units, default celsius" | fahrenheit / celsius |
+| `tc15-extract-search-query` | "Extract the file search query" | Search string |
+| `tc15-extract-person` | "Extract the person's name" | Name |
+| `tc15-extract-translate` | "Extract source_lang, target_lang, text" | Pipe-separated triple |
+
+**Not compiled (handled by code):**
+
+Some tasks are better handled by regular code than by a neural function:
+
+- **Date/time parsing** (regex): "next Monday at 9:30am" → `2026-03-23T09:30`
+- **Direct answers** (lookup table): "15% of 200" → `30`
+- **Response synthesis** (template): format tool results into natural language
+- **Protocol formatting**: build OpenAI-compatible `tool_calls` JSON
+
+This is a general design principle: **use PAW for fuzzy judgment, use code for structured logic.**
+
+## The full specs
+
+Here are the actual specs the agent compiled. Each is a standalone PAW function.
+
+### tc15-tool-router
+
+```
+Classify which tool should be called for a user request.
+Output EXACTLY one of these tool names:
+web_search, get_weather, calculator, send_email, search_files,
+read_file, create_calendar_event, get_contacts, translate_text,
+get_stock_price, set_reminder, run_code, NONE.
+Output NONE if the user's question can be answered from general
+knowledge without any tool.
+
+Input: What's the weather in Berlin?
+Output: get_weather
+
+Input: What is the price of AAPL stock?
+Output: get_stock_price
+
+Input: What year did WWII end?
+Output: NONE
+
+Input: What is 15% of 200?
+Output: NONE
+
+Input: Delete all my emails
+Output: NONE
+
+Input: Translate hello to Spanish
+Output: translate_text
+
+Input: Find the Q3 budget report
+Output: search_files
+
+Input: Send an email to Sarah
+Output: get_contacts
+
+Input: Schedule a meeting for Monday
+Output: create_calendar_event
+
+Input: Remind me to bring an umbrella
+Output: set_reminder
+```
+
+### tc15-needs-tool
+
+```
+Determine if a user request needs an external tool or can be
+answered directly. Output YES if the request needs real-time data,
+file access, sending messages, scheduling, or looking up current
+information. Output NO if it's basic knowledge, simple math, or
+an impossible request with no available tool.
+
+Input: What's the weather?
+Output: YES
+
+Input: What year did WWII end?
+Output: NO
+
+Input: What is 15% of 200?
+Output: NO
+
+Input: Delete all my emails
+Output: NO
+
+Input: What's AAPL stock price?
+Output: YES
+
+Input: Find the budget report
+Output: YES
+```
+
+### tc15-extract-location
+
+```
+Extract the location/city name from a weather or location request.
+Output just the city or location name, nothing else.
+
+Input: What's the weather in Berlin?
+Output: Berlin
+
+Input: Temperature in Tokyo in Fahrenheit
+Output: Tokyo
+
+Input: How's the weather in Paris right now?
+Output: Paris
+
+Input: London weather forecast
+Output: London
+```
+
+### tc15-extract-ticker
+
+```
+Extract the stock ticker symbol from a stock price request.
+Output just the uppercase ticker symbol.
+
+Input: What is the price of AAPL stock?
+Output: AAPL
+
+Input: MSFT stock price
+Output: MSFT
+
+Input: How much is Apple stock?
+Output: AAPL
+
+Input: Microsoft stock price?
+Output: MSFT
+```
+
+### tc15-extract-units
+
+```
+Extract temperature units from a weather request.
+Output 'fahrenheit' if Fahrenheit is mentioned, otherwise
+output 'celsius'.
+
+Input: Temperature in Tokyo in Fahrenheit
+Output: fahrenheit
+
+Input: Weather in Berlin
+Output: celsius
+
+Input: Paris weather in F
+Output: fahrenheit
+
+Input: London temperature celsius
+Output: celsius
+```
+
+### tc15-extract-search-query
+
+```
+Extract the file search query from a user request about finding
+documents or files. Output a concise search query string.
+
+Input: Find the Q3 budget report
+Output: Q3 budget report
+
+Input: Find the Johnson proposal document
+Output: Johnson proposal
+
+Input: Search for the annual review
+Output: annual review
+```
+
+### tc15-extract-person
+
+```
+Extract the person's name from a request about contacting or
+looking up someone. Output just the person's name.
+
+Input: I need to let Sarah know about the meeting
+Output: Sarah
+
+Input: Send an email to John
+Output: John
+
+Input: Contact my manager
+Output: manager
+
+Input: Email the report to Jamie
+Output: Jamie
+```
+
+### tc15-extract-translate
+
+```
+Extract translation parameters from a translate request.
+Output in the format: source_lang|target_lang|text.
+If multiple target languages, output one line per target language.
+
+Input: Translate hello to Spanish
+Output: English|Spanish|hello
+
+Input: Translate Where is the hospital from English to Japanese
+Output: English|Japanese|Where is the hospital
+```
+
+### tc15-impossible-check
+
+```
+Determine if a request asks for something that cannot be done
+with standard tools (web search, weather, calculator, email,
+file search/read, calendar, contacts, translate, stocks,
+reminders, code execution).
+Output IMPOSSIBLE if no tool can fulfill it (like deleting
+emails, modifying files, accessing databases, etc).
+Output POSSIBLE otherwise.
+
+Input: Delete all my emails from last month
+Output: IMPOSSIBLE
+
+Input: What's the weather?
+Output: POSSIBLE
+
+Input: Hack into the server
+Output: IMPOSSIBLE
+
+Input: Send an email
+Output: POSSIBLE
+```
+
+### tc15-second-tool
+
+```
+Given a user request that may need multiple tools, identify if
+a SECOND tool is needed after the first. Output the second tool
+name or NONE.
+
+Input: Weather in London and MSFT stock price
+Output: get_stock_price
+
+Input: Find the report and email it to my manager
+Output: read_file
+
+Input: Check weather, if raining remind me about umbrella
+Output: set_reminder
+
+Input: What's the weather in Berlin?
+Output: NONE
+
+Input: Let Sarah know the meeting moved to 3pm
+Output: send_email
+```
+
+## Pipeline architecture
+
+The 10 PAW functions are orchestrated by a FastAPI proxy server that speaks the OpenAI `/v1/chat/completions` protocol. Here's the flow:
+
+```
+User message arrives at proxy
+│
+├─ detect_parallel_tools (regex)
+│  "weather AND stock price" → call both tools in parallel
+│  └─ if match: extract params for each → return tool_calls → done
+│
+├─ tc15-needs-tool → "Does this need a tool?"
+│  └─ NO → tc15-impossible-check → explain refusal or answer directly
+│
+├─ tc15-tool-router → "Which tool?"
+│  └─ NONE → answer from knowledge (lookup table or simple math)
+│
+├─ extract_params_for_tool(tool_name):
+│  ├─ get_weather    → tc15-extract-location + tc15-extract-units
+│  ├─ get_stock_price → tc15-extract-ticker
+│  ├─ search_files   → tc15-extract-search-query
+│  ├─ get_contacts   → tc15-extract-person
+│  ├─ translate_text  → tc15-extract-translate (may produce 2 calls)
+│  ├─ calendar_event  → regex date/time parsing
+│  └─ set_reminder   → regex date/time parsing
+│
+└─ Return tool_calls in OpenAI JSON format
+
+──── On follow-up turns (tool results come back) ────
+
+├─ decide_follow_up_tool:
+│  ├─ Error result? → explain to user
+│  ├─ Empty results? → retry with broader query
+│  ├─ tc15-second-tool → "Is another tool needed?"
+│  │  └─ Chain logic: search → read → contacts → send_email
+│  └─ No more tools → synthesize_response from all results
+│
+└─ Return either more tool_calls or final text response
+```
+
+The proxy handles multi-turn conversations by tracking which tools have already been called and threading data between steps (e.g., using a `file_id` from a search result in a subsequent `read_file` call).
+
+## How it was built
+
+A single prompt to Cursor (Claude Opus 4.6):
+
+> "Build a PAW-powered proxy for the ToolCall-15 benchmark. Read AGENTS.md for PAW usage. Analyze the benchmark scenarios and create specialized PAW functions for tool routing and parameter extraction. Wire them into an OpenAI-compatible endpoint."
+
+The agent:
+
+1. Read the benchmark methodology and all 15 scenario definitions
+2. Identified which decisions need fuzzy judgment (PAW) vs. structured logic (code)
+3. Designed and compiled 10 PAW functions with specs tailored to the scenarios
+4. Built a FastAPI proxy server (~700 lines) with conversation state management
+5. Ran the benchmark, debugged failures, and iterated
+
+### Iteration 1: 70% (21/30)
+
+The first run had several failures:
+
+- **TC-01, TC-02 (Tool Selection): Infinite loop.** After calling the correct tool and receiving results, the proxy called the same tool again instead of synthesizing a response. Root cause: the conversation state handler didn't properly detect "I already have results, stop calling tools."
+- **TC-08 (Conditional Branching): Wrong first tool.** The router sent "check weather, if raining set reminder" to `create_calendar_event` instead of `get_weather`. The tool-router spec didn't have an example for conditional chains.
+- **TC-15 (Data Integrity): Wrong second tool.** After `web_search` returned Iceland's population, the proxy called `get_stock_price("Iceland")` instead of `calculator`. The `tc15-second-tool` function didn't have enough examples for search-then-calculate chains.
+- **TC-07 (Multi-Step Chain): Partial.** The 4-step chain (search → read → contacts → email) partially worked but didn't thread data correctly between all steps.
+
+### Iteration 2: fix the orchestration
+
+The agent fixed the proxy code (not the PAW specs):
+
+- Added proper conversation state tracking to detect when tool results are present
+- Added routing overrides for common patterns (conditional weather checks, search-then-calculate)
+- Fixed data threading in multi-step chains (extracting `file_id` from search results, email from contacts)
+- Recompiled 2 functions that had HuggingFace download failures
+
+### Final result: 93% (28/30)
+
+| Category | Score | Details |
+|---|---|---|
+| A: Tool Selection | 100% (6/6) | TC-01 pass, TC-02 pass, TC-03 pass |
+| B: Parameter Precision | 100% (6/6) | TC-04 pass, TC-05 pass, TC-06 pass |
+| C: Multi-Step Chains | 100% (6/6) | TC-07 pass, TC-08 pass, TC-09 pass |
+| D: Restraint & Refusal | 100% (6/6) | TC-10 pass, TC-11 pass, TC-12 pass |
+| E: Error Recovery | 67% (4/6) | TC-13 **fail**, TC-14 pass, TC-15 pass |
+
+**14 out of 15 scenarios passed.** The single failure:
+
+### Why TC-13 failed
+
+**TC-13 (Empty Results Retry):** The user asks "Find the Johnson proposal document." The proxy correctly calls `search_files(query="Johnson proposal")`, gets empty results, but instead of retrying with a broader query like `"Johnson"`, it immediately responds "No results were found."
+
+**Root cause:** The proxy has loop-prevention logic that blocks re-calling a tool already in `tools_called`. This prevents infinite loops (which was a real problem in iteration 1) but also prevents the legitimate retry pattern where you call the same tool with different parameters. The agent's fix for the infinite loop was too aggressive — it stopped all same-tool retries.
+
+This is an **architectural limitation**, not a PAW function limitation. The `tc15-extract-search-query` function works fine. The issue is in the orchestration code's retry policy.
+
+## Contamination disclosure
+
+The AI agent that built this pipeline had full access to:
+
+- The 15 scenario definitions (user messages, expected tools, scoring logic)
+- The mocked tool responses
+- The benchmark source code
+
+This means:
+
+1. The specs were tuned for these 15 specific scenarios (e.g., the tool-router examples map closely to the actual test inputs)
+2. The routing overrides in the proxy code target specific scenario patterns
+3. The direct-answer lookup table contains exact answers for TC-10 and TC-11
+
+**What this proves:** The architecture works — 10 small PAW specialists + Python glue can implement a tool-calling system that scores 93% on a multi-category benchmark.
+
+**What this does NOT prove:** That these exact specs would generalize to unseen tool-calling scenarios. A principled evaluation would compile specs without seeing the test cases, then evaluate on held-out scenarios.
+
+## Full code
+
+### compile_functions.py
+
+The compilation script that creates all 10 functions:
+
+```python
+import json, sys
+import programasweights as paw
+
+COMPILER = "paw-4b-qwen3-0.6b"
+
+FUNCTIONS_TO_COMPILE = [
+    {"slug": "tc15-tool-router",         "spec": "Classify which tool..."},
+    {"slug": "tc15-needs-tool",          "spec": "Determine if request needs tool..."},
+    {"slug": "tc15-extract-location",    "spec": "Extract city name..."},
+    {"slug": "tc15-extract-ticker",      "spec": "Extract stock ticker..."},
+    {"slug": "tc15-extract-units",       "spec": "Extract temperature units..."},
+    {"slug": "tc15-extract-search-query","spec": "Extract file search query..."},
+    {"slug": "tc15-extract-person",      "spec": "Extract person's name..."},
+    {"slug": "tc15-extract-translate",   "spec": "Extract translation parameters..."},
+    {"slug": "tc15-impossible-check",    "spec": "Is this request impossible?..."},
+    {"slug": "tc15-second-tool",         "spec": "Is a second tool needed?..."},
+]
+
+for fn_def in FUNCTIONS_TO_COMPILE:
+    program = paw.compile(fn_def["spec"], compiler=COMPILER)
+    print(f"{fn_def['slug']}: {program.id}")
+```
+
+### server.py (key routing logic)
+
+The proxy server's main decision flow:
+
+```python
+@app.post("/v1/chat/completions")
+async def chat_completions(request: ChatRequest):
+    messages = [m.model_dump() for m in request.messages]
+    user_msg = ""
+    for msg in messages:
+        if msg.get("role") == "user":
+            user_msg = msg.get("content", "")
+
+    # If we already have tool results, decide: more tools or synthesize?
+    if any(m.get("role") == "tool" for m in messages):
+        follow_up = decide_follow_up_tool(user_msg, messages)
+        if follow_up:
+            return make_response(tool_calls=follow_up)
+        return make_response(content=synthesize_response(user_msg, messages))
+
+    # First turn: should we use tools at all?
+    tool_calls = build_tool_calls_for_user(user_msg, messages)
+    if tool_calls:
+        return make_response(tool_calls=tool_calls)
+
+    # No tools needed — answer directly
+    return make_response(content=generate_direct_answer(user_msg))
+```
+
+The full source code is available in the [paw-proxy directory](https://github.com/stevibe/ToolCall-15/tree/main/paw-proxy) of the ToolCall-15 repository.
+
+## Takeaways
+
+- **Decomposition works.** 10 small specialists beat one monolithic prompt. Each function handles one decision — "which tool?", "what city?", "is this impossible?" — and does it well.
+- **PAW + code hybrid.** Fuzzy judgment (tool routing, entity extraction, intent classification) goes to PAW. Structured logic (date parsing, JSON formatting, conversation state) stays in Python. Neither alone would work.
+- **An AI agent can build the whole thing.** From analyzing the benchmark to compiling functions to writing the proxy server — one Cursor session, one prompt.
+- **Loop prevention vs. retry is hard.** The single failure (TC-13) came from the tension between preventing infinite tool loops and allowing legitimate retries with different parameters. This is an orchestration problem, not a model problem.
+- **0.6B is enough for focused tasks.** Tool routing across 12 options, entity extraction, yes/no classification — none of these need a large model when the spec is precise and examples are clear.
+- **Always evaluate on unseen data.** Our 93% includes benchmark leakage. The architecture is sound, but the specific accuracy is optimistic.

programasweights-0.2.8/docs/guide/writing-good-specs.md

@@ -0,0 +1,11 @@
+# Writing Good Specs
+
+The canonical spec-writing guide lives in [AGENTS.md](https://programasweights.com/agents) — the same document that AI coding assistants read when integrating PAW.
+
+See the **"Writing Good Specs"** section there for:
+
+- The #1 practice: iterate with test cases
+- Spec-tuning tips (role naming, output constraints, positive vs negative instructions, field ordering, and more)
+- Example specs with `Input: ... Output: ...` format
+
+You can also copy the full AGENTS.md from the [AI Agents page](https://programasweights.com/agents) and save it as `AGENTS.md` in your project root — AI assistants will read it automatically.

{programasweights-0.2.7 → programasweights-0.2.8}/mkdocs.yml

@@ -52,8 +52,10 @@ nav:
     - Publishing Programs: hub/publishing-programs.md
     - Feedback & Cases: hub/feedback-cases.md
   - Case Studies:
+    - Log Monitoring: case-studies/log-monitoring.md
     - Site Navigation: case-studies/site-navigation.md
     - Semantic Search: case-studies/semantic-search.md
+    - Tool Calling: case-studies/tool-calling.md
   - Advanced:
     - Architecture: advanced/architecture.md
     - Architecture Decision Records: advanced/adrs.md

{programasweights-0.2.7 → programasweights-0.2.8}/programasweights/__init__.py

@@ -27,7 +27,7 @@ try:
     from importlib.metadata import version as _meta_version
     __version__ = _meta_version("programasweights")
 except Exception:
-    __version__ = "0.2.7"
+    __version__ = "0.2.8"
 
 from .config import get_api_url, get_api_key, set_api_key
 
@@ -68,8 +68,15 @@ def compile(
         '{"name": "Alice"}'
     """
     from .client import PAWClient
+    from ._output import status
+    status("Compiling...")
     client = PAWClient(api_url=get_api_url(), api_key=get_api_key())
-    return client.compile(spec, compiler=compiler, name=name, tags=tags, public=public, slug=slug)
+    result = client.compile(spec, compiler=compiler, name=name, tags=tags, public=public, slug=slug)
+    label = f"{result.id}"
+    if result.slug:
+        label += f" ({result.slug})"
+    status(f"Compiled: {label}")
+    return result
 
 
 def function(
@@ -111,12 +118,15 @@ def function(
     if n_gpu_layers is None:
         n_gpu_layers = int(os.environ.get("PAW_GPU_LAYERS", "0"))
 
+    from ._output import status
+
     resolved_id = program_id
     if not re.fullmatch(r"[a-f0-9]{16,64}", program_id):
         cached = get_cached_slug(program_id)
         if cached:
             resolved_id = cached
         else:
+            status(f"Resolving {program_id}...")
             from .client import PAWClient
             client = PAWClient(api_url=get_api_url(), api_key=get_api_key())
             resolved_id = client.resolve_slug(program_id)

programasweights-0.2.8/programasweights/_output.py

@@ -0,0 +1,34 @@
+"""User-facing status messages for the PAW SDK.
+
+All messages go to stderr so they don't pollute stdout when users pipe output.
+Set PAW_QUIET=1 to suppress all status messages.
+"""
+from __future__ import annotations
+
+import os
+import sys
+
+
+def _quiet() -> bool:
+    return os.environ.get("PAW_QUIET", "").strip() in ("1", "true", "yes")
+
+
+def status(msg: str, **kwargs) -> None:
+    """Print a status message to stderr."""
+    if _quiet():
+        return
+    print(msg, file=sys.stderr, flush=True, **kwargs)
+
+
+def status_inline(msg: str) -> None:
+    """Print a status message inline (no newline) for progress updates."""
+    if _quiet():
+        return
+    print(f"\r{msg}", end="", file=sys.stderr, flush=True)
+
+
+def status_end() -> None:
+    """End an inline status line."""
+    if _quiet():
+        return
+    print(file=sys.stderr, flush=True)

{programasweights-0.2.7 → programasweights-0.2.8}/programasweights/cache.py

@@ -54,9 +54,10 @@ def get_base_model_path(interpreter: str = "Qwen/Qwen3-0.6B") -> Path:
             f"Available: {list(INTERPRETER_TO_GGUF.keys())}"
         )
 
-    print(f"Downloading base model {gguf_name} (one-time download)...")
+    from ._output import status
+    status(f"Downloading interpreter {gguf_name} (one-time download)...")
     _download_file(url, gguf_path)
-    print(f"Saved to {gguf_path}")
+    status(f"Saved to {gguf_path}")
     return gguf_path
 
 
@@ -87,8 +88,10 @@ def _download_file(url: str, dest: Path):
                     if total > 0:
                         pct = downloaded / total * 100
                         mb = downloaded / 1024 / 1024
-                        print(f"\r  {mb:.1f} MB ({pct:.0f}%)", end="", flush=True)
-            print()
+                        from ._output import status_inline, status_end
+                        status_inline(f"  {mb:.1f} MB ({pct:.0f}%)")
+            from ._output import status_end
+            status_end()
         os.replace(str(tmp), str(dest))
     except BaseException:
         tmp.unlink(missing_ok=True)

{programasweights-0.2.7 → programasweights-0.2.8}/programasweights/client.py

@@ -114,9 +114,13 @@ class PAWClient:
         if (program_dir / "prompt_template.txt").exists():
             return program_dir
 
+        from ._output import status
+
+        status(f"Downloading program {program_id[:12]}...")
         max_wait = 30
         elapsed = 0
         resp = None
+        waiting_logged = False
         while elapsed < max_wait:
             resp = httpx.get(
                 f"{self._api_url}/api/v1/programs/{program_id}/download",
@@ -125,6 +129,9 @@ class PAWClient:
                 follow_redirects=True,
             )
             if resp.status_code == 202:
+                if not waiting_logged:
+                    status("Waiting for program to be ready...")
+                    waiting_logged = True
                 retry_after = int(resp.headers.get("Retry-After", "3"))
                 time.sleep(retry_after)
                 elapsed += retry_after
@@ -137,6 +144,9 @@ class PAWClient:
                 if "not found" in detail.lower():
                     raise RuntimeError(f"Program {program_id} not found on server.")
                 if elapsed < max_wait - 3:
+                    if not waiting_logged:
+                        status("Waiting for program to be ready...")
+                        waiting_logged = True
                     time.sleep(3)
                     elapsed += 3
                     continue

{programasweights-0.2.7 → programasweights-0.2.8}/programasweights/runtime_llamacpp.py

@@ -51,9 +51,12 @@ class PawFunction:
             raise FileNotFoundError(f"No prompt_template.txt in {program_dir}")
         self._template = template_path.read_text()
 
+        from ._output import status
         interpreter = self._meta.get("interpreter", "Qwen/Qwen3-0.6B")
         base_model_path = cache.get_base_model_path(interpreter)
 
+        status("Loading interpreter...")
+
         if not verbose:
             import os as _os, sys as _sys
             _stderr_fd = _sys.stderr.fileno()
@@ -103,6 +106,7 @@ class PawFunction:
         self._n_prefix = len(self._prefix_tokens)
 
         self._load_or_eval_prefix()
+        status("Ready.")
 
     def _apply_adapter(self, scale: float):
         """Apply LoRA adapter, compatible with both old and new llama.cpp API."""

{programasweights-0.2.7 → programasweights-0.2.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "programasweights"
-version = "0.2.7"
+version = "0.2.8"
 description = "Compile natural language specifications into neural programs that run locally via llama.cpp."
 readme = "PYPI_README.md"
 requires-python = ">=3.9"

programasweights-0.2.7/docs/guide/writing-good-specs.md

@@ -1,36 +0,0 @@
-# Writing Good Specs
-
-A specification is a **self-contained** natural-language description of what the function should do. There is no separate “examples” API field: everything the compiler sees must live in that single spec string.
-
-## What to include
-
-Write clearly enough that someone unfamiliar with your project could implement the behavior from the text alone.
-
-### Good practices
-
-- **Output shape:** Say how results should look. Examples: “return a JSON list of strings”, “respond with only `positive` or `negative`”, “output valid CSV with a header row”.
-- **Edge cases:** State behavior for empty input, missing data, or failure modes. Example: “if no emails are found, return an empty list”.
-- **Ambiguity:** Explain how to handle unclear inputs (e.g., multiple interpretations, partial data).
-- **Length:** Keep the spec under **8000 characters**; the API enforces this limit.
-
-### Examples inside the spec
-
-You may embed example pairs directly in the spec text, for example:
-
-```text
-Examples:
-Input: hello
-Output: greeting
-```
-
-Use a consistent pattern so the compiler can treat them as demonstrations of desired behavior.
-
-## What to avoid
-
-- **Vague goals:** Phrases like “do something useful” or “be smart about it” do not constrain behavior.
-- **Excessive length:** Long essays dilute the task definition and hit limits unnecessarily.
-- **Contradictions:** Conflicting instructions produce unreliable pseudo-programs and weaker adapters.
-
-## How the compiler uses your spec
-
-The compiler **generates a pseudo-program** from your spec. That artifact often **rephrases and expands** the spec with structured examples. You can inspect what was produced in the playground under **“View compiled program internals”** to verify alignment with your intent.