@ssheleg/agent-stack 0.3.0

package/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md

@@ -0,0 +1,255 @@
+# Reselling LLM access — metering, wallets and guardrails
+
+When your product gives users LLM access and bills for it, you are running a
+proxy with a wallet behind it. The failure modes are not model failures: they
+are **double-credited transfers**, **spend you discovered after it happened**,
+and **a runaway loop that emptied a balance overnight**. This reference is the
+provider-neutral shape of that problem, drawn from a production system built on
+OpenRouter's Management API — the API calls are named where they are concrete,
+the patterns hold for any upstream that issues per-tenant keys with limits.
+
+## Contents
+
+- [The tiered wallet](#the-tiered-wallet)
+- [Two-phase commit across a DB and an external API](#two-phase-commit-across-a-db-and-an-external-api)
+- [Serializing concurrent transfers](#serializing-concurrent-transfers)
+- [Optimistic concurrency for reclaims](#optimistic-concurrency-for-reclaims)
+- [Discovering spend you do not control](#discovering-spend-you-do-not-control)
+- [Guardrails: budgets, loops, auto-pause](#guardrails-budgets-loops-auto-pause)
+- [Key lifecycle and healing](#key-lifecycle-and-healing)
+- [The refund waterfall](#the-refund-waterfall)
+- [Model routing and fallbacks](#model-routing-and-fallbacks)
+
+---
+
+## The tiered wallet
+
+Money does not move in one hop. Model it as tiers, and be explicit about **which
+boundary applies the markup** — this is the single most common source of
+accounting drift.
+
+| Tier | Holds | Denominated in |
+|---|---|---|
+| Account wallet | what the user paid you | user-facing USD |
+| Tenant reserve | allocated to one bot / workspace / project | user-facing USD, 1:1 with the account |
+| Upstream key limit | what the provider will actually let them spend | provider USD, after markup |
+
+```
+markup = 0.30                          // your cut, from config, never hardcoded
+
+toKeyAmount(userUsd)  = userUsd * (1 - markup)   // $50 user → $35 on the key
+toUserAmount(keyUsd)  = keyUsd  / (1 - markup)   // $35 key  → $50 shown back
+```
+
+**Apply markup at exactly one boundary** — reserve → key — and keep account ↔
+reserve at 1:1. Two boundaries applying a cut is how a balance silently shrinks
+every time a user moves money around without spending anything.
+
+Keep the same two functions on the client. A UI that recomputes the conversion
+with its own copy of the constant will disagree with the server the first time
+the constant changes.
+
+**Thresholds worth naming rather than inlining:**
+
+| Constant | Typical | Purpose |
+|---|---|---|
+| `LOW_BALANCE_THRESHOLD` | $10 | trigger reserve → key transfer + notify |
+| `CRITICAL_BALANCE_THRESHOLD` | $1 | permit auto-topup from the account wallet |
+| `AUTO_TOPUP_AMOUNT` | $15 | ceiling pulled per auto-topup |
+
+---
+
+## Two-phase commit across a DB and an external API
+
+You have a database you can roll back and an HTTP API you cannot. Order matters,
+and so does what you do when step 2 fails.
+
+**DB first, API second, compensate on failure:**
+
+1. Acquire the lock (below).
+2. Read fresh balances **inside** the transaction — not before it.
+3. Compute the transfer and apply the markup once.
+4. Zero the source tier, increment the destination, write an audit row.
+5. Commit.
+6. Call the provider to raise the key limit.
+7. **On API failure: a compensating transaction restores every DB value and
+   writes a `compensation` audit row.**
+
+The alternative — API first, DB second — leaves money on the key that your
+ledger does not know about, and no amount of retrying finds it again. The
+compensating transaction is not optional politeness; it is the only thing that
+makes step 6 recoverable.
+
+Log both the intent and the compensation. An audit trail that records only
+successes cannot answer "where did the $35 go" six weeks later.
+
+---
+
+## Serializing concurrent transfers
+
+Two requests topping up the same tenant at the same time will both read the same
+starting balance and both write their own total. Take a lock keyed on the tenant,
+in the same transaction:
+
+```sql
+SELECT pg_advisory_xact_lock(hashtext(tenant_id || '_key'));
+```
+
+Transaction-scoped (`_xact_`) so it releases on commit **or** rollback — a
+session-scoped lock survives a failed transaction and deadlocks the retry.
+
+Every operation that moves this tenant's money takes the same lock: top-up,
+reclaim, pull-back, refund. A single unlocked path makes the other five
+pointless.
+
+---
+
+## Optimistic concurrency for reclaims
+
+Locking protects concurrent writers in your database. It does not protect you
+from a change the **provider** made while your transaction was open — a spend
+that landed, a limit an operator edited in their dashboard.
+
+For any operation that reads a provider value, acts, and writes back: snapshot
+the value before the external call and re-read it after. If it moved, abort
+rather than reconcile.
+
+```
+before = db.keyLimit
+info   = await provider.getKey(hash)
+if (info.limit !== before) throw new ConcurrentModification()   // do not guess
+```
+
+Aborting costs a retry. Guessing double-credits.
+
+---
+
+## Discovering spend you do not control
+
+The provider deducts from the key as calls happen. Nothing notifies you. You
+discover spend by **polling a cumulative counter and taking the delta**:
+
+```
+delta = currentUsage - lastRecordedUsage
+```
+
+Three cases, and only the first is obvious:
+
+- `lastRecordedUsage == 0 && currentUsage > 0` → **seed the baseline, record
+  nothing.** Recording it charges the tenant for everything spent before you
+  started watching.
+- `currentUsage > lastRecordedUsage` → record `delta`, then immediately enforce
+  budgets (below).
+- `currentUsage < lastRecordedUsage` → the key was recreated. **Resync the
+  baseline, record nothing.** A negative delta treated as spend credits money
+  that was never returned.
+
+Sync your stored limit from the provider's authoritative value on the same pass —
+under the lock, with a re-read, so the sync does not clobber a transfer that
+landed mid-poll.
+
+---
+
+## Guardrails: budgets, loops, auto-pause
+
+Three independent mechanisms, one shared action. Keep them separate in
+configuration and unified in effect.
+
+**Budget limits** — per-tenant daily and monthly caps with their own counters and
+reset timestamps. `recordSpend()` increments and calls `enforceBudgetLimit()`
+in the same breath; enforcement that runs on a schedule rather than on the write
+is enforcement that arrives after the money is gone.
+
+**Loop detection** — spend *velocity* over a rolling window against a
+configurable multiplier of the tenant's normal rate. This is what catches an
+agent that started calling itself; a daily cap will also catch it, tomorrow.
+
+**Auto-pause** — an absolute per-tenant threshold, plus a user-level aggregate
+across all their tenants. The aggregate exists because ten tenants each just
+under their limit is a bill nobody approved.
+
+All three converge on one function:
+
+```
+pauseBot(tenantId, cause, reason)
+  1. set the pause timestamp for THAT cause (budgetPausedAt / loopPausedAt / …)
+  2. disable the upstream key
+  3. write an audit row naming the cause
+  4. notify the user
+```
+
+Separate timestamps per cause, because resuming must know what paused it: a
+daily budget reset should not un-pause a tenant that a loop detector stopped.
+Reset jobs zero their own counters, clear **their own** timestamp, and re-enable
+only if no other pause is still set.
+
+---
+
+## Key lifecycle and healing
+
+Per-tenant keys go missing — deleted in a dashboard, orphaned by a failed
+provision, expired. Treat the key as cache, not truth.
+
+**Before every deploy**, validate and heal:
+
+1. Check the key against the provider's own auth endpoint (the raw key, not the
+   management hash).
+2. Dead → create a fresh one with the same limit, update your stored key, hash
+   and limit.
+3. If the tenant was running, trigger a redeploy so the new key takes effect.
+
+The polling job does the same when the Management API returns 404 for a hash it
+holds.
+
+**Management keys are not inference keys.** They do key CRUD and nothing else;
+a management key sent to a completions endpoint fails in a way that reads like
+an auth bug for an hour.
+
+**The full key is returned exactly once, at creation.** Store it then or issue a
+new one. Every provider does this and every integration learns it the same way.
+
+Name keys after the tenant (`tenant:{id}:{name}`) — the provider dashboard is
+where you will be debugging at 2am, and `sk-or-v1-…` tells you nothing.
+
+Wire the lifecycle explicitly, one row per event: enabled → create + fund;
+disabled → disable; subscription cancelled → disable + reclaim; deleted →
+delete + reclaim; balance depleted → disable + notify; budget reset → re-enable.
+A table like that in your own docs is what stops the eleventh event from being
+handled three different ways.
+
+---
+
+## The refund waterfall
+
+A payment refund has to come out of somewhere, and the money has usually moved.
+Pull in a fixed priority, most-liquid first:
+
+1. tenant reserve — purchased pool
+2. tenant reserve — subscription pool
+3. remaining balance on the upstream key (markup-adjusted, via the provider API)
+4. account wallet
+
+Any unrecoverable remainder is **logged for manual review**, not silently
+forgiven and not left to make a balance negative. A tenant who spent the money
+already is a business decision, not an arithmetic one.
+
+---
+
+## Model routing and fallbacks
+
+Map your public model names to provider ids in **one** function, and give every
+provider a default and a fallback:
+
+```
+toUpstreamModel(provider, model)   // "gpt-4o" → "openai/gpt-4o"
+getDefaultModel(provider)          // when the caller names none
+getFallbackModels(provider)        // ordered, tried on 5xx / overload
+```
+
+Three levels of model selection, in precedence order: the caller's explicit
+choice → the tenant's configured default → the application default. Resolve them
+in that order in one place, and log which level won — "why did it use that
+model" is otherwise unanswerable.
+
+See `patterns.md` for the retry, health-check and error-hierarchy patterns these
+routing calls sit inside.

package/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md

@@ -0,0 +1,365 @@
+# Agent Orchestrator — Reference Guide
+
+Extended patterns, data models, and implementation details.
+
+## Contents
+
+- [Data Models](#data-models)
+- [Pipeline Data Models](#pipeline-data-models)
+- [Validation Loop (SQL Execution)](#validation-loop-sql-execution)
+- [Context Window Sizes](#context-window-sizes)
+- [Learning Extraction Heuristics](#learning-extraction-heuristics)
+- [Confidence Management](#confidence-management)
+- [Fuzzy Deduplication Pattern](#fuzzy-deduplication-pattern)
+- [Conflict Resolution Pattern](#conflict-resolution-pattern)
+- [Cross-Resource Learning Transfer](#cross-resource-learning-transfer)
+- [Suggestion Engine (No LLM Cost)](#suggestion-engine-no-llm-cost)
+
+
+## Data Models
+
+### Message Protocol
+
+```python
+@dataclass
+class Message:
+    role: str          # system | user | assistant | tool
+    content: str
+    tool_call_id: str | None = None   # for role="tool" responses
+    name: str | None = None           # tool name
+    tool_calls: list[ToolCall] | None = None  # for role="assistant" with tools
+
+@dataclass
+class ToolCall:
+    id: str                    # unique ID per call
+    name: str                  # tool function name
+    arguments: dict[str, Any]  # parsed JSON arguments
+
+@dataclass
+class Tool:
+    name: str
+    description: str
+    parameters: list[ToolParameter]
+
+@dataclass
+class ToolParameter:
+    name: str
+    type: str          # "string", "number", "boolean"
+    description: str
+    required: bool = True
+    enum: list[str] | None = None
+
+@dataclass
+class LLMResponse:
+    content: str = ""
+    tool_calls: list[ToolCall] = field(default_factory=list)
+    usage: dict[str, int] = field(default_factory=dict)
+    model: str = ""
+    provider: str = ""
+    finish_reason: str = ""
+```
+
+### Agent Result Protocol
+
+```python
+@dataclass
+class AgentResult:
+    status: str = "success"    # success | error | no_result
+    token_usage: dict[str, int] = field(default_factory=dict)
+    error: str | None = None
+    metadata: dict[str, Any] = field(default_factory=dict)
+
+# Typed subclass example:
+@dataclass
+class SQLAgentResult(AgentResult):
+    query: str | None = None
+    query_explanation: str | None = None
+    results: QueryResult | None = None
+    attempts: list[QueryAttempt] = field(default_factory=list)
+    insights: list[dict] = field(default_factory=list)
+```
+
+### Orchestrator Response
+
+```python
+@dataclass
+class AgentResponse:
+    answer: str = ""
+    query: str | None = None
+    query_explanation: str | None = None
+    results: QueryResult | None = None
+    viz_type: str = "text"
+    viz_config: dict = field(default_factory=dict)
+    knowledge_sources: list[RAGSource] = field(default_factory=list)
+    error: str | None = None
+    workflow_id: str | None = None
+    token_usage: dict = field(default_factory=dict)
+    llm_provider: str = ""
+    llm_model: str = ""
+    response_type: str = "text"       # text|sql_result|knowledge|error
+                                      # |clarification_request|stage_checkpoint
+                                      # |stage_failed|pipeline_complete
+    tool_call_log: list[dict] = field(default_factory=list)
+    suggested_followups: list[str] = field(default_factory=list)
+    insights: list[dict] = field(default_factory=list)
+    context_usage_pct: int = 0
+    staleness_warning: str | None = None
+```
+
+---
+
+## Pipeline Data Models
+
+```python
+@dataclass
+class PlanStage:
+    stage_id: str              # unique ID
+    description: str           # human-readable
+    tool: str                  # "query_database" | "search_codebase" | "analyze_results"
+    question: str              # sub-question for the stage
+    depends_on: list[str]      # stage IDs this depends on
+    checkpoint: bool = False   # pause for user review after this stage
+    validation: StageValidation | None = None
+
+@dataclass
+class StageValidation:
+    min_rows: int | None = None
+    max_rows: int | None = None
+    required_columns: list[str] = field(default_factory=list)
+
+@dataclass
+class ExecutionPlan:
+    question: str
+    stages: list[PlanStage]
+    def to_json(self) -> str: ...
+    @classmethod
+    def from_json(cls, raw: str) -> ExecutionPlan: ...
+
+@dataclass
+class StageResult:
+    stage_id: str
+    status: str = "success"    # success | error | skipped
+    query: str | None = None
+    query_result: QueryResult | None = None
+    answer: str | None = None
+    error: str | None = None
+```
+
+---
+
+## Validation Loop (SQL Execution)
+
+The full SQL execution cycle with pre/post validation and repair:
+
+```
+for attempt in range(1, max_retries + 1):
+    1. PRE-VALIDATE
+       └─ Schema check: tables exist? columns exist? types compatible?
+       └─ If fails → repair query via LLM → continue
+
+    2. SAFETY CHECK
+       └─ Read-only enforcement, DML blocking
+       └─ If unsafe → fail immediately (no retry)
+
+    3. EXPLAIN DRY-RUN (optional)
+       └─ Run EXPLAIN, check for full table scans on huge tables
+       └─ If problematic → repair query → continue
+
+    4. EXECUTE
+       └─ Run query against user's database
+       └─ Classify errors: table_not_found, column_not_found, syntax_error,
+          timeout, permission_denied, connection_error
+       └─ If retryable error → enrich context + repair → continue
+       └─ If fatal → fail
+
+    5. POST-VALIDATE
+       └─ Sanity checks on results
+       └─ If fails → repair → continue
+
+    6. SUCCESS → extract learnings from attempt history
+```
+
+---
+
+## Context Window Sizes
+
+```python
+MODEL_CONTEXT_WINDOWS = {
+    "gpt-4o": 128_000,
+    "gpt-4o-mini": 128_000,
+    "gpt-4-turbo": 128_000,
+    "gpt-4": 8_192,
+    "gpt-3.5-turbo": 16_385,
+    "claude-sonnet-4-20250514": 200_000,
+    "claude-3-5-sonnet-20241022": 200_000,
+    "claude-3-haiku-20240307": 200_000,
+    "claude-3-opus-20240229": 200_000,
+}
+DEFAULT_CONTEXT_WINDOW = 16_000
+
+# Token estimation: tiktoken for OpenAI models, ~4 chars/token fallback
+def estimate_tokens(text):
+    try:
+        import tiktoken
+        return len(tiktoken.get_encoding("cl100k_base").encode(text))
+    except Exception:
+        return max(1, len(text) // 4)
+```
+
+---
+
+## Learning Extraction Heuristics
+
+```python
+class LearningAnalyzer:
+    def _detect_table_preference(attempts, question):
+        # For consecutive attempt pairs where attempt[i] failed and attempt[i+1] succeeded:
+        # Compare tables used. If old_tables - new_tables and new_tables - old_tables:
+        # → "Use `new_table` instead of `old_table` for {topic}"
+
+    def _detect_column_correction(attempts):
+        # If attempt[i] has column_not_found error with column name X
+        # and attempt[i+1] no longer uses X:
+        # → "Column `X` doesn't exist on `table`. Use `Y` instead."
+
+    def _detect_format_discovery(attempts):
+        # If fixed query adds "/ 100" or "/ 1000" that wasn't in failed query:
+        # → "Column `amount` stores cents. Divide by 100."
+
+    def _detect_schema_gotcha(attempts):
+        # If fixed query adds "deleted_at IS NULL" or schema prefix:
+        # → "Table uses soft-delete. Always filter active records."
+
+    def _detect_performance_hint(attempts):
+        # If timeout error fixed by adding LIMIT or date filter:
+        # → "Table can timeout. Always add LIMIT and date filter."
+```
+
+---
+
+## Confidence Management
+
+```
+LEARNING CONFIDENCE:
+  Initial:     0.6
+  Confirmed:   +0.1 (cap 1.0)
+  Applied:     tracked (times_applied counter)
+  Contradicted: -0.3
+  Stale (30d): -0.02/month
+  Deactivated: below 0.2
+
+SESSION NOTE CONFIDENCE:
+  Initial:     0.7
+  Confirmed:   +0.1 (cap 1.0)
+  Verified:    +0.15 (exempt from decay)
+  Stale (60d): -0.1 per cycle
+  Floor:       0.1
+
+INSIGHT CONFIDENCE:
+  Initial:     0.5
+  Resurfaced:  +0.05
+  Confirmed:   +0.15
+  Dismissed:   -0.2
+  Stale (30d): -0.05 per cycle
+  Expired:     below 0.15
+```
+
+---
+
+## Fuzzy Deduplication Pattern
+
+Used across all memory layers:
+
+```python
+from difflib import SequenceMatcher
+
+THRESHOLD = 0.75  # learnings/notes; 0.80 for insights
+
+async def find_similar(session, connection_id, category, subject, text):
+    candidates = await load_existing(session, connection_id, category, subject)
+    text_lower = text.strip().lower()
+    best_match, best_ratio = None, 0.0
+    for c in candidates:
+        ratio = SequenceMatcher(None, c.text.strip().lower(), text_lower).ratio()
+        if ratio >= THRESHOLD and ratio > best_ratio:
+            best_match, best_ratio = c, ratio
+    return best_match
+
+# On match: bump confidence +0.1, keep longer text, set is_active=True
+# On no match: create new entry
+```
+
+---
+
+## Conflict Resolution Pattern
+
+Detect when new learning contradicts existing ones:
+
+```python
+CONFLICT_INDICATORS = {"use", "prefer", "always", "never", "should",
+                       "instead", "not", "avoid", "correct", "wrong"}
+
+def resolve_conflicts(existing_learnings, new_lesson, new_confidence):
+    new_keywords = {w for w in new_lesson.lower().split() if w in CONFLICT_INDICATORS}
+    for old in existing_learnings:
+        old_keywords = {w for w in old.lesson.lower().split() if w in CONFLICT_INDICATORS}
+        shared = new_keywords & old_keywords
+        if not shared: continue
+
+        has_negation_flip = (
+            ("not" in new_keywords) != ("not" in old_keywords) or
+            ("never" in new_keywords) != ("never" in old_keywords) or
+            ("avoid" in new_keywords) != ("avoid" in old_keywords))
+
+        if has_negation_flip and old.confidence <= new_confidence:
+            old.is_active = False  # superseded
+```
+
+---
+
+## Cross-Resource Learning Transfer
+
+```python
+async def get_cross_connection_learnings(session, connection_id, exclude_hashes):
+    project_id = get_project_for_connection(connection_id)
+    sibling_ids = get_sibling_connections(project_id, exclude=connection_id)
+    # Only transfer schema_gotcha and performance_hint (universally applicable)
+    transferable = await load_learnings(sibling_ids, categories={"schema_gotcha", "performance_hint"},
+                                        min_confidence=0.6)
+    return [f"- [from sibling] {l.lesson} [{int(l.confidence*100)}%]"
+            for l in transferable if l.lesson_hash not in exclude_hashes][:8]
+
+async def promote_global_patterns(session, connection_id):
+    # Find learnings appearing on 2+ independent connections
+    patterns = await query(
+        SELECT lesson_hash, MAX(lesson), MAX(confidence), COUNT(DISTINCT connection_id)
+        WHERE is_active AND confidence >= 0.7
+        GROUP BY lesson_hash HAVING COUNT(DISTINCT connection_id) >= 2)
+    # Exclude already-known patterns, format as prompt lines
+    return [f"- [global, seen on {p.conn_count} DBs] {p.lesson}" for p in patterns][:5]
+```
+
+---
+
+## Suggestion Engine (No LLM Cost)
+
+Template-based follow-up suggestions:
+
+```python
+FOLLOWUP_TEMPLATES = [
+    "Show this as a pie chart",
+    "Break this down by month",
+    "Compare with the previous period",
+    "Show only the top 5 results",
+    "What is the trend over time?",
+]
+
+def generate_followups(query, columns, row_count) -> list[str]:
+    pool = list(FOLLOWUP_TEMPLATES)
+    if has_aggregate_keywords(query):
+        pool.extend(["Show percentage breakdown", "Average instead of count?"])
+    if row_count > 1 and len(columns) >= 2:
+        pool.append(f"Sort by {columns[-1]} descending")
+    random.shuffle(pool)
+    return pool[:3]
+```