opencode-llmstack 0.7.3__tar.gz → 0.9.0__tar.gz

{opencode_llmstack-0.7.3 → opencode_llmstack-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: opencode-llmstack
-Version: 0.7.3
+Version: 0.9.0
 Summary: Multi-tier local LLM stack: llama-swap + FastAPI auto-router + opencode wiring.
 Author: llmstack
 License: MIT
@@ -542,8 +542,8 @@ All knobs are env vars; defaults are picked up by `llmstack start`.
 | `ROUTER_ULTRA_MODEL` | `code-ultra` | short-context top tier → here (gated on availability) |
 | `ROUTER_PLAN_MODEL` | `plan` | design/discussion verbs → here |
 | `ROUTER_UNCENSORED_MODEL` | `plan-uncensored` | `[nofilter]` triggers → here |
-| `ROUTER_HIGH_FIDELITY_CEILING` | `8000` | tokens; at or below this, route to top tier (ultra → smart fallback) |
-| `ROUTER_MID_FIDELITY_CEILING` | `32000` | tokens; at or below this, route to `code-smart`; beyond, step down to `code-fast` |
+| `ROUTER_HIGH_FIDELITY_CEILING` | `12000` | tokens; at or below this, route to top tier (ultra → smart fallback). Paired with `code-ultra.ctx_size = 24000` (2x). |
+| `ROUTER_MID_FIDELITY_CEILING` | `32000` | tokens; at or below this, route to `code-smart`; beyond, step down to `code-fast`. Paired with `code-smart.ctx_size = 64000` (2x). |
 | `ROUTER_MULTI_TURN` | `6` | turn count that floors the long-context rung at `code-smart` |
 | `ROUTER_HOST` / `ROUTER_PORT` | `127.0.0.1` / `10101` | listen address |
 | `LOG_LEVEL` | `info` | router log level |

{opencode_llmstack-0.7.3 → opencode_llmstack-0.9.0}/README.md

@@ -507,8 +507,8 @@ All knobs are env vars; defaults are picked up by `llmstack start`.
 | `ROUTER_ULTRA_MODEL` | `code-ultra` | short-context top tier → here (gated on availability) |
 | `ROUTER_PLAN_MODEL` | `plan` | design/discussion verbs → here |
 | `ROUTER_UNCENSORED_MODEL` | `plan-uncensored` | `[nofilter]` triggers → here |
-| `ROUTER_HIGH_FIDELITY_CEILING` | `8000` | tokens; at or below this, route to top tier (ultra → smart fallback) |
-| `ROUTER_MID_FIDELITY_CEILING` | `32000` | tokens; at or below this, route to `code-smart`; beyond, step down to `code-fast` |
+| `ROUTER_HIGH_FIDELITY_CEILING` | `12000` | tokens; at or below this, route to top tier (ultra → smart fallback). Paired with `code-ultra.ctx_size = 24000` (2x). |
+| `ROUTER_MID_FIDELITY_CEILING` | `32000` | tokens; at or below this, route to `code-smart`; beyond, step down to `code-fast`. Paired with `code-smart.ctx_size = 64000` (2x). |
 | `ROUTER_MULTI_TURN` | `6` | turn count that floors the long-context rung at `code-smart` |
 | `ROUTER_HOST` / `ROUTER_PORT` | `127.0.0.1` / `10101` | listen address |
 | `LOG_LEVEL` | `info` | router log level |

{opencode_llmstack-0.7.3 → opencode_llmstack-0.9.0}/llmstack/app.py

@@ -74,21 +74,35 @@ Routing decision tree (first match wins):
   2. Explicit "ultra" trigger (``[ultra]``, ``[opus]``,
      ``ultra:``, ``opus:``) AND ultra tier configured -> code-ultra
   3. PLAN signal words AND no code-block / agent verbs / tools
-     (design discussion, no implementation pending)   -> plan
+     AND estimated tokens <= ``[plan]`` tier's ctx_size
+     (pure design discussion that fits the planner's
+     window)                                          -> plan
+                                                         (if the planner's
+                                                          ctx_size is breached
+                                                          we fall through to
+                                                          the coding ladder
+                                                          rather than send a
+                                                          request that won't
+                                                          fit -- the coding
+                                                          tiers cover larger
+                                                          windows by design)
   4. Estimated input tokens <= HIGH_FIDELITY_CEILING
      ("reasonable context still being built")         -> code-ultra
                                                          (else code-smart)
   5. Estimated input tokens <= MID_FIDELITY_CEILING   -> code-smart
-  6. Otherwise (long context, top-tier becomes
-     expensive/slow, fast tier's 128k window is the
-     best fit and it's free)                          -> code-fast
+   6. Otherwise (long context, top-tier becomes
+      expensive/slow, fast tier's 128k window is the
+      best fit and it's free)                          -> code-fast
                                                          (floored at
                                                           code-smart when
-                                                          ``tools[]`` is set
-                                                          or n_turns >=
-                                                          MULTI_TURN_THRESHOLD,
-                                                          since 3B models
-                                                          tool-call unreliably)
+                                                          n_turns >=
+                                                          MULTI_TURN_THRESHOLD)
+
+The auto router's effective max context window is
+``[code-fast].ctx_size`` -- fast is the bottom of the step-down
+ladder, so any context that would overflow the tiers above lands on
+fast. Inputs longer than fast's window have no safe home and should
+be considered out of scope for ``model = auto``.
 
 Ultra-tier routing is gated on availability: rule (2) and the
 "high-fidelity" rung of (4) first check that the tier is loaded
@@ -137,14 +151,21 @@ UNCENSORED_MODEL = os.getenv("ROUTER_UNCENSORED_MODEL", "plan-uncensored")
 #   est <= MID_FIDELITY_CEILING   -> code-smart
 #   est >  MID_FIDELITY_CEILING   -> code-fast (or smart with tools/loop)
 #
+# Each ceiling is half of the corresponding tier's ``ctx_size`` in
+# models.ini -- the ceiling marks where the tier still has comfortable
+# headroom, and double the ceiling is where the router has already
+# stepped down to the next tier (so the upper tier never has to handle
+# inputs at its own limit).
+#
 # Defaults:
-#   HIGH 8000  - "reasonable context built": a couple of files loaded,
+#   HIGH 12000 - "reasonable context built": a couple of files loaded,
 #                instructions clear, top-tier still cheap+fast here.
-#   MID  32000 - half of code-smart's 65k window; past this, hosted
+#                Pairs with a 24k ctx_size on code-ultra.
+#   MID  32000 - half of code-smart's 64k window; past this, hosted
 #                top-tier latency/$cost balloons and code-smart starts
 #                getting cramped, while code-fast's 128k YaRN window
 #                still has comfortable headroom.
-HIGH_FIDELITY_CEILING = int(os.getenv("ROUTER_HIGH_FIDELITY_CEILING", "8000"))
+HIGH_FIDELITY_CEILING = int(os.getenv("ROUTER_HIGH_FIDELITY_CEILING", "12000"))
 MID_FIDELITY_CEILING = int(os.getenv("ROUTER_MID_FIDELITY_CEILING", "32000"))
 # Floor the long-context rung at code-smart whenever a tool-call
 # protocol is in play -- 3B models tool-call unreliably regardless of
@@ -332,25 +353,36 @@ def classify(body: dict[str, Any]) -> tuple[str, str]:
                     ULTRA_MODEL, AGENT_MODEL)
         return AGENT_MODEL, f"ultra-trigger->agent ({ULTRA_MODEL} unavailable)"
 
-    has_tools = bool(body.get("tools"))
     n_turns = len(messages) if messages else 0
     has_code_signal = (
         _matches(CODE_BLOCK, messages, prompt)
         or _matches(AGENT_SIGNALS, messages, prompt)
     )
 
+    est = _estimate_tokens(messages, prompt)
+
     # Plan track is orthogonal to the code fidelity ladder: ``plan`` is a
     # chat-tuned model meant for design / "should we" discussions. Only
     # take it when nothing about the request says "I'm about to write
-    # code" (no triple-backticks, no agent verbs, no tool calls).
+    # code" (no triple-backticks, no agent verbs). Tools are stripped
+    # from the request body before dispatch (see ``_handle_completion``),
+    # so their presence here does not block plan routing.
+    # Only route to plan if the input fits in the planner's ctx_size --
+    # past that we fall through to the coding ladder which has tiers
+    # (smart, fast) explicitly sized for larger contexts.
     if (
-        not has_tools
-        and not has_code_signal
+        not has_code_signal
         and _matches(PLAN_SIGNALS, messages, prompt)
     ):
-        return PLAN_MODEL, "plan-signal"
-
-    est = _estimate_tokens(messages, prompt)
+        plan_tier = TIER_BY_ALIAS.get(PLAN_MODEL)
+        plan_ctx = plan_tier.ctx_size if plan_tier else 0
+        if not plan_ctx or est <= plan_ctx:
+            return PLAN_MODEL, "plan-signal"
+        log.info(
+            "plan-signal but tokens~%d > %s.ctx_size %d; "
+            "falling through to coding ladder",
+            est, PLAN_MODEL, plan_ctx,
+        )
 
     # Rung 1: short context -- start at the top.
     if est <= HIGH_FIDELITY_CEILING:
@@ -365,12 +397,12 @@ def classify(body: dict[str, Any]) -> tuple[str, str]:
     if est <= MID_FIDELITY_CEILING:
         return AGENT_MODEL, f"mid-fidelity tokens~{est}<={MID_FIDELITY_CEILING}"
 
-    # Rung 3: long context -- step down to fast (128k YaRN, free,
-    # always-resident). Floor at smart when tools/agent loop is in
-    # play; the 3B coder doesn't tool-call reliably.
-    if has_tools or n_turns >= MULTI_TURN_THRESHOLD:
-        why = "tools" if has_tools else f"turns={n_turns}"
-        return AGENT_MODEL, f"long-context tokens~{est}>{MID_FIDELITY_CEILING} ({why} floor)"
+    # Rung 3: long context -- step down to fast. Floor at smart only
+    # when the multi-turn threshold is hit; tools alone no longer
+    # prevent the step-down (plan tiers strip tools before dispatch,
+    # and code-fast is a hosted model that tool-calls reliably).
+    if n_turns >= MULTI_TURN_THRESHOLD:
+        return AGENT_MODEL, f"long-context tokens~{est}>{MID_FIDELITY_CEILING} (turns={n_turns} floor)"
     return FAST_MODEL, f"long-context tokens~{est}>{MID_FIDELITY_CEILING}"
 
 
@@ -590,6 +622,11 @@ async def _handle_completion(req: Request, path: str) -> Response:
         mutated = True
 
     chosen_name = body.get("model")
+    if chosen_name in {PLAN_MODEL, UNCENSORED_MODEL} and body.get("tools"):
+        log.info("plan tier %s: stripping tools from request", chosen_name)
+        body.pop("tools")
+        body.pop("tool_choice", None)
+        mutated = True
     tier = _resolve_tier(chosen_name)
     if tier is not None and _inject_sampler(body, tier):
         mutated = True

{opencode_llmstack-0.7.3 → opencode_llmstack-0.9.0}/llmstack/generators/opencode.py

@@ -172,9 +172,22 @@ def build_config(
 
     tier_sections = [s for s in cfg.sections() if s != "ROUTING"]
 
-    # `auto` context = MIN across all tiers so opencode never packs a prompt
-    # that overflows the tier the router actually picks.
-    auto_ctx = min(
+    # `auto` context = the fast tier's ctx_size. The router runs a
+    # step-DOWN ladder (ultra -> smart -> fast as context grows), so
+    # the largest window in the ladder is fast's, and that's the
+    # effective ceiling for `model = auto` -- anything bigger has no
+    # tier to land on. Using `min(...)` here would clip opencode to
+    # the smallest tier's window even though the router would never
+    # actually send a long prompt to that tier.
+    fast_ctx = next(
+        (
+            _int(cfg[s].get("ctx_size", ""), 0)
+            for s in tier_sections
+            if (cfg[s].get("role") or "").strip() == "fast"
+        ),
+        0,
+    )
+    auto_ctx = fast_ctx or max(
         (_int(cfg[s].get("ctx_size", ""), 0) for s in tier_sections),
         default=8192,
     ) or 8192

{opencode_llmstack-0.7.3 → opencode_llmstack-0.9.0}/llmstack/models.ini

@@ -67,7 +67,7 @@ tier         = code
 role         = fast
 hf_repo      = bartowski/Qwen2.5-Coder-3B-Instruct-GGUF
 hf_file      = Qwen2.5-Coder-3B-Instruct-Q5_K_M.gguf
-ctx_size     = 131072       ; native 32k extended via YaRN (factor 4)
+ctx_size     = 131072       ; native 32k extended via YaRN (factor 4); also defines the auto router's effective max -- contexts longer than this have nowhere to land, since fast is the bottom of the step-down ladder
 rope_scaling = yarn (scale=4, orig_ctx=32768)
 size_gb      = 2.5
 quant        = Q5_K_M
@@ -97,7 +97,7 @@ role         = agent
 hf_repo      = unsloth/Qwen3-Coder-Next-GGUF
 hf_file      = Qwen3-Coder-Next-Q4_K_M.gguf
 hf_file_next = Qwen3-Coder-Next-UD-Q4_K_XL.gguf
-ctx_size     = 65536
+ctx_size     = 64000        ; 2x mid_fidelity_ceiling -- router steps down to code-fast past this, so a larger window would never get used
 size_gb      = 45
 size_gb_next = 50
 quant        = Q4_K_M
@@ -117,7 +117,7 @@ description  = Qwen3-Coder-Next 80B-A3B MoE - heavy coder for agent loops
 ; aws_model_id = eu.anthropic.claude-sonnet-4-6
 ; aws_region   = eu-central-1
 ; aws_profile  = bedrock-prod
-; ctx_size     = 200000
+; ctx_size     = 64000        ; 2x mid_fidelity_ceiling -- intentionally well below Sonnet 4.6's 200k native window; router steps down to code-fast past this rather than paying Sonnet's long-context $cost/latency
 ; sampler      = temp=0.5     ; Sonnet 4.6 accepts ONE of temp / top_p; pick `temp` for agent work
 ; description  = Claude Sonnet 4.6 on Bedrock - heavy coder for agent loops
 
@@ -137,7 +137,7 @@ description  = Qwen3-Coder-Next 80B-A3B MoE - heavy coder for agent loops
 ; aws_model_id = global.anthropic.claude-opus-4-7   ; global.* cross-region inference profile
 ; aws_region   = eu-central-1                       ; API anchor region; global.* auto-routes inference cross-region (set EU as the anchor for residency)
 ; aws_profile  = bedrock-prod                       ; conventional profile name; configure once with `aws configure --profile bedrock-prod` (or change to your own and run `llmstack install`)
-; ctx_size     = 200000
+; ctx_size     = 24000        ; 2x high_fidelity_ceiling -- intentionally well below Opus 4.7's 200k native window; we only invoke ultra for short prompts, where it's still cheap+fast, then step down past this
 ; ; NB: no `sampler =` line. Claude Opus 4.7 explicitly rejects all
 ; ;     sampler params (temperature, top_p, top_k) -- per the Bedrock
 ; ;     model card, "the recommended migration path is to omit these
@@ -260,7 +260,11 @@ description  = Mistral-Small 3.2 24B Heretic - no-filter planning
 ;   2. "[ultra]" / "[opus]" / "ultra:" trigger AND code-ultra
 ;      tier configured                                            -> code-ultra
 ;   3. PLAN signal words AND no code-block / agent verbs / tools
-;      (pure design discussion)                                   -> plan
+;      AND tokens <= [plan].ctx_size (pure design discussion that
+;      still fits the planner's window)                           -> plan
+;      ...if the plan tier's ctx_size is breached, the request
+;      falls through to the coding ladder below rather than being
+;      sent to a planner whose window can't hold the input.
 ;   4. tokens <= high_fidelity_ceiling AND code-ultra configured  -> code-ultra
 ;      tokens <= high_fidelity_ceiling AND no code-ultra          -> code-smart
 ;   5. tokens <= mid_fidelity_ceiling                             -> code-smart
@@ -268,6 +272,24 @@ description  = Mistral-Small 3.2 24B Heretic - no-filter planning
 ;        - if tools[] OR turns >= multi_turn (3B tool-calls badly) -> code-smart
 ;        - else                                                    -> code-fast
 ;
+; AUTO ROUTER MAX CONTEXT = [code-fast].ctx_size. The fast tier sits at
+; the bottom of the step-down ladder, so any context too big for the
+; tiers above lands on fast. Anything beyond fast's window has no
+; safe home and should be considered out of scope for `model = auto`
+; -- callers with such payloads should pass an explicit model name
+; instead of relying on the router.
+;
+; FIDELITY-CEILING <-> CTX-SIZE COUPLING. Each "fidelity" rung's
+; ceiling is half of the corresponding tier's ctx_size:
+;   high_fidelity_ceiling x 2 == [code-ultra].ctx_size
+;   mid_fidelity_ceiling  x 2 == [code-smart].ctx_size
+; This is deliberate: the ceiling marks where the tier still has
+; comfortable headroom; double the ceiling is where we'd be packing
+; the tier to its limit (and where the router has already stepped
+; down to the next tier). If you bump a ceiling, bump the matching
+; ctx_size in the tier section too -- otherwise the router will
+; route requests up to a tier whose window can't hold them.
+;
 ; The "high-fidelity" rung is gated on availability: when the
 ; [code-ultra] section is absent (or fails to load), rules (2) and (4)
 ; silently fall back to code-smart instead of routing to a tier that
@@ -276,8 +298,8 @@ description  = Mistral-Small 3.2 24B Heretic - no-filter planning
 ; ROUTER_HIGH_FIDELITY_CEILING / ROUTER_MID_FIDELITY_CEILING /
 ; ROUTER_MULTI_TURN.
 ;
-high_fidelity_ceiling = 8000     ; tokens; below this, top-tier model is still cheap+fast
-mid_fidelity_ceiling  = 32000    ; tokens; smart's sweet spot up to here, then step down to fast
+high_fidelity_ceiling = 12000    ; tokens; below this, top-tier model is still cheap+fast (and ultra ctx_size = 2 * this)
+mid_fidelity_ceiling  = 32000    ; tokens; smart's sweet spot up to here, then step down to fast (smart ctx_size = 2 * this)
 multi_turn            = 6        ; turn count that floors the long-context rung at code-smart
 agent_signal_words    = implement, fix bug, write a function, refactor, edit, patch, debug, run tests, build it
 plan_signal_words     = design, architect, approach, trade-off, should we, how would you, explain why, think through, compare options, brainstorm, root cause

{opencode_llmstack-0.7.3 → opencode_llmstack-0.9.0}/opencode_llmstack.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: opencode-llmstack
-Version: 0.7.3
+Version: 0.9.0
 Summary: Multi-tier local LLM stack: llama-swap + FastAPI auto-router + opencode wiring.
 Author: llmstack
 License: MIT
@@ -542,8 +542,8 @@ All knobs are env vars; defaults are picked up by `llmstack start`.
 | `ROUTER_ULTRA_MODEL` | `code-ultra` | short-context top tier → here (gated on availability) |
 | `ROUTER_PLAN_MODEL` | `plan` | design/discussion verbs → here |
 | `ROUTER_UNCENSORED_MODEL` | `plan-uncensored` | `[nofilter]` triggers → here |
-| `ROUTER_HIGH_FIDELITY_CEILING` | `8000` | tokens; at or below this, route to top tier (ultra → smart fallback) |
-| `ROUTER_MID_FIDELITY_CEILING` | `32000` | tokens; at or below this, route to `code-smart`; beyond, step down to `code-fast` |
+| `ROUTER_HIGH_FIDELITY_CEILING` | `12000` | tokens; at or below this, route to top tier (ultra → smart fallback). Paired with `code-ultra.ctx_size = 24000` (2x). |
+| `ROUTER_MID_FIDELITY_CEILING` | `32000` | tokens; at or below this, route to `code-smart`; beyond, step down to `code-fast`. Paired with `code-smart.ctx_size = 64000` (2x). |
 | `ROUTER_MULTI_TURN` | `6` | turn count that floors the long-context rung at `code-smart` |
 | `ROUTER_HOST` / `ROUTER_PORT` | `127.0.0.1` / `10101` | listen address |
 | `LOG_LEVEL` | `info` | router log level |

{opencode_llmstack-0.7.3 → opencode_llmstack-0.9.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "opencode-llmstack"
-version = "0.7.3"
+version = "0.9.0"
 description = "Multi-tier local LLM stack: llama-swap + FastAPI auto-router + opencode wiring."
 readme = "README.md"
 requires-python = ">=3.11"