blq-cli 1.2.2__tar.gz → 2.0.0__tar.gz

{blq_cli-1.2.2 → blq_cli-2.0.0}/CHANGELOG.md

@@ -1,5 +1,133 @@
 # Changelog
 
+## v2.0.0
+
+### Fixed (breaking) — a run and the read of it now agree on the workspace (#55)
+
+`config(active_root=...)` steered only the read tools. `_run_impl` and
+`_exec_impl` passed no `cwd` to their child process, so `blq exec`/`blq run`
+walked up from the *server's* directory to find `.bird/` and recorded the run
+there, whatever root the caller had named. `_get_storage()` honoured
+`active_root` and read a different workspace. A run went into one store and was
+looked for in another.
+
+Two more places resolved from the process cwd, and both matter once a call can
+name its own workspace:
+
+* `BlqConfig.find()` supplied the registered-command registry, so a prefix
+  match on `exec(command=..., cwd=X)` would reroute through whatever the
+  *server's* workspace had registered under that prefix and run that
+  definition instead;
+* the same lookup supplied `mcp.disabled_tools`, so a call aimed at project X
+  ran under a policy file belonging to another project;
+* `_clean_impl` walked up from `Path.cwd()` with its own loop. `clean` deletes
+  directories, so a call aimed at one project could clear another's `.bird/`.
+
+Every workspace tool now takes `cwd=`, and the resolution order is the same
+everywhere with no third step:
+
+1. the `cwd=` argument,
+2. the session's `active_root`,
+3. otherwise the call is refused.
+
+**This breaks bare calls**, which is the point: a wrong-workspace answer
+becomes a loud failure instead of a confident one. Pass `cwd=`, or set the root
+once with `config(set={"active_root": "..."})`, or seed `BLQ_ACTIVE_ROOT` in the
+launch environment. The retritis plugin seeds it from `${PWD}`.
+
+The refusal **raises** rather than returning an error dict. Every tool declares
+a TypedDict return and fastmcp coerces the value to that schema, so an error
+dict arrived as `{"sources": None}` with its own keys dropped — a refusal
+indistinguishable from an empty answer. Raising matches `_check_tool_enabled`,
+which is already how this module refuses a call.
+
+Ordering inside each tool is server-level policy, then argument validation,
+then the root, then the workspace's own policy. The cached server-level set
+(CLI flags, env, `safe_mode`) is still checked first, so a caller cannot point
+`cwd=` at a permissive project to escape it.
+
+`root` is a **required keyword-only** parameter on the 29 helpers that reach a
+store or a config, so a missed call site is a `TypeError` and a mypy error
+rather than a silent read of the server's workspace. mypy found all 19
+`_get_storage()` sites that way. The MCP resources and prompts pass
+`root=None` explicitly: a `blq://` URI cannot carry a workspace, so they keep
+the cwd fallback on purpose.
+
+Both sides of the store lookup now walk up through `BlqConfig`, so a `cwd=`
+that names a subdirectory resolves to the same `.bird/` the child process
+finds.
+
+Verified: 12 new tests, measured red before the change. The decisive one is a
+round trip — a run made through workspace X is read back through X, *and* the
+server's own workspace is asserted to have gained no run, because a `pwd`
+check passes while the two halves still disagree. Suite: 1373 passed before,
+1385 after, with a failure set identical to the baseline's 11. mypy unchanged
+at its 14 pre-existing errors; `ruff check` on `serve.py` clean.
+
+### Fixed — 10 MCP tools were deleting keys they returned
+
+fastmcp coerces a tool's value to its declared `outputSchema`, so a key the
+impl returns but the TypedDict omits is **removed before the caller sees it**.
+The TypedDicts added in #47 were incomplete, which made that silent data loss
+rather than a loose type.
+
+Measured 2026-11-03, by calling each impl and comparing its top-level keys
+against its declared schema:
+
+| tool | keys the caller never received |
+|---|---|
+| `inspect` | all 13 of the single-event shape — `message`, `severity`, `ref_file`, `ref_line`, `log_context`, … |
+| `output` | `content` — the tool whose purpose is returning captured text returned none |
+| `exec` | `cmd`, `duration_sec`, `output_stats`, `preview`, and `matched_command` on a reroute |
+| `register_command` | `existing`, `command`, `format_detected` |
+| `ci_check` | `has_new_errors` — the schema declared `has_errors`, which the impl does not return |
+| `run` | `extension_data`, `output_tail` |
+
+**This was the cause of all 11 of the suite's failing tests.** They assert on
+keys like `result["existing"]` and `result["message"]`, which arrived deleted.
+Completing the schemas took the suite from 11 failed / 1385 passed to **1400
+passed**, with no change to any implementation.
+
+Two of the types were wrong in the first attempt, and the schema caught both
+loudly rather than silently: `recommendation` is a dict carrying
+`message`/`suggested_call`/`reason`, not a string, and `log_context` is
+nullable. Every single-event `inspect` field comes from
+`_to_json_safe(event_data.get(...))`, so each one is now declared nullable.
+
+`tests/test_output_schemas.py` gains the direction it was missing. Its existing
+tests assert that a *declared* key is published; they cannot see a *returned*
+key that is undeclared. Four new tests pin the keys that were lost, measured
+red against the schemas as #47 left them.
+
+### Fixed — mypy is clean
+
+The 14 remaining errors were all one pattern: a tool declares a TypedDict while
+its impl returns `dict[str, Any]`. Each now casts at the tool boundary, which
+is honest only because the schemas above are complete — a cast onto an
+incomplete TypedDict would have asserted a shape that drops data.
+
+### Added — an optional Slurm route for `run` and `exec` (#58)
+
+blq can now run a command as one Slurm job, with `srun`. The output still
+streams to blq, so the run is captured and queryable in the same way as a local
+run.
+
+The route stays off until a policy file exists at `~/.config/blq/slurm.toml`.
+A command takes the Slurm route only when its working directory is inside one of
+the policy's `roots`. A bad value in the file produces a warning, and the
+command uses the local route.
+
+A command in `.bird/commands.toml` can set `slurm = false` to stay local, or a
+`slurm` table to change its CPUs, memory and partition. `local_memory_max`
+caps the memory of a command on the local route.
+
+`srun` prints its task status line at any position in the merged output, and it
+reorders the task's stdout and stderr. On the Slurm route the task merges the
+two streams before the command runs, and blq moves the status line to the end of
+the output, so a multi-line record stays whole for the parser.
+
+See `docs/slurm.md`.
+
 ## v1.2.2
 
 ### Fixed — Suppress FastMCP banners on stdio transport

{blq_cli-1.2.2 → blq_cli-2.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: blq-cli
-Version: 1.2.2
+Version: 2.0.0
 Summary: Build Log Query - capture and query build/test logs with DuckDB
 Project-URL: Homepage, https://blq-cli.readthedocs.io/
 Project-URL: Repository, https://github.com/teaguesterling/blq-cli
@@ -20,7 +20,7 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Software Development :: Build Tools
 Classifier: Topic :: Software Development :: Testing
 Requires-Python: >=3.10
-Requires-Dist: duckdb>=1.4.4
+Requires-Dist: duckdb<2,>=1.4.4
 Requires-Dist: jinja2>=3.0.0
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: pyyaml>=6.0

blq_cli-2.0.0/docs/slurm.md

@@ -0,0 +1,94 @@
+# Slurm Routing
+
+blq can run commands through a Slurm queue with `srun`. Output still streams to blq, so the run is captured and queryable as usual.
+
+This guide uses two terms:
+
+- **Slurm route**: blq runs the command as one Slurm job.
+- **Local route**: blq runs the command as a local process.
+
+With no policy file, blq uses the local route and does not call any Slurm tool.
+
+## Policy File
+
+The policy file is `~/.config/blq/slurm.toml`. It applies to all projects for this user.
+
+```toml
+roots = ["~/code"]
+partition = "batch"
+cpus = 4
+mem = "8G"
+local_memory_max = "16G"
+```
+
+| Key | Default | Meaning |
+|-----|---------|---------|
+| `enabled` | `true` | Set to `false` to turn off all Slurm routing. |
+| `roots` | `[]` | Directories whose commands use the Slurm route. |
+| `partition` | `"batch"` | The Slurm partition for each job. |
+| `cpus` | `4` | The CPUs for each job. This must be an integer above 0. |
+| `mem` | `"8G"` | The memory for each job, for example `512M` or `16G`. |
+| `local_memory_max` | not set | The memory limit for commands on the local route. |
+
+If the file has a bad value, blq writes a warning and ignores the whole file. The command then uses the local route.
+
+## Roots
+
+A command uses the Slurm route only if its working directory is in one of the `roots`. Subdirectories count. A `~` at the start of a root expands to the home directory.
+
+## Settings for One Command
+
+Add a `slurm` value to a command in `.bird/commands.toml`.
+
+Use `slurm = false` to keep a command on the local route:
+
+```toml
+[commands.serve]
+cmd = "npm run dev"
+slurm = false
+```
+
+Use a table to change the resources of a command. A key that you do not set comes from the policy file:
+
+```toml
+[commands.build]
+cmd = "make -j8"
+slurm = { cpus = 8, mem = "16G", partition = "batch" }
+```
+
+If the value is not valid, blq writes a warning and ignores the value. The command still loads. `blq commands register --force` keeps the existing `slurm` value.
+
+## Turn Off Routing for One Run
+
+Set `BLQ_SLURM=0` to use the local route. The values `false`, `no` and `off` have the same effect.
+
+```bash
+BLQ_SLURM=0 blq run build
+```
+
+A command that blq starts inside a Slurm job also uses the local route.
+
+## When the Queue Is Down
+
+Before each run on the Slurm route, blq runs `scontrol ping`. If the check fails, blq writes this line to stderr and uses the local route:
+
+```text
+[blq] Slurm unavailable, running locally: <reason>
+```
+
+If `srun` then cannot contact the controller, blq writes the same warning and runs the command again on the local route. This rerun occurs only if the command did not start. Captured runs store the warning in their output. Runs with `--no-capture` do not have this rerun.
+
+## Limits
+
+These limits apply on the Slurm route:
+
+- **Stdin is empty.** The command cannot read input from the terminal or from a pipe.
+- **Background daemons die with the job.** A process that the command leaves behind stops when the job ends.
+- **Interactive commands need `slurm = false`.** These commands need a terminal and stdin.
+- **GPU and NPU commands need `slurm = false`.** blq does not request these devices from Slurm.
+- **A timeout includes the time in the queue.** A job that waits in the queue can use all of its timeout before it starts.
+- **Signal exit codes appear as 128+N.** For example, a command that receives SIGSEGV gives 139, and SIGKILL gives 137. They do not appear as negative numbers.
+
+This limit applies on the local route:
+
+- **A memory kill can show two results.** A command above `local_memory_max` shows `Terminated` with exit code 143, or `Killed` with exit code 137.

{blq_cli-1.2.2 → blq_cli-2.0.0}/mkdocs.yml

@@ -66,6 +66,7 @@ nav:
       - Python API: python-api.md
       - Integration: integration.md
       - MCP Server: mcp.md
+      - Slurm Routing: slurm.md
   - Development:
       - CI/CD Setup: ci-cd.md
       - Design - Sync: design-sync.md

{blq_cli-1.2.2 → blq_cli-2.0.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "blq-cli"
-version = "1.2.2"
+version = "2.0.0"
 description = "Build Log Query - capture and query build/test logs with DuckDB"
 readme = "README.md"
 license = "MIT"
@@ -27,7 +27,12 @@ classifiers = [
     "Topic :: Software Development :: Testing",
 ]
 dependencies = [
-    "duckdb>=1.4.4",
+    # Upper bound: duckdb 2.0 removed DuckDBPyRelation.fetchone() and tightened
+    # connection reuse, which fails 71 of this suite's tests (24 AttributeError,
+    # 18 ResourceInUseException). Measured 2026-11-03 on py3.12.3 and py3.13.13
+    # with a clean HOME. CI never caught it: the 3.10 leg resolves 1.5.6, fails
+    # first, and matrix fail-fast cancels the 3.11/3.12 legs that do get 2.0.
+    "duckdb>=1.4.4,<2",
     "jinja2>=3.0.0",
     "pandas>=2.0.0",
     "pyyaml>=6.0",
@@ -81,6 +86,12 @@ packages = ["src/blq", "src/blq_sandbox", "src/blq_sandbox_systemd", "src/blq_sa
 [tool.ruff]
 target-version = "py310"
 line-length = 100
+# ruff formats Python code blocks inside markdown too. In this repo that
+# collapses the aligned trailing comments the docs use as a two-column
+# listing, so `ruff format --check .` reported 19 documentation files as
+# unformatted. Excluding markdown keeps the docs as written; ruff's lint
+# rules never applied to them anyway.
+extend-exclude = ["*.md"]
 
 [tool.ruff.lint]
 select = ["E", "F", "I", "N", "W", "UP"]

{blq_cli-1.2.2 → blq_cli-2.0.0}/src/blq/__init__.py

@@ -14,7 +14,8 @@ Example usage:
     events = LogQuery.from_file("build.log").filter(severity="error").df()
 """
 
-from importlib.metadata import PackageNotFoundError, version as _dist_version
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version as _dist_version
 
 # Derived, never written by hand. The literal here read "1.0.1" against a
 # distribution version of 1.2.1 — two releases stale, and immune to

{blq_cli-1.2.2 → blq_cli-2.0.0}/src/blq/commands/core.py

@@ -30,6 +30,7 @@ from blq.config_format import (
 )
 
 # Git integration - re-exported for backward compatibility
+from blq.ext.slurm_policy import MEM_FORMAT
 from blq.git import GitInfo, capture_git_info  # noqa: F401
 
 if TYPE_CHECKING:
@@ -1109,6 +1110,7 @@ class RegisteredCommand:
     suppress: list[str] = field(default_factory=list)  # Fingerprints to suppress in reports
     lines: str | None = None  # Default line selection for run/exec output (e.g., "+20-")
     lock: str | None = None  # Lock name for resource contention (shared across commands)
+    slurm: bool | dict[str, Any] | None = None  # False opts out; a table sets resources
     _extra: dict[str, Any] = field(default_factory=dict)
 
     @property
@@ -1207,6 +1209,8 @@ class RegisteredCommand:
             d["lines"] = self.lines
         if self.lock is not None:
             d["lock"] = self.lock
+        if self.slurm is not None:
+            d["slurm"] = self.slurm
         d.update(self._extra)
         return d
 
@@ -1396,9 +1400,44 @@ _KNOWN_COMMAND_KEYS = {
     "suppress",
     "lines",
     "lock",
+    "slurm",
 }
 
 
+def _slurm_override_problem(value: Any) -> str | None:
+    """Return why a command's `slurm` value is invalid, or None if it is valid."""
+    if isinstance(value, bool):
+        return None
+    if not isinstance(value, dict):
+        return "must be a bool or a table"
+    for key, item in value.items():
+        if key == "cpus":
+            # bool is an int subclass, so `cpus = true` needs its own check.
+            if isinstance(item, bool) or not isinstance(item, int) or item <= 0:
+                return "cpus must be a positive integer"
+        elif key == "mem":
+            if not isinstance(item, str) or not MEM_FORMAT.fullmatch(item):
+                return "mem must look like 16G"
+        elif key == "partition":
+            if not isinstance(item, str) or not item:
+                return "partition must be a non-empty string"
+        else:
+            return f"unknown key {key!r}"
+    return None
+
+
+def _parse_slurm_override(name: str, value: Any) -> bool | dict[str, Any] | None:
+    """Validate a command's `slurm` value. A bad value warns and counts as unset."""
+    if value is None:
+        return None
+    problem = _slurm_override_problem(value)
+    if problem:
+        logger.warning("Ignoring slurm setting of command %r: %s", name, problem)
+        return None
+    override: bool | dict[str, Any] = value
+    return override
+
+
 def _load_commands_impl(lq_dir: Path) -> dict[str, RegisteredCommand]:
     """Internal implementation of load_commands."""
     commands_path = lq_dir / COMMANDS_FILE
@@ -1444,6 +1483,7 @@ def _load_commands_impl(lq_dir: Path) -> dict[str, RegisteredCommand]:
                 suppress=suppress,
                 lines=config.get("lines"),  # Default line selection for output
                 lock=config.get("lock"),
+                slurm=_parse_slurm_override(name, config.get("slurm")),
                 _extra=extra,
             )
     return commands

{blq_cli-1.2.2 → blq_cli-2.0.0}/src/blq/commands/execution.py

@@ -17,6 +17,7 @@ import subprocess
 import sys
 import threading
 import time
+import uuid
 from datetime import datetime
 from pathlib import Path
 from types import FrameType
@@ -45,10 +46,12 @@ from blq.commands.core import (
     parse_log_content,
     write_run_parquet,
 )
-from blq.ext import CommandSpec, capture_env
+from blq.ext import CommandSpec, Executor, capture_env
 from blq.ext.discovery import load_extensions, order_extensions
 from blq.ext.local_executor import LocalExecutor
 from blq.ext.pipeline import run_pipeline
+from blq.ext.slurm_executor import SlurmExecutor, memory_capped, srun_command
+from blq.ext.slurm_policy import decide_route, load_policy
 from blq.locks import LockHeldError, acquire_lock, release_lock
 
 # Logger for lq status messages
@@ -245,6 +248,7 @@ def _execute_with_live_output(
     lock_name: str | None = None,
     no_lock: bool = False,
     wait_lock: int | None = None,
+    slurm: bool | dict[str, Any] | None = None,
 ) -> RunResult:
     """Execute command with live output streaming (attempts/outcomes pattern).
 
@@ -403,7 +407,13 @@ def _execute_with_live_output(
         ordered = order_extensions(extensions)
 
         # Create executor with live output path
-        executor = LocalExecutor(quiet=quiet, live_output_path=live_output_path)
+        executor: Executor
+        policy = load_policy()
+        if policy is None:
+            executor = LocalExecutor(quiet=quiet, live_output_path=live_output_path)
+        else:
+            route = decide_route(policy, cmd_spec.cwd, slurm, {**os.environ, **cmd_spec.env})
+            executor = SlurmExecutor(route, policy, quiet=quiet, live_output_path=live_output_path)
 
         # Run the pipeline: prepare -> execute -> collect
         exec_result = run_pipeline(cmd_spec, ordered, executor)
@@ -665,6 +675,7 @@ def _execute_command(
     lock_name: str | None = None,
     no_lock: bool = False,
     wait_lock: int | None = None,
+    slurm: bool | dict[str, Any] | None = None,
 ) -> RunResult:
     """Execute a command and capture its output.
 
@@ -687,6 +698,8 @@ def _execute_command(
         lock_name: Optional lock name for resource contention.
         no_lock: If True, bypass lock even if lock_name is set.
         wait_lock: If set, wait up to this many seconds for the lock.
+        slurm: The command's Slurm override (False opts out, a table sets resources).
+            Only BIRD mode routes through Slurm; legacy parquet mode always runs locally.
 
     Returns:
         RunResult with execution details and parsed events
@@ -709,6 +722,7 @@ def _execute_command(
             lock_name=lock_name,
             no_lock=no_lock,
             wait_lock=wait_lock,
+            slurm=slurm,
         )
 
     # Legacy parquet mode - write everything at the end
@@ -1094,17 +1108,35 @@ def _parse_command_args(
     return named_args, positional_args, extra_args
 
 
-def _run_no_capture(command: str, quiet: bool = False) -> int:
+def _route_no_capture(command: str, slurm: bool | dict[str, Any] | None) -> str:
+    """Return `command`, wrapped in `srun` if the Slurm route says so."""
+    policy = load_policy()
+    if policy is None:
+        return command
+    cwd = Path.cwd()
+    route = decide_route(policy, cwd, slurm, os.environ)
+    if route.kind == "slurm" and route.resources is not None:
+        return srun_command(command, cwd, route.resources, uuid.uuid4().hex, None)
+    if route.warn:
+        print(f"[blq] Slurm unavailable, running locally: {route.reason}", file=sys.stderr)
+    return memory_capped(command, policy)
+
+
+def _run_no_capture(
+    command: str, quiet: bool = False, slurm: bool | dict[str, Any] | None = None
+) -> int:
     """Run a command without capturing output to parquet.
 
     Args:
         command: Shell command to run
         quiet: If True, don't stream output
+        slurm: The command's Slurm override (False opts out, a table sets resources)
 
     Returns:
         Exit code from the command
     """
     started_at = datetime.now()
+    command = _route_no_capture(command, slurm)
 
     # Track subprocess for signal handling
     process: subprocess.Popen[str] | None = None
@@ -1284,9 +1316,11 @@ def cmd_run(args: argparse.Namespace) -> None:
     else:
         logger.setLevel(logging.WARNING)
 
+    slurm = registered_commands[cmd_name].slurm if cmd_name in registered_commands else None
+
     # No-capture mode: just run and exit with the command's exit code
     if not should_capture:
-        exit_code = _run_no_capture(command, quiet)
+        exit_code = _run_no_capture(command, quiet, slurm)
         sys.exit(exit_code)
 
     # Determine timeout: CLI flag overrides command config
@@ -1326,6 +1360,7 @@ def cmd_run(args: argparse.Namespace) -> None:
             lock_name=lock_name,
             no_lock=no_lock,
             wait_lock=wait_lock,
+            slurm=slurm,
         )
     except LockHeldError as e:
         print(f"Error: {e}", file=sys.stderr)
@@ -1564,7 +1599,7 @@ def cmd_exec(args: argparse.Namespace) -> None:
 def _write_imported_run(
     events: list[dict[str, Any]],
     run_meta: dict[str, Any],
-    config: "BlqConfig",
+    config: BlqConfig,
     lq_dir: Path,
 ) -> Path:
     """Persist an imported/captured run to the ACTIVE storage backend.

{blq_cli-1.2.2 → blq_cli-2.0.0}/src/blq/commands/registry.py

@@ -218,8 +218,10 @@ def cmd_register(args: argparse.Namespace) -> None:
                     )
                     break  # Only show first match
 
-    # Register new command
+    # Register new command. The Slurm setting survives an overwrite: dropping `slurm = false`
+    # would silently put the command back on the Slurm route.
     capture = not getattr(args, "no_capture", False)
+    slurm = commands[name].slurm if name in commands else None
 
     # Detect format if not specified
     format_hint = args.format
@@ -240,6 +242,7 @@ def cmd_register(args: argparse.Namespace) -> None:
             timeout=args.timeout,
             format=format_hint,
             capture=capture,
+            slurm=slurm,
         )
         defaults_note = f" (defaults: {defaults})" if defaults else ""
         format_note = f" [format: {format_hint}]" if format_detected else ""
@@ -255,6 +258,7 @@ def cmd_register(args: argparse.Namespace) -> None:
             timeout=args.timeout,
             format=format_hint,
             capture=capture,
+            slurm=slurm,
         )
         capture_note = " (no capture)" if not capture else ""
         format_note = f" [format: {format_hint}]" if format_detected else ""

{blq_cli-1.2.2 → blq_cli-2.0.0}/src/blq/commands/watch_cmd.py

@@ -264,6 +264,7 @@ class WatchController:
                     format_hint=reg_cmd.format,
                     quiet=self._quiet,
                     session_id=self._session.session_id,
+                    slurm=reg_cmd.slurm,
                 )
 
                 self._session.run_count += 1

{blq_cli-1.2.2 → blq_cli-2.0.0}/src/blq/ext/__init__.py

@@ -6,13 +6,12 @@ Extension.prepare() → Executor.execute() → Collector.collect().
 
 from __future__ import annotations
 
+import os
 from dataclasses import dataclass, field
 from datetime import datetime
 from pathlib import Path
 from typing import Any, Protocol
 
-import os
-
 
 @dataclass
 class CommandSpec: