by-framework 0.2.2.dev14__tar.gz → 0.2.2.dev16__tar.gz

{by_framework-0.2.2.dev14 → by_framework-0.2.2.dev16}/CLAUDE.md

@@ -151,6 +151,22 @@ Pre-commit hooks are configured in `.pre-commit-config.yaml` and run isort, ruff
   plugin/trace integration point must degrade to a no-op (logged) rather
   than raising.
 
+- **A rule enforced only behind an optional dependency is not enforced.** If
+  the check short-circuits when a package is missing, then whatever controls
+  the environment controls the rule — and a green run stops meaning anything.
+  Incident: `tests/metrics/test_catalog.py` asserts every exported metric has a
+  `metrics/catalog.py` entry, but returned early when `prometheus-client` was
+  absent, while `make test` synced with `--extra dev` and dropped the root
+  `observability` extra. The assertion ran against an empty registry and
+  reported success for months; four uncatalogued metrics landed on main with CI
+  green (#142, then the CI fix). Correct form: enforce the rule where the
+  optional dependency cannot reach it — here `scripts/check-metric-catalog.sh`
+  reads the source, so **every new Prometheus `Counter`/`Gauge`/`Histogram` in
+  `src/` needs a `catalog.py` entry** regardless of what is installed — and
+  make the dependent test `pytest.skip()` rather than `return`, so degrading is
+  at least visible. Test commands must sync the extras the tests exercise
+  (`make test` uses `--all-extras`); narrowing them silently deletes coverage.
+
 ## Iron rules
 
 <!-- No process rules (test invocation, release steps) have caused an
@@ -166,6 +182,7 @@ Pre-commit hooks are configured in `.pre-commit-config.yaml` and run isort, ruff
 | Worker deployment/production-readiness — README's 部署 section, `__main__.py` CLI flags, `run_worker()`'s signature, or shutdown/signal handling | `docs/architecture/production-deployment.md` |
 | Worker readiness/health-check endpoint (`WorkerHealthServer`, `/readyz`, `--health-port`) — building it, or touching anything that changes what "ready" means | `docs/architecture/worker-readiness-endpoint.md` |
 | suspend/resume liveness — the wait index, its idempotency gate, or the sweep that resolves callers whose reply never arrives | `docs/architecture/suspend-resume-liveness.md` |
+| Langfuse tracing state held in worker memory — `_active_workflows`, trace-output uploads, or anything that queues/retains per-task telemetry | `docs/architecture/langfuse-tracing-memory.md` |
 
 ## Maintaining this map
 

{by_framework-0.2.2.dev14 → by_framework-0.2.2.dev16}/Makefile

@@ -37,11 +37,22 @@ list-projects:
 		echo $$project; \
 	done
 
+# All ten projects share the workspace venv, so syncing them in a loop makes
+# each one PRUNE the previous one's dependencies — the loop used to end with a
+# venv holding only the last project's deps (and it uninstalled prometheus-client
+# on the second iteration). --all-packages resolves the whole workspace once
+# instead. The loop survives only for an explicit PROJECTS override, where
+# narrowing the venv is what the caller asked for.
 install:
-	@for project in $(PROJECTS); do \
-		echo "==> Syncing $$project"; \
-		(cd $$project && uv sync --all-extras); \
-	done
+	@if [ "$(strip $(PROJECTS))" = "$(strip $(PYTHON_PROJECTS))" ]; then \
+		echo "==> Syncing workspace (all packages, all extras)"; \
+		uv sync --all-extras --all-packages; \
+	else \
+		for project in $(PROJECTS); do \
+			echo "==> Syncing $$project (narrowed: PROJECTS override)"; \
+			(cd $$project && uv sync --all-extras); \
+		done; \
+	fi
 
 format:
 	@PROJECTS="$(PROJECTS)" ./scripts/python_quality.sh format $(FILES)
@@ -80,6 +91,18 @@ lint-changed:
 		$(MAKE) lint PROJECTS="$(PROJECTS)" FILES="$$changed_files"; \
 	fi
 
+# --all-extras, not --extra dev: `uv run` only ever adds what it is asked for,
+# it never restores an extra it was not told about. So once `install`'s old
+# per-project loop had uninstalled prometheus-client, `--extra dev` left it
+# uninstalled and the root project's whole `observability` extra stayed missing
+# for the entire test run. tests/metrics then ran against a build with no
+# Prometheus registry, where the metric catalog check short-circuits and reports
+# success without inspecting anything. State what the tests need, rather than
+# inheriting whatever `install` happened to leave behind.
+#
+# Deliberately per-project rather than --all-packages: each project must be
+# testable with only its own declared dependencies, or a package that quietly
+# leans on a sibling's deps would still pass here and break on install.
 test:
 	@set -e; for project in $(PROJECTS); do \
 		if ! find "$$project/tests" -type f \( -name 'test_*.py' -o -name '*_test.py' \) -print -quit 2>/dev/null | grep -q .; then \
@@ -87,7 +110,7 @@ test:
 			continue; \
 		fi; \
 		echo "==> Testing $$project"; \
-		(cd $$project && uv run --extra dev pytest); \
+		(cd $$project && uv run --all-extras pytest); \
 	done
 
 clean:

{by_framework-0.2.2.dev14 → by_framework-0.2.2.dev16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: by-framework
-Version: 0.2.2.dev14
+Version: 0.2.2.dev16
 Summary: 分布式 Agent 调度框架
 License-File: LICENSE
 Requires-Python: >=3.12

{by_framework-0.2.2.dev14 → by_framework-0.2.2.dev16}/docs/architecture/KEY_FILES.md

@@ -314,6 +314,22 @@ Entry anatomy:
   best-effort — a background component that fails to construct must never
   prevent the worker from consuming — and every one of them must be
   cancelled and awaited in `_shutdown()`, or shutdown hangs on it.
+  `_run_once()` acquires a concurrency slot via `_acquire_slot()`, never a
+  bare `await self.semaphore.acquire()` — a saturated pool (all
+  `max_concurrency` slots held by legitimately long-running tasks) must not
+  be indistinguishable from a genuinely stuck consume loop. `_acquire_slot()`
+  polls `len(self._running_tasks) >= self.max_concurrency` and calls
+  `_mark_consumer_tick()` on every poll while waiting, so `_is_consumer_healthy()`
+  (whose own comparison logic is untouched) keeps seeing fresh ticks for as
+  long as the loop is merely busy, not stalled. `self._slot_wait_poll_seconds`
+  is derived from `_consumer_health_timeout_seconds` (not hardcoded) so the
+  two stay proportional under any lease-TTL/stream-block-ms configuration.
+  Trade-off this does *not* solve, by design: it cannot distinguish a
+  legitimately long task from a deadlocked one that will never return — both
+  present as `_running_tasks` stuck at `max_concurrency`; task-level SLA
+  enforcement (detecting and cancelling an individual stuck task) is a
+  separate concern from consumer-loop liveness and is not implemented here —
+  see `CancelTaskCommand` for the existing per-task cancel path.
 
 - `src/by_framework/worker/health_server.py` — `WorkerHealthServer`: the
   `/readyz` readiness HTTP endpoint, on its own daemon thread (mirrors

by_framework-0.2.2.dev16/docs/architecture/langfuse-tracing-memory.md

@@ -0,0 +1,130 @@
+# Langfuse tracing: what the worker holds, and for how long
+
+## What this covers
+
+`libs/by-framework-trace-langfuse` is the only plugin that keeps **live
+per-task state in worker memory across hook calls**. Everything else in the
+trace layer is either fire-and-forget (`RedisSpanExporter`) or bounded by the
+SDK (`LangfuseSpanProcessor` wraps OTel's `BatchSpanProcessor`, whose queue
+tops out at 2048 spans and drops the overflow).
+
+This doc exists because three separate unbounded-growth bugs shipped in that
+plugin and none of them were visible from reading a single function. They
+shared one root cause, stated below, and the fixes are only correct as long
+as that framing holds.
+
+## The rule everything follows from
+
+**Langfuse reporting is best-effort telemetry, not business data.** A slow or
+unreachable Langfuse must cost trace fidelity — never worker memory, never
+worker shutdown, never a shared executor.
+
+Every degraded path therefore **sheds load**: drop the work, count it, log it
+sparsely. The failure mode to avoid is the intuitive one — "queue it, we'll
+send it when Langfuse recovers" — because a worker completing tasks faster
+than Langfuse accepts them turns that queue into an OOM with the whole
+serialized task output pinned in each entry.
+
+This is a stricter rule than the repo-wide fail-soft invariant in `CLAUDE.md`.
+Fail-soft says *don't raise into the task path*. This says *don't accumulate
+in the task path either* — a plugin that swallows every exception can still
+kill the worker by growing.
+
+## The three things the plugin holds
+
+### 1. `_active_workflows` — durable workflow spans
+
+An `agent.workflow` observation spans the **logical** task, which outlives any
+single worker execution: `ask_user` and `call_agent` suspend the execution and
+a later `RESUME` reattaches to it (see `suspend-resume-liveness.md`). So the
+handle cannot be closed when the first execution returns — it lives in an
+`OrderedDict` keyed by `(session_id, message_id)` until the task reaches a
+terminal status.
+
+Three things can end an entry, and all three must exist:
+
+| Exit | Trigger |
+|---|---|
+| Normal | terminal `on_task_complete` / `on_task_error` / `on_task_cancel` |
+| TTL | `_expire_active_workflows()`, for a suspend whose resume never came |
+| LRU | `_evict_active_workflows_if_needed()`, at `max_active_workflows` |
+
+**Invariant — LRU order equals expiry order.** `_expire_active_workflows()`
+scans from the head and stops at the first live entry, which is only valid
+because every write goes through `_track_active_workflow()`, which refreshes
+`expires_at` *and* moves the entry to the tail. A future write that bypasses
+that helper silently turns expiry into a no-op for everything behind it.
+
+**Ownership is claimed at registration, not at the end of the hook.**
+`on_task_start` registers the workflow and immediately sets
+`LANGFUSE_WORKFLOW_OBSERVATION_ATTR` on the context, because the tracer calls
+that follow it can raise — and `PluginRegistry._execute_hook()` swallows that.
+`_end_workflow_observation()` additionally falls back to the
+`(session_id, message_id)` key when the context attribute is missing. Both
+halves are load-bearing: without the fallback, a mid-hook SDK error orphaned
+100% of entries until LRU eviction, each one a live span that would never
+export.
+
+For the same reason `on_task_error` / `on_task_cancel` must **not** return
+early when the context has no agent/worker observations. The workflow entry
+can outlive them, and bailing is what stranded it.
+
+### 2. `_pending_trace_output_updates` — in-flight trace-output uploads
+
+`update_trace_output` is a *synchronous* HTTP POST — it patches the trace-list
+`output` field, which the OTel span pipeline does not carry. It runs on the
+plugin's **own** `ThreadPoolExecutor`, never the asyncio default executor:
+that pool is shared with the rest of the framework's blocking calls, and a
+stalled Langfuse would starve unrelated work.
+
+The bound is enforced **before submitting**, not by trimming the tracking set.
+Dropping only the future leaves the work item — and the payload it closes
+over — sitting in the executor queue, which is exactly the growth being
+guarded. At the limit the upload is discarded and counted.
+
+`on_worker_shutdown` waits on the remainder with a deadline and abandons
+whatever misses it. The discard callback catches `BaseException`, not
+`Exception`: shutdown cancels the stragglers, and `CancelledError` is a
+`BaseException`, so a narrower catch dumps one traceback per abandoned upload.
+
+### 3. Serialized output
+
+`_serialize_value` deep-copies the whole task result. The end-of-task path
+serializes **once**, at the top of `on_task_complete`; `_end_observation`,
+`_end_workflow_observation` and `_update_trace_output` all take an
+already-serialized value and must not re-serialize it. Their docstrings state
+this precondition — it is a contract, not an optimization, because copy #3 is
+what an in-flight upload pins.
+
+## Tunables
+
+All have safe defaults; unset behaves no worse than the built-in values.
+Explicit constructor kwargs always win over the environment, per `CLAUDE.md`'s
+"explicitly-passed kwarg" invariant.
+
+| Env | Default | What it bounds |
+|---|---|---|
+| `BYAI_LANGFUSE_MAX_ACTIVE_WORKFLOWS` | 10000 | live workflow spans |
+| `BYAI_LANGFUSE_WORKFLOW_TTL_SECONDS` | 3600 | how long a suspended workflow is held |
+| `BYAI_LANGFUSE_MAX_PENDING_TRACE_OUTPUTS` | 128 | in-flight trace-output uploads |
+| `BYAI_LANGFUSE_SHUTDOWN_FLUSH_TIMEOUT_SECONDS` | 5 | shutdown flush deadline |
+
+## Accepted trade-off
+
+A workflow reclaimed by TTL or LRU whose resume arrives later gets a *new*
+observation built from the same deterministic span id
+(`str_to_uint64(f"{execution_anchor}:agent.workflow")`), so Langfuse sees a
+duplicate span id. This is pre-existing LRU behaviour, not new; the 1-hour
+default TTL keeps it rare. Trace-output drops under load are likewise
+deliberate — the trace and its observations still ship via the SDK's own
+batch pipeline, so only the trace-list `output` preview is lost.
+
+## One more thing that is not a leak
+
+`build_langchain_callback` subclasses the Langfuse `CallbackHandler` to stop
+LangChain runs being promoted to trace roots. That subclass is cached per base
+class in a `WeakKeyDictionary`. Minting one per call was *not* a hard leak —
+the classes are reclaimable — but it ran once per task, and type objects are
+only collected by the generational GC while every creation invalidates
+CPython's type method cache. Per-callback state lives on the instance
+(`_by_framework_metadata`), so one shared subclass is safe.

{by_framework-0.2.2.dev14 → by_framework-0.2.2.dev16}/docs/architecture/worker-readiness-endpoint.md

@@ -216,7 +216,15 @@ regardless.
 2. `draining` — `SIGTERM`/`SIGINT` received, shutdown in progress
 3. `evicted` — `admin_lifecycle == "evicted"`
 4. `suspended` — `admin_lifecycle == "suspended"`
-5. `consumer_stalled` — consume loop stale past `_consumer_health_timeout_seconds`
+5. `consumer_stalled` — consume loop stale past `_consumer_health_timeout_seconds`.
+   "Stale" means the loop itself has stopped making progress (crashed task,
+   or an unexpected hang outside slot-waiting) — it deliberately excludes a
+   consume loop that is merely waiting for every `max_concurrency` slot to
+   free up while legitimately long-running tasks are in flight. A saturated-
+   but-alive loop keeps refreshing its liveness signal while it waits (see
+   `runner.py`'s `_acquire_slot()` in `docs/architecture/KEY_FILES.md`), so
+   it never surfaces here; only a loop that has actually stopped responding
+   does.
 6. `serving` — none of the above; `ready=true`
 
 `ready` is `true` if and only if `reason == "serving"`.

{by_framework-0.2.2.dev14 → by_framework-0.2.2.dev16}/libs/by-framework-langgraph/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "by-framework-langgraph"
-version = "0.0.3.dev4"
+version = "0.0.3.dev5"
 description = "LangGraph integration for by-framework"
 readme = "README.md"
 requires-python = ">=3.12"

{by_framework-0.2.2.dev14 → by_framework-0.2.2.dev16}/libs/by-framework-trace-langfuse/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "by-framework-trace-langfuse"
-version = "0.0.3.dev4"
+version = "0.0.3.dev5"
 description = "Langfuse trace provider for by-framework"
 readme = "README.md"
 requires-python = ">=3.12"