npm - @kontourai/flow-agents - Versions diffs - 3.9.0 → 3.10.0 - Mend

@kontourai/flow-agents 3.9.0 → 3.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (106) hide show

package/.github/workflows/ci.yml +8 -0
package/CHANGELOG.md +17 -0
package/README.md +1 -1
package/build/src/cli/assignment-provider.js +4 -18
package/build/src/cli/continuation-adapter.d.ts +10 -8
package/build/src/cli/continuation-adapter.js +22 -4
package/build/src/cli/validate-workflow-artifacts.js +21 -1
package/build/src/cli/workflow-sidecar.js +15 -20
package/build/src/cli/workflow.js +58 -11
package/build/src/continuation-driver.d.ts +27 -3
package/build/src/continuation-driver.js +97 -14
package/build/src/index.d.ts +2 -2
package/build/src/index.js +1 -1
package/build/src/tools/build-universal-bundles.js +52 -25
package/build/src/tools/validate-source-tree.js +5 -2
package/context/scripts/hooks/lib/continuation-turn-authority.js +386 -0
package/context/scripts/hooks/stop-goal-fit.js +244 -73
package/context/scripts/hooks/workflow-steering.js +24 -15
package/context/scripts/telemetry/lib/pricing.sh +18 -6
package/context/scripts/telemetry/lib/transport.sh +139 -37
package/context/scripts/telemetry/lib/usage.sh +282 -32
package/context/scripts/telemetry/lib/usage_model_guard.sh +42 -0
package/context/scripts/telemetry/telemetry.sh +76 -1
package/docs/coordination-guide.md +21 -5
package/docs/fixture-ownership.md +2 -1
package/docs/public-workflow-cli.md +44 -0
package/docs/spec/runtime-hook-surface.md +8 -4
package/evals/acceptance/prove-capture-teeth-declared.sh +13 -0
package/evals/acceptance/prove-capture-teeth.sh +13 -0
package/evals/ci/run-baseline.sh +2 -0
package/evals/fixtures/codex-legacy-agents/5273878130bdafc8a024a650bb5b66c9b003f1f859b5dc6e5b588cbf4ab23228.md +25 -0
package/evals/fixtures/telemetry/usage-transcript-adversarial.jsonl +2 -0
package/evals/fixtures/telemetry/usage-transcript-malformed-sandwich.jsonl +3 -0
package/evals/fixtures/telemetry/usage-transcript-negative-tokens.jsonl +1 -0
package/evals/fixtures/telemetry/usage-transcript-oversized-prefix.jsonl +201 -0
package/evals/fixtures/telemetry/usage-transcript-oversized-tokens.jsonl +1 -0
package/evals/fixtures/telemetry/usage-transcript-prefixed-secret-lower.jsonl +1 -0
package/evals/fixtures/telemetry/usage-transcript-prefixed-secret-upper.jsonl +1 -0
package/evals/fixtures/telemetry/usage-transcript-secret-model-jwt.jsonl +1 -0
package/evals/fixtures/telemetry/usage-transcript-secret-model.jsonl +1 -0
package/evals/fixtures/telemetry/usage-transcript-torn-trailing.jsonl +2 -0
package/evals/integration/test_actor_identity.sh +63 -0
package/evals/integration/test_assignment_provider_local_file.sh +34 -0
package/evals/integration/test_builder_entry_enforcement.sh +5 -1
package/evals/integration/test_bundle_install.sh +106 -17
package/evals/integration/test_bundle_lifecycle.sh +17 -3
package/evals/integration/test_captured_fail_reconciliation.sh +11 -0
package/evals/integration/test_ci_actor_identity.sh +2 -2
package/evals/integration/test_command_log_concurrency.sh +11 -0
package/evals/integration/test_command_log_fork_classification.sh +11 -0
package/evals/integration/test_command_log_integrity.sh +11 -0
package/evals/integration/test_current_json_per_actor.sh +416 -21
package/evals/integration/test_enforcer_expects_driven.sh +11 -0
package/evals/integration/test_ensure_session_ownership_guard.sh +39 -0
package/evals/integration/test_evidence_capture_hook.sh +25 -6
package/evals/integration/test_fixture_retirement_audit.sh +2 -2
package/evals/integration/test_flowdef_union_floor_regression.sh +11 -0
package/evals/integration/test_gate_bypass_chain.sh +11 -0
package/evals/integration/test_gate_lockdown.sh +10 -0
package/evals/integration/test_goal_fit_ghost_session.sh +11 -0
package/evals/integration/test_goal_fit_hook.sh +257 -9
package/evals/integration/test_goal_fit_rederive.sh +11 -0
package/evals/integration/test_install_merge.sh +141 -3
package/evals/integration/test_liveness_conflict_injection.sh +39 -3
package/evals/integration/test_liveness_heartbeat.sh +46 -9
package/evals/integration/test_phase_map_and_gate_claim.sh +10 -1
package/evals/integration/test_public_workflow_cli.sh +6 -6
package/evals/integration/test_session_resume_roundtrip.sh +30 -0
package/evals/integration/test_stop_hook_release.sh +111 -11
package/evals/integration/test_telemetry_tool_usage.sh +394 -0
package/evals/integration/test_verify_hold.sh +64 -1
package/evals/integration/test_workflow_sidecar_writer.sh +10 -1
package/evals/integration/test_workflow_steering_hook.sh +20 -4
package/evals/run.sh +2 -0
package/evals/static/test_universal_bundles.sh +123 -0
package/package.json +1 -1
package/packaging/README.md +21 -0
package/packaging/codex-legacy-agents-fingerprints.json +33 -0
package/scripts/README.md +2 -1
package/scripts/audit-codex-legacy-agents.js +43 -0
package/scripts/classify-codex-legacy-agents.js +71 -0
package/scripts/hooks/evidence-capture.js +18 -6
package/scripts/hooks/lib/actor-identity.js +62 -29
package/scripts/hooks/lib/continuation-turn-authority.js +386 -0
package/scripts/hooks/lib/current-pointer.js +114 -12
package/scripts/hooks/lib/liveness-heartbeat.js +15 -12
package/scripts/hooks/stop-goal-fit.js +244 -73
package/scripts/hooks/workflow-steering.js +24 -15
package/scripts/install-codex-home.sh +8 -7
package/scripts/telemetry/lib/pricing.sh +18 -6
package/scripts/telemetry/lib/transport.sh +71 -0
package/scripts/telemetry/lib/usage.sh +282 -32
package/scripts/telemetry/lib/usage_model_guard.sh +42 -0
package/scripts/telemetry/telemetry.sh +76 -1
package/src/cli/assignment-provider.ts +5 -26
package/src/cli/builder-flow-runtime.test.mjs +93 -6
package/src/cli/continuation-adapter.ts +33 -7
package/src/cli/continuation-driver.test.mjs +741 -2
package/src/cli/public-api.test.mjs +1 -0
package/src/cli/validate-workflow-artifacts.ts +19 -1
package/src/cli/workflow-sidecar.ts +18 -23
package/src/cli/workflow.ts +71 -11
package/src/continuation-driver.ts +125 -19
package/src/index.ts +4 -0
package/src/tools/build-universal-bundles.ts +61 -25
package/src/tools/validate-source-tree.ts +5 -2

package/context/scripts/telemetry/lib/transport.sh CHANGED Viewed

@@ -2,6 +2,9 @@
 # transport.sh — Dual-channel telemetry transport
 source "${TELEMETRY_DIR}/lib/redact.sh"
+# Shared usage.model / numeric-bound validation constants (security review
+# HIGH + LOW findings) -- single source, also used by usage.sh's extractor.
+source "${TELEMETRY_DIR}/lib/usage_model_guard.sh"
 console_telemetry_endpoint_url() {
   if [[ -n "${CONSOLE_TELEMETRY_ENDPOINT_URL:-}" ]]; then
@@ -46,6 +49,66 @@ console_telemetry_timeout_seconds() {
   fi
 }
+# console_post_json <endpoint_url> <body> [connect_timeout] [max_time] [tmp_dir]
+# Shared best-effort JSON POST to a Console records endpoint. BOTH the telemetry mirror
+# (console_telemetry_emit) and the liveness relay (#295, scripts/liveness/relay.sh) go through this
+# ONE core so endpoint-allow, auth (Bearer + tenant), timeouts, temp-file handling, and the detached
+# fire can never drift between the two paths (the #356 shared-not-forked discipline). Reads
+# CONSOLE_TELEMETRY_TOKEN / CONSOLE_TENANT_ID from the environment. Never blocks or fails the caller.
+console_post_json() {
+  local endpoint_url="$1"
+  local body="$2"
+  local connect_timeout max_time tmp_dir curl_config curl_body
+  [[ -z "$endpoint_url" ]] && return
+  if ! console_telemetry_endpoint_allowed "$endpoint_url"; then
+    # Dropped silently before today's change (no signal that the event never
+    # left the machine). Warn once per shell process (guarded by a plain,
+    # non-local var so it survives across repeated console_post_json calls
+    # within the same sourced shell) without changing the drop itself.
+    if [[ -z "${_CONSOLE_TELEMETRY_ENDPOINT_WARNED:-}" ]]; then
+      printf 'warning: transport.sh: console endpoint dropped by the allowlist (must be https://, or http://localhost|127.0.0.1): %s\n' "$endpoint_url" >&2
+      _CONSOLE_TELEMETRY_ENDPOINT_WARNED=1
+    fi
+    return
+  fi
+  connect_timeout=$(console_telemetry_timeout_seconds "${3:-2}" 2 30)
+  max_time=$(console_telemetry_timeout_seconds "${4:-5}" 5 60)
+  tmp_dir="${5:-${TELEMETRY_SESSION_DIR:-${TMPDIR:-/tmp}}}"
+  curl_config=$(mktemp "${tmp_dir%/}/console-curl.XXXXXX") || return
+  curl_body=$(mktemp "${tmp_dir%/}/console-body.XXXXXX") || {
+    rm -f "$curl_config"
+    return
+  }
+  chmod 600 "$curl_config" "$curl_body" 2>/dev/null
+  printf '%s' "$body" > "$curl_body" || {
+    rm -f "$curl_config" "$curl_body"
+    return
+  }
+  {
+    printf 'url = "%s"\n' "$endpoint_url"
+    printf 'request = "POST"\n'
+    printf 'connect-timeout = "%s"\n' "$connect_timeout"
+    printf 'max-time = "%s"\n' "$max_time"
+    printf 'header = "Content-Type: application/json"\n'
+    if [[ -n "${CONSOLE_TELEMETRY_TOKEN:-}" ]] && console_telemetry_safe_token "$CONSOLE_TELEMETRY_TOKEN"; then
+      printf 'header = "Authorization: Bearer %s"\n' "$CONSOLE_TELEMETRY_TOKEN"
+    fi
+    if [[ -n "${CONSOLE_TENANT_ID:-}" ]] && console_telemetry_safe_tenant "$CONSOLE_TENANT_ID"; then
+      printf 'header = "x-console-tenant-id: %s"\n' "$CONSOLE_TENANT_ID"
+    fi
+    printf 'data-binary = "@%s"\n' "$curl_body"
+  } > "$curl_config" || {
+    rm -f "$curl_config" "$curl_body"
+    return
+  }
+  (
+    curl -s --proto =https,http --proto-redir =https,http --config "$curl_config" >/dev/null 2>&1
+    rm -f "$curl_config" "$curl_body"
+  ) &
+}
 # Derive a coarse, path-free project label for console attribution, most-stable-first so the SAME
 # project resolves to the SAME label across developers and machines (folder names differ between
 # clones and worktrees; the project manifest and git remote do not). Precedence:
@@ -118,12 +181,79 @@ console_project_label() {
   printf '%s' "$label"
 }
+# Defense-in-depth sanitize backstop (#568 slice 1 security review, MEDIUM,
+# hardened per HIGH + LOW follow-up findings): nulls any
+# usage.{input_tokens,output_tokens,cache_creation_input_tokens,
+# cache_read_input_tokens,estimated_cost_usd} that isn't a JSON number within
+# [USAGE_NUMERIC_MIN, USAGE_NUMERIC_MAX] (negative/absurd-magnitude values are
+# as untrusted as a wrong-typed one), and bounds/validates usage.model against
+# the SAME shared vendor-prefix allowlist (USAGE_MODEL_REGEX/
+# USAGE_MODEL_MAX_LEN, from usage_model_guard.sh) usage_last_turn_usage
+# enforces at extraction time -- a charset-only check here would be, and
+# previously was, a strict superset of secret/credential shapes (API keys,
+# JWTs) that a regression in the primary extractor could otherwise relay
+# verbatim. FAILS CLOSED: if `.usage` is present but is not a JSON object (a
+# raw string/number/array -- e.g. a future caller populating it wrong), the
+# whole `.usage` is nulled rather than left unsanitized (a naive
+# `.usage? != null` + `?`-chained field check on a non-object silently
+# produces NO jq output at all, which the caller would then treat as "sanitize
+# failed" and fall through to the ORIGINAL unsanitized event -- exactly the
+# fail-OPEN bug this branch exists to close). This exists so a FUTURE
+# extractor regression (a new caller populating .usage from a less-trusted
+# source, a refactor that drops a type/shape guard, etc.) can never again
+# relay an untyped/oversized/malformed/secret-shaped value to the console; it
+# is not the primary defense (usage_last_turn_usage already type-guards and
+# allowlists every field) but a last-resort net right before the POST. One jq
+# pass, no extra process spawned beyond the jq call already needed here.
+console_telemetry_sanitize_usage() {
+  local event="$1"
+  # FAIL CLOSED if the shared guard constants didn't resolve (e.g. a partial
+  # dist/ sync dropped usage_model_guard.sh): null .usage entirely rather than
+  # running a broken/empty jq that would return nothing and let the caller relay
+  # the ORIGINAL unsanitized event. Never fail open on a packaging defect.
+  if [[ -z "$USAGE_MODEL_REGEX" || -z "$USAGE_MODEL_MAX_LEN" || -z "$USAGE_NUMERIC_MIN" || -z "$USAGE_NUMERIC_MAX" ]]; then
+    printf '%s' "$event" | jq -c 'if has("usage") then .usage = null else . end'
+    return
+  fi
+  printf '%s' "$event" | jq -c \
+    --arg model_regex "$USAGE_MODEL_REGEX" \
+    --argjson model_max_len "$USAGE_MODEL_MAX_LEN" \
+    --argjson numeric_min "$USAGE_NUMERIC_MIN" \
+    --argjson numeric_max "$USAGE_NUMERIC_MAX" '
+    def bounded_number($v):
+      if ($v | type) == "number" and $v >= $numeric_min and $v <= $numeric_max then $v else null end;
+    if (.usage? == null) then .
+    elif ((.usage | type) != "object") then (.usage = null)
+    else
+      .usage.input_tokens = bounded_number(.usage.input_tokens?)
+      | .usage.output_tokens = bounded_number(.usage.output_tokens?)
+      | .usage.cache_creation_input_tokens = bounded_number(.usage.cache_creation_input_tokens?)
+      | .usage.cache_read_input_tokens = bounded_number(.usage.cache_read_input_tokens?)
+      | .usage.estimated_cost_usd = bounded_number(.usage.estimated_cost_usd?)
+      | .usage.model = (
+          if ((.usage.model? | type) == "string")
+             and ((.usage.model | length) <= $model_max_len)
+             and (.usage.model | test($model_regex))
+          then .usage.model
+          else null
+          end
+        )
+    end
+  ' 2>/dev/null
+}
 console_telemetry_emit() {
   local event="$1"
   local endpoint_url
   endpoint_url=$(console_telemetry_endpoint_url)
   [[ -z "$endpoint_url" ]] && return
-  console_telemetry_endpoint_allowed "$endpoint_url" || return
+  # Sanitize backstop runs before anything else so labeling/redaction/POST all
+  # see the already-bounded event; a failed/empty sanitize pass (bad JSON)
+  # falls back to the original event unchanged (never blocks relay).
+  local sanitized_event
+  sanitized_event=$(console_telemetry_sanitize_usage "$event") || sanitized_event=""
+  [[ -n "$sanitized_event" ]] && event="$sanitized_event"
   # Attribution: stamp a coarse, path-free project label (see console_project_label) before redaction
   # so the console buckets by project consistently across developers. The full context.cwd is still
@@ -143,42 +273,14 @@ console_telemetry_emit() {
   local processed_event
   processed_event=$(redact_event "$event" "${CONSOLE_TELEMETRY_REDACT:-${TELEMETRY_CHANNEL_ANALYTICS_REDACT:-}}")
-  local curl_config curl_body connect_timeout max_time
-  connect_timeout=$(console_telemetry_timeout_seconds "${CONSOLE_TELEMETRY_CONNECT_TIMEOUT_SECONDS:-2}" 2 30)
-  max_time=$(console_telemetry_timeout_seconds "${CONSOLE_TELEMETRY_MAX_TIME_SECONDS:-5}" 5 60)
-  curl_config=$(mktemp "${TELEMETRY_SESSION_DIR}/console-curl.XXXXXX") || return
-  curl_body=$(mktemp "${TELEMETRY_SESSION_DIR}/console-body.XXXXXX") || {
-    rm -f "$curl_config"
-    return
-  }
-  chmod 600 "$curl_config" "$curl_body" 2>/dev/null
-  printf '%s' "$processed_event" > "$curl_body" || {
-    rm -f "$curl_config" "$curl_body"
-    return
-  }
-  {
-    printf 'url = "%s"\n' "$endpoint_url"
-    printf 'request = "POST"\n'
-    printf 'connect-timeout = "%s"\n' "$connect_timeout"
-    printf 'max-time = "%s"\n' "$max_time"
-    printf 'header = "Content-Type: application/json"\n'
-    if [[ -n "${CONSOLE_TELEMETRY_TOKEN:-}" ]] && console_telemetry_safe_token "$CONSOLE_TELEMETRY_TOKEN"; then
-      printf 'header = "Authorization: Bearer %s"\n' "$CONSOLE_TELEMETRY_TOKEN"
-    fi
-    if [[ -n "${CONSOLE_TENANT_ID:-}" ]] && console_telemetry_safe_tenant "$CONSOLE_TENANT_ID"; then
-      printf 'header = "x-console-tenant-id: %s"\n' "$CONSOLE_TENANT_ID"
-    fi
-    printf 'data-binary = "@%s"\n' "$curl_body"
-  } > "$curl_config" || {
-    rm -f "$curl_config" "$curl_body"
-    return
-  }
-  (
-    curl -s --proto =https,http --proto-redir =https,http --config "$curl_config" >/dev/null 2>&1
-    rm -f "$curl_config" "$curl_body"
-  ) &
+  # Delegate the endpoint-allow gate, auth, timeouts, temp files, and detached POST to the shared
+  # core. Timeouts/tmp-dir are passed explicitly so telemetry behavior is byte-for-byte unchanged.
+  console_post_json \
+    "$endpoint_url" \
+    "$processed_event" \
+    "${CONSOLE_TELEMETRY_CONNECT_TIMEOUT_SECONDS:-2}" \
+    "${CONSOLE_TELEMETRY_MAX_TIME_SECONDS:-5}" \
+    "${TELEMETRY_SESSION_DIR:-}"
 }
 transport_emit() {

package/context/scripts/telemetry/lib/usage.sh CHANGED Viewed

@@ -6,6 +6,9 @@ USAGE_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # Single-source pricing registry loader (local / remote / bundled).
 source "${USAGE_LIB_DIR}/pricing.sh"
+# Shared usage.model / numeric-bound validation constants (security review
+# HIGH + LOW findings) -- single source, also used by transport.sh's backstop.
+source "${USAGE_LIB_DIR}/usage_model_guard.sh"
 # Resolve model from agent-spec.json
 usage_get_model() {
@@ -89,6 +92,50 @@ usage_count_delegations() {
 # if the on-disk schema changes so drift is logged rather than silently zeroed.
 USAGE_TRANSCRIPT_SCHEMA="message.usage.input_tokens"
+# Bounded tail-read size for usage_last_turn_usage (per-tool-call join), in
+# bytes. Overridable via env for tuning; default 256 KiB is large enough to
+# reliably contain the most recent assistant usage block even with sizeable
+# tool-output content nearby, small enough to bound per-call latency (this
+# function runs once per tool call, unlike usage_parse_transcript which runs
+# once per session at stop).
+TELEMETRY_USAGE_TAIL_BYTES="${TELEMETRY_USAGE_TAIL_BYTES:-262144}"
+# Shared pricing/cache-multiplier formula — the ONE place the per-model cost
+# arithmetic is defined. Both usage_parse_transcript (whole-session aggregate,
+# at `stop`) and usage_last_turn_usage (single-turn, per tool call) prepend
+# this jq `def` to their program text and call price_one(...), rather than
+# each hand-copying the formula (see #356 shared-not-forked discipline in
+# transport.sh — a duplicated formula is exactly the kind of drift that
+# invites). Given a model id ($m), a token breakdown ($u — object with
+# input/output/cache_creation/cache_read keys), the resolved pricing registry
+# ($registry), whether it's usable ($has_registry), and a requested pricing
+# version ($version, empty string = registry's current_version), returns
+# {estimated_cost_usd, pricing_version} — both null when no registry/version
+# resolves (never invents a cost from unpriced data).
+USAGE_PRICE_ONE_JQ_DEF='
+def price_one($m; $u; $registry; $has_registry; $version):
+  ($has_registry and ($registry != null)) as $has_reg
+  | (if $has_reg then (if $version == "" then $registry.current_version else $version end) else null end) as $ver
+  | (if $has_reg and ($ver != null) then ($registry.versions[$ver]) else null end) as $p
+  | ($p != null) as $priced
+  | (if $priced then $p.cache_multipliers else null end) as $cm
+  | (if $priced then (($p.models[$m]) // $p.default) else null end) as $rate
+  | (if $priced then (if ([$m] | inside($p.zero_cost_models)) then 0 else 1 end) else null end) as $billable
+  | {
+      estimated_cost_usd: (
+        if $priced then
+          $billable * (
+            ($u.input // 0)          * $rate.input
+            + ($u.output // 0)         * $rate.output
+            + ($u.cache_creation // 0) * $rate.input * $cm.write_5m
+            + ($u.cache_read // 0)     * $rate.input * $cm.read
+          ) / 1000000
+        else null end
+      ),
+      pricing_version: (if $priced then $ver else null end)
+    };
+'
 # Append a one-line schema-drift warning (transcript carried usage data we could
 # not parse). Goes to TELEMETRY_DRIFT_LOG if set, else stderr. Never fatal.
 usage_log_drift() {
@@ -118,6 +165,54 @@ usage_log_debug() {
   fi
 }
+# Shared registry resolution (DRY fix, review finding MEDIUM/code): both
+# usage_parse_transcript and usage_last_turn_usage need {registry, has_registry}
+# before invoking price_one() -- resolve it in exactly one place instead of
+# hand-copying the resolve+validate block in each caller.
+#   $1 = name of the caller's variable to receive the registry JSON (or the
+#        literal string "null" when unavailable/unparseable)
+#   $2 = name of the caller's variable to receive has_registry ("true"/"false")
+#   $3 = mode: "" (default -- pricing_registry() may hit network/local/bundled
+#        sources, used by usage_parse_transcript's once-per-session stop path)
+#        | "cache_only" (never perform a network fetch -- see
+#        usage_last_turn_usage's per-tool-call latency/race note below; a
+#        cold/stale cache degrades straight to the bundled snapshot)
+#   $4 = optional log context (e.g. transcript path) for usage_log_debug parity
+#        with the pre-refactor per-caller messages
+usage_resolve_registry() {
+  local __usage_reg_var="$1" __usage_has_var="$2" __usage_mode="${3:-}" __usage_log_ctx="${4:-}"
+  # Double-underscore-prefixed locals throughout (matching this function's own
+  # parameter convention, code review LOW finding) to harden against a future
+  # third caller's variable-name collision -- these never leak into the
+  # caller's scope (output is returned only via printf -v into the caller's
+  # named variables above), but a plain `_registry`/`_has_registry` name is
+  # exactly the kind of generic local name a future edit could accidentally
+  # shadow.
+  local __registry __has_registry
+  if [[ "$__usage_mode" == "cache_only" ]]; then
+    __registry="$(TELEMETRY_PRICING_CACHE_ONLY=1 pricing_registry 2>/dev/null)"
+  else
+    __registry="$(pricing_registry 2>/dev/null)"
+  fi
+  if [[ $? -eq 0 && -n "$__registry" ]]; then
+    __has_registry=true
+  else
+    __has_registry=false
+    __registry='null'
+    [[ -n "$__usage_log_ctx" ]] && usage_log_debug "pricing registry unavailable (${__usage_log_ctx}) — extracting tokens without cost"
+  fi
+  # A non-empty registry can still be malformed (corrupt/truncated pricing.json,
+  # or a bad remote 200) — validate it parses before handing it to `jq -n
+  # --argjson`, which would otherwise abort the whole parse and discard tokens.
+  if [[ "$__has_registry" == true ]] && ! jq -e . >/dev/null 2>&1 <<<"$__registry"; then
+    __has_registry=false
+    __registry='null'
+    [[ -n "$__usage_log_ctx" ]] && usage_log_debug "pricing registry unparseable (${__usage_log_ctx}) — extracting tokens without cost"
+  fi
+  printf -v "$__usage_reg_var" '%s' "$__registry"
+  printf -v "$__usage_has_var" '%s' "$__has_registry"
+}
 usage_parse_transcript() {
   local transcript="$1" version="${2:-}"
   if [[ -z "$transcript" ]]; then
@@ -136,33 +231,17 @@ usage_parse_transcript() {
   # Pricing registry is best-effort: when unavailable, tokens are still
   # extracted below and only the cost fields degrade to null (defect #2 fix —
   # previously this hard-returned 1 here, discarding token extraction
-  # entirely whenever pricing_registry() failed).
+  # entirely whenever pricing_registry() failed). Resolution itself is shared
+  # with usage_last_turn_usage via usage_resolve_registry (DRY fix).
   local registry has_registry
-  registry="$(pricing_registry 2>/dev/null)"
-  if [[ $? -eq 0 && -n "$registry" ]]; then
-    has_registry=true
-  else
-    has_registry=false
-    registry='null'
-    usage_log_debug "pricing registry unavailable (${transcript}) — extracting tokens without cost"
-  fi
-  # A non-empty registry can still be malformed (corrupt/truncated pricing.json,
-  # or a bad remote 200) — validate it parses before handing it to `jq -n
-  # --argjson`, which would otherwise abort the whole parse and discard tokens
-  # (the exact intermittent-null-tokens defect this function exists to fix).
-  if [[ "$has_registry" == true ]] && ! jq -e . >/dev/null 2>&1 <<<"$registry"; then
-    has_registry=false
-    registry='null'
-    usage_log_debug "pricing registry unparseable (${transcript}) — extracting tokens without cost"
-  fi
+  usage_resolve_registry registry has_registry "" "$transcript"
   local out
-  out="$(jq -n --argjson registry "$registry" --argjson has_registry "$has_registry" --arg version "$version" '
+  out="$(jq -n --argjson registry "$registry" --argjson has_registry "$has_registry" --arg version "$version" "${USAGE_PRICE_ONE_JQ_DEF}"'
     ($has_registry and ($registry != null)) as $has_reg
     | (if $has_reg then (if $version == "" then $registry.current_version else $version end) else null end) as $ver
     | (if $has_reg and ($ver != null) then ($registry.versions[$ver]) else null end) as $p
     | ($p != null) as $priced
-    | (if $priced then $p.cache_multipliers else null end) as $cm
     | (reduce inputs as $l ({};
         ($l.message.usage) as $u
         | if $u then
@@ -175,24 +254,14 @@ usage_parse_transcript() {
     | ($agg | to_entries
         | map(
             .key as $m | .value as $u
-            | (if $priced then (($p.models[$m]) // $p.default) else null end) as $rate
-            | (if $priced then (if ([$m] | inside($p.zero_cost_models)) then 0 else 1 end) else null end) as $billable
+            | price_one($m; $u; $registry; $has_registry; $version) as $price
             | {
                 model: $m,
                 input_tokens: ($u.input // 0),
                 output_tokens: ($u.output // 0),
                 cache_creation_input_tokens: ($u.cache_creation // 0),
                 cache_read_input_tokens: ($u.cache_read // 0),
-                estimated_cost_usd: (
-                  if $priced then
-                    $billable * (
-                      ($u.input // 0)          * $rate.input
-                      + ($u.output // 0)         * $rate.output
-                      + ($u.cache_creation // 0) * $rate.input * $cm.write_5m
-                      + ($u.cache_read // 0)     * $rate.input * $cm.read
-                    ) / 1000000
-                  else null end
-                )
+                estimated_cost_usd: $price.estimated_cost_usd
               })) as $by_model
     | {
         by_model: $by_model,
@@ -228,3 +297,184 @@ usage_parse_transcript() {
   printf '%s\n' "$out"
 }
+# Bounded, single-turn usage extractor — the tool-event (preToolUse/
+# postToolUse) counterpart to usage_parse_transcript's whole-session
+# aggregate (used only at `stop`). This runs once per tool call, so unlike
+# usage_parse_transcript it MUST NOT scan the whole transcript: a large
+# session can fire hundreds of tool calls, and a full-file jq scan per call
+# would add real, cumulative latency to every tool invocation.
+#
+# Reads only the last TELEMETRY_USAGE_TAIL_BYTES bytes of the transcript,
+# drops the (possibly partial) first line of that tail window, and returns
+# the LAST well-formed `type=="assistant"` line with a `message.usage` object.
+# Because tool execution is synchronous — the CLI appends the requesting
+# assistant message (including its tool_use blocks) before firing PreToolUse
+# for each block, and no new assistant turn can be appended mid-tool-call —
+# that last assistant/usage entry at hook-fire time is exactly the turn that
+# produced this specific tool call, for both PreToolUse and PostToolUse of the
+# same call.
+#
+# Hardened line selection (security review findings CRITICAL/HIGH): the tail
+# window is parsed in `jq -R` RAW mode, one line at a time, with `fromjson?`
+# per line — a torn/malformed line ANYWHERE in the window (e.g. a concurrent
+# writer's partial trailing write, or noise before/after the real line) is
+# SKIPPED, never fatal, and never masks a real usage line elsewhere in the
+# window (previously a single malformed line anywhere in the tail aborted the
+# whole `jq -n` JSON-mode parse, silently discarding a present, findable usage
+# line). Every extracted field is independently type-guarded AND bound-checked
+# before it is ever emitted: a token/cost field is emitted ONLY if it is a
+# JSON number within [USAGE_NUMERIC_MIN, USAGE_NUMERIC_MAX] (never a string,
+# object, array, negative, or absurd magnitude like 1e308 — never
+# "0-as-fabrication" when the source type/range is wrong; see
+# usage_model_guard.sh). `model` is emitted ONLY if it is a string, <=
+# USAGE_MODEL_MAX_LEN chars, AND matches USAGE_MODEL_REGEX — a small,
+# case-insensitive VENDOR-PREFIX ALLOWLIST (claude/gpt/o<digit>/gemini/glm/
+# llama/mistral/deepseek/qwen/grok/command/codestral), NOT a bare charset
+# check (security review HIGH finding: a charset-only check like
+# `^[A-Za-z0-9._:-]+$` is a strict SUPERSET of common secret/credential
+# shapes — an Anthropic API key `sk-ant-api03-...`, an AWS key `AKIA...`, or a
+# JWT `eyJ...` all satisfy that charset and would pass through verbatim).
+# Anything that doesn't match the allowlist resolves to "unknown". This is
+# what stops a crafted transcript line (e.g. {"message":{"model":"<secret or
+# arbitrary text>","usage":{"input_tokens":"<arbitrary text>"}}}) from ever
+# landing a raw string in the emitted record (previously `// 0` /
+# `// "unknown"` copied a wrong-typed value through verbatim, which then
+# relayed to the console unredacted since usage.* isn't on the redact
+# deny-list). estimated_cost_usd/pricing_version are computed ONLY when every
+# token field for that turn is a valid, bounded number — a turn with any
+# invalid/out-of-range token field degrades to null cost too, never a cost
+# computed from partly-zeroed untrusted data; the computed cost itself is
+# then also bound-checked before being emitted (defense-in-depth against an
+# implausible pricing-registry rate).
+#
+# Prints a compact JSON object on success:
+#   {model, input_tokens, output_tokens, cache_creation_input_tokens,
+#    cache_read_input_tokens, estimated_cost_usd, pricing_version}
+# and returns 0. Prints nothing and returns 1 when the transcript is
+# missing/unreadable, jq is unavailable, or no assistant/usage line is found
+# within the tail window (e.g. transcript not Anthropic-message-shaped, or a
+# huge preceding blob pushed the last assistant entry outside the bound) —
+# the caller degrades to its own model-only/null fallback; this function
+# never invents a number. Reuses the exact pricing/cache-multiplier formula
+# usage_parse_transcript uses, via the shared USAGE_PRICE_ONE_JQ_DEF/
+# price_one() helper above (one source, two call sites), and the shared
+# usage_resolve_registry() registry-resolution helper (also one source, two
+# call sites — review finding MEDIUM/code DRY fix).
+#
+# Pricing resolution here is deliberately CACHE-ONLY (review finding
+# MEDIUM/code latency+race): this function runs once per tool call, so a
+# cold/stale local pricing cache must never trigger a blocking
+# `curl --max-time 5` (and race pricing.sh's fixed `${cache}.tmp` path)
+# per tool call. `usage_resolve_registry ... cache_only` skips the network
+# branch entirely; session.usage (usage_parse_transcript, once per session at
+# `stop`) remains the authoritative cost source and still resolves the
+# registry normally (network-eligible). A cold cache here just means
+# estimated_cost_usd/pricing_version are null for this tool event — tokens
+# still survive.
+usage_last_turn_usage() {
+  local transcript="$1" version="${2:-}"
+  if [[ -z "$transcript" ]]; then
+    usage_log_debug "usage_last_turn_usage: no usage — reason: empty transcript_path"
+    return 1
+  fi
+  if [[ ! -f "$transcript" || ! -r "$transcript" ]]; then
+    usage_log_debug "usage_last_turn_usage: no usage — reason: transcript file missing/unreadable (${transcript})"
+    return 1
+  fi
+  if ! command -v jq >/dev/null 2>&1; then
+    usage_log_debug "usage_last_turn_usage: no usage — reason: jq unavailable"
+    return 1
+  fi
+  local file_size
+  file_size="$(wc -c < "$transcript" 2>/dev/null | tr -d ' ')"
+  [[ "$file_size" =~ ^[0-9]+$ ]] || file_size=0
+  # Bounded tail-read: only the last N bytes when the file exceeds the bound,
+  # then drop the first (possibly partial/truncated-mid-JSON) line of that
+  # window. This is a cheap belt-and-suspenders trim, not a correctness
+  # requirement any more — the jq program below tolerates a malformed line
+  # anywhere in the window (see the hardened-parsing note above) — but there
+  # is no reason to hand jq a line we already know is certainly truncated.
+  local tail_text
+  if [[ "$file_size" -gt "$TELEMETRY_USAGE_TAIL_BYTES" ]]; then
+    tail_text="$(tail -c "$TELEMETRY_USAGE_TAIL_BYTES" "$transcript" 2>/dev/null | tail -n +2)"
+  else
+    tail_text="$(cat "$transcript" 2>/dev/null)"
+  fi
+  if [[ -z "$tail_text" ]]; then
+    usage_log_debug "usage_last_turn_usage: no usage — reason: empty tail window (${transcript})"
+    return 1
+  fi
+  # Pricing registry — cache-only (see function header): never a network
+  # fetch per tool call. Missing/unparseable registry degrades cost/
+  # pricing_version to null but never blocks token extraction.
+  local registry has_registry
+  usage_resolve_registry registry has_registry cache_only "$transcript"
+  local jq_program
+  jq_program="${USAGE_PRICE_ONE_JQ_DEF}"'
+    def bounded_number($v):
+      if ($v | type) == "number" and $v >= $numeric_min and $v <= $numeric_max then $v else null end;
+    (
+      [inputs]
+      | map(try fromjson catch null)
+      | map(select(. != null))
+      | map(select(
+          (.type? == "assistant")
+          and ((.message?.usage?) != null)
+          and ((.message.usage | type) == "object")
+        ))
+      | last
+    ) as $turn
+    | if $turn == null then empty else
+        ($turn.message.usage) as $raw
+        | (
+            (($turn.message.model?) | type) == "string"
+            and (($turn.message.model | length) <= $model_max_len)
+            and ($turn.message.model | test($model_regex))
+          ) as $model_valid
+        | (if $model_valid then $turn.message.model else "unknown" end) as $m
+        | bounded_number($raw.input_tokens?) as $input
+        | bounded_number($raw.output_tokens?) as $output
+        | bounded_number($raw.cache_creation_input_tokens?) as $cache_creation
+        | bounded_number($raw.cache_read_input_tokens?) as $cache_read
+        | ($input != null and $output != null and $cache_creation != null and $cache_read != null) as $tokens_all_valid
+        | (if $tokens_all_valid then
+             price_one($m; {input: $input, output: $output, cache_creation: $cache_creation, cache_read: $cache_read}; $registry; $has_registry; $version)
+           else
+             {estimated_cost_usd: null, pricing_version: null}
+           end) as $price
+        | bounded_number($price.estimated_cost_usd?) as $cost_bounded
+        | {
+            model: $m,
+            input_tokens: $input,
+            output_tokens: $output,
+            cache_creation_input_tokens: $cache_creation,
+            cache_read_input_tokens: $cache_read,
+            estimated_cost_usd: $cost_bounded,
+            pricing_version: $price.pricing_version
+          }
+      end
+  '
+  local out
+  out="$(jq -R -n -c \
+    --argjson registry "$registry" \
+    --argjson has_registry "$has_registry" \
+    --arg version "$version" \
+    --arg model_regex "$USAGE_MODEL_REGEX" \
+    --argjson model_max_len "$USAGE_MODEL_MAX_LEN" \
+    --argjson numeric_min "$USAGE_NUMERIC_MIN" \
+    --argjson numeric_max "$USAGE_NUMERIC_MAX" \
+    "$jq_program" <<<"$tail_text" 2>/dev/null)"
+  if [[ -z "$out" ]]; then
+    usage_log_debug "usage_last_turn_usage: no usage — reason: no assistant/usage entry in tail window (${transcript})"
+    return 1
+  fi
+  printf '%s\n' "$out"
+}

package/context/scripts/telemetry/lib/usage_model_guard.sh ADDED Viewed

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+# usage_model_guard.sh — shared usage.model/usage.numeric validation constants
+# (#568 slice 1 security review, HIGH + LOW findings).
+#
+# HIGH: a charset-only check (e.g. `^[A-Za-z0-9._:-]+$`, <=128 chars) is a
+# STRICT SUPERSET of common secret/credential shapes -- an Anthropic API key
+# (`sk-ant-api03-...`), an AWS access key (`AKIA...`), or a JWT (`eyJ...`,
+# often with an embedded email) all satisfy that charset and would pass
+# through verbatim into usage.model. A model id must instead match a small,
+# known VENDOR-PREFIX ALLOWLIST (case-insensitive) and stay under a much
+# tighter length bound. Sourced by BOTH usage.sh (the primary extractor,
+# usage_last_turn_usage) and transport.sh (the defense-in-depth backstop,
+# console_telemetry_sanitize_usage) so the pattern is defined in exactly ONE
+# place -- never hand-copied into two jq programs that could drift apart.
+#
+# LOW: a `type=="number"` check alone accepts negative and absurd-magnitude
+# values (e.g. -1, 1e308) as if they were real token counts / costs. Every
+# numeric usage field must additionally satisfy a sane [min, max] bound.
+# Same shared-constant discipline: defined once, used by both usage.sh and
+# transport.sh.
+USAGE_MODEL_MAX_LEN=40
+# jq regex source. IMPORTANT: matched CASE-SENSITIVELY (plain `test($regex)`,
+# NO "i" flag) at both call sites — the suffix is `[a-z0-9]` (lowercase+digit)
+# ON PURPOSE, so an uppercase run (AWS `AKIA...`, mixed-case JWT `eyJ...`)
+# cannot appear even when a secret is prefixed with a vendor token.
+#
+# The suffix is also STRUCTURED like a real version string, not free text: a
+# known vendor prefix, then zero or more `[-._:]`-separated tokens each at most
+# 16 chars. This closes the earlier "prefix-only allowlist" gap where a secret
+# wearing a vendor prefix (`claude-UPPERCASE-REJECTED`, or a long lowercase
+# token) still passed a bare `[a-z0-9._:-]*` suffix. Legitimate model ids this
+# keeps matching: claude-opus-4-8, claude-3-5-sonnet-20241022, claude-fable-5,
+# gpt-5, gpt-4o-2024-08-06, o1, gemini-2.0, glm-5.2. Anything else → "unknown".
+#
+# Residual (accepted, tracked): a caller that FULLY controls the transcript
+# could still pack <=40 chars of lowercase dash-separated text into .model.
+# That requires complete transcript forgery (already in-session code exec),
+# .model is display-only, and the transport backstop is defense-in-depth.
+USAGE_MODEL_REGEX='^(claude|gpt|o[0-9]|gemini|glm|llama|mistral|deepseek|qwen|grok|command|codestral)([._:-][a-z0-9]{1,16})*$'
+USAGE_NUMERIC_MIN=0
+USAGE_NUMERIC_MAX=1000000000000