bigquery-agent-analytics 0.5.1__tar.gz → 0.5.2__tar.gz

{bigquery_agent_analytics-0.5.1 → bigquery_agent_analytics-0.5.2}/CHANGELOG.md

@@ -7,6 +7,401 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.5.2] - 2026-09-05
+
+### Release highlights
+
+An EvalBench-centred wheel update and a reorganised evaluation API. The
+wheel gains the versioned EvalBench import pipeline — immutable
+`(job_id, import_version)` snapshots with the W0.4 failed-session contract,
+the version-pinned `evalbench_failed_sessions` view, `bq-agent-sdk
+evalbench-score`, and the native `agent_events` snapshot writer that needs
+no EvalBench tables at all — plus the failure taxonomy frozen at G1 v0.1.0
+(`failure_taxonomy`) and its span-level localisation layer
+(`span_taxonomy`, persisted on the native snapshot with
+`--span-labels-table`). Evaluation is unified behind `PerformanceEvaluator`
+with compatibility aliases for every existing evaluator name (#123), and
+the canonical rubrics gain metric factories and an opt-in policy-compliance
+scorecard (#91). Around the wheel: `scripts/skill_evolution.py` grows
+host-integration hooks and auditable patch provenance (#395, #477) with a
+scheduled Cloud Run Job wrapper (#472), and the AgentForensics Week 0 gates
+are frozen and the preregistration sealed (#435) — **the six-week clock has
+still not started**.
+
+### Added
+
+- **Canonical metric factories and opt-in policy scorecard (#91)** — four
+  public factories exported from the SDK and `evaluation_rubrics`:
+  `response_usefulness_metric`, `task_grounding_metric`,
+  `policy_compliance_metric`, and `three_pillar_scorecard_metrics`.
+  Usefulness and grounding come from the same canonical interpreter
+  `quality_report` uses (with `scope_context` / `has_scope` forwarded, so
+  scope adds `declined` to usefulness); the new opt-in `policy_compliance`
+  rubric (`compliant` / `violation`) covers confidentiality, data
+  minimization, and requested redaction or masking across international
+  PII identifiers — a model-judged checklist, not legal certification.
+- **Span-level G1 labels persisted on the native snapshot (#469, parent
+  #435)** — the existing `bq-agent-sdk evalbench-native-import` command
+  gains a thin `--span-labels-table` flag (no new command family): when
+  set, the #466 localization layer's labels for every failed session are
+  published as BQAA-owned rows via
+  `NativeAgentEventsRun.to_span_label_rows` / `_publish_span_labels`,
+  reusing `label_native_run` under the frozen
+  `EvalScorePolicy({"goal_completion": 1.0})` gate — resolved as the ONE
+  effective policy the manifest `view_policy` and the failed-sessions
+  view also record, so the session denominator and the span rows can
+  never disagree. Opting in is durable: the dataset's
+  `evalbench_span_bindings` registry (fixed name, one row per job)
+  records the span table, the resolved policy, and the synchronized
+  manifest `generation_id`, and every later native publish of a bound
+  job re-resolves the same policy and re-synchronizes the span rows — or
+  fails closed before the denominator advances — so an ordinary call can
+  neither drop the committed gate nor leave stale span rows behind. Rows
+  are keyed by the `(job_id, import_version)` pin plus the exact
+  manifest `generation_id` they were synchronized under (checked, with
+  the canonical `view_policy`, inside the lock-serialized replace
+  transaction that also upserts the binding), and a companion
+  `<span-labels-table>_pinned` view is pinned to that exact generation
+  with a rendered latest-generation guard, so any base/span skew (for
+  example a span sync that fails after a changed-source `replace`
+  committed) yields an empty view instead of stale labels joined onto
+  the new session snapshot. The widget-stock anchor
+  holds: `eval_id` `7e352c34` / `span_id` `b7ad6b7169203331` /
+  `target_kind` `gap_after_span` with the three frozen G1 names
+  (`task/planning`, `finalization`, `tool blockers`); session-level
+  `failed_sessions` + G1 stays the denominator. Publication fails closed
+  without a real `span_id` (no synthetic span identifiers), identity
+  stays the frozen first-8 `eval_id` with the full session id on
+  collision, **the six-week clock has still NOT started**, tests are
+  offline fixture tests only, and the `evalbench-import` adapter (#97)
+  stays. The registry boundary is hardened end to end: the publish-side
+  re-validation is driven by private structured binding state rendered
+  into two fixed, parameterized predicates — `materialize` exposes no
+  SQL-string hook, and any non-structured argument is rejected before
+  anything runs — the unchanged-path policy recommit runs as its own
+  import-lock-claiming transaction so it serializes against the
+  span-binding transaction instead of racing it from a mutually stale
+  snapshot, and the span sync rejects, before any row or binding DML, a
+  span table already bound to another job, so a losing concurrent job
+  stays unbound and simply retries with its own table.
+- **Span-G1 e2e team demo (#466, parent #435)** — new
+  `examples/evalbench_span_g1_e2e.sh --fixture` (speaker notes
+  `examples/evalbench_span_g1_e2e.md`, offline tests
+  `tests/test_evalbench_span_g1_e2e.py`) narrates the `span_taxonomy`
+  library below on the widget-stock silence session `7e352c34` as a
+  six-act presenter story: session-level `failed_sessions` + G1 stays
+  the denominator, and span-level G1 localizes all three tripped frozen
+  categories onto the real native `AGENT_STARTING` span
+  `b7ad6b7169203331` (`target_kind="gap_after_span"`), shown as
+  `label_native_run` / `label_failed_session_spans` sample JSON — no new
+  CLI, no EvalBench source tables, no live BigQuery, `--fixture` argv
+  only (anything else exits 2), and **the six-week clock has still NOT
+  started**. Consolidates the native-import setup from #465 into these
+  speaker notes and the frozen-category JSON from #462 into the existing
+  `evalbench_mvp_e2e` walkthrough.
+- **EvalBench CLI discoverability (#435)** — `bq-agent-sdk --help` lists
+  `evalbench-import`, `evalbench-failed-sessions`, and `evalbench-score`;
+  the installed `bq-agent-sdk` console script (`pyproject.toml`
+  `[project.scripts]`) is the surface after `pip install`. Tests pin the
+  help listing, the command registry, and the console-script mapping.
+- **Week 0 preregistration sealed (#435, PR #473)** — adopts the v4
+  freeze-candidate as the sealed plan of record
+  (`docs/week0_preregistration.md` and
+  `examples/fixtures/week0_real_preregistration.json`, `sealed: true`,
+  `freeze_candidate: false`, `clock_started: false`, sealed 2026-09-02 by
+  the D4 consumer). Fills the three numbers the freeze-candidate still
+  left as placeholders: the value gate is **≥50%** of completed,
+  non-investigator-adjudicated counterfactual investigations (applied to
+  completed investigations only if volume slips); the noisy-small-n
+  localization rule **fails** the gate when the point estimate clears but
+  the 95% CI spans zero (no localization, no uplift claim, week-6 judgment
+  may not override); and there is **no separate absolute hit@1 floor**
+  (the paired CI lower bound >0 and +10pp point uplift stay the hit@1
+  gates). Every other floor is unchanged from v4, partner / D4 / G1 stay
+  frozen, `tests/test_week0_real_freeze.py` pins the sealed values, and
+  **the six-week clock has still NOT started** — it starts only when the
+  first Week 1 snapshot job is kicked.
+- **Span-level G1 taxonomy on `span_id` (#466, parent #435)** — new
+  `bigquery_agent_analytics.span_taxonomy` localizes the G1-frozen failure
+  categories of a *failed* session onto the span where the failure is
+  observable, as `(trace_id, span_id, failure_category, evidence,
+  confidence)` rows (`SpanFailureLabel.as_tuple()`).
+  `label_failed_session_spans` takes one session's native
+  `agent_events`-shaped rows plus the landed three-flag verdict;
+  `label_native_run` covers every failed session of a
+  `NativeAgentEventsRun` offline. Session-level `failed_sessions` + G1
+  stays the denominator (this layer never classifies, only localizes),
+  the taxonomy stays frozen at v0.1.0 — only the three mechanically
+  emittable frozen names (`task/planning`, `finalization`, `tool
+  blockers`) are accepted at construction; the other five frozen names
+  and any non-frozen string are rejected — and `eval_id` keeps the frozen
+  first-8-with-full-id-on-collision identity so span labels join the
+  session-level contract. No synthetic span identifiers: the silence case
+  targets the last existing span with `target_kind="gap_after_span"`
+  (the widget-stock session `7e352c34` localizes to its `AGENT_STARTING`
+  span with evidence that no `TOOL_STARTING` / `check_inventory` /
+  `AGENT_COMPLETED` followed), a raw `status=ERROR` row is targeted
+  directly by `tool blockers` (end-of-trace claims stay anchored to the
+  last real span, so mid-stream errors never yield false silence
+  evidence), and rows without a `span_id` fail closed. Missing-tool
+  evidence comes only from the completed sibling that asked the same
+  prompt, never from a run-wide pool, and labels carry no turn index (the
+  #429 conversation coordinate has no importable package mapping yet, and
+  a lookalike ordinal would fork it). Pure and deterministic — no
+  BigQuery, no LLM judge, offline fixture tests only, and **the six-week
+  clock has still NOT started**.
+- **Native `agent_events` snapshot writer — the EvalBench-adapter exit ramp
+  (#463, parent #435)** — new
+  `bigquery_agent_analytics.native_events.NativeAgentEventsRun` (and thin
+  CLI `bq-agent-sdk evalbench-native-import`) starts from production ADK
+  `agent_events` rows and publishes the same BQAA-owned contract
+  `EvalBenchRun.materialize` produces — an immutable
+  `(job_id, import_version)` snapshot (events + deterministic
+  `goal_completion` scores + manifest with the `view_policy` pin) plus the
+  `failed_sessions` view pinned to the latest successful publication —
+  with **no EvalBench `configs`/`results`/`scores` tables anywhere in the
+  path**. Identity stays joinable (scenario id = first eight characters of
+  the ADK session id, full id on collision; event/score rows keep the real
+  `session_id`), scores are derived from the session alone (1.0 iff
+  `AGENT_COMPLETED` — *completed*, not *passed*; only the
+  `EvalScorePolicy` gate decides passed), and failed sessions carry the
+  frozen G1 v0.1.0 names — the widget-stock silence session `7e352c34`
+  yields `task/planning`, `finalization`, `tool blockers`. The
+  `evalbench-import` adapter (#97) stays as an optional on-ramp, the
+  production `agent_events` table is never written (reserved-name guard),
+  and **the six-week clock has still NOT started**. Offline fixture tests
+  only; no live BigQuery.
+- **REAL Week 0 freeze: partner, D4 boundary, G1 taxonomy v0.1 (#435)** —
+  the Week 0 gates are frozen for real, distinct from the EXAMPLE pack
+  below (which stays illustrative). `docs/week0_partner.md` records the
+  real pilot partner — Google Cloud BigQuery Agent Analytics (this SDK),
+  piloting the ADK `support_agent` traces as EvalBench job
+  `mvp-e2e-real-traces`; SANA-adjacent, not a SANA fork and not a named
+  collaboration with SANA authors, not duplicating LakeQA/KramaBench.
+  `docs/week0_d4_memo.md` lands the fail-closed D4 boundary for exactly
+  this pilot's datasets with one named report consumer (Hai-Yuan Cao) and
+  a text-only grants policy (no IAM API calls); fixtures/synthetic runs
+  validate ingestion, taxonomy mechanics, and stability ONLY and never
+  produce a Part II funding recommendation. `docs/week0_g1_taxonomy.md`
+  documents the G1 freeze (below) and `docs/week0_preregistration.md`
+  copies the v4 floors as the freeze-candidate. Machine-readable copies in
+  `examples/fixtures/week0_real_*.json` (`example: false`,
+  `clock_started: false`), guarded by `tests/test_week0_real_freeze.py`.
+  **The six-week clock has NOT started**: it starts only when the first
+  Week 1 snapshot job is kicked, not at merge of this freeze.
+- **EXAMPLE Week 0 scenario pack (#435 slice 10)** — new
+  `examples/evalbench_week0_full_idea.md` / `.sh --fixture` walk all five
+  Week 0 human gates of `docs/agentforensics_mvp_plan.md` (partner + SANA
+  relationship, runtime + route, pilot-benchmark rubric, D4 boundary memo,
+  preregistration) as one concrete story on the widget-stock failed session,
+  ending in the mechanical `taxonomy_categories` row and an EXAMPLE mapping
+  onto SANA-seeded names in `examples/fixtures/week0_example_*.json`.
+  Offline only (no BigQuery, no network) and entirely illustrative: not a
+  freeze, `g1_frozen` stays false, the six-week clock has not started, no
+  real partner is named, and `src/` is untouched — Week 0 remains
+  human-gated.
+- **Failed-session rows carry scaffold taxonomy categories (#435 slice 9)**
+  — `failed_sessions()` / `bq-agent-sdk evalbench-failed-sessions` now
+  attach `taxonomy_categories` to each session row: `EvalBenchSession`
+  exposes it as a property computed from the row's three mechanical flags
+  via `failure_taxonomy.categorize_failed_session`, and `to_dict()` (what
+  the CLI JSON/text output serializes) includes it as a list. All-flags-false
+  rows (`--include-passed`) emit `[]` — no invented `unknown` bucket. Pure
+  Python over the flags already returned; no extra BigQuery work, and still
+  NOT the G1 taxonomy (ids stay the unfrozen flag names).
+- **Mechanical failure-taxonomy scaffold (#435 slice 8)** — new
+  `failure_taxonomy` module: a versioned scaffold config
+  (`taxonomy_version: 0.0.0-scaffold`, `g1_frozen: false`) in the #431
+  `{"metrics": [...]}` schema shape whose one core metric carries the three
+  mechanical categories of the landed failed-session contract
+  (`process_failed` / `missing_completion` / `score_failed`), an empty D2
+  `dialects` slot (per-benchmark extension categories on the same core), and
+  a pure `categorize_failed_session()` that maps one failed-session row
+  (dict or `SessionVerdict`) to the tripped categories — no BigQuery, no
+  LLM, no network. NOT the G1 taxonomy: names are unfrozen and the six-week
+  clock has not started.
+- **EvalBench MVP e2e demo tells one session's story (#435 slice 5, #97)**
+  — the `--fixture` walkthrough of `examples/evalbench_mvp_e2e.sh` now
+  follows `support_agent` session `7e352c34` ("How many widgets are in
+  stock?", never answered; trace stops after `AGENT_STARTING`) through
+  import → its one `failed_sessions` row → score, ending on the punchline
+  `goal_completion=0.0`, instead of touring CLI flags; fixture defaults are
+  job `mvp-e2e-real-traces` / 7 scenarios, matching the `--synth` run.
+- **AgentForensics MVP plan of record (#435)** — `docs/agentforensics_mvp_plan.md`
+  lands the FINAL v4 execution plan: Week 0 pre-clock gate, the six-week MVP
+  table with preregistered exit criteria, labeler ledger, staged Part II, and
+  standing invariants. Docs only; Week-0 items stay human-gated.
+- **EvalBench MVP end-to-end demo (#435 slice 5, #97)** —
+  `examples/evalbench_mvp_e2e.sh` walks `bq-agent-sdk evalbench-import` →
+  `evalbench-failed-sessions` → `evalbench-score` in order on one
+  EvalBench job, printing a banner per step; `--fixture` (or
+  `EVALBENCH_FIXTURE=1`) prints annotated sample output for all three
+  steps without calling BigQuery so the demo is recordable and CI-safe,
+  and live mode calls the three CLIs from `BQ_AGENT_*` / `EVALBENCH_*`
+  environment variables. `--synth` (or `EVALBENCH_SYNTH=1`) is live mode
+  without an EvalBench run: step 0 runs the new
+  `examples/evalbench_synth_from_traces.py`, which folds a real BQAA
+  `agent_events` table into EvalBench-shaped `configs`/`results`/`scores`
+  tables (one scenario per trace, grouped on the full
+  `(session_id, user_id, root_agent_name)` identity taken as exact strings
+  — whitespace kept, `NULL` distinct from `""` — with percent-escaped,
+  collision-proof scenario ids when session ids are reused; prompts and
+  `LLM_RESPONSE` / `AGENT_COMPLETED` responses are the real trace text,
+  never invented; `goal_completion` = 1.0 when the trace reached
+  `AGENT_COMPLETED`, else 0.0; every source/target name is validated as a
+  plain identifier before any SQL is built), then steps 1–3 run on that job
+  with the demo's default dataset names. Walkthrough in
+  `examples/evalbench_mvp_e2e.md`; `tests/test_evalbench_mvp_e2e.py` runs
+  the fixture path and the offline `--synth` guards, and
+  `tests/test_evalbench_synth_from_traces.py` checks the synthesizer's
+  mapping in memory and round-trips its rows through
+  `EvalBenchRun.to_agent_event_rows` / `to_score_rows`. A source trace
+  that reached `AGENT_COMPLETED` with a non-tool `status=ERROR` /
+  `error_message` is written with the importer-recognized `results.error`
+  field (alongside `error_message` for provenance), so it imports as
+  `status=ERROR` and stays `process_failed` in `failed_sessions`;
+  `returncode` / `goal_completion` remain tied to completion only. No CLI
+  or Python library behavior changes; `examples/evalbench_score_gate.sh`
+  stays the CI gate.
+- **EvalBench LLM-judge scoring of one import version (#435 slice 3, #97)**
+  — new `bq-agent-sdk evalbench-score` command: a thin wrapper over the
+  existing `Client.evaluate` + `LLMAsJudge` (`correctness` |
+  `hallucination` | `sentiment`, `--threshold`, `--strict`, `--exit-code`
+  with the same exit codes and `FAIL` lines as `evaluate`) that points the
+  client at the mirror events table and scores exactly one published
+  version of `--job-id` — the latest successful import unless
+  `--import-version` pins one. `evalbench.import_sessions()` resolves the
+  version from the manifest and lists its session identities;
+  `EvalBenchImportSessions.trace_filter()` renders them as
+  `TraceFilter(experiment_id=job_id, session_ids=..., limit=...)`, so the
+  version pin reaches `Client.evaluate` without changing it. The report's
+  `details.evalbench` names the scored job, version, table, and pinned
+  session count. Reference in `docs/evalbench.md`; CI gate example in
+  `examples/evalbench_score_gate.sh`.
+- **EvalBench failed-session view and version-pinned consumer (#435, slice 2)**
+  — `EvalBenchRun.materialize()` now keeps an `evalbench_failed_sessions`
+  view in the target dataset pinned (as literals) to the job's latest
+  successful import, so SQL consumers never scan another `import_version`;
+  `failed_sessions()` / `bq-agent-sdk evalbench-failed-sessions` list the
+  failed sessions of exactly one published version, resolving the version
+  from the manifest before reading events, and each row's versioned
+  `session_id`/`trace_id` (`EvalBenchSession.trace_selector()`) drills into
+  `Client.get_session_trace` without mixing versions. `failed_sessions_sql()`
+  accepts `job_id=`/`import_version=` to render the pin as literals. The
+  gate and generation are committed manifest state: every publish mints
+  an opaque `generation_id` (caller-supplied `imported_at` may repeat and
+  is not the generation) and records the score policy as `view_policy`,
+  a same-version `unchanged` re-import that adds, changes, or drops the
+  policy commits it to the row it found under a new generation, and the
+  view is always rendered from the latest manifest row. The view's pin
+  comment names the generation and the rendered policy, and ownership is
+  proven by re-rendering the definition from the committed manifest row
+  and comparing byte-for-byte (a copied pin over other SQL, an edited or
+  reformatted managed view, or a canonical rendering of the current
+  generation under a gate the manifest does not record, is refused, never
+  replaced). The view is written with create-if-absent plus
+  ETag-conditional replace after re-reading the view and latest manifest,
+  so concurrent imports cannot overwrite a newer pin or attach an older
+  caller's policy to a newer generation.
+- **EvalBench versioned snapshots and failed-session contract (#435, slice 1; #97)**
+  — `EvalBenchRun.materialize()` publishes one immutable, versioned import
+  of an EvalBench job into BQAA-owned tables — `evalbench_agent_events`
+  (`agent_events` columns plus `job_id`, `import_version`),
+  `evalbench_scores_imported`, and `evalbench_import_manifest` — and never
+  writes the ADK plugin's production `agent_events` table (that name is
+  rejected before any BigQuery call). Rows are loaded into per-import
+  staging tables and published by a single multi-statement transaction
+  keyed on `(job_id, import_version)`, with the manifest re-checked inside
+  the transaction and a lock sentinel so two first-time imports of the same
+  version cannot both commit. The W0.4 failed-session contract lands with
+  it: `returncode == 0` means the scenario *completed*, not *passed*;
+  non-zero or non-numeric `returncode`, usable `stderr`, and `*_error`
+  columns surface as `status = 'ERROR'` plus `error_message` on the
+  terminal row, never as a clean `OK`. CLI: `bq-agent-sdk
+  evalbench-import`. Reference in `docs/evalbench.md`.
+
+### Changed
+
+- **Evaluator API unification (#123)** — `PerformanceEvaluator` is the
+  one entry point for deterministic trajectory/response checks, one-sided
+  LLM judging, and side-by-side reference comparisons; every existing
+  evaluator name remains a compatibility alias.
+  - `PerformanceEvaluator` now supports one-sided judging and side-by-side
+    comparisons, including deterministic trajectory and response checks.
+  - Deterministic metrics and shared report models live in `system_evaluator`;
+    trial aggregation lives in `multi_trial_performance_evaluator` and
+    `aggregate_grader`. Existing evaluator names remain compatibility aliases.
+  - `LLMAsJudge` keeps its factories, custom criteria API, and existing
+    `Client.evaluate` execution order: BigQuery `AI.GENERATE`, then BQML
+    `ML.GENERATE_TEXT`, then Gemini API fallback. Configured BigQuery endpoints,
+    connections, and execution/fallback metadata remain supported. The API
+    fallback uses bounded concurrency and preserves per-session failures.
+  - `PerformanceEvaluator` provides opt-in API-only judging with bounded
+    concurrency. Existing `LLMAsJudge` callers retain their BigQuery path.
+  - The deprecated BigQuery `AI.GENERATE` and `ML.GENERATE_TEXT` SQL helpers remain
+    in `evaluators` for compatibility. `PerformanceEvaluator` uses the Gemini API;
+    it does not execute those SQL templates.
+
+- **`failure_taxonomy` frozen at G1 v0.1.0 (#435)** — the scaffold era
+  ends: `taxonomy_version` is `0.1.0` and `g1_frozen` is `true`. The
+  frozen vocabulary (`FROZEN_CATEGORY_NAMES`, also exported as
+  `CORE_CATEGORY_IDS`) is the SANA-neighborhood seven plus `unknown` —
+  `task/planning`, `wrong source`, `execution/computation`,
+  `incomplete evidence`, `turn-waste`, `finalization`, `tool blockers`,
+  `unknown` — replacing the three flag ids, which live on as the mapper's
+  input contract `MECHANICAL_FLAGS`. `categorize_failed_session` now
+  returns frozen names in frozen order (`missing_completion` →
+  `finalization`, `process_failed` → `tool blockers`, `score_failed` →
+  `task/planning`; all flags false still returns `()`, never `unknown`),
+  so `EvalBenchSession.taxonomy_categories` and the
+  `evalbench-failed-sessions` CLI output carry frozen names. The config
+  accessor is renamed `taxonomy_config()`; `scaffold_taxonomy_config()`
+  remains as a compatibility wrapper returning the same frozen config.
+  Assignment stays mechanical until the labeler study; the six-week clock
+  has not started.
+
+### Skill evolution (repo side, not in the wheel)
+
+- **Auditable skill-evolution host contract (#397)** —
+  `scripts/skill_evolution.py` now declares its supported integration surface
+  with `__all__` and documents the additive quality-report session contract;
+  incompatible session-schema changes must be recorded here. The host analyst
+  contract also marks user corrections as hypotheses that require tool
+  verification. Evolution patch artifacts now include `source: builtin|host`
+  for every quality-gated patch, so runs using `error_analyst_fn` can be audited
+  without changing the existing `collect_patches() -> list[str]` API.
+- **Skill-evolution host integration hooks (#395)** — `scripts/skill_evolution.py`
+  gains three additive seams so a host can adopt the engine instead of
+  forking it: `error_analyst_fn` (a host-supplied analyst for failure
+  trajectories; success trajectories keep the built-in one),
+  `incumbent_score` (reuse an already-measured base-skill score instead of
+  re-measuring), and richer trajectory context for the analyst. Existing
+  callers are unchanged.
+
+### Deploy (repo side, not in the wheel)
+
+- **Scheduled skill-evolution Cloud Run Job (#472)** —
+  `deploy/skill_evolution_job/` wraps `scripts/skill_evolution.py` in an
+  agentic orchestrator (ADK agent + tools) that any agent repo can run on a
+  weekly Cloud Scheduler cron: quality report from BigQuery `agent_events`
+  → bottleneck analysis → candidate evolution → publish gate → GitHub PR.
+  Ships the job package, `Dockerfile`, `deploy.sh` (job + scheduler +
+  least-privilege IAM), an example agent registry, offline tests, and the
+  user guide `docs/guides/scheduled-skill-evolution.md`.
+
+### Examples
+
+- **Live ADK+BQAA observe trace → derived OKF adapter (#474)** —
+  `examples/okf_bqaa_adapter/`: a real ADK observe agent writes
+  retrieve/receipt-shaped rows through `BigQueryAgentAnalyticsPlugin`, and a
+  Python port of observe + adapt + computeIdentities (`okf-bqaa-adapter:v0`)
+  derives identities from the committed live fixtures with fail-closed
+  `lookup(context_ref)`. Derived demo only; no catalog or Dataplex writes.
+
+### Infrastructure
+
+- `dashboard_v2` dependency bump: browserslist 4.28.2 → 4.28.9 (#476).
+
 ## [0.5.1] - 2026-08-29
 
 ### Release highlights

{bigquery_agent_analytics-0.5.1 → bigquery_agent_analytics-0.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: bigquery-agent-analytics
-Version: 0.5.1
+Version: 0.5.2
 Summary: SDK for analyzing and evaluating agent traces stored in BigQuery.
 Author: Google LLC
 License-Expression: Apache-2.0
@@ -83,10 +83,9 @@ regressions — all through BigQuery SQL or Python.
 - Observability dashboards (Looker Studio, SQL, and BigFrames)
 
 **Evaluation**
-- Code-based metrics (latency, turn count, error rate, token efficiency, cost)
-- LLM-as-Judge scoring (correctness, hallucination, sentiment)
-- Trajectory matching (exact, in-order, any-order)
-- Multi-trial evaluation with pass@k / pass^k
+- System metrics (latency, turn count, tool call error rate, token efficiency, time to first token, cost)
+- Performance Metrics (correctness, hallucination, sentiment, efficiency, etc)
+- Multi-trial system and performance metrics
 - Grader composition (weighted, binary, majority strategies)
 - Eval suite lifecycle management with graduation and saturation detection
 - Static quality validation (ambiguous tasks, class imbalance, suspicious thresholds)
@@ -334,10 +333,10 @@ src/bigquery_agent_analytics/
 │   └── formatter.py               # Output formatting (json/text/table)
 │
 ├── Evaluation
-│   ├── evaluators.py              # SystemEvaluator + LLMAsJudge
-│   ├── trace_evaluator.py         # Trajectory matching & replay
-│   ├── multi_trial.py             # Multi-trial runner + pass@k
-│   ├── grader_pipeline.py         # Grader composition pipeline
+│   ├── system_evaluator.py        # SystemEvaluator
+│   ├── performance_evaluator.py   # PerformanceEvaluator
+│   ├── multi_trial_performance_evaluator.py # MultiTrialPerformanceEvaluator
+│   └── aggregate_grader.py        # AggregateGrader
 │   ├── eval_suite.py              # Eval suite lifecycle management
 │   └── eval_validator.py          # Static validation checks
 │

{bigquery_agent_analytics-0.5.1 → bigquery_agent_analytics-0.5.2}/README.md

@@ -25,10 +25,9 @@ regressions — all through BigQuery SQL or Python.
 - Observability dashboards (Looker Studio, SQL, and BigFrames)
 
 **Evaluation**
-- Code-based metrics (latency, turn count, error rate, token efficiency, cost)
-- LLM-as-Judge scoring (correctness, hallucination, sentiment)
-- Trajectory matching (exact, in-order, any-order)
-- Multi-trial evaluation with pass@k / pass^k
+- System metrics (latency, turn count, tool call error rate, token efficiency, time to first token, cost)
+- Performance Metrics (correctness, hallucination, sentiment, efficiency, etc)
+- Multi-trial system and performance metrics
 - Grader composition (weighted, binary, majority strategies)
 - Eval suite lifecycle management with graduation and saturation detection
 - Static quality validation (ambiguous tasks, class imbalance, suspicious thresholds)
@@ -276,10 +275,10 @@ src/bigquery_agent_analytics/
 │   └── formatter.py               # Output formatting (json/text/table)
 │
 ├── Evaluation
-│   ├── evaluators.py              # SystemEvaluator + LLMAsJudge
-│   ├── trace_evaluator.py         # Trajectory matching & replay
-│   ├── multi_trial.py             # Multi-trial runner + pass@k
-│   ├── grader_pipeline.py         # Grader composition pipeline
+│   ├── system_evaluator.py        # SystemEvaluator
+│   ├── performance_evaluator.py   # PerformanceEvaluator
+│   ├── multi_trial_performance_evaluator.py # MultiTrialPerformanceEvaluator
+│   └── aggregate_grader.py        # AggregateGrader
 │   ├── eval_suite.py              # Eval suite lifecycle management
 │   └── eval_validator.py          # Static validation checks
 │