bigquery-agent-analytics 0.5.4__tar.gz → 0.6.0__tar.gz

{bigquery_agent_analytics-0.5.4 → bigquery_agent_analytics-0.6.0}/CHANGELOG.md

@@ -7,6 +7,185 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-10-08
+
+### Release highlights
+
+A minor release: it adds public API, and the per-event views change what
+they select and which tables they deploy over (see Changed). In the wheel:
+span-level OpenTelemetry correlation (#312 / #510). `Span` gains
+`otel_span_id`, `otel_trace_id` and `source_event_id`; every per-event
+view selects `otel_span_id` and `otel_trace_id`; and
+`ViewManager(denied_columns=...)`, with a matching `--denied-column` flag
+on the `views` CLI, deploys views over an events table written with the
+plugin's `payload_column_denylist` (#321). Two view changes matter when
+you upgrade: the event-specific columns now come two positions later, and
+over an events table with no `attributes` column the per-event views
+deploy only with `denied_columns=["attributes"]`. Unused imports are
+dropped from 11 modules (#504). No public API is removed or renamed, and
+the wheel's dependencies are unchanged. Around the wheel: an agent memory
+demo built from `agent_events` (#512) and a BigQuery Conversational
+Analytics (BQCA) Prompt & Response Logging starter notebook and manual
+(#507) land under `examples/` and `docs/`; SDK.md gains a trace and span
+correlation guide (#510) and a `compaction_windows` limitation note
+(#502), and the EvalBench guide documents its `import_version` pin (#508);
+two deploy scripts keep IAM-retry stderr in a private temp file (#506);
+and tests, a hash lock and the tracing producer get small cleanups (#505,
+#509, #513).
+
+### Added
+
+- **Span-level OpenTelemetry correlation (#312 / #510)** — `Span` gains
+  `otel_span_id`, `otel_trace_id` and `source_event_id` after its
+  existing fields, so positional construction is unchanged.
+  `Span.from_bigquery_row()` reads them from same-named columns, or else
+  from `attributes.otel` and `attributes.adk` (or the legacy
+  `attributes.source_event_id`), ignoring non-string values in
+  `attributes`; they stay `None` on rows without those columns or keys,
+  and the span tree is still built from `span_id` / `parent_span_id`.
+  Every per-event view also projects
+  `otel_span_id` and `otel_trace_id`, which are `NULL` unless the ADK
+  plugin runs with `BigQueryLoggerConfig(enable_otel_correlation=True)`
+  (`google-adk>=2.4.0`).
+- **`ViewManager(denied_columns=...)` (#321 / #510)** — a keyword-only
+  argument for an events table written with the plugin's
+  `payload_column_denylist`. Denying `attributes` drops the two
+  OpenTelemetry columns, and a view whose own SQL reads a denied column
+  is skipped by `create_all_views()` (with a warning) and rejected by
+  `create_view()` with `ValueError` before any query runs. With
+  `denied_columns=["attributes"]` the six per-event views that read
+  `attributes` (`LLM_REQUEST`, `LLM_RESPONSE`, `TOOL_COMPLETED`,
+  `STATE_DELTA`, `A2A_INTERACTION`, `TOOL_PAUSED`) and
+  `compaction_windows` are skipped; the other 19 per-event views deploy.
+  Only `content`, `content_parts`, `attributes` and `latency_ms` (exact
+  case) are accepted: any other name raises `ValueError`, and a single
+  string instead of a collection raises `TypeError`.
+  `bq-agent-sdk views create-all` and `views create` take the same names
+  as a repeatable `--denied-column` flag and exit 2 on an invalid name.
+
+### Changed
+
+- **Per-event views select two more columns (#312 / #510)** — every
+  per-event `CREATE OR REPLACE VIEW` statement now selects the nullable
+  `otel_span_id` and `otel_trace_id` between the standard headers and the
+  event-specific columns. Existing columns keep their names and types,
+  but the event-specific columns now come two positions later: update
+  anything that reads view columns by position, such as
+  `INSERT INTO ... SELECT *` from a view. Re-run `create_all_views()` to
+  pick up the new columns. Over an events table without an `attributes`
+  column, every per-event view now fails to deploy unless you pass
+  `ViewManager(denied_columns=["attributes"])` or
+  `bq-agent-sdk views create-all --denied-column=attributes`; with it,
+  the same 19 per-event views deploy as before and the other six are
+  skipped.
+
+### Removed
+
+- **Unused imports (#504)** — import lines only, in 11 modules; no logic,
+  signature or exported name changes. Module attributes that existed only
+  as these imports are gone: `ai_ml_integration.timedelta`,
+  `eval_validator.field`, `extractor_compilation.diagnostics.Any`,
+  `extractor_compilation.measurement.CompileResult` (import it from
+  `extractor_compilation` or the package root), `feedback.dataclass`,
+  `feedback.field`, `feedback.json`, `insights.asyncio`,
+  `insights.dataclass`, `insights.json`,
+  `ontology_materializer.ResolvedProperty` (defined in `resolved_spec`),
+  `ontology_models.Any`, `ttl_importer.XSD` (present only with `rdflib`
+  installed; use `rdflib.XSD`) and
+  `bigquery_ontology.graph_ddl_compiler.PropertyBinding` (exported from
+  `bigquery_ontology`). None of these names was exported from the
+  package or documented.
+
+### Examples and guides (repo side, not in the wheel)
+
+- **Agent memory demo (#511 / #512)** — `examples/agent_memory/` reads
+  three memory layers back from the `agent_events` rows that the ADK
+  `BigQueryAgentAnalyticsPlugin` writes, with `Client.list_traces`:
+  short-term (per-session conversations), long-term (the version history
+  of ADK `user:` state, plus entities) and reasoning (one trace per
+  invocation with its model turns, tool calls and outcome). It combines
+  them in a `get_context()` prompt block whose lines name their source
+  rows. The demo runs offline on a committed synthetic fixture, through
+  the real `Client` and a fail-closed stand-in for `bigquery.Client`, or
+  live with `--project-id` / `--dataset-id`. `analyst_agent.py` runs a
+  live ADK data-analyst agent (`gemini-3.8-flash`) over
+  `bigquery-public-data.thelook_ecommerce`; `memory_consolidation.py`
+  extracts facts and entities from each logged user message with
+  `AI.GENERATE` and embeds it with `AI.EMBED`, and recall ranks similar
+  past tasks with `ML.DISTANCE`. The recorded run of 2026-10-07 (36
+  sessions with memory and 6 without, 1,606 logged rows), a static web
+  view and a narrated video are committed. Its tests in
+  `tests/examples/test_agent_memory_*.py` (182 cases) run without network
+  or credentials.
+- **BQCA Prompt & Response Logging starter notebook and manual (#507)** —
+  `examples/bqca_prompt_response_logging_customer_notebook.ipynb` and
+  `docs/guides/bqca-prompt-response-logging-manual.md` evaluate BigQuery
+  Conversational Analytics Prompt & Response Logging tables with this SDK,
+  in eight sections: schema and Vertex AI health (`doctor()` plus an
+  `AI.GENERATE` ping), a conversation explorer, sentiment and UX triage,
+  question topics, semantic drift and golden-question coverage, golden Q&A
+  grading, FinOps, and operational health. Two non-destructive compatibility
+  views, one for the SDK and one for SQL and BI, flatten BQCA's nested
+  `AGENT_RESPONSE` JSON. The notebook's SDK adapters are notebook code, not
+  wheel changes, and its install cell installs
+  `bigquery-agent-analytics[llm]==0.5.4` when the SDK is missing or older
+  than 0.5.4. `README.md` and `examples/README.md` link both.
+
+### Documentation
+
+- **Trace and span correlation guide (#209, #220, #320 / #510)** — SDK.md
+  documents the three correlation layers, the plugin's
+  `enable_otel_correlation`, `custom_metadata_allowlist` and
+  `payload_column_denylist` options, the OpenTelemetry span each plugin
+  callback records, and two partition-pruned SQL recipes that join the
+  events table to an `otel_spans` table.
+- **`compaction_windows` partition limitation in SDK.md (#502)** — the
+  view's reference section now states the known limitation recorded in
+  0.5.4 (no event `timestamp` column, so no partition pruning, and a
+  failure on tables that set `require_partition_filter`) and points
+  time-bounded queries at the `adk_event_compactions` per-event view,
+  which is not a drop-in replacement. A test ties the note to the view.
+- **EvalBench drill-down `import_version` pin (#508)** —
+  `docs/evalbench.md` now says that `EvalBenchSession.trace_selector()`
+  also returns `import_version` and that
+  `EvalBenchImportSessions.trace_filter()` sets
+  `TraceFilter(import_version=...)`. The pin is redundant for imports
+  converted from EvalBench source tables, but it is what isolates one
+  version of a native `agent_events` import (#463).
+
+### Security (repo side, not in the wheel)
+
+- **IAM-retry stderr goes to a private temp file (#506)** — `_retry_iam` in
+  `deploy/skill_evolution_job/deploy.sh` and
+  `examples/context_graph/periodic_materialization/deploy_cloud_run_job.sh`
+  wrote each IAM grant's stderr to `/tmp/_iam_err.$$`, a name another local
+  user can predict: the redirect truncated the target of a symlink planted
+  there, and the file was world-readable under a 022 umask. Both scripts now
+  capture it in a `mktemp` file (mode 0600, random name) that is removed on
+  every return path; retries are unchanged.
+
+### CI, tests and build locks (repo side, not in the wheel)
+
+- **Compiled-only deploy-script test stays offline (#505)** —
+  `test_compiled_only_now_accepted_by_validator` shadows `python3`, `bq`
+  and `gcloud` on the subprocess `PATH`, so `deploy_cloud_run_job.sh` no
+  longer pip-installs from PyPI or calls BigQuery during the unit test,
+  and the test now asserts that the script reached its first deploy
+  step. The deploy script is unchanged.
+- **Browser smoke readiness budget (#513)** — `browser_smoke.sh` reads
+  its readiness budget from `SMOKE_READY_POLLS` (default 20 polls, 0.5 s
+  apart, so the real check is unchanged) and rejects an invalid value
+  before it starts a server or browser. The self-test's
+  alive-but-unready fixture uses 4 polls, about 8 s less per self-test
+  run.
+- **Dead private helpers and a drifted hash lock (#509)** — removes the
+  tracing producer's unused OTLP writer helper `_str` (under
+  `producers/`, which ships on `tracing-v*` tags; no tracing release
+  includes it yet) and an unused test helper, with no behavior change,
+  and regenerates `deploy/otlp_receiver/requirements.lock` for upstream
+  PyPI drift (newer pins such as `google-cloud-bigquery` 3.46.1,
+  `google-auth` 2.60.0, `cryptography` 50.0.2 and OpenTelemetry 1.45.1).
+
 ## [0.5.4] - 2026-09-29
 
 ### Release highlights

{bigquery_agent_analytics-0.5.4 → bigquery_agent_analytics-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: bigquery-agent-analytics
-Version: 0.5.4
+Version: 0.6.0
 Summary: SDK for analyzing and evaluating agent traces stored in BigQuery.
 Author: Google LLC
 License-Expression: Apache-2.0
@@ -350,6 +350,7 @@ with a runnable ADK agent.
 | Resource | Description |
 |----------|-------------|
 | [SDK Feature Reference](SDK.md) | Complete API walkthrough with working code examples |
+| [BQCA Prompt & Response Logging Manual](docs/guides/bqca-prompt-response-logging-manual.md) | Self-hosting setup, capability matrix, customization, and self-debugging guide for the [BQCA Customer Starter Notebook](examples/bqca_prompt_response_logging_customer_notebook.ipynb) ([Open in Colab](https://colab.research.google.com/github/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/blob/main/examples/bqca_prompt_response_logging_customer_notebook.ipynb)) |
 | [Looker Studio Dashboard](dashboard/looker_studio/README.md) | Published 37-chart BQAA observability template with project/dataset/table configurator |
 | [Dashboard User Manual](dashboard/looker_studio/USER_MANUAL.md) | End-user guide to the Looker Studio dashboard: setup in three steps, page guide, sharing, troubleshooting |
 | [Agent Context Graph Codelab](docs/codelabs/periodic_materialization.md) | Extract decision traces from your agent's context graph, end to end (~35 min) |

{bigquery_agent_analytics-0.5.4 → bigquery_agent_analytics-0.6.0}/README.md

@@ -277,6 +277,7 @@ with a runnable ADK agent.
 | Resource | Description |
 |----------|-------------|
 | [SDK Feature Reference](SDK.md) | Complete API walkthrough with working code examples |
+| [BQCA Prompt & Response Logging Manual](docs/guides/bqca-prompt-response-logging-manual.md) | Self-hosting setup, capability matrix, customization, and self-debugging guide for the [BQCA Customer Starter Notebook](examples/bqca_prompt_response_logging_customer_notebook.ipynb) ([Open in Colab](https://colab.research.google.com/github/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/blob/main/examples/bqca_prompt_response_logging_customer_notebook.ipynb)) |
 | [Looker Studio Dashboard](dashboard/looker_studio/README.md) | Published 37-chart BQAA observability template with project/dataset/table configurator |
 | [Dashboard User Manual](dashboard/looker_studio/USER_MANUAL.md) | End-user guide to the Looker Studio dashboard: setup in three steps, page guide, sharing, troubleshooting |
 | [Agent Context Graph Codelab](docs/codelabs/periodic_materialization.md) | Extract decision traces from your agent's context graph, end to end (~35 min) |

{bigquery_agent_analytics-0.5.4 → bigquery_agent_analytics-0.6.0}/SDK.md

@@ -158,6 +158,209 @@ traces = client.list_traces(
 )
 ```
 
+### Trace & Span Correlation (`trace_id`, `span_id`, `attributes.otel.span_id`, `attributes.adk.source_event_id`)
+
+Rows written by the ADK `BigQueryAgentAnalyticsPlugin` carry three layers of
+correlation identifiers:
+
+| Layer | Where to read it | What it identifies |
+|---|---|---|
+| 1. Trace and execution tree | `trace_id`, `span_id`, `parent_span_id` (columns and `Span` fields) | `trace_id` is the OpenTelemetry trace ID whenever the run has an active span, so it matches Cloud Trace. `span_id` / `parent_span_id` are the plugin's own execution tree: they link rows of the events table to each other and drive `Trace` (for example `trace.render()`). They are **not** OpenTelemetry span IDs; only the root invocation row may reuse the active span ID. |
+| 2. OpenTelemetry span | `attributes.otel.span_id` / `attributes.otel.trace_id`; the `otel_span_id` / `otel_trace_id` [view columns](#otel-correlation-columns-and-denied_columns) and `Span` fields | The OpenTelemetry span that was active when the row was written. Recorded only when the plugin runs with `enable_otel_correlation=True`. A best-effort join key into Cloud Trace or an `otel_spans` table, not a foreign key: spans dropped by head sampling are never exported. |
+| 3. Source event | `attributes.adk.source_event_id` (and the legacy `attributes.source_event_id` on `AGENT_RESPONSE` rows); the `Span.source_event_id` field | The ADK `Event.id` behind a row written from the event stream. ADK 2.0 `invoke_node` spans list the IDs of the events their node yielded in the span attribute `gcp.vertex.agent.associated_event_ids`. |
+
+#### Producer configuration
+
+Three `BigQueryLoggerConfig` options of the ADK plugin control these layers.
+All three require `google-adk>=2.4.0`; their defaults keep the earlier
+behavior.
+
+| Option | Default | Effect |
+|---|---|---|
+| `enable_otel_correlation` | `False` | When `True`, each row records the active OpenTelemetry span context in `attributes.otel.span_id` (16 hex digits) and `attributes.otel.trace_id` (32 hex digits). A row written while no valid span is active gets no `attributes.otel`. The plugin does not create spans of its own. |
+| `custom_metadata_allowlist` | `None` | Keys of `Event.custom_metadata` to copy into `attributes.custom_metadata`: exact keys, or prefixes ending in `*` (for example `"a2a:*"`). `None` or empty keeps only the built-in `a2a:*` capture. |
+| `payload_column_denylist` | `None` | Payload columns to leave out of the events table. Only `content`, `content_parts`, `attributes` and `latency_ms` are accepted; any other name raises `ValueError`. Denying `attributes` removes layers 2 and 3 (`attributes.otel`, `attributes.adk`) as well as `attributes.custom_metadata`, and combining it with a non-empty `custom_metadata_allowlist` raises `ValueError`. |
+
+```python
+from google.adk.plugins.bigquery_agent_analytics_plugin import BigQueryAgentAnalyticsPlugin
+from google.adk.plugins.bigquery_agent_analytics_plugin import BigQueryLoggerConfig
+
+plugin = BigQueryAgentAnalyticsPlugin(
+    project_id="my-project",
+    dataset_id="my_dataset",
+    config=BigQueryLoggerConfig(enable_otel_correlation=True),
+)
+```
+
+Span-level joins also need the spans themselves, exported to Cloud Trace or to
+an `otel_spans` table (for example by the
+[OTLP receiver](docs/otlp_receiver_design.md)).
+
+#### Reading correlation IDs in the SDK
+
+`Span` exposes layers 2 and 3 as `otel_span_id`, `otel_trace_id` and
+`source_event_id`. `Span.from_bigquery_row()` takes them from same-named
+columns when the row has them (for example a row read from a per-event view)
+and otherwise from `attributes`. They are `None` on rows that do not record
+them, such as rows written with `enable_otel_correlation=False` or by an older
+plugin. The span tree is still built from `span_id` / `parent_span_id` alone.
+
+```python
+trace = client.get_trace("trace-abc-123")
+for span in trace.spans:
+    print(span.event_type, span.span_id, span.otel_span_id, span.source_event_id)
+```
+
+#### Which OpenTelemetry span a row records
+
+`attributes.otel.span_id` is the span that was active when the plugin callback
+ran. The table describes google-adk 2.11, where the plugin's 14 callbacks fall
+into six groups. Older releases write fewer event types:
+`on_agent_error_callback` and `on_run_error_callback` (`AGENT_ERROR`,
+`INVOCATION_ERROR`) arrived in 2.5.0, `AGENT_RESPONSE` rows from
+`after_tool_callback` (`final_response_tool_names`) in 2.6.0, and
+`NODE_OUTPUT` / `NODE_ERROR` in 2.7.0.
+
+| Plugin callback(s) | `event_type`(s) | Active OpenTelemetry span | Fidelity |
+|---|---|---|---|
+| `on_user_message_callback` | `USER_MESSAGE_RECEIVED`, `HITL_*_REQUEST_COMPLETED`, resume-side `TOOL_COMPLETED` | The runner's root span: `invocation` under ADK telemetry schema v1 (the default), `invoke_workflow {root agent name}` under schema v2 (the default on Agent Engine; `ADK_TELEMETRY_SCHEMA_VERSION_OPT_IN` pins either). Under schema v2 a `Workflow` root opens no runner span, so the row records the caller's span, if any. | Exact, except for a `Workflow` root under schema v2; the row's `span_id` may equal it too |
+| `before_run_callback`, `after_run_callback`, `on_run_error_callback` | `INVOCATION_STARTING`, `INVOCATION_COMPLETED`, `INVOCATION_ERROR` | The runner's root span: `invocation` under ADK telemetry schema v1 (the default), `invoke_workflow {root agent name}` under schema v2 (the default on Agent Engine; `ADK_TELEMETRY_SCHEMA_VERSION_OPT_IN` pins either). Under schema v2 a `Workflow` root opens no runner span, so the row records the caller's span, if any. | Exact, except for a `Workflow` root under schema v2; the row's `span_id` may equal it too |
+| `before_agent_callback`, `after_agent_callback`, `on_agent_error_callback` | `AGENT_STARTING`, `AGENT_COMPLETED`, `AGENT_ERROR` | `invoke_agent {agent_name}` | Exact; the row's `span_id` is a plugin-only ID |
+| `before_model_callback`, `after_model_callback`, `on_model_error_callback` | `LLM_REQUEST`, `LLM_RESPONSE` (streaming partials and final), `LLM_ERROR` | `call_llm` (parent of `generate_content {model}`) as of google-adk 2.11. ADK's telemetry schema migration plans to remove `call_llm`; re-check this row after that change. | Exact; one ID for the request, every partial and the final response or error |
+| `before_tool_callback`, `after_tool_callback`, `on_tool_error_callback` | `TOOL_STARTING`, `TOOL_COMPLETED`, `TOOL_ERROR`, `AGENT_RESPONSE` (tools in `final_response_tool_names`) | `execute_tool {tool_name}`; for a tool used as a node in a `Workflow` (google-adk 2.11 runs these callbacks there too), the node's `invoke_node {node_name}` span | Exact; distinct for each parallel tool call |
+| `on_event_callback` | `STATE_DELTA`, `NODE_OUTPUT`, `NODE_ERROR`, `AGENT_TRANSFER`, `EVENT_COMPACTION`, `AGENT_STATE_CHECKPOINT`, `TOOL_PAUSED`, `HITL_*_REQUEST`, `A2A_INTERACTION`, `AGENT_RESPONSE` | The runner's root span, as in the rows above: the runner writes these rows outside the span of the node that produced the event | Coarse; for events yielded by function nodes or other nodes that are neither agents nor workflows, join on `source_event_id` instead (second recipe below) |
+
+#### Join recipes for an `otel_spans` table
+
+Both recipes read an `otel_spans` table with the OTLP receiver's schema
+(`timestamp`, `end_timestamp`, `trace_id`, `span_id`, `span_name`,
+`span_attributes`, ...). Set the `@trace_id`, `@start_ts` and `@end_ts` query
+parameters. Both tables are partitioned on `timestamp`, so each CTE filters on
+the constant `@start_ts` / `@end_ts` bounds, which prune partitions and satisfy
+`require_partition_filter`. The ±1 hour window in the join condition refers to
+the other table, so it narrows the matches but prunes nothing.
+
+Rows from the synchronous callbacks join directly on
+`attributes.otel.span_id`. Use a `LEFT JOIN`, since unsampled spans are missing
+from `otel_spans`:
+
+```sql
+-- Direct join: a row written inside a model, tool, agent or invocation
+-- callback carries the OTel span that encloses that callback.
+WITH events AS (
+  SELECT
+    timestamp,
+    event_type,
+    agent,
+    trace_id,
+    span_id,
+    parent_span_id,
+    JSON_VALUE(attributes, '$.otel.span_id') AS otel_span_id
+  FROM `my-project.my_dataset.agent_events`
+  WHERE timestamp BETWEEN @start_ts AND @end_ts
+    AND trace_id = @trace_id
+),
+spans AS (
+  -- Constant bounds prune otel_spans partitions before the join.
+  SELECT timestamp, end_timestamp, trace_id, span_id, span_name, span_attributes
+  FROM `my-project.my_dataset.otel_spans`
+  WHERE timestamp BETWEEN @start_ts AND @end_ts
+    AND trace_id = @trace_id
+)
+SELECT
+  e.timestamp,
+  e.event_type,
+  e.agent,
+  e.span_id AS bqaa_span_id,
+  e.parent_span_id AS bqaa_parent_span_id,
+  e.otel_span_id,
+  s.span_name AS otel_span_name,
+  TIMESTAMP_DIFF(s.end_timestamp, s.timestamp, MILLISECOND) AS otel_span_ms,
+  s.span_attributes
+FROM events AS e
+-- LEFT JOIN: spans dropped by head sampling are absent from otel_spans.
+LEFT JOIN spans AS s
+  ON s.trace_id = e.trace_id
+  AND s.span_id = e.otel_span_id
+  AND s.timestamp BETWEEN TIMESTAMP_SUB(e.timestamp, INTERVAL 1 HOUR)
+                      AND TIMESTAMP_ADD(e.timestamp, INTERVAL 1 HOUR)
+ORDER BY e.timestamp;
+```
+
+Rows written by `on_event_callback` carry at most the runner's root span in
+`attributes.otel`. For an event yielded by an ADK 2.0 function node or other
+node that is neither an agent nor a workflow, match the row's source event ID
+against the `gcp.vertex.agent.associated_event_ids` attribute of that node's
+`invoke_node` span instead:
+
+```sql
+-- Event-queue join: rows written by on_event_callback carry at most the
+-- runner's root span in attributes.otel, so match their source event ID
+-- against the ADK node span that emitted the event instead.
+WITH events AS (
+  SELECT
+    timestamp,
+    event_type,
+    agent,
+    trace_id,
+    JSON_VALUE(attributes, '$.adk.source_event_id') AS source_event_id
+  FROM `my-project.my_dataset.agent_events`
+  WHERE timestamp BETWEEN @start_ts AND @end_ts
+    AND trace_id = @trace_id
+    AND JSON_VALUE(attributes, '$.adk.source_event_id') IS NOT NULL
+),
+spans AS (
+  -- Constant bounds prune otel_spans partitions before the join.
+  SELECT timestamp, trace_id, span_id, span_name, span_attributes
+  FROM `my-project.my_dataset.otel_spans`
+  WHERE timestamp BETWEEN @start_ts AND @end_ts
+    AND trace_id = @trace_id
+)
+SELECT
+  e.timestamp,
+  e.event_type,
+  e.agent,
+  e.source_event_id,
+  s.span_id AS node_otel_span_id,
+  s.span_name AS node_otel_span_name
+FROM events AS e
+JOIN spans AS s
+  ON s.trace_id = e.trace_id
+  AND s.timestamp BETWEEN TIMESTAMP_SUB(e.timestamp, INTERVAL 1 HOUR)
+                      AND TIMESTAMP_ADD(e.timestamp, INTERVAL 1 HOUR)
+  AND e.source_event_id IN UNNEST(
+    JSON_VALUE_ARRAY(
+      s.span_attributes, '$."gcp.vertex.agent.associated_event_ids"'
+    )
+  )
+ORDER BY e.timestamp;
+```
+
+These joins are best-effort:
+
+- Rows written before `enable_otel_correlation` was turned on, or by
+  `google-adk<2.4.0`, have no `attributes.otel` and never match the first
+  recipe. `attributes.adk.source_event_id` does not depend on that option.
+- The second recipe finds only events that an ADK 2.0 function node or other
+  node that is neither an agent nor a workflow yields itself; `NODE_ERROR` rows
+  and events a `Workflow` emits never match. Agents, including an
+  `LlmAgent` root, list their events on no span (`invoke_agent` spans carry no
+  `gcp.vertex.agent.associated_event_ids`), so their rows have no match;
+  narrow those by `trace_id` and time instead.
+- Before google-adk 2.8.0, a root agent that is not an `LlmAgent` (for
+  example a `SequentialAgent`) runs on the runner's classic path, where
+  `on_event_callback` rows record the span that was active where the agent
+  yielded the event, such as an `invoke_agent` span, instead of the runner's
+  root span.
+- A span that starts before `@start_ts`, such as a long root span, is
+  filtered out of `spans`. Subtract a margin from `@start_ts` in that CTE; a
+  constant expression still prunes partitions. The join condition compares
+  the span's start `timestamp` with the row's, so a row written more than an
+  hour after its span started finds no match: for runs longer than an hour,
+  widen the ±1 hour window along with that margin.
+- Never join on the top-level `span_id`: apart from the root invocation row it
+  is a plugin-only ID with no counterpart in Cloud Trace.
+
 ---
 
 ## 3. Deterministic System Metrics
@@ -1417,6 +1620,56 @@ tool completions they are **null**. `pause_orphan` is reserved for the pause
 registry and stays null until that ships — treat a null `pause_orphan` as
 "not yet determined", not as "not an orphan".
 
+#### OTel correlation columns and `denied_columns`
+
+Every per-event view also projects `otel_span_id` and `otel_trace_id`, right
+after the standard headers, from `attributes.otel` (see
+[Trace & Span Correlation](#trace--span-correlation-trace_id-span_id-attributesotelspan_id-attributesadksource_event_id)).
+Both are `NULL` on rows written without `enable_otel_correlation=True`.
+
+If the plugin writes the events table with `payload_column_denylist`, pass the
+same column names as `denied_columns` so the views read only columns that
+exist:
+
+```python
+vm = ViewManager(
+    project_id="my-project",
+    dataset_id="analytics",
+    denied_columns=["attributes"],  # same list as payload_column_denylist
+)
+created = vm.create_all_views()
+```
+
+- Denying `attributes` drops `otel_span_id` / `otel_trace_id` from every
+  per-event view.
+- A view whose own SQL reads a denied column is skipped by
+  `create_all_views()` with a warning and left out of the returned dict, and
+  `create_view()` raises `ValueError` for it without issuing a query;
+  `get_view_sql()` still returns its SQL. With `denied_columns=["attributes"]`
+  that skips the `LLM_REQUEST`, `LLM_RESPONSE`, `TOOL_COMPLETED`,
+  `STATE_DELTA`, `A2A_INTERACTION` and `TOOL_PAUSED` views and the
+  `compaction_windows` cross-event view; the other 19 per-event views deploy.
+- `denied_columns` is keyword-only and accepts only `content`,
+  `content_parts`, `attributes` and `latency_ms`, matched exactly as the plugin
+  matches them: any other name raises `ValueError`, and a bare string raises
+  `TypeError`.
+- `bq-agent-sdk views create-all` and `views create` take the same names as a
+  repeatable `--denied-column` flag and validate them the same way:
+
+  ```bash
+  bq-agent-sdk views create-all --project-id=my-project --dataset-id=analytics \
+    --denied-column=attributes
+  ```
+
+- `Client` read methods such as `get_trace()` select `content`,
+  `content_parts`, `attributes` and `latency_ms` by name, so they fail on an
+  events table written with `payload_column_denylist`; `denied_columns` applies
+  only to the views.
+
+The plugin's own views (`BigQueryLoggerConfig(create_views=True)`) handle a
+denylist differently: they drop only the derived columns that read a denied
+column instead of skipping the view.
+
 #### Cross-event views
 
 Some analytical views span several event types — pairing a `TOOL_PAUSED` with
@@ -1473,6 +1726,13 @@ uses safe multiplication/casts and `SAFE.TIMESTAMP_MICROS`, preserving
 sub-second windows such as `1.234` to `1.235` seconds. The existing
 `adk_event_compactions` per-event view remains available unchanged.
 
+Known limitation: the view exposes no event `timestamp` column, so a query
+through it reads every partition of the events table (no partition pruning)
+and fails on a table that sets `require_partition_filter`. For time-bounded
+queries, the `adk_event_compactions` per-event view keeps the `timestamp`
+header, but it is not a drop-in replacement: it does not deduplicate windows
+or skip malformed boundaries, and it has `agent` rather than `app_name`.
+
 A registry entry maps a view key to `(view_suffix, query_sql)`. `query_sql` is
 the full `SELECT` body, rendered with `str.format` (double any literal braces)
 with `{project}`, `{dataset}`, `{table}` (the events table) and
@@ -2149,6 +2409,10 @@ bq-agent-sdk categorical-views --project-id=P --dataset-id=D --prefix=adk_
 
 ```bash
 bq-agent-sdk views create-all --project-id=P --dataset-id=D --prefix=adk_
+
+# Events table written with payload_column_denylist=["attributes"]
+# (repeat --denied-column for more columns; views create takes it too)
+bq-agent-sdk views create-all --project-id=P --dataset-id=D --denied-column=attributes
 ```
 
 #### `views create` — Create a Single View

{bigquery_agent_analytics-0.5.4 → bigquery_agent_analytics-0.6.0}/dashboard/looker_studio/tools/browser_smoke.sh

@@ -45,6 +45,10 @@
 # Env: CHROME_BIN, SMOKE_PORT, SMOKE_DOCS_DIR override discovery. A pinned
 # SMOKE_PORT disables the port retry (exactly one attempt on that port),
 # so the occupied-port negative fixture keeps failing as it must.
+# SMOKE_READY_POLLS is the readiness budget per server attempt: that many
+# nonce polls, 0.5 s apart (default 20, about 10 s). The self-test lowers
+# it only for the alive-but-unready fixture, whose expected outcome is the
+# budget running out.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -74,6 +78,13 @@ fail() {
   exit 1
 }
 
+READY_POLLS="${SMOKE_READY_POLLS:-20}"
+case "$READY_POLLS" in
+  *[!0-9]* | 0*)
+    fail "SMOKE_READY_POLLS must be a positive integer, got '$READY_POLLS'"
+    ;;
+esac
+
 find_chrome() {
   if [ -n "${CHROME_BIN:-}" ]; then
     echo "$CHROME_BIN"
@@ -483,7 +494,11 @@ SHIM
   #     a readiness timeout with a LIVE child must fail immediately — a
   #     retry on a fresh port could only mask a real startup hang. The
   #     shim would serve normally on a second spawn, so the spawn counter
-  #     catches any retry.
+  #     catches any retry. The expected outcome is the budget running
+  #     out, and the code after the poll loop is the same for any budget,
+  #     so a 4-poll budget (about 2 s) replaces the default 20 (about
+  #     10 s). The shim logs its line and execs sleep within milliseconds
+  #     of the spawn, well inside 2 s.
   make_shim "$OUT_DIR/shim-alive"
   ALIVE_COUNT="$OUT_DIR/alive-spawns.txt"
   ALIVE_ERR="$OUT_DIR/fixture13-stderr.txt"
@@ -493,6 +508,7 @@ SHIM
      SMOKE_SHIM_MODE=alive-unready-first \
      SMOKE_SHIM_COUNTER="$ALIVE_COUNT" \
      SMOKE_DOCS_DIR="$FIXTURE" \
+     SMOKE_READY_POLLS=4 \
      "$SCRIPT_PATH" >/dev/null 2>"$ALIVE_ERR"; then
     fail "self-test 13 FAILED: an alive-but-unready server passed"
   fi
@@ -635,7 +651,7 @@ for ATTEMPT in $(seq 1 "$ATTEMPTS"); do
   SERVER_LOG="$OUT_DIR/server-attempt-$ATTEMPT.log"
   python3 -m http.server "$PORT" --directory "$SITE" >"$SERVER_LOG" 2>&1 &
   SERVER_PID=$!
-  for _ in $(seq 1 20); do
+  for _ in $(seq 1 "$READY_POLLS"); do
     BODY="$(curl -fsS --max-time 2 "http://127.0.0.1:$PORT/$NONCE.txt" 2>/dev/null || true)"
     if [ "$BODY" = "$NONCE" ]; then
       READY=1