bigquery-agent-analytics 0.5.3__tar.gz → 0.6.0__tar.gz

{bigquery_agent_analytics-0.5.3 → bigquery_agent_analytics-0.6.0}/.github/workflows/ci.yml

@@ -37,6 +37,15 @@ jobs:
         with:
           persist-credentials: false
 
+      - name: Reject SDLC working notes (intent.md / plan.md / spec.md)
+        run: |
+          found="$(find . -path ./.git -prune -o -type f \( -name intent.md -o -name plan.md -o -name spec.md \) -print)"
+          if [ -n "$found" ]; then
+            echo "$found"
+            echo "::error::intent.md / plan.md / spec.md must not be checked in. Keep SDLC working notes outside this repo."
+            exit 1
+          fi
+
       - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065  # v5
         with:
           python-version: "3.12"

{bigquery_agent_analytics-0.5.3 → bigquery_agent_analytics-0.6.0}/.gitignore

@@ -44,3 +44,9 @@ dashboards/grafana/.local/
 grafana/*.json
 dashboards/grafana/*.json
 dashboards/streamlit/*.json
+
+# SDLC working notes (intent / spec / plan) live outside this repo.
+# Basename patterns match at any depth; CI "Format check" also enforces this.
+intent.md
+plan.md
+spec.md

{bigquery_agent_analytics-0.5.3 → bigquery_agent_analytics-0.6.0}/CHANGELOG.md

@@ -7,6 +7,341 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-10-08
+
+### Release highlights
+
+A minor release: it adds public API, and the per-event views change what
+they select and which tables they deploy over (see Changed). In the wheel:
+span-level OpenTelemetry correlation (#312 / #510). `Span` gains
+`otel_span_id`, `otel_trace_id` and `source_event_id`; every per-event
+view selects `otel_span_id` and `otel_trace_id`; and
+`ViewManager(denied_columns=...)`, with a matching `--denied-column` flag
+on the `views` CLI, deploys views over an events table written with the
+plugin's `payload_column_denylist` (#321). Two view changes matter when
+you upgrade: the event-specific columns now come two positions later, and
+over an events table with no `attributes` column the per-event views
+deploy only with `denied_columns=["attributes"]`. Unused imports are
+dropped from 11 modules (#504). No public API is removed or renamed, and
+the wheel's dependencies are unchanged. Around the wheel: an agent memory
+demo built from `agent_events` (#512) and a BigQuery Conversational
+Analytics (BQCA) Prompt & Response Logging starter notebook and manual
+(#507) land under `examples/` and `docs/`; SDK.md gains a trace and span
+correlation guide (#510) and a `compaction_windows` limitation note
+(#502), and the EvalBench guide documents its `import_version` pin (#508);
+two deploy scripts keep IAM-retry stderr in a private temp file (#506);
+and tests, a hash lock and the tracing producer get small cleanups (#505,
+#509, #513).
+
+### Added
+
+- **Span-level OpenTelemetry correlation (#312 / #510)** — `Span` gains
+  `otel_span_id`, `otel_trace_id` and `source_event_id` after its
+  existing fields, so positional construction is unchanged.
+  `Span.from_bigquery_row()` reads them from same-named columns, or else
+  from `attributes.otel` and `attributes.adk` (or the legacy
+  `attributes.source_event_id`), ignoring non-string values in
+  `attributes`; they stay `None` on rows without those columns or keys,
+  and the span tree is still built from `span_id` / `parent_span_id`.
+  Every per-event view also projects
+  `otel_span_id` and `otel_trace_id`, which are `NULL` unless the ADK
+  plugin runs with `BigQueryLoggerConfig(enable_otel_correlation=True)`
+  (`google-adk>=2.4.0`).
+- **`ViewManager(denied_columns=...)` (#321 / #510)** — a keyword-only
+  argument for an events table written with the plugin's
+  `payload_column_denylist`. Denying `attributes` drops the two
+  OpenTelemetry columns, and a view whose own SQL reads a denied column
+  is skipped by `create_all_views()` (with a warning) and rejected by
+  `create_view()` with `ValueError` before any query runs. With
+  `denied_columns=["attributes"]` the six per-event views that read
+  `attributes` (`LLM_REQUEST`, `LLM_RESPONSE`, `TOOL_COMPLETED`,
+  `STATE_DELTA`, `A2A_INTERACTION`, `TOOL_PAUSED`) and
+  `compaction_windows` are skipped; the other 19 per-event views deploy.
+  Only `content`, `content_parts`, `attributes` and `latency_ms` (exact
+  case) are accepted: any other name raises `ValueError`, and a single
+  string instead of a collection raises `TypeError`.
+  `bq-agent-sdk views create-all` and `views create` take the same names
+  as a repeatable `--denied-column` flag and exit 2 on an invalid name.
+
+### Changed
+
+- **Per-event views select two more columns (#312 / #510)** — every
+  per-event `CREATE OR REPLACE VIEW` statement now selects the nullable
+  `otel_span_id` and `otel_trace_id` between the standard headers and the
+  event-specific columns. Existing columns keep their names and types,
+  but the event-specific columns now come two positions later: update
+  anything that reads view columns by position, such as
+  `INSERT INTO ... SELECT *` from a view. Re-run `create_all_views()` to
+  pick up the new columns. Over an events table without an `attributes`
+  column, every per-event view now fails to deploy unless you pass
+  `ViewManager(denied_columns=["attributes"])` or
+  `bq-agent-sdk views create-all --denied-column=attributes`; with it,
+  the same 19 per-event views deploy as before and the other six are
+  skipped.
+
+### Removed
+
+- **Unused imports (#504)** — import lines only, in 11 modules; no logic,
+  signature or exported name changes. Module attributes that existed only
+  as these imports are gone: `ai_ml_integration.timedelta`,
+  `eval_validator.field`, `extractor_compilation.diagnostics.Any`,
+  `extractor_compilation.measurement.CompileResult` (import it from
+  `extractor_compilation` or the package root), `feedback.dataclass`,
+  `feedback.field`, `feedback.json`, `insights.asyncio`,
+  `insights.dataclass`, `insights.json`,
+  `ontology_materializer.ResolvedProperty` (defined in `resolved_spec`),
+  `ontology_models.Any`, `ttl_importer.XSD` (present only with `rdflib`
+  installed; use `rdflib.XSD`) and
+  `bigquery_ontology.graph_ddl_compiler.PropertyBinding` (exported from
+  `bigquery_ontology`). None of these names was exported from the
+  package or documented.
+
+### Examples and guides (repo side, not in the wheel)
+
+- **Agent memory demo (#511 / #512)** — `examples/agent_memory/` reads
+  three memory layers back from the `agent_events` rows that the ADK
+  `BigQueryAgentAnalyticsPlugin` writes, with `Client.list_traces`:
+  short-term (per-session conversations), long-term (the version history
+  of ADK `user:` state, plus entities) and reasoning (one trace per
+  invocation with its model turns, tool calls and outcome). It combines
+  them in a `get_context()` prompt block whose lines name their source
+  rows. The demo runs offline on a committed synthetic fixture, through
+  the real `Client` and a fail-closed stand-in for `bigquery.Client`, or
+  live with `--project-id` / `--dataset-id`. `analyst_agent.py` runs a
+  live ADK data-analyst agent (`gemini-3.8-flash`) over
+  `bigquery-public-data.thelook_ecommerce`; `memory_consolidation.py`
+  extracts facts and entities from each logged user message with
+  `AI.GENERATE` and embeds it with `AI.EMBED`, and recall ranks similar
+  past tasks with `ML.DISTANCE`. The recorded run of 2026-10-07 (36
+  sessions with memory and 6 without, 1,606 logged rows), a static web
+  view and a narrated video are committed. Its tests in
+  `tests/examples/test_agent_memory_*.py` (182 cases) run without network
+  or credentials.
+- **BQCA Prompt & Response Logging starter notebook and manual (#507)** —
+  `examples/bqca_prompt_response_logging_customer_notebook.ipynb` and
+  `docs/guides/bqca-prompt-response-logging-manual.md` evaluate BigQuery
+  Conversational Analytics Prompt & Response Logging tables with this SDK,
+  in eight sections: schema and Vertex AI health (`doctor()` plus an
+  `AI.GENERATE` ping), a conversation explorer, sentiment and UX triage,
+  question topics, semantic drift and golden-question coverage, golden Q&A
+  grading, FinOps, and operational health. Two non-destructive compatibility
+  views, one for the SDK and one for SQL and BI, flatten BQCA's nested
+  `AGENT_RESPONSE` JSON. The notebook's SDK adapters are notebook code, not
+  wheel changes, and its install cell installs
+  `bigquery-agent-analytics[llm]==0.5.4` when the SDK is missing or older
+  than 0.5.4. `README.md` and `examples/README.md` link both.
+
+### Documentation
+
+- **Trace and span correlation guide (#209, #220, #320 / #510)** — SDK.md
+  documents the three correlation layers, the plugin's
+  `enable_otel_correlation`, `custom_metadata_allowlist` and
+  `payload_column_denylist` options, the OpenTelemetry span each plugin
+  callback records, and two partition-pruned SQL recipes that join the
+  events table to an `otel_spans` table.
+- **`compaction_windows` partition limitation in SDK.md (#502)** — the
+  view's reference section now states the known limitation recorded in
+  0.5.4 (no event `timestamp` column, so no partition pruning, and a
+  failure on tables that set `require_partition_filter`) and points
+  time-bounded queries at the `adk_event_compactions` per-event view,
+  which is not a drop-in replacement. A test ties the note to the view.
+- **EvalBench drill-down `import_version` pin (#508)** —
+  `docs/evalbench.md` now says that `EvalBenchSession.trace_selector()`
+  also returns `import_version` and that
+  `EvalBenchImportSessions.trace_filter()` sets
+  `TraceFilter(import_version=...)`. The pin is redundant for imports
+  converted from EvalBench source tables, but it is what isolates one
+  version of a native `agent_events` import (#463).
+
+### Security (repo side, not in the wheel)
+
+- **IAM-retry stderr goes to a private temp file (#506)** — `_retry_iam` in
+  `deploy/skill_evolution_job/deploy.sh` and
+  `examples/context_graph/periodic_materialization/deploy_cloud_run_job.sh`
+  wrote each IAM grant's stderr to `/tmp/_iam_err.$$`, a name another local
+  user can predict: the redirect truncated the target of a symlink planted
+  there, and the file was world-readable under a 022 umask. Both scripts now
+  capture it in a `mktemp` file (mode 0600, random name) that is removed on
+  every return path; retries are unchanged.
+
+### CI, tests and build locks (repo side, not in the wheel)
+
+- **Compiled-only deploy-script test stays offline (#505)** —
+  `test_compiled_only_now_accepted_by_validator` shadows `python3`, `bq`
+  and `gcloud` on the subprocess `PATH`, so `deploy_cloud_run_job.sh` no
+  longer pip-installs from PyPI or calls BigQuery during the unit test,
+  and the test now asserts that the script reached its first deploy
+  step. The deploy script is unchanged.
+- **Browser smoke readiness budget (#513)** — `browser_smoke.sh` reads
+  its readiness budget from `SMOKE_READY_POLLS` (default 20 polls, 0.5 s
+  apart, so the real check is unchanged) and rejects an invalid value
+  before it starts a server or browser. The self-test's
+  alive-but-unready fixture uses 4 polls, about 8 s less per self-test
+  run.
+- **Dead private helpers and a drifted hash lock (#509)** — removes the
+  tracing producer's unused OTLP writer helper `_str` (under
+  `producers/`, which ships on `tracing-v*` tags; no tracing release
+  includes it yet) and an unused test helper, with no behavior change,
+  and regenerates `deploy/otlp_receiver/requirements.lock` for upstream
+  PyPI drift (newer pins such as `google-cloud-bigquery` 3.46.1,
+  `google-auth` 2.60.0, `cryptography` 50.0.2 and OpenTelemetry 1.45.1).
+
+## [0.5.4] - 2026-09-29
+
+### Release highlights
+
+A small patch release. In the wheel: a `compaction_windows` cross-event
+view deploys beside the per-event views (#216 / #494); the
+extractor-compilation smoke test decodes its child process's pickled
+output through an allowlist, so compiled-extractor code can no longer
+run code in the parent (#492); and unreferenced private helpers and
+imports are removed (#490, #496). No public API is removed or renamed,
+and the wheel's dependencies are unchanged. Around the wheel: the
+separately released tracing producer keeps its spool, state and log
+private to the user, with compatibility notes below (#499); SDK docs
+and examples are corrected against the code (#493, #495); and CI,
+tests and hash locks are tidied (#490, #491, #497, #500).
+
+### Added
+
+- **Compaction windows view (#216 / #494)** — `ViewManager` and the
+  `views` CLI deploy `compaction_windows` (`adk_compaction_windows` by
+  default, `<view_prefix>compaction_windows` otherwise) from the
+  cross-event registry: `create_all_views()` and
+  `bq-agent-sdk views create-all` now create it after the per-event
+  views, and `bq-agent-sdk views create compaction_windows` creates it
+  alone. It returns one row per distinct compaction range for each full
+  telemetry identity (`app_name`, `user_id`, `session_id`,
+  `invocation_id`), keeps fractional epoch seconds at microsecond
+  precision, and excludes incomplete identities and invalid boundaries.
+  Per-event view SQL is unchanged. Known limitation: the view exposes no
+  event `timestamp` column, so a query through it scans the whole events
+  table (no partition pruning) and fails on a table that sets
+  `require_partition_filter`.
+
+### Security
+
+- **Smoke-test harness unpickles child output against an allowlist
+  (#492)** — `run_smoke_test_in_subprocess` decodes the child process's
+  pickled result with a restricted unpickler that admits only the SDK's
+  extraction result and model classes (`StructuredExtractionResult`,
+  `ExtractorException`, `ExtractedNode`, `ExtractedEdge`,
+  `ExtractedProperty`) plus a few inert standard-library value types
+  (`datetime`, `date`, `time`, `timedelta`, `timezone`, `Decimal`,
+  `bytearray`). A `__reduce__` gadget in untrusted compiled-extractor
+  code can no longer run in the parent process: any other global fails
+  closed and is reported as a smoke-test harness failure. An extractor
+  whose `ExtractedProperty.value` holds another type that pickle looks
+  up by name, such as a custom class instance or a `uuid.UUID`, now
+  fails the gate the same way; `str`, `int`, `float`, `bool`, `None`,
+  lists, dicts and sets are unaffected.
+
+### Removed
+
+- **Unreferenced private helpers and incidental imports (#490, #496)** —
+  #490 deletes unreferenced private helpers and unused imports from seven
+  `src/` modules (59 lines), and #496 drops two unused imports from
+  `system_evaluator.py`. Supported behavior is unchanged, but module
+  attributes that existed only as incidental imports are gone:
+  `client.CATEGORICAL_AI_GENERATE_QUERY` (import it from
+  `categorical_evaluator`), `system_evaluator.strip_markdown_fences`
+  (still exported from `utils` and `evaluators`),
+  `system_evaluator._parse_json_from_text` (a private helper that stays
+  in `utils`), `system_evaluator.udf_kernels` and
+  `_streaming_evaluation.udf_kernels` (import
+  `bigquery_agent_analytics.udf_kernels`),
+  `ontology_schema_compiler.ResolvedEntity` / `ResolvedRelationship`
+  (defined in `resolved_spec`), `insights.dc_field` (use
+  `dataclasses.field`) and `trace.functools`. The deleted private
+  helpers are `ontology_schema_compiler._compile_entity_schema`,
+  `_compile_relationship_schema` and `ontology_runtime._first_string_value`.
+  None of these names was exported from the package or documented.
+
+### Tracing producer (released separately, not in this wheel)
+
+- **Spool, state and log are private to the user (#499)** — this changes
+  `bigquery-agent-analytics-tracing` under `producers/`, which ships on
+  `tracing-vX.Y.Z` tags; no tracing release includes it yet. The spool,
+  state and log now default to a per-user
+  `<tempdir>/bqaa-agent-tracing-<user>/` directory created with mode
+  0700 (`<user>` is the numeric uid; without `os.getuid`, a sanitized
+  login name, else a per-process tag) instead of the shared
+  `/tmp/bqaa-agent-tracing/` and `/tmp/bqaa-agent-tracing.log`. Spool
+  and state files are created with mode 0600 and, where the platform
+  supports it, opened relative to the checked directory with
+  `O_NOFOLLOW`. Compatibility:
+  - Spool, state and dead-letter directories, default or set through
+    `BQAA_SPOOL_DIR` / `BQAA_STATE_DIR`, are refused if they are a
+    symlink, are owned by another user, are group- or world-writable
+    (for a directory you own, remove both write bits with
+    `chmod go-w`), or sit under a parent owned by another non-root
+    user. A shared directory such as `/tmp` itself is refused.
+  - An explicit `BQAA_LOG_FILE` must be a regular file you own or a
+    character device; `/dev/stdout`, `/dev/stderr`, `/dev/null` and
+    terminals keep working. Its directory and every physical parent
+    must be owned by root or by you, and a directory of yours must not
+    be group- or world-writable; a missing directory is created with
+    mode 0700. Symlinks leading to an existing log directory outside
+    the state directory are still followed (a log inside the state
+    directory gets the spool and state rules above), but a symlink at
+    the log file name is refused, and in a root-owned shared directory
+    such as `/tmp` a log with other hard links is refused. An existing
+    log with group or other permission bits is changed to mode 0600.
+  - Refusals fail closed. The drainer exits 1 without draining when its
+    spool directory is refused. Hooks swallow the error and return 0;
+    they try to log it, but a refused log path drops log lines silently,
+    so nothing is recorded when the refused directory also holds the log
+    (the default layout).
+  - The drainer uploads only spool entries that are regular files you
+    own; anything else is left in place and logged once per drainer run.
+  - The old `/tmp` paths are no longer the defaults and queued rows are
+    not migrated: drain or copy any rows you need from the old spool
+    before deleting it. Explicit `BQAA_*` paths keep working when they
+    pass the checks above.
+  - Where a platform lacks descriptor-relative calls or `O_NOFOLLOW`,
+    the checks run by path as a best effort; without `os.geteuid`, the
+    ownership and permission-mode checks are skipped, and tightening an
+    existing log's mode needs `os.fchmod`. None of this adds Windows
+    support: the producer still imports `fcntl`.
+  - The Claude Code plugin's vendored copy (`plugins/claude_code_dist`)
+    is unchanged; it picks this up at the next tracing release sync.
+
+### Documentation
+
+- **SDK docs and examples match the code (#493, #495)** — `SDK.md` and
+  the example snippets call `get_section()` with section titles, stop
+  reading `.key` / `.value` from `search_memory()` entries, and pass
+  `extract_response_text()` its single argument. The README counts six
+  deployment surfaces, the periodic-materialization README documents the
+  shipped `--backfill` mode, and the `sdk_feature` label lists name all
+  19 values, including `eval-performance` and the four EvalBench labels.
+  The conversational-analytics guide points at its codelab's
+  `agent_analytics_demo` dataset with its own checkpoint key, and the
+  docs index links the `gm scaffold` design page.
+
+### CI and tests (repo side, not in the wheel)
+
+- **Checked-in SDLC working notes are rejected (#490)** — six committed
+  `intent.md` / `plan.md` / `spec.md` files are deleted, `.gitignore`
+  lists those names, and the CI Format check fails if one is committed
+  again.
+- **Categorical retry tests stop sleeping (#491)** — two tests record the
+  backoff instead of waiting it out, saving about four seconds per
+  Python version in the CI matrix. Production retry code is unchanged.
+- **Browser smoke test retries its server port (#497)** — the Looker
+  Studio configurator's `browser_smoke.sh` tries up to five random
+  ports, each verified with a nonce, instead of failing on the first
+  collision. Runs pinned with `SMOKE_PORT` still make one attempt.
+
+### Build locks (repo side, not in the wheel)
+
+- **Hash locks regenerated (#500)** —
+  `deploy/otlp_receiver/pip-tools.lock`,
+  `deploy/otlp_receiver/requirements.lock` and
+  `producers/build-requirements.lock` are replaced with CI's
+  `regen-locks.sh` output to clear upstream PyPI drift. No `.in`,
+  `pyproject.toml`, Dockerfile or script changes.
+
 ## [0.5.3] - 2026-09-20
 
 ### Release highlights

{bigquery_agent_analytics-0.5.3 → bigquery_agent_analytics-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: bigquery-agent-analytics
-Version: 0.5.3
+Version: 0.6.0
 Summary: SDK for analyzing and evaluating agent traces stored in BigQuery.
 Author: Google LLC
 License-Expression: Apache-2.0
@@ -350,13 +350,14 @@ with a runnable ADK agent.
 | Resource | Description |
 |----------|-------------|
 | [SDK Feature Reference](SDK.md) | Complete API walkthrough with working code examples |
+| [BQCA Prompt & Response Logging Manual](docs/guides/bqca-prompt-response-logging-manual.md) | Self-hosting setup, capability matrix, customization, and self-debugging guide for the [BQCA Customer Starter Notebook](examples/bqca_prompt_response_logging_customer_notebook.ipynb) ([Open in Colab](https://colab.research.google.com/github/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/blob/main/examples/bqca_prompt_response_logging_customer_notebook.ipynb)) |
 | [Looker Studio Dashboard](dashboard/looker_studio/README.md) | Published 37-chart BQAA observability template with project/dataset/table configurator |
 | [Dashboard User Manual](dashboard/looker_studio/USER_MANUAL.md) | End-user guide to the Looker Studio dashboard: setup in three steps, page guide, sharing, troubleshooting |
 | [Agent Context Graph Codelab](docs/codelabs/periodic_materialization.md) | Extract decision traces from your agent's context graph, end to end (~35 min) |
 | [Scheduled Deploy Runbook](docs/guides/scheduled-context-graph-deploy.md) | Keep the context graph fresh on a Cloud Run + Cloud Scheduler cron |
 | [Design Documents](docs/README.md) | Architecture decisions and design rationale |
 | [Examples](examples/README.md) | Notebooks, SQL scripts, and demos |
-| [Deployment Guides](deploy/README.md) | Four deployment surfaces for Google Cloud |
+| [Deployment Guides](deploy/README.md) | Six deployment surfaces for Google Cloud |
 
 ## Architecture
 

{bigquery_agent_analytics-0.5.3 → bigquery_agent_analytics-0.6.0}/README.md

@@ -277,13 +277,14 @@ with a runnable ADK agent.
 | Resource | Description |
 |----------|-------------|
 | [SDK Feature Reference](SDK.md) | Complete API walkthrough with working code examples |
+| [BQCA Prompt & Response Logging Manual](docs/guides/bqca-prompt-response-logging-manual.md) | Self-hosting setup, capability matrix, customization, and self-debugging guide for the [BQCA Customer Starter Notebook](examples/bqca_prompt_response_logging_customer_notebook.ipynb) ([Open in Colab](https://colab.research.google.com/github/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/blob/main/examples/bqca_prompt_response_logging_customer_notebook.ipynb)) |
 | [Looker Studio Dashboard](dashboard/looker_studio/README.md) | Published 37-chart BQAA observability template with project/dataset/table configurator |
 | [Dashboard User Manual](dashboard/looker_studio/USER_MANUAL.md) | End-user guide to the Looker Studio dashboard: setup in three steps, page guide, sharing, troubleshooting |
 | [Agent Context Graph Codelab](docs/codelabs/periodic_materialization.md) | Extract decision traces from your agent's context graph, end to end (~35 min) |
 | [Scheduled Deploy Runbook](docs/guides/scheduled-context-graph-deploy.md) | Keep the context graph fresh on a Cloud Run + Cloud Scheduler cron |
 | [Design Documents](docs/README.md) | Architecture decisions and design rationale |
 | [Examples](examples/README.md) | Notebooks, SQL scripts, and demos |
-| [Deployment Guides](deploy/README.md) | Four deployment surfaces for Google Cloud |
+| [Deployment Guides](deploy/README.md) | Six deployment surfaces for Google Cloud |
 
 ## Architecture