bigquery-data-science 0.7.0__py3-none-any.whl

bigquery_data_science/__init__.py

@@ -0,0 +1,58 @@
+"""BigQuery Data Science Agent Skills, CLI, and MCP Server."""
+
+from __future__ import annotations
+
+from importlib.metadata import PackageNotFoundError, version
+
+from bigquery_data_science.catalog import (
+    LookupResult,
+    SearchHit,
+    TriageResult,
+    get_walkthrough,
+    lookup,
+    lookup_symbol,
+    search,
+    search_catalog,
+    triage,
+    triage_task,
+)
+from bigquery_data_science.installer import (
+    available_skills,
+    bundled_skills_root,
+    discover_federated_skill_dirs,
+    get_skill_path,
+    get_skills_dir,
+    install_kernel,
+    install_skills,
+    list_bundled_skills,
+)
+
+try:
+    __version__ = version("bigquery-data-science")
+except PackageNotFoundError:
+    __version__ = "0.7.0"
+
+list_skills = list_bundled_skills
+
+__all__ = [
+    "LookupResult",
+    "SearchHit",
+    "TriageResult",
+    "__version__",
+    "available_skills",
+    "bundled_skills_root",
+    "discover_federated_skill_dirs",
+    "get_skill_path",
+    "get_skills_dir",
+    "get_walkthrough",
+    "install_kernel",
+    "install_skills",
+    "list_bundled_skills",
+    "list_skills",
+    "lookup",
+    "lookup_symbol",
+    "search",
+    "search_catalog",
+    "triage",
+    "triage_task",
+]

bigquery_data_science/_skills/PLANS.md

@@ -0,0 +1,76 @@
+# Agent Skills — Operating Manual & 4-Lens Architecture (`skills/PLANS.md`)
+
+Operating manual for the `bigquery-data-science` Agent Skills bundle: the 4-Lens progressive context-loading architecture, the authoring standard, the change checklists, and the verification audit log. Mirrors the `README.md` + `PLANS.md` + `RESOURCES.md` convention used in `bigquery-ml` and `bigquery-ai`.
+
+## Vision & 4-Lens Progressive Architecture
+
+Package this repository's live-verified BigQuery ML (`bigquery-ml`) and BigQuery AI (`bigquery-ai`) content as zero-dependency Agent Skills usable across Claude Code, Gemini CLI, Cursor, Windsurf, OpenAI Codex, and any MCP client.
+
+To prevent context-window bloat while keeping 100% of syntax tables, empirical gotchas, and end-to-end walkthroughs reachable offline, every skill in `skills/` follows a **4-Lens progressive disclosure hierarchy**:
+
+- **Lens 1 (`SKILL.md`, < 500 lines)**: High-signal entry router, decision tree, 5 Canonical Skill Patterns (Router, Tool Wrapper, Generator, Reviewer, Inversion), and cross-cutting gotchas.
+- **Lens 2 (`reference/<bucket>.md`)**: Use-case decision guides (or the cross-domain prerequisite matrix `reference/namespace-matrix.md` in `bigquery-data-science`) with comparison tables, verified empirical gotchas, canonical SQL/BigFrames snippets, and links into Lens 3 and Lens 4. Files exceeding 100 lines open with a `Contents:` table of contents in the first 10 lines.
+- **Lens 3 (`syntax/<slug>.md` + `syntax/INDEX.md`, each card < 25 KB)**: Deterministic, per-statement syntax and option cards auto-sliced at `## ` (H2) boundaries from `bigquery-ml/reference/*.md` (93 cards) and `bigquery-ai/reference/*.md` (44 cards) via `bigquery-data-science sync-syntax`. Every card carries a clickable GitHub `Source:` blockquote linking back to its exact heading in the domain reference page.
+- **Lens 4 (`narrative/<topic>.md`, 104 walkthroughs)**: Complete, ceremony-stripped markdown walkthroughs extracted from all 61 `bigquery-ml` topic notebooks and 43 `bigquery-ai` topic notebooks via `bigquery-data-science convert-notebook`. Every narrative starts with a clickable GitHub provenance banner linking to the source `.ipynb` and companion `.sql` script.
+
+```mermaid
+flowchart TD
+    User["User / Agent Task"] --> L1_Router["Lens 1: Umbrella Router<br/>skills/bigquery-data-science/SKILL.md"]
+    L1_Router --> Matrix["Lens 2: Prerequisite & Namespace Matrix<br/>skills/bigquery-data-science/reference/namespace-matrix.md"]
+    L1_Router --> L1_ML["Lens 1: Trained Models & Feature Store<br/>skills/bigquery-ml/SKILL.md"]
+    L1_Router --> L1_AI["Lens 1: Foundation Models & Cloud AI<br/>skills/bigquery-ai/SKILL.md"]
+
+    L1_ML --> L2_ML["Lens 2: Use-Case Decision Guides<br/>skills/bigquery-ml/reference/*.md (6 guides)"]
+    L1_AI --> L2_AI["Lens 2: Use-Case Decision Guides<br/>skills/bigquery-ai/reference/*.md (8 guides)"]
+
+    L2_ML --> L3_ML["Lens 3: Per-Statement Syntax Cards (< 25 KB)<br/>skills/bigquery-ml/syntax/*.md (93 cards + INDEX.md)"]
+    L2_AI --> L3_AI["Lens 3: Per-Statement Syntax Cards (< 25 KB)<br/>skills/bigquery-ai/syntax/*.md (44 cards + INDEX.md)"]
+
+    L2_ML --> L4_ML["Lens 4: Worked Walkthroughs<br/>skills/bigquery-ml/narrative/*.md (61 walkthroughs)"]
+    L2_AI --> L4_AI["Lens 4: Worked Walkthroughs<br/>skills/bigquery-ai/narrative/*.md (43 walkthroughs)"]
+```
+
+## Authoring Standard
+
+A skill is a directory under `skills/<name>/` (symlinked into `.agents/skills/<name>` and `.claude/skills/<name>`):
+
+1. **`SKILL.md` (Lens 1)** — YAML frontmatter (`name` <= 64 chars lowercase/digits/hyphens; `description` <= 1024 chars; `metadata.version: "0.7.0"`) + decision tree + Tool Wrapper & Generator patterns + Reviewer checklist + cross-cutting gotchas + pointers to `reference/*.md` and `syntax/INDEX.md`. Strictly < 500 lines.
+2. **`reference/<bucket>.md` (Lens 2)** — One file per use-case bucket, organized by what the practitioner is trying to accomplish. Each carries a single H1 heading, a `Contents:` TOC in the first 10 lines when > 100 lines, an options table, selection guidance, verified empirical gotchas, a canonical snippet, and relative links into `../syntax/<slug>.md` and `../narrative/<topic>.md`.
+3. **`syntax/<slug>.md` + `syntax/INDEX.md` (Lens 3)** — Auto-generated from `<domain>/reference/*.md` via `uv run bigquery-data-science sync-syntax`. Never hand-edit `syntax/*.md`; edit the source `<domain>/reference/*.md` page and re-run `sync-syntax`. Every card is verified `< 25,000` bytes.
+4. **`narrative/<topic>.md` (Lens 4)** — Auto-generated from `<domain>/{models,functions,workflows,pipelines}/**/*.ipynb` via `uv run bigquery-data-science sync-narratives` (or `convert-notebook`). Strips the HTML tracker header, `def install(*packages):` helper, Colab auth shim, and trailing `## Cleanup` section while prepending a clickable GitHub provenance banner to the source `.ipynb` and `.sql`.
+5. **`skill.manifest.json`** — Generated via `uv run bigquery-data-science manifest --all --version 0.7.0`.
+
+## Change Types and Checklists
+
+#### Adding or updating a model, function, or workflow
+- [ ] Update the ground-truth notebook/SQL and `<domain>/reference/*.md` in `bigquery-ml/` or `bigquery-ai/`.
+- [ ] Update the corresponding Lens 2 decision guide in `skills/<domain>/reference/<bucket>.md` (and `skills/bigquery-data-science/reference/namespace-matrix.md` if adding a new symbol).
+- [ ] Run `uv run bigquery-data-science sync-syntax` to regenerate Lens 3 syntax cards and `syntax/INDEX.md`.
+- [ ] Run `uv run bigquery-data-science sync-narratives` to regenerate Lens 4 narrative markdown files.
+- [ ] Run `uv run bigquery-data-science manifest --all --version 0.7.0` to refresh `skill.manifest.json`.
+- [ ] Run `uv run bigquery-data-science sweep` and confirm all 6 verification stages exit `0`.
+
+#### Adopting or retiring a pinned model version
+- [ ] Update `skills/pinned-models.json` (`*`, `bigquery-ml`, `bigquery-ai`).
+- [ ] Run `uv run bigquery-data-science check-pinned-models bigquery-ml` and `uv run bigquery-data-science check-pinned-models bigquery-ai`.
+
+#### Maintainer tooling change (`src/bigquery_data_science/`)
+- [ ] Update the module under `src/bigquery_data_science/`.
+- [ ] Prove every new check fails on a broken fixture in `tests/` before trusting a green pass.
+- [ ] Run the full verification gate:
+  ```bash
+  uv run ruff check src tests
+  uv run mypy src
+  uv run pytest -q
+  uv run bigquery-data-science sweep
+  python3 architectures/headers/sync_headers.py --check
+  ./scripts/check_install.sh
+  ```
+
+## Audit Log
+
+- **2026-07-29** — Phase 1 kickoff: built notebook-to-narrative extraction (`convert_notebook.py`), skill validator (`validate.py`), and manifest generator (`manifest.py`).
+- **2026-09-04** — Added `check-reference` (1:1 `RESOURCES.md` <-> `reference/*.md` coverage and heading-depth enforcement), `check-narratives` (regenerate-and-diff against `source_project` notebooks), and GitHub-verified `#anchor` resolution in `check-links` via `markdown_anchors.py`.
+- **2026-09-11** — Added point-in-time feature retrieval (`ML.FEATURES_AT_TIME`, `ML.ENTITY_FEATURES_AT_TIME`), `narrative_sources` collision disambiguation (`feature_store.md` vs `feature_store_workflow.md`), SQL image preprocessing (`ML.DECODE_IMAGE`, `ML.RESIZE_IMAGE`, `ML.CONVERT_COLOR_SPACE`, `ML.CONVERT_IMAGE_TYPE`), and the 5 pre-trained Cloud AI service models (`ML.PROCESS_DOCUMENT`, `ML.TRANSLATE`, `ML.UNDERSTAND_TEXT`, `ML.ANNOTATE_IMAGE`, `ML.TRANSCRIBE`).
+- **2026-10-09** — Added `check-pinned-models` (`skills/pinned-models.json`), lazy `nbformat` imports so CLI and MCP commands run with zero third-party dependencies, and the single-command `sweep` gate.
+- **2026-10-19** — Consolidated standalone `statmike/bigquery-data-science` v0.7.0 release: unified 1:1 naming (`bigquery-data-science`, `bigquery-ml`, `bigquery-ai`), added `skills/bigquery-data-science/reference/namespace-matrix.md`, added Lens 3 H2-sliced syntax cards (`sync-syntax`, 93 cards in `bigquery-ml/syntax/` and 44 cards in `bigquery-ai/syntax/`, all < 25 KB), expanded Lens 4 to all 104 topic notebooks (`61` in `bigquery-ml/narrative/` + `43` in `bigquery-ai/narrative/`) with clickable GitHub `.ipynb`/`.sql` provenance banners, and shipped the zero-dependency CLI (`list`, `path`, `install`, `kernel`, `lookup`, `search`, `triage`, `sweep`) and stdio MCP server (`triage_bigquery_task`, `lookup_bigquery_syntax`, `search_bigquery_gotchas`, `get_bigquery_walkthrough`).

bigquery_data_science/_skills/bigquery-ai/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: bigquery-ai
+description: Use when calling Gemini, TimesFM, TabFM, or pre-trained Cloud AI services from BigQuery SQL or BigFrames — generation (AI.GENERATE*, AI.COUNT_TOKENS), managed scoring/classification (AI.IF, AI.SCORE, AI.CLASSIFY, AI.AGG), embeddings and vector/hybrid search (AI.EMBED, VECTOR_SEARCH, AI.SEARCH), forecasting/prediction (AI.FORECAST, AI.PREDICT, AI.KEY_DRIVERS), and Cloud AI service models (ML.PROCESS_DOCUMENT, ML.TRANSLATE, ML.TRANSCRIBE).
+metadata:
+  version: "0.7.0"
+---
+
+# BigQuery AI (`bigquery-ai`)
+
+BigQuery AI brings foundation models (Gemini, TimesFM, TabFM, EmbeddingGemma, Text/Multimodal Embeddings) and pre-trained Google Cloud AI services directly into SQL and Python (`bigframes.bigquery.ai`) with zero model training.
+
+This skill packages a verified, field-tested 4-Lens reference distilled from live-executing all 29 functions and 14 composed workflows in `bigquery-ai/`.
+
+## Decision Tree (Lens 1 → Lens 2)
+
+1. **Do you want free-form or structured schema output per row** (summaries, multi-field JSON/STRUCT extraction, typed scalars, token counting)? → see [`reference/general-purpose-generation.md`](reference/general-purpose-generation.md)
+2. **Do you want a managed, zero-config judgment per row or group** (boolean filter `AI.IF`, numeric rating `AI.SCORE`, fixed-category label `AI.CLASSIFY`, or group synthesis `AI.AGG`)? → see [`reference/managed-functions.md`](reference/managed-functions.md)
+3. **Do you need to embed text/multimodal content, compute pairwise similarity, or run vector/hybrid RRF search?** → see [`reference/embeddings-and-search.md`](reference/embeddings-and-search.md)
+4. **Do you need zero-training predictions** — forecast a time series (`AI.FORECAST` on TimesFM 2.5 / 3.0), detect anomalies (`AI.DETECT_ANOMALIES`), or predict tabular targets without training (`AI.PREDICT` + `AI.EVALUATE` on TabFM)? → see [`reference/predictive-ai.md`](reference/predictive-ai.md)
+5. **Do you need to explain why a summable metric moved (`AI.KEY_DRIVERS`), or estimate the causal impact of an intervention (`AI.CAUSAL_EFFECT`)?** → see [`reference/augmented-analytics.md`](reference/augmented-analytics.md) (which also routes `AI.CAUSAL_EFFECT` to [`../bigquery-ml/reference/workflows-and-pipelines.md`](../bigquery-ml/reference/workflows-and-pipelines.md))
+6. **Are you extracting structured records from PDFs/images or passing `ObjectRef` media into `AI.*` functions?** → see [`reference/document-processing.md`](reference/document-processing.md)
+7. **Do you need a pre-trained Cloud AI service (`REMOTE_SERVICE_TYPE`) with a fixed API schema and per-row status column** (`ML.TRANSLATE`, `ML.UNDERSTAND_TEXT`, `ML.ANNOTATE_IMAGE`, `ML.TRANSCRIBE`, `ML.PROCESS_DOCUMENT`)? → see [`reference/cloud-ai-service-models.md`](reference/cloud-ai-service-models.md)
+8. **Are you composing multiple capabilities into an end-to-end application** (RAG, hybrid catalog search, document intelligence, log triage, content moderation, metric diagnostics)? → see [`reference/workflows.md`](reference/workflows.md)
+
+If the task is ambiguous between zero-training foundation functions (`bigquery-ai`) and trained models (`bigquery-ml`), consult [`../bigquery-data-science/SKILL.md`](../bigquery-data-science/SKILL.md) first.
+
+## Tool Wrapper & Generator Rules (SQL + `bigframes.bigquery.ai`)
+
+- **Dual Surface & Pinned Models**: Always pin the explicit `endpoint` or `model` parameter (`'gemini-2.5-flash'`, `'TimesFM 2.5'`, `'TimesFM 3.0'`, `'embeddinggemma-300m'`, `'text-embedding-005'`, `'gemini-embedding-001'`) from `skills/pinned-models.json` so silent default upgrades never shift outputs. Pair SQL calls with `bigframes.bigquery.ai` / `bigframes.ml.llm` wrappers when generating Python notebooks.
+- **Per-Statement Syntax Lookup (Lens 3)**: Before emitting `AI.*`, `VECTOR_SEARCH`, or `OBJ.*` calls, inspect [`syntax/INDEX.md`](syntax/INDEX.md) (or run `bigquery-data-science lookup <SYMBOL>`) to read the exact `< 25 KB` syntax card in `syntax/*.md`.
+- **Direct `ObjectRef` Input**: Pass `ObjectRef` columns (from an Object Table's `ref` column, `OBJ.LIST(...)`, or `OBJ.MAKE_REF(uri, connection)`) directly into `AI.GENERATE*`, `AI.CLASSIFY`, `AI.SCORE`, `AI.IF`, `AI.AGG`, and `AI.EMBED` without wrapping in `OBJ.GET_ACCESS_URL`.
+
+## Reviewer Verification Checklist & Rubric
+
+Before returning any `bigquery-ai` SQL or Python code, verify against this checklist:
+
+- [ ] **No Retired `AI.PARSE_DOCUMENT`**: Reject any query using `AI.PARSE_DOCUMENT` (offline since 2026-06-01); replace with `ML.PROCESS_DOCUMENT` or `AI.GENERATE` + `output_schema`.
+- [ ] **No Non-Existent `HYBRID_SEARCH` TVF**: Use `VECTOR_SEARCH(..., options => '{"lexical_search_columns": [...]}')` or `AI.SEARCH(..., mode => 'HYBRID')` and size `top_k >= R/10` for target semantic rank `R`.
+- [ ] **`output_schema` Field Access**: When `output_schema` is passed to `AI.GENERATE` or `AI.GENERATE_TABLE`, select the schema field names directly (never `.result`, which disappears when `output_schema` is specified).
+- [ ] **Subquery Sampling Before `LIMIT`**: Materialize or subquery-limit input rows *before* invoking per-row `AI.*` scalar functions so `LIMIT` does not evaluate and bill the entire table.
+- [ ] **Per-Row Status Check on Cloud AI TVFs**: Queries calling `ML.PROCESS_DOCUMENT`, `ML.TRANSLATE`, `ML.UNDERSTAND_TEXT`, `ML.ANNOTATE_IMAGE`, or `ML.TRANSCRIBE` inspect `<function>_status` because per-row failures do not fail the BigQuery job.
+
+## Cross-Cutting Gotchas (Apply Across Functions)
+
+- **`AI.PARSE_DOCUMENT` is offline and its documentation has been withdrawn** (since 2026-06-01). Always use `ML.PROCESS_DOCUMENT` (via a `CLOUD_AI_DOCUMENT_V1` remote model) or `AI.GENERATE` over `ObjectRef`.
+- **`output_schema` replaces the `result` field entirely** in `AI.GENERATE` and `AI.GENERATE_TABLE`.
+- **`LIMIT` / `OFFSET` on an outer `SELECT` does not reduce billed LLM calls** — filter or `LIMIT` inside a CTE/subquery before calling scalar `AI.*` functions.
+- **Pass `ObjectRef` directly to multimodal `AI.*` functions** — `OBJ.GET_ACCESS_URL` is only needed when generating short-lived signed URLs (`<= 6` hours; never persist signed URLs).
+- **There is no standalone `HYBRID_SEARCH` function** — hybrid RRF retrieval (`distance = 1 - (1/(60 + rank_vector) + 1/(61 + rank_lexical))`) is enabled via `lexical_search_columns` on `VECTOR_SEARCH` or `mode => 'HYBRID'` on `AI.SEARCH` over a `10 * top_k` semantic pool.
+- **Pin `TimesFM 2.5` or `TimesFM 3.0` on `AI.FORECAST`, `AI.DETECT_ANOMALIES`, and `AI.EVALUATE`**; materialize `AI.PREDICT` (TabFM) outputs because TabFM has no version pin and is non-deterministic across runs.
+- **Use `AI.COUNT_TOKENS` as a free pre-flight check** before running large paid generative batches.
+- **Cloud AI service models (`REMOTE_SERVICE_TYPE`) are handles, not trained models**: `CREATE MODEL` does not validate IAM permissions (`roles/serviceusage.serviceUsageConsumer` on the connection's service account is mandatory), `ML.EVALUATE` is rejected, and errors surface per row in `<function>_status`.
+
+## 4-Lens Index
+
+- **Lens 2 — Use-Case Decision Guides (`reference/`)**:
+  - [`reference/general-purpose-generation.md`](reference/general-purpose-generation.md) — `AI.GENERATE`, `AI.GENERATE_TEXT`, `AI.GENERATE_TABLE`, `AI.GENERATE_BOOL`, `AI.GENERATE_DOUBLE`, `AI.GENERATE_INT`, `ML.GENERATE_TEXT`, `AI.COUNT_TOKENS`
+  - [`reference/managed-functions.md`](reference/managed-functions.md) — `AI.IF`, `AI.SCORE`, `AI.CLASSIFY`, `AI.AGG`
+  - [`reference/embeddings-and-search.md`](reference/embeddings-and-search.md) — `AI.EMBED`, `AI.GENERATE_EMBEDDING`, `ML.GENERATE_EMBEDDING`, `AI.SIMILARITY`, `VECTOR_SEARCH`, `AI.SEARCH`, and hybrid RRF search
+  - [`reference/predictive-ai.md`](reference/predictive-ai.md) — `AI.FORECAST`, `AI.DETECT_ANOMALIES`, `AI.PREDICT`, `AI.EVALUATE` (TimesFM & TabFM)
+  - [`reference/augmented-analytics.md`](reference/augmented-analytics.md) — `AI.KEY_DRIVERS` and routing for `AI.CAUSAL_EFFECT`
+  - [`reference/document-processing.md`](reference/document-processing.md) — `ML.PROCESS_DOCUMENT`, `AI.PARSE_DOCUMENT` (retired), Object Tables, `OBJ.LIST`, `OBJ.MAKE_REF`, `OBJ.FETCH_METADATA`, `OBJ.GET_ACCESS_URL`
+  - [`reference/cloud-ai-service-models.md`](reference/cloud-ai-service-models.md) — `ML.TRANSLATE`, `ML.UNDERSTAND_TEXT`, `ML.ANNOTATE_IMAGE`, `ML.TRANSCRIBE`, `ML.PROCESS_DOCUMENT`
+  - [`reference/workflows.md`](reference/workflows.md) — 14 composed end-to-end workflows (RAG, hybrid catalog search, document intelligence, tabular prediction, etc.)
+- **Lens 3 — Per-Statement Syntax Cards (`syntax/`, < 25 KB each)**: [`syntax/INDEX.md`](syntax/INDEX.md) (44 H2-sliced syntax cards)
+- **Lens 4 — Worked Walkthroughs (`narrative/`)**: 43 extracted notebook narratives in `narrative/*.md` (e.g., [`narrative/ai_forecast.md`](narrative/ai_forecast.md), [`narrative/ai_generate.md`](narrative/ai_generate.md), [`narrative/catalog_search.md`](narrative/catalog_search.md))