bigquery-data-science 0.7.0__py3-none-any.whl

bigquery_data_science/_skills/bigquery-ai/syntax/objget_read_url.md

@@ -0,0 +1,40 @@
+# `OBJ.GET_READ_URL`
+
+> **Source:** [`bigquery-ai/reference/unstructured-data-infrastructure.md#objget_read_url`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/unstructured-data-infrastructure.md#objget_read_url)
+
+- **Description:** (Preview) Scalar function that returns a `STRUCT<url STRING, status STRING>` holding a read URL for a Cloud Storage object. The URL expires after **45 minutes**.
+- **Use cases:** Render an object inline in BigQuery Studio query results — the `url` column displays the image rather than the link text. A shorter-lived, read-only alternative to `OBJ.GET_ACCESS_URL` when all you need is to look at the object.
+- [documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/objectref_functions)
+- **Type:** Scalar function (returns STRUCT) — Preview
+
+**Syntax:**
+```sql
+OBJ.GET_READ_URL(objectref)
+```
+
+**Inputs:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `objectref` | ObjectRef STRUCT | Required | Must use **delegated access** — the ObjectRef has to carry an `authorizer`. A direct-access ObjectRef is rejected. |
+
+**Outputs:** `STRUCT<url STRING, status STRING>`. On success `url` holds the read URL and `status` is `NULL`; on failure `url` is `NULL` and `status` holds the error message. Nothing raises — check `status`.
+
+```sql
+SELECT OBJ.GET_READ_URL(poster) AS read_url
+FROM `mydataset.films`;
+```
+
+**How it differs from `OBJ.GET_ACCESS_URL`:**
+
+| | `OBJ.GET_READ_URL` | `OBJ.GET_ACCESS_URL` |
+|---|---|---|
+| Returns | `STRUCT<url, status>` | `ObjectRefRuntime` JSON |
+| Modes | Read only | `'r'` or `'rw'` |
+| TTL | Fixed 45 minutes | 30 min – 6 hours, caller-set |
+| Access | Delegated access required | Either |
+| Errors | Reported in `status` | Reported in `runtime_errors` |
+
+**Locations:** All BigQuery regions supporting Cloud resource connections.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/objlist.md

@@ -0,0 +1,34 @@
+# `OBJ.LIST`
+
+> **Source:** [`bigquery-ai/reference/unstructured-data-infrastructure.md#objlist`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/unstructured-data-infrastructure.md#objlist)
+
+- **Description:** (GA) Table-valued function that lists Cloud Storage objects matching a URI with one `*` wildcard per path segment and returns their metadata plus an `ObjectRef` for each — ad-hoc discovery of unstructured data without creating an object table. Works across all Cloud Storage storage classes.
+- **Use cases:** Point an `AI.*` function at "every PDF in this prefix" in one query, feed `AI.AGG` a set of images, build an extract-and-load pipeline over a landing bucket without DDL, prototype against a public sample bucket.
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/objectref_functions#objlist)
+- **Type:** Table-valued function — GA
+
+**Syntax:**
+```sql
+SELECT uri, content_type, size, ref
+FROM OBJ.LIST(
+  uri
+  [, authorizer]
+)
+```
+
+**Inputs:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `uri` | STRING literal | Required | Cloud Storage path with at most one `*` per segment, e.g. `gs://bucket/images/*.png`. Must be a literal — scalar subqueries and `CONCAT` are rejected. Must not end in `/` or contain `//` (`gs://bucket/images/` fails with *"GCS paths cannot end with a slash"* and suggests the `/*` form). |
+| `authorizer` | STRING | Optional | Cloud resource connection for **delegated access**, e.g. `project.us.connection`. Omit it and every returned `ref` uses **direct access** — fetched as the identity running the query. |
+
+**Outputs:** one row per object: `uri` STRING, `content_type` STRING, `size` INT64, `md5_hash` STRING, `updated` TIMESTAMP, `metadata` ARRAY\<STRUCT\<name STRING, value STRING\>\>, `generation` INT64, and `ref` — an `ObjectRef` whose `authorizer` is populated only when you passed one.
+
+**Measured 2026-10-09** against `gs://cloud-samples-data/bigquery/tutorials/cymbal-pets/images/*.png` (205 objects): the no-authorizer form and the authorizer form both feed `AI.GENERATE`, the typed variants, `AI.IF`/`AI.SCORE`/`AI.CLASSIFY`, `AI.AGG`, `AI.EMBED` and `AI.SIMILARITY` directly via `ref`, with an empty `status` on every row. Direct access worked here because the bucket is public; on a private bucket it works only if the query runner holds `storage.objectViewer`, and inside a VPC Service Controls perimeter it is the form that works (see the note under [`OBJ.MAKE_REF`](objmake_ref.md)).
+
+**Limitations:** Shares the object-table limitations because it generates the same metadata dynamically. The listing is computed per query — there is no persistence and no change tracking; use an object table for that. `ML.PROCESS_DOCUMENT` still reads object tables only.
+
+**Locations:** All BigQuery regions supporting Cloud resource connections.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/objmake_ref.md

@@ -0,0 +1,58 @@
+# `OBJ.MAKE_REF`
+
+> **Source:** [`bigquery-ai/reference/unstructured-data-infrastructure.md#objmake_ref`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/unstructured-data-infrastructure.md#objmake_ref)
+
+- **Description:** (Preview) Scalar function that creates an `ObjectRef` value containing reference information for a Cloud Storage object. This is the entry point for creating ObjectRef values from raw URI strings — no validation is performed on JSON input.
+- **Use cases:** Create ObjectRef values for standard table columns without object tables, build ad-hoc ObjectRef values inline in queries for one-off multimodal analysis, reference transformation outputs saved to Cloud Storage.
+- [documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/objectref_functions)
+- **Type:** Scalar function (returns ObjectRef STRUCT) — Preview
+
+**Syntax (URI, optional authorizer):**
+```sql
+OBJ.MAKE_REF(
+  uri
+  [, authorizer]
+  [, version => version_value]
+  [, details => gcs_metadata_json]
+)
+```
+
+**Syntax (JSON input):**
+```sql
+OBJ.MAKE_REF(
+  objectref_json
+)
+```
+
+**Syntax (re-authorize an existing ObjectRef):**
+```sql
+OBJ.MAKE_REF(
+  objectref,
+  authorizer
+)
+```
+The top-level `authorizer` overwrites whatever the input `objectref` carried.
+
+**Inputs:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `uri` | STRING | Required (overload 1) | Cloud Storage URI, e.g., `gs://mybucket/file.jpg`. Can be a column reference. |
+| `authorizer` | STRING | Optional (overload 1) | Cloud resource connection, e.g., `us.myconnection`, used for **delegated access**. Must be in the same project and region as the query. Omit it and the ObjectRef uses **direct access** instead. |
+| `version_value` | STRING | Optional | Cloud Storage object version. |
+| `gcs_metadata_json` | JSON | Optional | Pre-supplied metadata: `{"content_type", "md5_hash", "size", "updated"}`. Fills `details` without a Cloud Storage round-trip. |
+| `objectref_json` | JSON | Required (overload 2) | JSON with schema `{"uri": "string", "authorizer": "string"}`. No validation performed. |
+
+**Outputs:** Returns `STRUCT<uri STRING, version STRING, authorizer STRING, details JSON>`. Only `uri` and `authorizer` are populated unless you supply `version` / `details`; otherwise use `OBJ.FETCH_METADATA` to fill them.
+
+> **Dropping the authorizer changes who reads the object, not whether it works.** `OBJ.MAKE_REF('gs://bucket/file.pdf')` is legal and succeeds against private buckets — but only because the identity running the query holds `storage.objectViewer` on that bucket. That is **direct access**: the object is fetched as the query runner. Passing the connection gives **delegated access**, where the object is fetched as the connection's service account and the grant is auditable and independent of whoever runs the query. The examples throughout this project pass the connection for that reason.
+>
+> **Inside a VPC Service Controls perimeter the recommendation inverts.** Delegated access mints a signed HTTPS URL, and Agent Platform blocks HTTP/HTTPS fetches for projects inside a perimeter — so a delegated-access `ObjectRef`, **including an object table's `ref` column**, fails with `INVALID_ARGUMENT: HTTP links are not supported for requests restricted by VPCSC.` in the output's `status` field. In that environment use the single-argument `OBJ.MAKE_REF(uri)`: direct access sends the Cloud Storage URI to the model with no signed URL in between. This is also a reason to prefer inline `OBJ.MAKE_REF` over object tables inside a perimeter — an object table has no direct-access form.
+
+**Limitations:**
+- Maximum **20 connections** per project+region for queries referencing ObjectRef values.
+- Connection must be in the same project and region as the query.
+
+**Locations:** All BigQuery regions supporting Cloud resource connections.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/per-row-failures-and-the-retry-pattern.md

@@ -0,0 +1,19 @@
+# Per-row failures and the retry pattern
+
+> **Source:** [`bigquery-ai/reference/cloud-ai-service-models.md#per-row-failures-and-the-retry-pattern`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/cloud-ai-service-models.md#per-row-failures-and-the-retry-pattern)
+
+Every member reports failure in a status column while the **job succeeds**. Measured examples: a non-audio file in a `ML.TRANSCRIBE` object table, and an unsupported language in `ML.UNDERSTAND_TEXT`. The standard shape is to persist results and re-run only the failures:
+
+```sql
+CREATE OR REPLACE TABLE `PROJECT_ID.DATASET.results` AS
+SELECT * FROM ML.TRANSCRIBE(MODEL ..., TABLE ..., recognition_config => JSON '...');
+
+-- what failed, and why
+SELECT uri, ml_transcribe_status
+FROM `PROJECT_ID.DATASET.results`
+WHERE ml_transcribe_status != '';
+```
+
+Whether the retry can succeed depends on the cause: a `RESOURCE EXHAUSTED` row failed for load and will likely pass on a second attempt; a row that failed for an unsupported language or an unsupported encoding will fail identically forever. Read the status before retrying.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/setup-apis-connection-roles.md

@@ -0,0 +1,39 @@
+# Setup: APIs, connection, roles
+
+> **Source:** [`bigquery-ai/reference/cloud-ai-service-models.md#setup-apis-connection-roles`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/cloud-ai-service-models.md#setup-apis-connection-roles)
+
+All five run through a **Cloud Resource connection**. The connection's auto-provisioned service account — not your user account — is the identity that calls the service and reads Cloud Storage.
+
+**The APIs to enable**, beyond `bigquery.googleapis.com` and `bigqueryconnection.googleapis.com`:
+
+| Function | Service API | Also needs |
+|---|---|---|
+| `ML.TRANSLATE` | `translate.googleapis.com` | — |
+| `ML.UNDERSTAND_TEXT` | `language.googleapis.com` | — |
+| `ML.ANNOTATE_IMAGE` | `vision.googleapis.com` | `storage.googleapis.com` |
+| `ML.TRANSCRIBE` | `speech.googleapis.com` | `storage.googleapis.com` |
+| `ML.PROCESS_DOCUMENT` | `documentai.googleapis.com` | `storage.googleapis.com` |
+
+**The roles to grant the connection's service account**, at project level:
+
+| Role | `TRANSLATE` | `UNDERSTAND_TEXT` | `ANNOTATE_IMAGE` | `TRANSCRIBE` | `PROCESS_DOCUMENT` |
+|---|---|---|---|---|---|
+| `roles/serviceusage.serviceUsageConsumer` | ✅ | ✅ | ✅ | ✅ | ✅ |
+| `roles/bigquery.connectionUser` | ✅ | ✅ | ✅ | ✅ | ✅ |
+| `roles/storage.objectViewer` (object tables) | — | — | ✅ | ✅ | ✅ |
+| Service-specific role | `roles/cloudtranslate.user` | *none exists* | *none exists* | `roles/speech.client` | `roles/documentai.apiUser` + `roles/documentai.viewer` |
+| `roles/aiplatform.user` | comparison examples only | comparison examples only | comparison examples only | comparison examples only | comparison examples only |
+
+Three things this table is for:
+
+- **The service-specific role is not uniform, and its absence is not an oversight.** Cloud Vision and Cloud Natural Language have no predefined IAM role — `serviceusage.services.use` is the permission that gates them, so their column stops at the BigQuery-side rows. Translation and Speech-to-Text each add one; Document AI adds two. Copying one member's setup to another either over-grants or fails at call time.
+- **`roles/serviceusage.serviceUsageConsumer` is the one people miss.** Without it every call fails, no matter which service. The `ML.TRANSLATE` notebook shows the exact message, which names the missing role — more than most IAM failures give you, but it arrives at call time.
+- **`roles/aiplatform.user` is not part of any of these functions.** Each notebook ends by running an `AI.*` function over the same rows for comparison, and *that* is what needs it. A deployment that only calls the Cloud AI service model can leave it off.
+
+**`CREATE MODEL` succeeding tells you nothing about permissions.** Model creation does not contact the service, so it completes happily against a connection that cannot call anything. The first failure arrives at call time.
+
+**Grants do not take effect instantly.** A freshly granted binding can take a minute to become usable. Each of the four notebooks on this page ends its setup with a probe on the smallest possible input, in a wait loop, so the first real example runs against a connection that has been shown to work — worth copying into any script that creates the connection and then uses it.
+
+See the [Setup Reference](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/setup) for connection creation, and each notebook's Setup section for a runnable, idempotent version that also prints which roles it granted.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/two-input-shapes.md

@@ -0,0 +1,22 @@
+# Two input shapes
+
+> **Source:** [`bigquery-ai/reference/cloud-ai-service-models.md#two-input-shapes`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/cloud-ai-service-models.md#two-input-shapes)
+
+**Object tables** — `ML.ANNOTATE_IMAGE`, `ML.TRANSCRIBE`, `ML.PROCESS_DOCUMENT`. An [object table](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/unstructured-data-infrastructure.md) is an external table whose rows are files. It reads through the same connection, which is why the service account needs `roles/storage.objectViewer`:
+
+```sql
+CREATE OR REPLACE EXTERNAL TABLE `PROJECT_ID.DATASET.OBJECT_TABLE`
+  WITH CONNECTION `PROJECT_ID.REGION.CONNECTION_ID`
+  OPTIONS (object_metadata = 'SIMPLE', uris = ['gs://BUCKET/PREFIX/*']);
+```
+
+**A `text_content` column** — `ML.TRANSLATE`, `ML.UNDERSTAND_TEXT`. Both read a column named exactly `text_content`. A table without one errors, and the fix is an alias in a subquery rather than a rename of your table:
+
+```sql
+FROM ML.TRANSLATE(
+  MODEL `PROJECT_ID.DATASET.MODEL_NAME`,
+  (SELECT review_id, comment AS text_content FROM `PROJECT_ID.DATASET.reviews`),
+  STRUCT('TRANSLATE_TEXT' AS translate_mode, 'en' AS target_language_code))
+```
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/vector_search.md

@@ -0,0 +1,78 @@
+# `VECTOR_SEARCH`
+
+> **Source:** [`bigquery-ai/reference/embedding-generation-and-semantic-search.md#vector_search`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/embedding-generation-and-semantic-search.md#vector_search)
+
+- **Description:** Table-valued function that searches embeddings to find the top-K closest embeddings from a base table to a given query embedding. Supports both batch searches (multiple query rows) and single searches (one embedding value). Can use vector indexes for approximate nearest neighbor (ANN) search. The single search syntax additionally supports **hybrid search** -- combining semantic similarity with lexical (keyword) matching -- via `lexical_search_columns` and `lexical_search_query_value`.
+- **Use cases:** Semantic search, hybrid semantic + keyword search, recommendation systems, classification, clustering, retrieval augmented generation (RAG).
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/search_functions#vector_search)
+- **Type:** Table-valued function (TVF) -- GA (single search syntax is Preview)
+
+**Syntax (batch search):**
+```sql
+VECTOR_SEARCH(
+  { TABLE base_table | (base_table_query) },
+  column_to_search,
+  { TABLE query_table | (query_table_query) },
+  [, query_column_to_search => query_column_to_search_value]
+  [, top_k => top_k_value]
+  [, distance_type => distance_type_value]
+  [, options => options_value]
+)
+```
+
+**Syntax (single search -- Preview; also the hybrid search syntax):**
+```sql
+VECTOR_SEARCH(
+  { TABLE base_table | (base_table_query) },
+  column_to_search,
+  query_value => single_query_value,
+  [, lexical_search_columns => lexical_search_columns_value]
+  [, lexical_search_query_value => single_lexical_search_query_value]
+  [, top_k => top_k_value]
+  [, distance_type => distance_type_value]
+  [, options => options_value]
+)
+```
+
+Supplying a non-empty `lexical_search_columns` is what makes the search hybrid. Hybrid search is only available in the single search syntax -- **there is no batch hybrid search**. See [Hybrid Search](hybrid-search-capability.md) below for how the fused score is computed.
+
+**Inputs:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `base_table` / `base_table_query` | Table/Query | Required | The table to search. Only SELECT, FROM, WHERE clauses allowed in query. Don't filter the embedding column. Can't use logical views. |
+| `column_to_search` | STRING | Required | Base table column to search. Must be ARRAY\<FLOAT64\> or STRING (Preview, requires autonomous embedding generation). If indexed, BigQuery attempts to use it. |
+| `query_table` / `query_table_query` | Table/Query | Required for batch | Query embeddings to find nearest neighbors for. |
+| `query_column_to_search` | Named arg, STRING | Optional (batch only) | Column in query table containing embeddings. ARRAY\<FLOAT64\> or STRING. |
+| `query_value` | Named arg, ARRAY\<FLOAT64\> or STRING | Required for single search | Single embedding or string to search for. |
+| `lexical_search_columns` | Named arg, ARRAY\<STRING\> | Optional (single search only) | Base table columns to match lexically. Non-empty value switches the call to hybrid search. If the base table has a vector index with `lexical_search_columns`, these columns must be a subset of the indexed ones. |
+| `lexical_search_query_value` | Named arg, STRING | Optional (single search only) | The keyword query text for the lexical leg. Defaults to `query_value` when that is a STRING. |
+| `top_k` | Named arg, INT64 | Optional | Number of nearest neighbors per query. Default: 10. Negative = return all. |
+| `distance_type` | Named arg, STRING | Optional | `EUCLIDEAN` (default), `COSINE`, or `DOT_PRODUCT`. |
+| `options` | Named arg, JSON STRING | Optional | `fraction_lists_to_search` (0.0-1.0, index only), `use_brute_force` (boolean). |
+
+**Outputs (batch search):**
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `query` | STRUCT | All selected columns from the query data |
+| `base` | STRUCT | All columns from base_table |
+| `distance` | FLOAT64 | Distance between base and query data |
+
+**Outputs (single search):** Same but without the `query` column.
+
+**Outputs (hybrid search):** Same shape as single search, but `distance` is **not a distance** -- it is a reciprocal rank fusion (RRF) score derived from the row's rank in the semantic and lexical result lists. Hybrid `distance` values are not comparable to semantic-only `distance` values. See [Hybrid Search](hybrid-search-capability.md).
+
+**Supported models:** VECTOR_SEARCH does not directly reference models. Operates on pre-computed ARRAY\<FLOAT64\> embeddings or STRING columns with autonomous embedding generation. Embeddings can come from any source (AI.EMBED, AI.GENERATE_EMBEDDING, external).
+
+**Best practices:** Use a vector index for large base tables. Use brute force for exact results. Use single search syntax for single queries (optimized performance).
+
+**Limitations:** Row-level and column-level security policies apply. Project running the query must match the project containing the base table. Subqueries in base_table_query might interfere with index usage. Logical views cannot be used. Hybrid search requires the single search syntax and therefore cannot be combined with batch search.
+
+**Locations:** Not specified specifically -- operates wherever BigQuery tables exist.
+
+**Provisioned throughput:** Not specified. When used with autonomous embedding generation on STRING columns, generative AI function limits apply.
+
+**BigFrames API:** `bigframes.bigquery.vector_search(base_table, column_to_search, query)` — Takes base table as string, query as DataFrame/Series. Supports `distance_type`, `top_k`, `fraction_lists_to_search`, `use_brute_force`. Also `bigframes.bigquery.create_vector_index()` for creating vector indexes.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/what-the-five-have-in-common.md

@@ -0,0 +1,27 @@
+# What the five have in common
+
+> **Source:** [`bigquery-ai/reference/cloud-ai-service-models.md#what-the-five-have-in-common`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/cloud-ai-service-models.md#what-the-five-have-in-common)
+
+**A model object that is a handle, not a model.** Every member needs `CREATE MODEL … REMOTE WITH CONNECTION … OPTIONS(REMOTE_SERVICE_TYPE = …)` first. Nothing trains and no data is read:
+
+```sql
+CREATE OR REPLACE MODEL `PROJECT_ID.DATASET.MODEL_NAME`
+  REMOTE WITH CONNECTION `PROJECT_ID.REGION.CONNECTION_ID`
+  OPTIONS (REMOTE_SERVICE_TYPE = 'CLOUD_AI_VISION_V1');
+```
+
+The model's own metadata says as much: `model_type` is unspecified, the single training run carries an empty `trainingOptions` and an empty `evaluationMetrics`, and the only substance in the record is `remoteModelInfo`. Each notebook prints it in its last SQL example.
+
+**`ML.EVALUATE` is rejected.** Asking for it returns *"ML.EVALUATE is not supported for remote models on Cloud AI APIs."* There is nothing to evaluate — the service is the model, and its quality is Google's version number, not your training run.
+
+**Models are not in `INFORMATION_SCHEMA`.** `INFORMATION_SCHEMA.TABLES` lists object tables but no models, and this project has no `INFORMATION_SCHEMA.MODELS` view. Read model metadata through the client library (`client.get_model(...).to_api_repr()`) or `bq show --model`.
+
+**Three output columns, named after the function.** Each returns the input columns plus `<function>_result` (`JSON`) and `<function>_status` (`STRING`, empty on success). `ML.TRANSCRIBE` adds a fourth, `transcripts` (`STRING`) — the only member that hands back a finished answer without a JSON accessor.
+
+**Failures are per row.** A row that the service rejects fills `<function>_status` and leaves the result empty **while the job succeeds**. A pipeline that does not select the status column records the failure as a result. See [Per-row failures](per-row-failures-and-the-retry-pattern.md).
+
+**They run on on-demand pricing.** Measured for `ML.TRANSCRIBE` in `INFORMATION_SCHEMA.JOBS_BY_PROJECT`: every call reports no `reservation_id` and an `edition` of `none` — on-demand. No BigQuery Editions reservation is required — unlike the [image preprocessing functions](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/reference/model-free-functions.md#image-preprocessing-functions-mldecode_image-mlresize_image-mlconvert_image_type-mlconvert_color_space) in the sibling `bigquery-ml` project, which fail outright without one. The BigQuery bill is bytes scanned; the service bill is separate and is priced by the service's own unit — characters for Translation and Natural Language, images (and features per image) for Vision, **seconds of audio** for Speech-to-Text, pages for Document AI.
+
+**They are the stable option.** Nothing about their output moves when a Gemini default changes. That is the reason to choose them over an `AI.*` function with a prompt, and it is not the same as being deterministic — `ML.TRANSCRIBE` disagrees with itself across identical calls, while `ML.TRANSLATE` and `ML.UNDERSTAND_TEXT` do not. Each notebook measures this directly with the query cache off.
+
+---

bigquery_data_science/_skills/bigquery-data-science/SKILL.md

@@ -0,0 +1,69 @@
+---
+name: bigquery-data-science
+description: Use FIRST when someone wants to predict, classify, forecast, embed, search, extract, or explain data in BigQuery and needs to choose between a trained model (bigquery-ml: CREATE MODEL + ML.*), a zero-training generative or foundation model function (bigquery-ai: AI.* + TimesFM + TabFM), or a pre-trained Google Cloud AI service model (ML.PROCESS_DOCUMENT, ML.TRANSLATE, ML.UNDERSTAND_TEXT, ML.ANNOTATE_IMAGE, ML.TRANSCRIBE).
+metadata:
+  version: "0.7.0"
+---
+
+# BigQuery Data Science — Umbrella Router & Triage (`bigquery-data-science`)
+
+BigQuery provides three distinct, composable surfaces for machine learning and AI in SQL and Python (`bigframes`):
+
+1. **Trained Models & Feature Engineering ([`bigquery-ml`](../bigquery-ml/SKILL.md))** — `CREATE MODEL` + `ML.*` across 23 model types, 49 model-free/lifecycle functions (including `ML.ENTITY_FEATURES_AT_TIME` and `ML.DECODE_IMAGE`), 18 end-to-end workflows (including `AI.CAUSAL_EFFECT` and econometric causal inference), and 8 production orchestration pipelines.
+2. **Foundation Models & Generative SQL ([`bigquery-ai`](../bigquery-ai/SKILL.md))** — zero-training per-row and table-valued foundation model functions (`AI.GENERATE*`, `AI.IF`, `AI.SCORE`, `AI.CLASSIFY`, `AI.AGG`, `AI.EMBED`, `AI.SEARCH`, `VECTOR_SEARCH`, `AI.FORECAST` on TimesFM, `AI.DETECT_ANOMALIES`, `AI.KEY_DRIVERS`, `AI.PREDICT` + `AI.EVALUATE` on TabFM).
+3. **Pre-Trained Cloud AI Service Models ([`bigquery-ai`](../bigquery-ai/SKILL.md))** — `CREATE MODEL ... OPTIONS(REMOTE_SERVICE_TYPE = '...')` consumed via `ML.PROCESS_DOCUMENT`, `ML.TRANSLATE`, `ML.UNDERSTAND_TEXT`, `ML.ANNOTATE_IMAGE`, and `ML.TRANSCRIBE`.
+
+Before writing SQL or Python, check the [Namespace & Prerequisite Matrix (`reference/namespace-matrix.md`)](reference/namespace-matrix.md) to verify whether a symbol requires a Cloud Resource Connection, IAM role, or BigQuery Enterprise Edition reservation.
+
+## 5 Canonical Skill Patterns
+
+1. **Router Pattern (Triage First)** — Run through the 9 diagnostic questions below to route the task to [`bigquery-ml`](../bigquery-ml/SKILL.md), [`bigquery-ai`](../bigquery-ai/SKILL.md), or a composed pipeline of both.
+2. **Inversion Pattern (Ask Before Guessing)** — When a user's request is underspecified (e.g., "forecast sales" without specifying horizon, covariates, holiday calendars, or reproducibility requirements), present 2–3 concrete options with their measured tradeoffs rather than silently picking one.
+3. **Tool Wrapper Pattern (Dual SQL & `bigframes`)** — Pair BigQuery SQL with the matching `bigframes.ml` / `bigframes.bigquery.ai` Python wrapper when a notebook or DataFrame workflow is requested, and use `bigquery-data-science lookup <SYMBOL>` or MCP `lookup_bigquery_syntax` to fetch exact option signatures.
+4. **Generator Pattern (Idempotent Templates)** — Emit self-contained SQL/Python templates with explicit model/endpoint pins (checked against `skills/pinned-models.json`), deterministic splits (`FARM_FINGERPRINT`), and point-in-time feature joins (`ML.ENTITY_FEATURES_AT_TIME`).
+5. **Reviewer Pattern (Verification Rubric)** — Audit generated queries against the cross-domain gotchas below (no retired `AI.PARSE_DOCUMENT`, no unpinned `endpoint`, no leaky `GROUP BY` feature joins, no missing reservation warnings for `MATRIX_FACTORIZATION` or `ML.DECODE_IMAGE`).
+
+## 9 Triage Questions — Ask Yourself (or the User) First
+
+1. **Do you have labeled historical data, and do you want a model that improves as data grows and retrains on a schedule?**
+   - Yes → [`bigquery-ml`](../bigquery-ml/SKILL.md).
+   - No / zero-setup baseline → [`bigquery-ai`](../bigquery-ai/SKILL.md).
+2. **Do you need coefficient-level interpretability, p-values, feature attribution, or a deterministic audit trail?**
+   - Yes → [`bigquery-ml`](../bigquery-ml/SKILL.md) (`ML.WEIGHTS`, `ML.ADVANCED_WEIGHTS`, `ML.GLOBAL_EXPLAIN`, `ML.FEATURE_IMPORTANCE`, `ML.EXPLAIN_PREDICT`).
+3. **Is the task naturally expressed as a natural-language instruction or schema extraction** ("is this compliant?", "extract line items", "score 1–10", "summarize per category")?
+   - Yes → [`bigquery-ai`](../bigquery-ai/SKILL.md) (`AI.IF`, `AI.SCORE`, `AI.CLASSIFY`, `AI.AGG`, `AI.GENERATE`, `AI.GENERATE_TABLE`).
+4. **For tabular regression or classification, do you need reproducible predictions, > 50 feature columns, or > 10 classes?**
+   - Yes → [`bigquery-ml`](../bigquery-ml/SKILL.md) (`LINEAR_REG`, `LOGISTIC_REG`, `BOOSTED_TREE_CLASSIFIER`, `BOOSTED_TREE_REGRESSOR`).
+   - No, and you want a zero-training baseline on a small table (<= 50 features, <= 10 classes) → `AI.PREDICT` + `AI.EVALUATE` (TabFM in [`bigquery-ai`](../bigquery-ai/SKILL.md)).
+5. **For time series, are you forecasting the future or decomposing the past?**
+   - **Forecasting with custom holidays, hierarchy, `ML.EXPLAIN_FORECAST` decomposition, or horizon > 1,024** → `ARIMA_PLUS` / `ARIMA_PLUS_XREG` ([`bigquery-ml`](../bigquery-ml/SKILL.md)).
+   - **Forecasting with zero training (univariate or multivariate/covariates up to 1,024 steps)** → `AI.FORECAST` with `model => 'TimesFM 2.5'` or `'TimesFM 3.0'` ([`bigquery-ai`](../bigquery-ai/SKILL.md)).
+   - **Decomposing past trend, seasonality, or structural level shifts with no forecast** → `ML.TREND`, `ML.SEASONALITY`, `ML.DETECT_CHANGE_POINTS` ([`bigquery-ml`](../bigquery-ml/SKILL.md)).
+6. **For root-cause metric attribution ("why did conversion drop?"), do you have > 12 dimensions or a ratio/category metric?**
+   - Yes → `CONTRIBUTION_ANALYSIS` + `ML.GET_INSIGHTS` ([`bigquery-ml`](../bigquery-ml/SKILL.md)).
+   - No (<= 12 dimensions, summable metric) → `AI.KEY_DRIVERS` ([`bigquery-ai`](../bigquery-ai/SKILL.md)).
+7. **Is the input unstructured text, PDFs, images, or audio needing embeddings, semantic/hybrid search, or multimodal generation?**
+   - Yes → [`bigquery-ai`](../bigquery-ai/SKILL.md) (`AI.EMBED`, `VECTOR_SEARCH`, `AI.SEARCH`, `AI.GENERATE*` over `ObjectRef`).
+8. **Is the task solved by a pre-trained Google Cloud AI API where you need a versioned schema and per-row status column?**
+   - Yes → Cloud AI service models in [`bigquery-ai`](../bigquery-ai/SKILL.md) (`ML.PROCESS_DOCUMENT`, `ML.TRANSLATE`, `ML.UNDERSTAND_TEXT`, `ML.ANNOTATE_IMAGE`, `ML.TRANSCRIBE`).
+9. **Do you need causal treatment-effect estimation or production MLOps retraining & drift monitoring?**
+   - Yes → [`bigquery-ml`](../bigquery-ml/SKILL.md) (`AI.CAUSAL_EFFECT`, `difference_in_differences`, `synthetic_control`, `propensity_score_matching`, `price_elasticity_dml`, `uplift_cate`, `ML.VALIDATE_DATA_DRIFT`, and 8 orchestration pipelines).
+
+## 10 Empirical Head-to-Head Comparisons
+
+1. **Forecasting (`ARIMA_PLUS` / `ARIMA_PLUS_XREG` vs `AI.FORECAST`)**: `AI.FORECAST` requires no `CREATE MODEL` and with `model => 'TimesFM 3.0'` (Preview) accepts `target_cols`, `past_covariate_cols`, and `future_covariate_cols` (capped at 1,024 horizon steps and 2,048 context points). `ARIMA_PLUS` and `ARIMA_PLUS_XREG` add custom holiday calendars, hierarchical reconciliation, `ML.EXPLAIN_FORECAST` decomposition, and `ML.ARIMA_COEFFICIENTS`.
+2. **Time-Series Decomposition (`ML.EXPLAIN_FORECAST` vs `ML.TREND` / `ML.SEASONALITY`)**: Tested on the same Citi Bike series, `ML.SEASONALITY` produces weekly and yearly components **bit-identical** to `ARIMA_PLUS`'s `ML.EXPLAIN_FORECAST` (`corr 1.0`, `mean abs diff 0.0`) with zero model training. `ML.TREND` defaults `adjust_step_changes => FALSE` whereas `ARIMA_PLUS` defaults to `TRUE`.
+3. **Change Points vs Anomalies vs Dataset Drift (`ML.DETECT_CHANGE_POINTS` vs `AI.DETECT_ANOMALIES` / `ML.DETECT_ANOMALIES` vs `ML.VALIDATE_DATA_DRIFT`)**: `ML.DETECT_CHANGE_POINTS` detects persistent level shifts (and gap-fills missing dates first — profile for ingestion outages before trusting change points!); `AI.DETECT_ANOMALIES` and `ML.DETECT_ANOMALIES` flag point spikes; `ML.VALIDATE_DATA_DRIFT` compares feature distributions across two tables.
+4. **Driver Attribution (`CONTRIBUTION_ANALYSIS` vs `AI.KEY_DRIVERS`)**: `AI.KEY_DRIVERS` (GA) is a single model-free TVF call for summable metrics with `<= 12` dimensions; `CONTRIBUTION_ANALYSIS` handles `> 12` dimensions plus summable-by-ratio and summable-by-category metrics.
+5. **Causal Counterfactuals (`AI.CAUSAL_EFFECT` vs Econometric Panel Estimators)**: `AI.CAUSAL_EFFECT` (documented in `bigquery-ml`) is a model-free interrupted time-series counterfactual on a single treated series (needs `> 1` year pre-history when annual seasonality is active). When untreated control units exist, prefer `difference_in_differences` or `synthetic_control`.
+6. **Tabular Prediction (`LINEAR_REG` / `BOOSTED_TREE_*` vs `AI.PREDICT` & `ML.EVALUATE` vs `AI.EVALUATE`)**: `AI.PREDICT` (TabFM) needs zero training but caps at 50 features and 10 classes, and its predictions are **non-deterministic** across identical runs. `AI.EVALUATE` scores its own internal pass rather than materialized `AI.PREDICT` outputs, whereas `ML.EVALUATE` and `ML.METRICS` deterministically score saved predictions.
+7. **Unstructured Classification (Trained Classifier vs `AI.CLASSIFY`)**: `AI.CLASSIFY` classifies text or `ObjectRef` media into a fixed category list with zero labeled data; `BOOSTED_TREE_CLASSIFIER` or `LOGISTIC_REG` over `AI.EMBED` features (`embeddings_classification`) is cheaper and faster at high row volume once labeled data exists.
+8. **Embeddings (`PCA` / `AUTOENCODER` / `MATRIX_FACTORIZATION` vs `AI.EMBED` / `AI.GENERATE_EMBEDDING`)**: Use `bigquery-ml` models for structured/tabular entity embeddings; use `AI.EMBED` (`model => 'embeddinggemma-300m'` or `'gemini-embedding-001'`) for text and multimodal embeddings paired with `VECTOR_SEARCH` or `AI.SEARCH`.
+9. **Document Extraction (`ML.PROCESS_DOCUMENT` vs `AI.GENERATE_TABLE` / `AI.GENERATE`)**: Never use `AI.PARSE_DOCUMENT` (retired 2026-06-01). Use `ML.PROCESS_DOCUMENT` with a Document AI processor when you need a deterministic layout/invoice schema and per-row `ml_process_document_status`; use `AI.GENERATE` with `output_schema` over `ObjectRef` when you need custom ad-hoc field extraction without provisioning a processor.
+10. **Pre-Trained Cloud AI Services vs Generative Prompts (`ML.TRANSLATE` / `ML.UNDERSTAND_TEXT` / `ML.ANNOTATE_IMAGE` / `ML.TRANSCRIBE` vs `AI.GENERATE`)**: Cloud AI service models provide fixed taxonomies and per-row error reporting (`ML.TRANSCRIBE` is the one non-deterministic member). Combine both by persisting the service output column and running `AI.GENERATE` or `AI.AGG` over it.
+
+## Routing Links
+
+- **Prerequisite & Namespace Matrix (Lens 2)**: [`reference/namespace-matrix.md`](reference/namespace-matrix.md)
+- **BigQuery ML Skill (`CREATE MODEL`, `ML.*`, Feature Store, Causal, Pipelines)**: [`../bigquery-ml/SKILL.md`](../bigquery-ml/SKILL.md)
+- **BigQuery AI Skill (`AI.*`, Cloud AI Service Models, Vector/Hybrid Search, RAG)**: [`../bigquery-ai/SKILL.md`](../bigquery-ai/SKILL.md)

bigquery_data_science/_skills/bigquery-data-science/reference/namespace-matrix.md

@@ -0,0 +1,112 @@
+# BigQuery Data Science Namespace & Prerequisite Matrix
+
+Contents:
+- [Why Namespace Prefix != Skill Folder](#why-namespace-prefix--skill-folder)
+- [BigQuery Edition Reservation Requirements](#bigquery-edition-reservation-requirements)
+- [Trained Models (`CREATE MODEL` in `bigquery-ml`)](#trained-models-create-model-in-bigquery-ml)
+- [Model Lifecycle & Model-Free Functions (`ML.*` in `bigquery-ml`)](#model-lifecycle--model-free-functions-ml-in-bigquery-ml)
+- [Generative, Foundation & Managed Functions (`AI.*` & `VECTOR_SEARCH` in `bigquery-ai`)](#generative-foundation--managed-functions-ai--vector_search-in-bigquery-ai)
+- [Pre-Trained Cloud AI Service Models & Object Infrastructure (`bigquery-ai`)](#pre-trained-cloud-ai-service-models--object-infrastructure-bigquery-ai)
+
+## Why Namespace Prefix != Skill Folder
+
+BigQuery's SQL prefix (`ML.*` vs `AI.*`) reflects when a feature was introduced and whether a `MODEL` DDL handle exists — **not** whether a model is trained:
+
+- **Cross-prefix placement in [`bigquery-ml`](../../bigquery-ml/SKILL.md)**: `AI.CAUSAL_EFFECT` uses the `AI.*` prefix (model-free interrupted time-series counterfactual TVF) but is documented in `bigquery-ml` alongside `ARIMA_PLUS` and the econometric causal inference workflows (`difference_in_differences`, `synthetic_control`, `propensity_score_matching`, `price_elasticity_dml`, `uplift_cate`).
+- **Cross-prefix placement in [`bigquery-ai`](../../bigquery-ai/SKILL.md)**: `ML.PROCESS_DOCUMENT`, `ML.TRANSLATE`, `ML.UNDERSTAND_TEXT`, `ML.ANNOTATE_IMAGE`, and `ML.TRANSCRIBE` use `CREATE MODEL ... OPTIONS(REMOTE_SERVICE_TYPE = '...')` and `ML.*` TVFs, plus legacy `ML.GENERATE_TEXT` and `ML.GENERATE_EMBEDDING`, and are documented in `bigquery-ai`.
+
+## BigQuery Edition Reservation Requirements
+
+Most BigQuery ML and BigQuery AI features run on standard on-demand pricing, **except** three surfaces that strictly require a BigQuery **`ENTERPRISE` (or `ENTERPRISE_PLUS`) Edition reservation**:
+
+1. **`MATRIX_FACTORIZATION` (`CREATE MODEL` in `bigquery-ml`)** — requires a `QUERY` assignment on an `ENTERPRISE` reservation.
+2. **SQL Image Preprocessing (`ML.DECODE_IMAGE`, `ML.RESIZE_IMAGE`, `ML.CONVERT_COLOR_SPACE`, `ML.CONVERT_IMAGE_TYPE` in `bigquery-ml`)** — all four functions fail on on-demand pricing even on inline `FROM_BASE64` bytes; require an `ENTERPRISE` reservation.
+3. **Vector Index Background Maintenance (`ALTER VECTOR INDEX` in `bigquery-ai`)** — requires a `BACKGROUND` assignment on an `ENTERPRISE` reservation in the dataset's region.
+
+## Trained Models (`CREATE MODEL` in `bigquery-ml`)
+
+| Symbol | Skill | Category | Connection Required | Reservation Required | Status | Walkthrough |
+| --- | --- | --- | --- | --- | --- | --- |
+| `LINEAR_REG` | `bigquery-ml` | Regression | None | On-demand OK | GA | `bigquery-ml/narrative/linear_regression.md` |
+| `LOGISTIC_REG` | `bigquery-ml` | Classification | None | On-demand OK | GA | `bigquery-ml/narrative/logistic_regression.md` |
+| `BOOSTED_TREE_CLASSIFIER` | `bigquery-ml` | Classification (XGBoost) | None | On-demand OK | GA | `bigquery-ml/narrative/boosted_tree_classifier.md` |
+| `BOOSTED_TREE_REGRESSOR` | `bigquery-ml` | Regression (XGBoost) | None | On-demand OK | GA | `bigquery-ml/narrative/boosted_tree_regressor.md` |
+| `RANDOM_FOREST_CLASSIFIER` | `bigquery-ml` | Classification | None | On-demand OK | GA | `bigquery-ml/narrative/random_forest_classifier.md` |
+| `RANDOM_FOREST_REGRESSOR` | `bigquery-ml` | Regression | None | On-demand OK | GA | `bigquery-ml/narrative/random_forest_regressor.md` |
+| `DNN_CLASSIFIER` | `bigquery-ml` | Deep Neural Network | None | On-demand OK | GA | `bigquery-ml/narrative/dnn_classifier.md` |
+| `DNN_REGRESSOR` | `bigquery-ml` | Deep Neural Network | None | On-demand OK | GA | `bigquery-ml/narrative/dnn_regressor.md` |
+| `WIDE_AND_DEEP_CLASSIFIER` | `bigquery-ml` | Wide & Deep Network | None | On-demand OK | GA | `bigquery-ml/narrative/wide_and_deep_classifier.md` |
+| `WIDE_AND_DEEP_REGRESSOR` | `bigquery-ml` | Wide & Deep Network | None | On-demand OK | GA | `bigquery-ml/narrative/wide_and_deep_regressor.md` |
+| `AUTOML_CLASSIFIER` | `bigquery-ml` | AutoML Tabular | None | On-demand OK | GA | `bigquery-ml/narrative/automl_classifier.md` |
+| `AUTOML_REGRESSOR` | `bigquery-ml` | AutoML Tabular | None | On-demand OK | GA | `bigquery-ml/narrative/automl_regressor.md` |
+| `ARIMA_PLUS` | `bigquery-ml` | Univariate Forecasting (`TIME_SERIES_TIMESTAMP_COL`, `TIME_SERIES_DATA_COL`) | None | On-demand OK | GA | `bigquery-ml/narrative/arima_plus.md` |
+| `ARIMA_PLUS_XREG` | `bigquery-ml` | Covariate Forecasting (`TIME_SERIES_TIMESTAMP_COL`, `TIME_SERIES_DATA_COL`) | None | On-demand OK | GA | `bigquery-ml/narrative/arima_plus_xreg.md` |
+| `KMEANS` | `bigquery-ml` | Clustering | None | On-demand OK | GA | `bigquery-ml/narrative/kmeans.md` |
+| `PCA` | `bigquery-ml` | Dimensionality Reduction | None | On-demand OK | GA | `bigquery-ml/narrative/pca.md` |
+| `AUTOENCODER` | `bigquery-ml` | Representation / Anomaly | None | On-demand OK | GA | `bigquery-ml/narrative/autoencoder.md` |
+| `MATRIX_FACTORIZATION` | `bigquery-ml` | Collaborative Filtering | None | **Enterprise Reservation (`QUERY`)** | GA | `bigquery-ml/narrative/matrix_factorization.md` |
+| `CONTRIBUTION_ANALYSIS` | `bigquery-ml` | Driver Attribution | None | On-demand OK | GA | `bigquery-ml/narrative/contribution_analysis.md` |
+| `TRANSFORM_ONLY` | `bigquery-ml` | Reusable Feature Pipeline | None | On-demand OK | GA | `bigquery-ml/narrative/transform_only.md` |
+| `TENSORFLOW` / `ONNX` / `XGBOOST` / `TENSORFLOW_LITE` | `bigquery-ml` | Imported GCS Models | Optional (for Object Tables) | On-demand OK | GA | `bigquery-ml/narrative/imported.md` |
+| `REMOTE WITH CONNECTION` (`endpoint`) | `bigquery-ml` | Vertex AI Endpoint | `CLOUD_RESOURCE` (`roles/aiplatform.user`) | On-demand OK | GA | `bigquery-ml/narrative/remote.md` |
+
+## Model Lifecycle & Model-Free Functions (`ML.*` in `bigquery-ml`)
+
+| Symbol | Skill | Category | Connection Required | Reservation Required | Status | Walkthrough |
+| --- | --- | --- | --- | --- | --- | --- |
+| `ML.PREDICT` | `bigquery-ml` | Batch Inference | Only for remote models | On-demand OK | GA | `bigquery-ml/narrative/logistic_regression.md` |
+| `ML.EVALUATE` | `bigquery-ml` | Model / Prediction Eval | None | On-demand OK | GA | `bigquery-ml/narrative/evaluation.md` |
+| `ML.METRICS` | `bigquery-ml` | Model-Free Eval | None | On-demand OK | Preview | `bigquery-ml/narrative/evaluation.md` |
+| `ML.ROC_CURVE` / `ML.CONFUSION_MATRIX` | `bigquery-ml` | Classification Eval | None | On-demand OK | GA | `bigquery-ml/narrative/evaluation.md` |
+| `ML.FORECAST` / `ML.EXPLAIN_FORECAST` | `bigquery-ml` | Time-Series Inference | None | On-demand OK | GA | `bigquery-ml/narrative/arima_plus.md` |
+| `ML.ARIMA_EVALUATE` / `ML.ARIMA_COEFFICIENTS` | `bigquery-ml` | Time-Series Diagnostics | None | On-demand OK | GA | `bigquery-ml/narrative/arima_plus.md` |
+| `ML.TREND` / `ML.SEASONALITY` / `ML.DETECT_CHANGE_POINTS` | `bigquery-ml` | Model-Free Time-Series TVFs | None | On-demand OK | Preview | `bigquery-ml/narrative/time_series.md` |
+| `AI.CAUSAL_EFFECT` | `bigquery-ml` | Interrupted Time-Series Causal TVF | None | On-demand OK | Preview | `bigquery-ml/narrative/causal_effect.md` |
+| `ML.DETECT_ANOMALIES` | `bigquery-ml` | Model-Based Anomaly Detection | None | On-demand OK | GA | `bigquery-ml/narrative/anomaly_fraud_detection.md` |
+| `ML.RECOMMEND` | `bigquery-ml` | Collaborative Filtering Eval | None | On-demand OK | GA | `bigquery-ml/narrative/matrix_factorization.md` |
+| `ML.GET_INSIGHTS` | `bigquery-ml` | Contribution Analysis Output | None | On-demand OK | GA | `bigquery-ml/narrative/contribution_analysis.md` |
+| `ML.EXPLAIN_PREDICT` / `ML.GLOBAL_EXPLAIN` / `ML.FEATURE_IMPORTANCE` / `ML.WEIGHTS` / `ML.ADVANCED_WEIGHTS` | `bigquery-ml` | Explainability & Weights | None | On-demand OK | GA | `bigquery-ml/narrative/evaluation.md` |
+| `ML.FEATURES_AT_TIME` / `ML.ENTITY_FEATURES_AT_TIME` | `bigquery-ml` | Point-in-Time Feature Store | None | On-demand OK | GA | `bigquery-ml/narrative/feature_store.md` |
+| `ML.DESCRIBE_DATA` / `ML.CORRELATION` / `ML.TF_IDF` / `ML.NGRAMS` / `ML.BAG_OF_WORDS` | `bigquery-ml` | EDA & Text Features | None | On-demand OK | GA / Preview | `bigquery-ml/narrative/exploration.md` |
+| `ML.STANDARD_SCALER` / `ML.MIN_MAX_SCALER` / `ML.ROBUST_SCALER` / `ML.MAX_ABS_SCALER` / `ML.NORMALIZER` | `bigquery-ml` | Numerical Scaling | None | On-demand OK | GA | `bigquery-ml/narrative/scalers.md` |
+| `ML.ONE_HOT_ENCODER` / `ML.LABEL_ENCODER` / `ML.TARGET_ENCODER` / `ML.FREQUENCY_ENCODER` / `ML.HASH_BUCKETIZE` | `bigquery-ml` | Categorical Encoding | None | On-demand OK | GA / Preview | `bigquery-ml/narrative/encoding.md` |
+| `ML.BUCKETIZE` / `ML.QUANTILE_BUCKETIZE` / `ML.IMPUTER` / `ML.POLYNOMIAL_EXPAND` / `ML.FEATURE_CROSS` | `bigquery-ml` | Bucketizing & Feature Crosses | None | On-demand OK | GA | `bigquery-ml/narrative/bucketizing.md` |
+| `ML.LP_NORM` / `ML.DISTANCE` / `ML.COSINE_DISTANCE` / `ML.EUCLIDEAN_DISTANCE` / `ML.MANHATTAN_DISTANCE` / `ML.EDIT_DISTANCE` | `bigquery-ml` | Vector & String Distances | None | On-demand OK | GA / Preview | `bigquery-ml/narrative/distance.md` |
+| `ML.DECODE_IMAGE` / `ML.RESIZE_IMAGE` / `ML.CONVERT_COLOR_SPACE` / `ML.CONVERT_IMAGE_TYPE` | `bigquery-ml` | SQL Image Preprocessing | Optional (Object Table) | **Enterprise Reservation (`QUERY`)** | Preview | `bigquery-ml/narrative/image.md` |
+| `ML.VALIDATE_DATA_DRIFT` / `ML.VALIDATE_DATA_SKEW` / `ML.TFDV_DESCRIBE` / `ML.TFDV_VALIDATE` | `bigquery-ml` | Data Drift & Skew Monitoring | None | On-demand OK | GA | `bigquery-ml/narrative/model_monitoring.md` |
+
+## Generative, Foundation & Managed Functions (`AI.*` & `VECTOR_SEARCH` in `bigquery-ai`)
+
+| Symbol | Skill | Category | Connection Required | Reservation Required | Status | Walkthrough |
+| --- | --- | --- | --- | --- | --- | --- |
+| `AI.GENERATE` | `bigquery-ai` | General-Purpose Scalar Gen / Schema | End-user creds (Connection optional) | On-demand OK | GA | `bigquery-ai/narrative/ai_generate.md` |
+| `AI.GENERATE_BOOL` / `AI.GENERATE_DOUBLE` / `AI.GENERATE_INT` | `bigquery-ai` | Typed Scalar Generation | End-user creds (Connection optional) | On-demand OK | Preview | `bigquery-ai/narrative/ai_generate_bool.md` |
+| `AI.GENERATE_TABLE` | `bigquery-ai` | Structured TVF Generation | `CLOUD_RESOURCE` (`roles/aiplatform.user`) | On-demand OK | GA | `bigquery-ai/narrative/ai_generate_table.md` |
+| `AI.GENERATE_TEXT` / `ML.GENERATE_TEXT` | `bigquery-ai` | Remote Model Text TVF | `CLOUD_RESOURCE` (`roles/aiplatform.user`) | On-demand OK | GA | `bigquery-ai/narrative/ai_generate_text.md` |
+| `AI.COUNT_TOKENS` | `bigquery-ai` | Local Token Estimator | None | On-demand OK | Preview | `bigquery-ai/narrative/ai_count_tokens.md` |
+| `AI.IF` | `bigquery-ai` | Managed Boolean Filter | End-user creds (Connection optional) | On-demand OK | GA | `bigquery-ai/narrative/ai_if.md` |
+| `AI.SCORE` | `bigquery-ai` | Managed Numeric Rubric Scorer | End-user creds (Connection optional) | On-demand OK | GA | `bigquery-ai/narrative/ai_score.md` |
+| `AI.CLASSIFY` | `bigquery-ai` | Managed Category Classifier | End-user creds (Connection optional) | On-demand OK | GA | `bigquery-ai/narrative/ai_classify.md` |
+| `AI.AGG` | `bigquery-ai` | Managed Aggregate Synthesis | End-user creds (Connection optional) | On-demand OK | Preview | `bigquery-ai/narrative/ai_agg.md` |
+| `AI.EMBED` | `bigquery-ai` | Scalar Text / Multimodal Embedding | None for `embeddinggemma-300m`; Connection for multimodal | On-demand OK | GA | `bigquery-ai/narrative/ai_embed.md` |
+| `AI.GENERATE_EMBEDDING` / `ML.GENERATE_EMBEDDING` | `bigquery-ai` | Remote Model Embedding TVF | `CLOUD_RESOURCE` (`roles/aiplatform.user`) | On-demand OK | GA | `bigquery-ai/narrative/ai_generate_embedding.md` |
+| `AI.SIMILARITY` | `bigquery-ai` | Scalar Pairwise Semantic Similarity | None for `embeddinggemma-300m`; Connection for multimodal | On-demand OK | GA | `bigquery-ai/narrative/ai_similarity.md` |
+| `VECTOR_SEARCH` | `bigquery-ai` | Vector & Hybrid RRF Search TVF | Only when query embedding uses remote model | On-demand OK (`ALTER VECTOR INDEX` needs Enterprise `BACKGROUND`) | GA / Preview (hybrid) | `bigquery-ai/narrative/vector_search.md` |
+| `AI.SEARCH` | `bigquery-ai` | Simplified Autonomous Search TVF | None (autonomous embedding column) | On-demand OK | Preview | `bigquery-ai/narrative/ai_search.md` |
+| `AI.FORECAST` | `bigquery-ai` | Zero-Training TimesFM Forecasting | None | On-demand OK | GA (2.5) / Preview (3.0) | `bigquery-ai/narrative/ai_forecast.md` |
+| `AI.DETECT_ANOMALIES` | `bigquery-ai` | Zero-Training TimesFM Anomalies | None | On-demand OK | GA | `bigquery-ai/narrative/ai_detect_anomalies.md` |
+| `AI.EVALUATE` | `bigquery-ai` | Zero-Training TimesFM / TabFM Eval | None | On-demand OK | GA (TS) / Preview (TabFM) | `bigquery-ai/narrative/ai_evaluate.md` |
+| `AI.PREDICT` | `bigquery-ai` | Zero-Training TabFM Prediction | None | On-demand OK | Preview | `bigquery-ai/narrative/ai_predict.md` |
+| `AI.KEY_DRIVERS` | `bigquery-ai` | Model-Free Driver Analysis TVF | None | On-demand OK | GA | `bigquery-ai/narrative/ai_key_drivers.md` |
+| `AI.PARSE_DOCUMENT` | `bigquery-ai` | **RETIRED (Offline 2026-06-01)** | Replaced by `ML.PROCESS_DOCUMENT` | N/A | Retired | `bigquery-ai/narrative/ai_parse_document.md` |
+
+## Pre-Trained Cloud AI Service Models & Object Infrastructure (`bigquery-ai`)
+
+| Symbol | Skill | Category | Connection Required | Reservation Required | Status | Walkthrough |
+| --- | --- | --- | --- | --- | --- | --- |
+| `ML.PROCESS_DOCUMENT` | `bigquery-ai` | Document AI (`CLOUD_AI_DOCUMENT_V1`) | `CLOUD_RESOURCE` (`roles/documentai.apiUser` + `serviceUsageConsumer`) | On-demand OK | GA | `bigquery-ai/narrative/ml_process_document.md` |
+| `ML.TRANSLATE` | `bigquery-ai` | Cloud Translation (`CLOUD_AI_TRANSLATE_V3`) | `CLOUD_RESOURCE` (`roles/cloudtranslate.user` + `serviceUsageConsumer`) | On-demand OK | GA | `bigquery-ai/narrative/ml_translate.md` |
+| `ML.UNDERSTAND_TEXT` | `bigquery-ai` | Cloud Natural Language (`CLOUD_AI_NATURAL_LANGUAGE_V1`) | `CLOUD_RESOURCE` (`roles/serviceusage.serviceUsageConsumer`) | On-demand OK | GA | `bigquery-ai/narrative/ml_understand_text.md` |
+| `ML.ANNOTATE_IMAGE` | `bigquery-ai` | Cloud Vision (`CLOUD_AI_VISION_V1`) | `CLOUD_RESOURCE` (`roles/serviceusage.serviceUsageConsumer`) | On-demand OK | GA | `bigquery-ai/narrative/ml_annotate_image.md` |
+| `ML.TRANSCRIBE` | `bigquery-ai` | Cloud Speech-to-Text (`CLOUD_AI_SPEECH_V1`) | `CLOUD_RESOURCE` (`roles/speech.client` + `serviceUsageConsumer` + Recon SA grant) | On-demand OK | GA | `bigquery-ai/narrative/ml_transcribe.md` |
+| `OBJ.MAKE_REF` / `OBJ.FETCH_METADATA` / `OBJ.GET_ACCESS_URL` / `OBJ.LIST` / `OBJ.READ` | `bigquery-ai` | Unstructured `ObjectRef` Functions | `CLOUD_RESOURCE` (`roles/storage.objectViewer`) | On-demand OK (Object Table + remote model joins may require Enterprise) | GA / Preview | `bigquery-ai/narrative/multimodal_analysis.md` |

bigquery_data_science/_skills/bigquery-data-science/skill.manifest.json

@@ -0,0 +1,13 @@
+{
+  "name": "bigquery-data-science",
+  "version": "0.7.0",
+  "source_project": null,
+  "narrative_sources": {},
+  "generated_date": "2026-10-19",
+  "source_commit": null,
+  "reference_files": [
+    "namespace-matrix.md"
+  ],
+  "syntax_files": [],
+  "narrative_files": []
+}