bigquery-data-science 0.7.0__py3-none-any.whl

bigquery_data_science/_skills/bigquery-ai/syntax/aipredict.md

@@ -0,0 +1,74 @@
+# `AI.PREDICT`
+
+> **Source:** [`bigquery-ai/reference/predictive-ai.md#aipredict`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/predictive-ai.md#aipredict)
+
+- **Description:** (Preview) Table-valued function that performs zero-shot regression and classification on structured data using TabFM, Google's pre-trained tabular foundation model. You pass a labeled training table and an unlabeled prediction table in a single call; the model learns in context. There is no `CREATE MODEL`, no training job, no connection, no endpoint, and no persisted model object.
+- **Use cases:** Predicting a numeric or categorical column without building a model, filling in missing structured attributes, quick baselines before investing in a trained model, prediction inside an analytics pipeline where a model artifact would be overhead.
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-predict)
+- **Type:** Table-valued function (TVF) -- Preview
+
+**Syntax:**
+```sql
+AI.PREDICT(
+  { TABLE TRAINING_TABLE | (TRAINING_QUERY) },
+  { TABLE PREDICTION_TABLE | (PREDICTION_QUERY) }
+  [, label_col => 'LABEL_COL' ]
+)
+```
+
+**Inputs:**
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `TRAINING_TABLE` / `TRAINING_QUERY` | Table/Query | Required | -- | Labeled training data. Must contain the label column; **every other column is treated as a feature**. |
+| `PREDICTION_TABLE` / `PREDICTION_QUERY` | Table/Query | Required | -- | Rows to predict. Must contain all training feature columns; extra columns are allowed. |
+| `label_col` | Named arg, STRING | Optional | `'label'` | Name of the label column in the training data. A column literally named `label` needs no argument. |
+
+Feature and label columns must be `STRING`, `BOOL`, `INT64`, `FLOAT64`, `NUMERIC`, or `BIGNUMERIC`. `DATE`, `TIMESTAMP`, `BYTES`, `JSON`, `GEOGRAPHY`, `ARRAY`, and `STRUCT` are rejected -- `EXTRACT` date parts into integers first.
+
+**Task selection is implicit and type-driven.** There is no `task_type` argument:
+
+| Label column type | Task | Added output columns |
+|---|---|---|
+| `INT64`, `FLOAT64`, `NUMERIC`, `BIGNUMERIC` | Regression | `predicted_<label>` (same type as the label) |
+| `BOOL`, `STRING` | Classification | `predicted_<label>`, plus `predicted_<label>_probs` as `ARRAY<STRUCT<label STRING, prob FLOAT64>>` |
+
+> **Cast your label.** A categorical encoded as INT64 (0/1, or a 1–5 rating) is silently treated as *regression* and you get fractional predictions with no probabilities. Cast it to `STRING` or `BOOL` to get classification. This consequence follows from the documented rule but is not itself documented.
+
+The `label` subfield of `predicted_<label>_probs` is typed `STRING` even when the label column is `BOOL`. Array element ordering is not documented.
+
+**Outputs:** The prediction input's columns plus the predicted column(s) above.
+
+> **Doc bug.** The reference page states that AI.PREDICT "returns the columns from the training table or query result". Both of Google's own worked examples contradict this -- the returned rows are the *prediction* rows. Trust the behavior, not the sentence.
+
+**Supported models:** TabFM only. Not selectable -- there is no `model` argument and no version pinning.
+
+**Best practices:** Keep the feature set tight; the 50-column cap is a hard limit, not a guideline. Split train/predict deterministically (e.g. `FARM_FINGERPRINT`) so that the same rows train and predict on every run -- note this pins the *split* only; the predictions themselves are not reproducible (see Limitations). Pair every AI.PREDICT call with `AI.EVALUATE` on a held-out set -- without a metric you have no idea whether the zero-shot prediction is any good. Use it as a baseline: if a trained `CREATE MODEL` beats it materially, the training cost is justified; if not, you have saved a model lifecycle.
+
+**Limitations:**
+- **Output is not deterministic.** Repeating a byte-identical `AI.PREDICT` call over the same `FARM_FINGERPRINT` split returns slightly different predictions -- consistent with the model averaging shuffled ensemble passes (`n_ensembles`, whose value Google does not document). Demonstrated by `functions/ai_predict/ai_predict.ipynb` cells 25 and 28, whose top-5 multisets differ (`5760` vs `5728` in one slot) despite identical SQL and an `ORDER BY value DESC LIMIT 5`, so tie-breaking cannot explain it. The `workflows/tabular_prediction/` notebook shows the downstream consequence: `AI.EVALUATE` scores its *own* fresh TabFM predictions rather than the rows you materialized, so the two disagree (measured across four runs: regression MAE 236.52 / 236.35 / 235.84, classification accuracy 0.9468 / 0.9362 / 0.9468 -- regression drift well under 1%, classification drift about 1pp, or 2 rows in 94). **Materialize results once and join to them; do not re-run the prediction to reproduce a number.** Drift appears to be input-dependent -- `workflows/data_enrichment/` reproduced exactly across two runs on a 6-row prediction relation -- so absence of drift on a small input is not evidence of determinism.
+- **50 feature columns** maximum (documented; was 20 until October 2026 — measured 2026-10-12: 50 runs, 51 fails with *The number of features 51 exceeds the maximum allowed number of features 50*). Escalation path is emailing bqml-feedback@google.com.
+- **10 classification categories** maximum (documented).
+- **Practical row ceiling, undocumented but observed:** a call with 10,000 training rows fails with `Resources exceeded ... allotted memory`; 8,000 rows succeeds. Budget for roughly 5,000 training rows on on-demand slots. Re-checked 2026-09-15: the two caps above are on **columns and classes**, not rows — the page still documents no row limit, so this remains an observation of ours rather than a published figure.
+- **Latency is flat and high:** 30--95 seconds per call regardless of input size. A notebook with several AI.PREDICT calls takes minutes, not seconds.
+- Max prediction rows, max input size, timeouts, concurrency, NULL handling, categorical encoding, and the `n_ensembles` value are all **undocumented**. AI.PREDICT does not appear in any table on the BigQuery quotas page.
+- Whether these limits also bind `AI.EVALUATE`'s TabFM branch is undocumented, though it runs the same model.
+
+**Locations:** Not stated on the AI.PREDICT page. The AI.EVALUATE page says TabFM is available in all supported BigQuery ML locations (non-remote models), plus the US and EU multi-regions.
+
+**Pricing:** Today, standard BigQuery slot / bytes-processed pricing. **From 2026-10-30**, TabFM moves to token-based pricing: you are charged for TabFM tokens plus slots/bytes for the non-inference parts of the query.
+
+```
+Input tokens  = (train_rows * columns + predict_rows * (columns - 1)) * n_ensembles
+Output tokens = predict_rows * n_ensembles
+```
+
+at $0.05 / mtok input and $0.20 / mtok output (re-read on the pricing page 2026-10-12; formula and rates unchanged). Note that `n_ensembles` -- the number of shuffled passes the model averages -- is a multiplier on your bill that Google does not document the value of.
+
+> **A `COUNT(*)` over `AI.PREDICT` does not run the model.** Measured 2026-10-12: `SELECT COUNT(*) FROM AI.PREDICT(...)` returned in under 2 seconds with the right row count, while the same call selecting `predicted_<label>` took the usual 45-50 seconds -- the planner prunes the inference when no predicted column is referenced. Two consequences: a row-count smoke test proves nothing about whether the call *works* (the 10-class cap, for one, is only enforced once inference runs), and a probe that references a predicted column is what you need to see an error or pay for a prediction.
+
+**Provisioned throughput:** Not specified.
+
+**BigFrames API:** No wrapper. `bigframes.bigquery.ai` has `forecast` but no `predict` -- checked against **bigframes 2.39.0**, this project's pin in `uv.lock` and the version that executed `functions/ai_predict/ai_predict.ipynb`. Use `%%bigquery` magics or `session.read_gbq_query()`.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/aiscore.md

@@ -0,0 +1,52 @@
+# `AI.SCORE`
+
+> **Source:** [`bigquery-ai/reference/managed-functions.md#aiscore`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/managed-functions.md#aiscore)
+
+- **Description:** (Preview) Scalar function that uses a Vertex AI Gemini model to rate inputs based on a scoring system that you describe and returns a FLOAT64 value. BigQuery rewrites your input prompt to generate a scoring rubric that can improve the consistency and quality of the results.
+- **Use cases:** Find the most negative customer reviews (retail), find the most qualified resumes (hiring), find the best customer support interactions, rating and ranking items. Commonly used with ORDER BY.
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-score)
+- **Type:** Scalar function -- returns a single FLOAT64 value per row (Preview)
+
+**Syntax:**
+```sql
+AI.SCORE(
+  [ prompt => ] 'PROMPT'
+  [, connection_id => 'CONNECTION' ]
+  [, endpoint => 'ENDPOINT' ]
+  [, max_error_ratio => MAX_ERROR_RATIO ]
+)
+```
+
+**Inputs:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `PROMPT` | STRING or STRUCT | Required (must be first argument) | The prompt describing the scoring criteria. STRING or STRUCT with STRING/ARRAY\<STRING\>/ObjectRef/ARRAY\<ObjectRef\> fields. At most one video object. For a document plus its criteria, the tuple form `('criteria text', ref)` is the usual shape. |
+| `CONNECTION` | STRING | Optional | Connection to use, format: `[PROJECT_ID].LOCATION.CONNECTION_ID`. |
+| `ENDPOINT` | STRING | Optional | Vertex AI endpoint. If not specified, BigQuery dynamically chooses a model. |
+| `MAX_ERROR_RATIO` | FLOAT64 | Optional | Range 0.0–1.0, default 1.0. If the error ratio exceeds this threshold, the query fails with an error. |
+
+**Outputs:**
+
+| Return Type | Description |
+|-------------|-------------|
+| FLOAT64 | The score assigned to the input. **There is no fixed default range for the score** -- provide a scoring range in your prompt. |
+| NULL | Returned if the call to Vertex AI is unsuccessful. |
+
+**Supported models:** Any GA or preview Gemini model. Uses various `gemini-2.5-*` models.
+
+**Best practices:**
+- Provide a scoring range in your prompt (e.g., "On a scale from 1 to 10") since there is no fixed default range.
+- AI.SCORE automatically rewrites your prompt to generate a scoring rubric.
+- Combine with AI.IF to filter results before or alongside scoring.
+- Use with ORDER BY and LIMIT for ranking.
+
+**Limitations:** Preview status. Returns NULL on error. No model parameter control. No fixed default score range.
+
+**Locations:** All regions supporting Gemini models, plus US and EU multi-regions.
+
+**Provisioned throughput:** Not supported. Uses dynamic shared quota (DSQ) only.
+
+**BigFrames API:** `bigframes.bigquery.ai.score(prompt)` — Returns a Series of FLOAT64 directly (not a struct). No endpoint param -- auto-selects model. Example: `bbq.ai.score(("Rate the negativity of: ", df["review"], " on scale 1-10"))`.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/aisearch.md

@@ -0,0 +1,66 @@
+# `AI.SEARCH`
+
+> **Source:** [`bigquery-ai/reference/embedding-generation-and-semantic-search.md#aisearch`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/embedding-generation-and-semantic-search.md#aisearch)
+
+- **Description:** Table-valued function for semantic and hybrid search on tables that have autonomous embedding generation enabled. Embeds the search query at runtime and searches the specified table. Uses vector indexes when available.
+- **Use cases:** Semantic search, hybrid semantic + keyword search, recommendation, classification, clustering, outlier detection.
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-search)
+- **Type:** Table-valued function (TVF) -- GA (the `mode` argument is Preview)
+
+**Syntax:**
+```sql
+AI.SEARCH(
+  { TABLE base_table | base_table_query },
+  column_to_search,
+  query_value
+  [, mode => mode_value]
+  [, top_k => top_k_value]
+  [, distance_type => distance_type_value]
+  [, options => options_value]
+)
+```
+
+**Inputs:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `base_table` / `base_table_query` | Table/Query | Required | Table to search. **Must have autonomous embedding generation enabled.** |
+| `column_to_search` | STRING literal | Required | Name of the **source string column** (not the generated embedding column). |
+| `query_value` | STRING literal | Required | Search query text. Embedded at runtime using the base table's connection and endpoint. |
+| `mode` | Named arg, STRING | Optional (Preview) | `AUTO` (default), `VECTOR`, or `HYBRID`. See below. |
+| `top_k` | Named arg, INT64 | Optional | Default: 10. Negative = return all. |
+| `distance_type` | Named arg, STRING | Optional | `EUCLIDEAN` (default), `COSINE`, or `DOT_PRODUCT`. Ignored in `HYBRID` mode. |
+| `options` | Named arg, JSON STRING | Optional | `fraction_lists_to_search`, `use_brute_force`. |
+
+**Search modes:**
+
+| Mode | Behavior |
+|------|----------|
+| `AUTO` | Default. Runs hybrid search **only if** the base table has a vector index configured with `lexical_search_columns`; otherwise silently runs semantic-only search. |
+| `VECTOR` | Semantic-only search. `distance` is a true distance in the chosen `distance_type`. |
+| `HYBRID` | Combines semantic and lexical matching on `column_to_search`. Runs with or without a vector index. `distance` is an RRF score, not a distance. |
+
+> **Gotcha:** `AUTO` does not mean "hybrid". Because a vector index cannot currently be built on an autonomous embedding generation column (see [Hybrid Search](hybrid-search-capability.md)), `AUTO` on an AI.SEARCH base table resolves to semantic-only. The query succeeds either way, so the fallback is silent. Specify `mode => 'HYBRID'` explicitly when you want hybrid behavior.
+
+**Outputs:**
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `base` | STRUCT | All columns from base_table |
+| `distance` | FLOAT64 | In `VECTOR`/`AUTO`-semantic mode, the distance between the query_value embedding and the base embedding. In `HYBRID` mode, a reciprocal rank fusion score -- not a distance, and not comparable across modes. |
+
+**Supported models:** Uses whatever embedding model and connection are configured for the base table's autonomous embedding generation.
+
+**AI.SEARCH vs VECTOR_SEARCH:** Use AI.SEARCH for simplified semantic or hybrid search when the base table has autonomous embedding generation and you want to search for a single string literal. Use VECTOR_SEARCH for batch queries, custom embeddings, control over which columns are matched lexically, or tables without autonomous embedding generation. In hybrid mode AI.SEARCH matches lexically only against `column_to_search`; VECTOR_SEARCH lets you name any set of columns in `lexical_search_columns`.
+
+**Best practices:** Create a vector index on the embedding column for better performance on large tables. State `mode` explicitly rather than relying on `AUTO`.
+
+**Limitations:** Base table must have autonomous embedding generation enabled. If embedding generation fails for query_value, the entire query fails. Rows with missing embeddings are skipped. Hybrid mode matches lexically only on `column_to_search`.
+
+**Locations:** All locations supporting Vertex AI embedding models, plus US and EU multi-regions.
+
+**Provisioned throughput:** Not specified in documentation.
+
+**BigFrames API:** No direct equivalent. Use `%%bigquery` magics or `session.read_gbq_query()` to execute AI.SEARCH SQL from BigFrames.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/aisimilarity.md

@@ -0,0 +1,79 @@
+# `AI.SIMILARITY`
+
+> **Source:** [`bigquery-ai/reference/embedding-generation-and-semantic-search.md#aisimilarity`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/embedding-generation-and-semantic-search.md#aisimilarity)
+
+- **Description:** (Preview) Scalar function that computes the cosine similarity between two inputs (text or images). Creates embeddings for both inputs at runtime and computes the cosine similarity between them. Values closer to 1 indicate more similar inputs.
+- **Use cases:** Semantic search without precomputed embeddings, recommendation, classification, prototyping similarity queries.
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-similarity)
+- **Type:** Scalar function (returns FLOAT64) -- Preview
+
+**Syntax (text with endpoint):**
+```sql
+AI.SIMILARITY(
+  content1 => 'CONTENT1',
+  content2 => 'CONTENT2',
+  endpoint => 'ENDPOINT'
+  [, model_params => MODEL_PARAMS]
+  [, connection_id => 'CONNECTION_ID']
+)
+```
+
+**Syntax (text with built-in model — Preview):**
+```sql
+AI.SIMILARITY(
+  content1 => 'CONTENT1',
+  content2 => 'CONTENT2',
+  model => 'MODEL'
+)
+```
+
+**Syntax (multimodal):**
+```sql
+AI.SIMILARITY(
+  content1 => 'CONTENT1',
+  content2 => 'CONTENT2',
+  connection_id => 'CONNECTION_ID',
+  endpoint => 'ENDPOINT'
+  [, model_params => MODEL_PARAMS]
+)
+```
+
+**Inputs:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `content1` | STRING, ObjectRef, or ObjectRefRuntime | Required | First value to compare. An `ObjectRef` is enough — no signing step. For `gemini-embedding-2`: can be a STRUCT (text, images, audio, video, PDFs). |
+| `content2` | STRING, ObjectRef, or ObjectRefRuntime | Required | Second value to compare. Same types as `content1`. |
+| `endpoint` | STRING | Required (unless `model` specified) | Embedding model endpoint (e.g., `text-embedding-005`, `multimodalembedding@001`, `gemini-embedding-2`). |
+| `model` | STRING | Optional (Preview) | Built-in text embedding model. Only supported value: `embeddinggemma-300m`. When specified, cannot use `endpoint`, `model_params`, or `connection_id`. No Vertex AI charges. |
+| `model_params` | JSON literal | Optional | For text: any parameters object fields including `outputDimensionality`. For multimodal: only `dimension`. |
+| `connection_id` | STRING | Optional for text; Required for multimodal | Connection in format `PROJECT_ID.LOCATION.CONNECTION_ID`. |
+
+**Outputs:** Returns FLOAT64 (cosine similarity). NULL on error.
+
+**AI.SIMILARITY vs VECTOR_SEARCH:**
+
+| Feature | AI.SIMILARITY | VECTOR_SEARCH |
+|---------|---------------|---------------|
+| Function type | Scalar | TVF |
+| Primary purpose | Cosine similarity between two specific inputs | Top-K nearest neighbors from a base table |
+| Embedding | Generates embeddings at runtime (2 per call) | Uses pre-computed embeddings |
+| Indexing | No vector indexes | Designed to use vector indexes for ANN search |
+| Use case | Small comparisons, prototyping | Large-scale semantic search, RAG |
+
+**Supported models:** Same models as AI.EMBED: `embeddinggemma-300m` (built-in, Preview), `gemini-embedding-001`, `text-embedding-005`, `text-multilingual-embedding-002`, `gemini-embedding-2` (multimodal, GA), `multimodalembedding@001`.
+
+**Best practices:**
+- Use for small comparisons and prototyping. Use VECTOR_SEARCH for large-scale searches.
+- Supports **cross-modal** comparisons: `content1` can be text while `content2` is an image (or vice versa) when using `multimodalembedding@001` or `gemini-embedding-2` (measured 2026-10-09 on `gemini-embedding-2`: image/image 0.79, text/image 0.28 on the same pet-store product photo). Text and image embeddings share the same vector space, so text descriptions can be matched against document images.
+- `multimodalembedding@001` supports images only (JPEG, PNG, BMP, GIF) — **not PDFs** (status: *"Unsupported content type application/pdf for AI.EMBED MULTIMODAL_EMBEDDING endpoint"*). Use `gemini-embedding-2` for PDF support, or render PDFs to images first.
+
+**Limitations:** Incurs charges for two embedding generations per call. For multimodal, `connection_id` is required and the connection's service account needs `roles/aiplatform.user` and `roles/storage.objectViewer`.
+
+**Locations:** All regions supporting Gemini models, plus US and EU multi-regions.
+
+**Provisioned throughput:** Not specified in documentation.
+
+**BigFrames API:** No direct equivalent. Use `%%bigquery` magics or `session.read_gbq_query()` to execute AI.SIMILARITY SQL from BigFrames.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/choosing-between-these-and-the-ai-functions.md

@@ -0,0 +1,11 @@
+# Choosing between these and the `AI.*` functions
+
+> **Source:** [`bigquery-ai/reference/cloud-ai-service-models.md#choosing-between-these-and-the-ai-functions`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/cloud-ai-service-models.md#choosing-between-these-and-the-ai-functions)
+
+Every member has a generative alternative — [`AI.GENERATE`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/general-purpose-functions.md) over an `ObjectRef` reads audio and images directly, and a prompt can translate, score sentiment or extract entities. Each notebook runs both over the same rows so the comparison is measured rather than argued. What the comparisons keep showing:
+
+**Reach for a Cloud AI service model** when the output is a record: a fixed API version, a published per-unit price, a closed vocabulary, and behavior that does not move when a model default does. When the service cannot do something, it says so — an unsupported language, an unsupported feature, an unreadable file all arrive as errors in a status column.
+
+**Reach for an `AI.*` function** when the deliverable is not the service's output: a summary rather than a transcript, your categories rather than the taxonomy's, translate-and-rewrite rather than translate. It will also answer questions the service refuses — and that is the trade, because it answers them the same confident way whether or not it is right, with no status column and no confidence beside the answer. A refusal is information; a fluent guess is not.
+
+**Use both** when the record and the analysis are different artifacts: the service output as the durable, priced, inspectable column, and the generative function over *that column* — where the input is text you can read instead of media you cannot.

bigquery_data_science/_skills/bigquery-ai/syntax/hybrid-search-capability.md

@@ -0,0 +1,165 @@
+# Hybrid Search *(capability)*
+
+> **Source:** [`bigquery-ai/reference/embedding-generation-and-semantic-search.md#hybrid-search-capability`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/embedding-generation-and-semantic-search.md#hybrid-search-capability)
+
+Hybrid search combines semantic (vector) retrieval with lexical (keyword) matching, so that exact tokens -- SKUs, error codes, model numbers, proper nouns -- can influence the ranking instead of being dissolved into embedding similarity. **BigQuery ships this as a capability of the two existing search functions, not as a separate `HYBRID_SEARCH` function.** There is no `HYBRID_SEARCH` in the SQL surface.
+
+> **Set the expectation correctly before you reach for it.** Hybrid search fuses two *rankings* of the same corpus; it does not union two result sets. How deep a lexical match can reach is decided entirely by `top_k`, through **two independent gates**: BigQuery hands the lexical leg only the top `10 * top_k` rows by semantic rank, and inside that pool a lexical hit is worth a bounded amount of score -- at most `1/62` = `0.0161`. Effective reach is the smaller of the two. At the page sizes most demos use (`top_k` under ~30) the score gate holds reach to ~110 rows and the token only nudges rows the semantic leg already placed near the top. At `top_k` = 64 the pool gate holds it to semantic rank 640, and no exact-token match reaches deeper than that no matter how perfect the match. Sizing `top_k` is part of the design, not an afterthought.
+
+| Route | How to invoke | Lexical columns | Batch? |
+|-------|---------------|-----------------|--------|
+| `VECTOR_SEARCH` | Single search syntax + `lexical_search_columns` (+ optional `lexical_search_query_value`) | Any set of base table columns you name | No -- single query only |
+| `AI.SEARCH` | `mode => 'HYBRID'` | Only `column_to_search` | No -- single query only |
+
+**Neither route requires a vector index.** Both run against an unindexed base table; the index is a speed optimization for the lexical leg, not a prerequisite.
+
+**Two gates, and `top_k` sets both.** A lexically-matched row appears on the result page only if it clears both of these:
+
+```
+Gate 1 (candidate pool):   rank_vector <= 10 * top_k
+Gate 2 (fusion score):     1/(60 + rank_vector) + 1/(61 + rank_lexical)
+                             must beat the row holding the last slot
+```
+
+**Effective reach = `min( score_reach(top_k), 10 * top_k )`.** Gate 1 binds for `top_k` >= 51; Gate 2 binds below that. Everything else in this section is a consequence of those two lines.
+
+**Gate 1: the lexical candidate pool is `10 * top_k` rows.** BigQuery hands the lexical (BM25) leg only the top `10 * top_k` rows by semantic rank. A row deeper than that receives **no lexical rank at all** -- BM25 never sees it, however exactly the token matches, and no amount of score can rescue it. This is why a page of 64 reaches semantic rank 640 and stops.
+
+The diagnostic signature of a query whose target sits outside the pool is that the whole result set comes back with `rank_lexical = rank_vector`, so every returned value is exactly `1 - ( 1/(60 + r) + 1/(61 + r) )`. Read the top row: a best `distance` of **`0.967478`** (= `1 - (1/61 + 1/62)`) means *nothing matched*, while **`0.967734`** (= `1 - (1/61 + 1/63)`) means a match fired somewhere and pushed the top semantic row to lexical rank 2. Those two numbers tell you whether to raise `top_k` or fix the token.
+
+**Gate 2 -- scoring: reciprocal rank fusion.** In hybrid mode the returned `distance` column is a fused rank score rather than a geometric distance. Each candidate is ranked separately by the semantic leg and the lexical leg, and the two ranks are combined. Reproduced against live results, the value matches reciprocal rank fusion with **k = 60 on the semantic leg and k = 61 on the lexical leg**, both ranks 1-based:
+
+```
+distance = 1 - ( 1/(60 + rank_vector) + 1/(61 + rank_lexical) )
+```
+
+**What is measured, and what is inferred.** `VECTOR_SEARCH` returns no rank column, so only one of the two ranks is ever read directly. `rank_vector` comes from a separate semantic-only run over the same rows; the lexical term is whatever remains after subtracting `1/(60 + rank_vector)` from `1 - distance`. What the arithmetic pins is therefore a set of *denominators*, not a pair of constants. Measured on a 30-row corpus, the semantic leg's top row contributes `1/61`, the lexical leg's top row contributes `1/62`, and the remaining lexical denominators run contiguously from there.
+
+Two readings fit identically, because `1/(61 + rank_lexical)` and `1/(60 + (rank_lexical + 1))` are the same number:
+
+| Reading | Semantic leg | Lexical leg |
+|---|---|---|
+| A -- two constants | k = 60, ranks `1..n` | k = 61, ranks `1..n` |
+| B -- one constant, offset ranks | k = 60, ranks `1..n` | k = 60, ranks `2..n+1` |
+
+Nothing observable from outside BigQuery separates them, but **B is the likelier**. k = 60 over 1-based ranks is the canonical RRF of Cormack, Clarke and Buettcher (2009) and the default wherever the constant is exposed -- Elasticsearch and OpenSearch both name it `rank_constant` and default it to 60, and Spanner and AlloyDB write 60 into their documented SQL. A constant of 61 appears in no published implementation. What is measured is that the semantic leg lands on the canonical `1/61` for its own top row -- so this document's rank convention already agrees with BigQuery's on one leg, and the extra `+1` sits on the lexical side specifically.
+
+**Why the lexical leg would start at 2 is a separate question, and it is open.** Reading B says *where* the extra `+1` sits, not *why*. Three mechanisms fit the observations equally well:
+
+- **A deliberate tie-break.** At equal ranks the vector term `1/(60 + r)` exceeds the lexical term `1/(61 + r)`, so the semantic leg wins every tie by a hair. That is what you would implement if hybrid should never reorder a tie against the embedding.
+- **A reserved slot.** If position 1 of the lexical list holds something that is not a result -- the query itself, a sentinel, a header row -- then real rows begin at 2 naturally.
+- **An off-by-one.** 61 is universal as the rank-1 *denominator*, `1/(60 + 1)`. Transcribing that as the constant, or applying `+1` to an already-1-based rank, produces exactly the observed `1/62` on a top row.
+
+Only the third is a defect, and it is the least charitable of the three. Nothing observable from outside BigQuery distinguishes them, so do not present any one of them as the explanation.
+
+One argument that looks decisive and is not: decoding with k = 60 on both legs yields lexical ranks `2 .. n+1`, and no n-row list hands out rank n+1. That rules out a shared base combined with ordinary 1-based lexical numbering. It does *not* rule out reading B, where `2 .. n+1` is precisely what an offset 1-based ranking looks like. A contiguous block starting at 2 is evidence *for* an offset, not against a shared constant.
+
+The 60/61 form is used throughout this document because it is the shortest expression that reproduces every observed value. Read it as arithmetic that holds, not as two deliberate design decisions.
+
+Three worked examples from that run, each matching a live returned value exactly:
+
+| rank_vector | rank_lexical | Arithmetic | Returned `distance` |
+|---|---|---|---|
+| 1 | 3 | `1 - (1/61 + 1/64)` | `0.967981557377` |
+| 2 | 1 | `1 - (1/62 + 1/62)` | `0.967741935484` |
+| 9 | 2 | `1 - (1/69 + 1/63)` | `0.969634230504` |
+
+The middle row is the one that trips people up: `1/62 + 1/62` is a value the measured arithmetic produces -- `1/(60 + 2)` and `1/(61 + 1)` happen to coincide at ranks (2, 1). It is also the `distance` on the `tiger` row of Google's published hybrid example, which decodes to exactly those ranks. Seeing two equal denominators is not evidence of a shared rank base: under either reading above, ranks (2, 1) produce `1/62` twice.
+
+Consequences worth internalizing:
+
+- **The best attainable score is `1 - (1/61 + 1/62)` = `0.9674775251189847`** -- rank 1 in *both* legs. An exact self-match does not return a near-zero "distance". (The oft-quoted `1 - 2/61` = `0.967213` is not attainable: the two denominators can never be equal at rank 1.)
+- **Each leg contributes at most ~`1/61` = `0.0164`.** A rank-1 lexical hit is worth `1/62` = `0.0161` and nothing more. That single number is the budget the whole capability operates inside.
+- **The lexical leg ranks the entire candidate pool, so no pooled row ever forfeits a term.** True BM25 matches take lexical ranks `1..m`; every remaining pooled row falls back to its *semantic* order behind them. Verified with a `lexical_search_query_value` matching zero rows -- every row still received a lexical term, with `rank_lexical = rank_vector`. Verified again by promoting one row to lexical rank 1: the row formerly at lexical rank 1 moved to 2, and every row below the promoted row was byte-identical. Within the pool there is no "single-list" state and no forfeiture ceiling; every row in the result is scored from both terms. Outside the pool there is no scoring at all -- that is Gate 1, not a forfeited term.
+- **Therefore an unmatched row is not penalized, only un-promoted.** It keeps a lexical rank -- its own semantic rank, pushed down one place for each row below it that did match. Where nothing matched at all, that reduces to `1 - ( 1/(60 + r) + 1/(61 + r) )` at semantic rank `r` -- at `r = 10`, `1 - (1/70 + 1/71)` = `0.97163`. A lexical match moves that row to lexical rank 1, replacing its `1/(61 + r)` with `1/62`. That single substitution is the entire mechanism.
+- **What limits hybrid is reach, and reach is a function of `top_k` twice over.** For the score gate: a matched row at semantic rank `R` scores `1/(60 + R) + 1/62`, and it has to displace the row holding the last slot, which sits at semantic rank `top_k` and -- pushed down one position by the match -- lexical rank `top_k + 1`, scoring `1/(60 + top_k) + 1/(61 + top_k + 1)`. For the pool gate the bound is simply `10 * top_k`. Taking the `min` of the two gives the deepest semantic rank a lexically-matched row can occupy and still make the page:
+
+  | `top_k` | 2 | 3 | 5 | 10 | 20 | 30 | 40 | 50 | 51 | 64 | 100 | 208 | 300 |
+  |---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+  | deepest semantic rank retrievable | 3 | 6 | 10 | 23 | 56 | 110 | 212 | 468 | 510 | 640 | 1,000 | 2,080 | 3,000 |
+  | binding gate | score | score | score | score | score | score | score | score | pool | pool | pool | pool | pool |
+
+  Below `top_k` = 51 the score gate is the binding one and growth is steeply non-linear -- nearly flat through the page sizes demos typically use, then accelerating. From `top_k` = 51 upward the pool gate takes over and reach becomes exactly linear at `10 * top_k`. There is no page size at which reach becomes unbounded. `top_k` >= `R/10` is therefore a floor for retrieving a row at semantic rank `R`, not a recipe: it is necessary everywhere, but sufficient only once the pool gate is the binding one. Below that crossover the score gate decides and `R/10` falls short -- a target at semantic rank 100 needs `top_k` of 29, not 10, and one at rank 200 needs 40, not 20. Read the real number off the row above.
+- **The evidence.** A 500-row corpus: 499 rows about office and desk items, semantically close to the query `"desk accessories for a computer setup"`, plus one target row `ZQX-9999` described as `"bulk compost bin for garden waste and autumn leaf mulching"` -- semantic rank **500 of 500**. Searching with `lexical_search_query_value => 'ZQX-9999'`:
+
+  | `top_k` | 10 | 49 | 50 | 51 | 63 | 64 |
+  |---|---|---|---|---|---|---|
+  | target returned | no | no | no | **yes** | yes | yes |
+
+  The arithmetic predicts the flip between 50 and 51 to the exact position. The target's returned `distance`, `0.9820852534562212`, is exactly `1 - (1/560 + 1/62)` -- semantic rank 500, lexical rank 1. Here the score gate is the binding one: at `top_k` = 51 the pool is 510 rows, comfortably deeper than the target.
+
+- **The pool gate, isolated.** Repeating the same construction on deeper corpora puts the target beyond any plausible score-gate reach, so the flip lands on `10 * top_k` exactly. Each of these was a sharp prediction made before the query ran; the 500-row probe is listed alongside them as the contrasting case, where the score gate is the binding one:
+
+  | corpus | index | target semantic rank | predicted flip | observed |
+  |---|---|---|---|---|
+  | 3,000 rows | none | 3,000 | `top_k` = 300 | 299 no match, **300 match** |
+  | 1,500 rows | none | 1,500 | `top_k` = 150 | 149 no match, **150 match** |
+  | 500 rows | none | 500 | `top_k` = 51 (score gate) | 50 absent, **51 present** |
+  | 7,275-row product catalog | `TREE_AH` hybrid, `ACTIVE`, 100% coverage | 2,072 | needs `top_k` >= 208 | `top_k` = 64 returned nothing |
+
+  The three synthetic probe tables are **unindexed** -- at 3,000, 1,500 and 500 rows they sit below BigQuery's 5,000-row `CREATE VECTOR INDEX` floor and are brute-force scanned, so no index existed that could have restricted anything. The product catalog is the opposite case: it carried a fully populated hybrid `TREE_AH` index declaring `lexical_search_columns`, status `ACTIVE` at 100% coverage, when the `top_k` = 64 observation was made. The pool gate behaves identically with and without an index, which makes it both *not* an index artifact and *not* something an index can avoid. At `top_k` = 64 on the catalog the pool was 640 rows and the target at rank 2,072 was never a candidate; the result set carried the zero-match signature (`rank_lexical = rank_vector`, best `distance` `0.967478`).
+- **This is why a hybrid demo at `top_k => 3` or `5` returns almost exactly the semantic-only rows:** at those page sizes reach is 6 and 10, deep enough to shuffle the top of the list, never deep enough to pull in something new. That is a property of the page size chosen, not a limit on the capability.
+- **Promotion is a swap.** The fused list is a fixed-length page, so a row hybrid adds displaces one semantic-only would have returned. Always measure both the added *and* the dropped set.
+- **When the token is all the user typed, use a predicate.** Hybrid *can* retrieve it, but only if `top_k` clears both gates for the target's semantic rank -- a tenth of that rank at the very least, and more than that whenever the score gate binds -- which you do not know in advance, and a fusion score is still a ranking that can be crowded. Use `WHERE sku = @sku` -- a predicate cannot be outranked and has no candidate pool. Hybrid earns its place on queries that carry descriptive words *and* an identifier.
+- Scores are bounded in a narrow band near 1 and are **not comparable** to `VECTOR`-mode distances. Do not threshold hybrid scores with a cutoff tuned on cosine or Euclidean distance.
+- Because the score depends only on ranks, `distance_type` has no effect in hybrid mode.
+
+**The sizing rule:** hybrid search re-ranks *and* widens, and `top_k` decides how much of each you get -- the widening is real but bounded, never unbounded. At `top_k` up to about 30 the token only nudges rows already near the top -- do not size a page that way and then expect recall. Beyond that, an exact token surfaces a row from about `10 * top_k` deep and no deeper, so for a target you expect at semantic rank `R`, **size `top_k` to at least `R/10`, and above that to whatever the reach table requires** -- the two gates cross at `top_k` = 51, so for anything shallower than a few hundred ranks the score gate is the one that binds and the floor alone will not retrieve the row. If the identifier *is* the query and you need certainty, use a `WHERE` predicate instead.
+
+> This formula is reverse-engineered from observed results, not documented by Google. Treat it as an explanation of the ordering you see, not as a contract -- it can change without notice.
+
+**Vector indexes for hybrid search.** To accelerate the lexical leg, create a vector index that declares the lexical columns:
+
+```sql
+CREATE [ OR REPLACE ] VECTOR INDEX [ IF NOT EXISTS ] index_name
+ON table_name(column_name)
+[STORING(stored_column_name [, ...])]
+[PARTITION BY partition_expression]
+OPTIONS(index_option_list);
+```
+
+```sql
+CREATE VECTOR INDEX IF NOT EXISTS my_hybrid_index
+ON my_table(embedding)
+STORING (product_name)
+OPTIONS (index_type = 'TREE_AH', distance_type = 'COSINE',
+         lexical_search_columns = ['product_name']);
+```
+
+Rules that are easy to get wrong, all confirmed against a live project:
+- Every column in `lexical_search_columns` **must also appear in `STORING(...)`**. Omitting the `STORING` clause fails with `Lexical search column x must be in the list of stored columns`.
+- A lexical column **may not also be the index key**. Naming the same column in `ON table(x)` and `STORING(x)` fails with `Column x found multiple times`.
+- The index key must be an `ARRAY<FLOAT64>` embedding column. **A vector index cannot be created on an autonomous embedding generation column**, because that column is a `STRUCT`. Demonstrating both autonomous embeddings and a vector index in one pipeline therefore requires two tables.
+- **Two separate size gates, and they fail differently.** (a) `CREATE VECTOR INDEX` is *rejected outright* below **5,000 rows**: `Total rows 30 is smaller than min allowed 5000 for CREATE VECTOR INDEX query with the IVF index type. Please use VECTOR_SEARCH table-valued function directly to perform the similarity search.` Verified 2026-09-02 on a 30-row table for **both** `IVF` and `TREE_AH` (the message names whichever `index_type` you passed), so this is not an IVF-only constraint. (b) Once created, **population** is asynchronous and does not begin until the base table exceeds roughly **10 MB**; below that the index sits at `coverage_percentage` 0 with `indexUnusedReasons` = `BASE_TABLE_TOO_SMALL` and queries silently fall back to brute force. A demo table must clear *both* bars to show a working index -- which is why the catalog workflow uses a real ~7,275-row public catalog rather than generated data. Note the 5,000-row floor is enforced by the engine but is **not** stated on the vector-index documentation page; only the 10 MB condition is documented.
+- `index_type` is `TREE_AH` or `IVF`.
+- **`PARTITION BY` must match the base table's partitioning expression exactly.** The clause does not let you pick an arbitrary partitioning for the index. On a table partitioned by `RANGE_BUCKET(id, GENERATE_ARRAY(0, 30000, 5000))`, repeating that same expression validates; naming any other column -- `PARTITION BY category` -- fails with `Invalid Partition By expression`, and so does `PARTITION BY category` against a table that is not partitioned at all.
+
+**Index lifecycle: coverage, drift, and rebuild.** Creating an index is not managing one. Three signals describe an index's health and they move *independently* -- all measured 2026-09-03 against a live 7,275-row `TREE_AH` hybrid index:
+
+| Signal | Where to read it | What it actually means |
+|---|---|---|
+| `coverage_percentage`, `unindexed_row_count` | `INFORMATION_SCHEMA.VECTOR_INDEXES` | how much of the base table the index currently covers |
+| `tfdv_drift` | `VECTOR_INDEX.STATISTICS(TABLE t)` | how far the *vector distribution* has moved since the index was built |
+| `last_model_build_time` | `INFORMATION_SCHEMA.VECTOR_INDEXES` | when the TreeAH model itself was last built -- distinct from `last_refresh_time` |
+
+`VECTOR_INDEX.STATISTICS` takes **exactly one argument** and it must be a relation: `VECTOR_INDEX.STATISTICS(TABLE my_table)`. Adding the index name as a second argument fails with `Signature accepts at most 1 argument, found 2 arguments`; passing a string instead of `TABLE t` fails with `argument 1 must be a relation (i.e. table subquery)`. It returns a single column, `tfdv_drift`, which is `null` until the index's first refresh and `"0.0"` at fresh 100% coverage. On a table carrying **no** index it returns **zero rows** rather than an error, and scans 0 bytes -- so it is safe to call unconditionally.
+
+**A negative `tfdv_drift` is a sentinel, not a magnitude.** Two freshly built indexes at 100% coverage reported different things: `"0.0"` from an unpartitioned hybrid index, and `-1.0` from a `PARTITION BY` index with no `lexical_search_columns`. Those two indexes differ in more than one way, so this does not isolate a cause -- partitioning, the absence of lexical columns, and how recently the first refresh completed are all still in play. Read a negative value as *no drift figure available for this index yet* and corroborate with `coverage_percentage` and `last_refresh_time` before acting on it. Partitioned indexes do populate normally otherwise: one on a 19,055-row / ~8.6 MB table reached 100% coverage about seven minutes after `CREATE`.
+
+**Coverage and drift are not the same alarm.** Inserting 11,780 rows whose vectors duplicate ones already indexed (7,275 -> 19,055 rows) dropped `coverage_percentage` from 100 to 38 and set `unindexed_row_count` to 11,780 -- while `tfdv_drift` moved only from `0.0` to `1.97e-5` and `last_model_build_time` did not change at all (held across six 90-second polls). That is the point of having both numbers: a large volume of *new but distributionally identical* rows is a coverage problem, not a drift problem. Coverage recovers on the automatic refresh; drift is what argues for a rebuild.
+
+**`ALTER VECTOR INDEX ... REBUILD` requires a BACKGROUND reservation.** With no reservation on the project the statement is rejected outright:
+
+```
+Cannot alter vector index on table my_table because there is no BACKGROUND reservation.
+```
+
+The same gate covers search indexes -- `ALTER SEARCH INDEX ... SET OPTIONS (...)` and `ALTER SEARCH INDEX ... ADD COLUMN` return the identical message with `search index` in place of `vector index`. **The check fires before validation, so these statements cannot even be dry-run**: `bq query --dry_run` on any of them returns the reservation error rather than `Query successfully validated`, and it does so even when the named index does not exist. That rules out the usual trick of parse-checking index-maintenance DDL for free. Index *maintenance* runs as background work, and background work needs slots assigned to it, so a project billed purely on demand has nowhere to run it. Exercising these statements requires an Enterprise-edition reservation carrying a `BACKGROUND` job-type assignment; an autoscale reservation with baseline 0 is the cheap shape, since it bills only while the rebuild actually runs. Creating an index, populating it, and querying through it all work with no reservation at all -- only the `ALTER` path is gated. Related: `last_index_alteration_info` stays `null` until an `ALTER` succeeds, and it is a `RECORD`, so `bq query --format=csv` refuses to print it (`Cannot print record field "last_index_alteration_info" in CSV format`) -- use `--format=prettyjson`.
+
+**Search indexes have a 10 GiB floor -- three orders of magnitude above the vector index gates.** `CREATE SEARCH INDEX` *succeeds* on a small table, and then the index never populates. `INFORMATION_SCHEMA.SEARCH_INDEXES` reports `index_status` = `TEMPORARILY DISABLED`, `disable_time` equal to `creation_time`, `coverage_percentage` 0, and this `disable_reason`:
+
+> The base table size is below the threshold of 10737418240 bytes for indexing. The index does not provide noticeable search performance gains when the base table is too small.
+
+10,737,418,240 bytes is exactly 10 GiB -- against the vector index's 5,000-row hard reject and ~10 MB population gate. No teaching-scale table can carry a populated search index, which is why this project creates none and reaches for `lexical_search_columns` on a vector index instead. Note also that the `ddl` column of `SEARCH_INDEXES` is normalized rather than a verbatim echo of what you ran: a statement written with `OPTIONS (analyzer = 'LOG_ANALYZER')` is recorded as `OPTIONS (data_types = ['STRING'])`, with the analyzer surfaced in the separate `analyzer` column.
+
+**Related:** [`VECTOR_SEARCH`](vector_search.md) · [`AI.SEARCH`](aisearch.md) · workflow [`catalog_search`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/workflows/catalog_search)

bigquery_data_science/_skills/bigquery-ai/syntax/mlannotate_image.md

@@ -0,0 +1,24 @@
+# `ML.ANNOTATE_IMAGE`
+
+> **Source:** [`bigquery-ai/reference/cloud-ai-service-models.md#mlannotate_image`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/cloud-ai-service-models.md#mlannotate_image)
+
+- **Description:** Table-valued function that annotates images in an object table with the Cloud Vision API. One model serves every feature; the feature list is an argument to the function, not an option on the model.
+- [documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-annotate-image) · [notebook](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/functions/ml_annotate_image/ml_annotate_image.ipynb) · [sql](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/functions/ml_annotate_image/ml_annotate_image.sql)
+
+```sql
+SELECT * FROM ML.ANNOTATE_IMAGE(
+  MODEL `PROJECT_ID.DATASET.MODEL_NAME`,
+  { TABLE `PROJECT_ID.DATASET.OBJECT_TABLE` | (QUERY_STATEMENT) },
+  STRUCT(['LABEL_DETECTION', 'OBJECT_LOCALIZATION'] AS vision_features));
+```
+
+The syntax page lists eight features: `LABEL_DETECTION`, `OBJECT_LOCALIZATION`, `TEXT_DETECTION`, `DOCUMENT_TEXT_DETECTION`, `LANDMARK_DETECTION`, `LOGO_DETECTION`, `FACE_DETECTION` and `IMAGE_PROPERTIES`.
+
+**Output:** `ml_annotate_image_result` (JSON), `ml_annotate_image_status`, plus the object table columns.
+
+**Worth knowing:**
+- **One call, many features.** Requesting several features in one call is one pass over the image; the result JSON gains a top-level key per feature.
+- **What the documented list covers and what the function parses are different sets.** Measured against the undocumented names: `SAFE_SEARCH_DETECTION` and `CROP_HINTS` are accepted and return `safe_search_annotation` and `crop_hints_annotation`; `WEB_DETECTION` is accepted and returns no keys at all; an invented name is rejected outright with `Found unsupported value … of setting field vision_features`. Accepted is not the same as supported — the silent-empty case is the one to guard against.
+- **A zero-valued JSON field is omitted from the response.** Landmark bounding boxes are in absolute pixels, and a vertex at the origin arrives as `{}`, so `INT64(vertex.x)` returns `NULL` rather than `0`. Coalesce it.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/mlgenerate_embedding.md

@@ -0,0 +1,34 @@
+# `ML.GENERATE_EMBEDDING`
+
+> **Source:** [`bigquery-ai/reference/embedding-generation-and-semantic-search.md#mlgenerate_embedding`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/embedding-generation-and-semantic-search.md#mlgenerate_embedding)
+
+- **Description:** Table-valued function identical in capability to `AI.GENERATE_EMBEDDING` but with `ml_generate_embedding_*` prefixed output column names and an additional `flatten_json_output` parameter. **Google recommends using `AI.GENERATE_EMBEDDING` instead for new queries.**
+- **Use cases:** Same as AI.GENERATE_EMBEDDING.
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-embedding)
+- **Type:** Table-valued function
+
+**Syntax:** Same patterns as AI.GENERATE_EMBEDDING, with the addition of `FLATTEN_JSON_OUTPUT AS flatten_json_output` in the STRUCT parameter.
+
+**Key difference from AI.GENERATE_EMBEDDING:**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `flatten_json_output` | BOOL | TRUE | Determines whether JSON content is parsed into separate columns. |
+
+All other inputs are the same as AI.GENERATE_EMBEDDING. Additionally, when using `RETRIEVAL_DOCUMENT` task type, you can include a `title` column in the input query.
+
+**Outputs (column naming difference):**
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `ml_generate_embedding_result` | ARRAY\<FLOAT64\> (flatten=TRUE) or JSON (flatten=FALSE) | Embedding vector or full JSON response |
+| `ml_generate_embedding_statistics` | JSON | Token count and truncated fields (flatten=TRUE only) |
+| `ml_generate_embedding_status` | STRING | API response status |
+| `ml_generate_embedding_start_sec` | INT64 | Video only (NULL for images) |
+| `ml_generate_embedding_end_sec` | INT64 | Video only (NULL for images) |
+
+**Supported models / Limitations / Locations / Provisioned throughput:** Same as AI.GENERATE_EMBEDDING. Note: `gemini-embedding-001` has a default output dimensionality of 3072 (vs 768 for `text-embedding-005`). `gemini-embedding-2` (GA) is also supported for multimodal, same as AI.GENERATE_EMBEDDING.
+
+**BigFrames API:** No direct `ML.GENERATE_EMBEDDING` wrapper. BigFrames routes through `AI.GENERATE_EMBEDDING` instead. Use `bbq.ai.generate_embedding()` or `TextEmbeddingGenerator`/`MultimodalEmbeddingGenerator` classes.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/mlgenerate_text.md

@@ -0,0 +1,65 @@
+# `ML.GENERATE_TEXT`
+
+> **Source:** [`bigquery-ai/reference/general-purpose-functions.md#mlgenerate_text`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/general-purpose-functions.md#mlgenerate_text)
+
+- **Description:** Table-valued function that performs generative natural language tasks. Functionally identical to `AI.GENERATE_TEXT` but with `ml_generate_text_*` prefixed column names and an additional `flatten_json_output` parameter. **Google recommends using `AI.GENERATE_TEXT` instead for new queries.**
+- **Use cases:** Same as AI.GENERATE_TEXT: classification, sentiment analysis, image captioning, transcription, text enrichment, audio analysis, PDF analysis, etc.
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-text)
+- **Type:** Table-valued function (returns a table with input columns plus output columns)
+
+**Syntax (Gemini, standard tables):**
+```sql
+ML.GENERATE_TEXT(
+  MODEL `PROJECT_ID.DATASET.MODEL`,
+  { TABLE `PROJECT_ID.DATASET.TABLE` | (QUERY_STATEMENT) },
+  STRUCT(
+    { { [MAX_OUTPUT_TOKENS AS max_output_tokens]
+        [, TOP_P AS top_p]
+        [, TEMPERATURE AS temperature]
+        [, STOP_SEQUENCES AS stop_sequences]
+        [, GROUND_WITH_GOOGLE_SEARCH AS ground_with_google_search]
+        [, SAFETY_SETTINGS AS safety_settings]
+      }
+      | [, MODEL_PARAMS AS model_params]
+    }
+    [, FLATTEN_JSON_OUTPUT AS flatten_json_output]
+    [, REQUEST_TYPE AS request_type]
+  )
+)
+```
+
+Other model type syntaxes (Claude, Llama, Mistral AI, Open models, Object tables) follow the same pattern as AI.GENERATE_TEXT but with the added `flatten_json_output` parameter.
+
+**Key difference from AI.GENERATE_TEXT -- The `flatten_json_output` parameter:**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `flatten_json_output` | BOOL | FALSE | When TRUE, JSON response is parsed into separate columns (changes column names). |
+
+All other parameters are identical to AI.GENERATE_TEXT.
+
+**Outputs (key difference -- column naming):**
+
+When `flatten_json_output` is FALSE (default):
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `ml_generate_text_result` | JSON | Full JSON response |
+| `ml_generate_text_status` | STRING | API status (empty = success) |
+
+When `flatten_json_output` is TRUE:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `ml_generate_text_llm_result` | STRING | Generated text |
+| `ml_generate_text_rai_result` | STRING | Safety ratings (Gemini only, when safety_settings specified) |
+| `ml_generate_text_grounding_result` | STRING | Grounding sources (Gemini only, when ground_with_google_search is TRUE) |
+| `ml_generate_text_status` | STRING | API status |
+
+**Supported models:** Same as AI.GENERATE_TEXT (remote models over Gemini, Claude, Llama, Mistral, open models). Also explicitly supports remote models over fine-tuned Gemini models.
+
+**Best practices / Limitations / Locations / Provisioned throughput:** Same as AI.GENERATE_TEXT.
+
+**BigFrames API:** No direct `ML.GENERATE_TEXT` wrapper. BigFrames routes through `AI.GENERATE_TEXT` instead. Use `bbq.ai.generate_text()` or `GeminiTextGenerator`/`Claude3TextGenerator` classes (which generate `AI.GENERATE_TEXT` SQL under the hood).
+
+---