bigquery-data-science 0.7.0__py3-none-any.whl

bigquery_data_science/_skills/bigquery-ai/reference/managed-functions.md

@@ -0,0 +1,72 @@
+# Managed Classification & Scoring Functions in BigQuery
+
+Contents: [Options](#options) · [Choosing among them](#choosing-among-them) · [Gotchas verified in this repo](#gotchas-verified-in-this-repo) · [Canonical snippet](#canonical-snippet) · [Go deeper](#go-deeper)
+
+## Options
+
+| Function | What it does | Use this when... | Key differentiator |
+|---|---|---|---|
+| `AI.IF` | Evaluates a natural-language condition, returns a plain BOOL | You need a fuzzy/semantic boolean test — a `WHERE` filter, a `JOIN` condition, or a branch — expressed in plain English | Returns BOOL directly (not a struct); relates to `AI.GENERATE_BOOL` but with automatic prompt optimization and less control |
+| `AI.SCORE` | Rates/ranks input against a scale you describe, returns a FLOAT64 | You need a numeric quality/relevance/priority score to sort or threshold on (e.g. "most negative review", "best resume") | Auto-generates a scoring rubric from your prompt; no fixed default range — you must state the scale in the prompt |
+| `AI.CLASSIFY` | Buckets input into a fixed, user-supplied category list, returns STRING or ARRAY<STRING> | You know the category set in advance and want single- or multi-label classification without writing an `output_schema` | Only managed function with no general-purpose counterpart; categories can carry descriptions and few-shot examples |
+| `AI.AGG` | Aggregate function (like `SUM`/`COUNT`) that summarizes/synthesizes many rows per group into one STRING | You need a GROUP BY-level synthesis (sentiment across all reviews for a product, a summary of a log group) that may exceed a single prompt's context window | Only aggregate function of the four; auto-batches multi-level so it can process data beyond the Gemini context window |
+
+## Choosing among them
+
+- **"I want to filter/branch on a fuzzy condition."** → `AI.IF`. Put ordinary (non-AI) filters in the same `WHERE` clause alongside it — BigQuery evaluates non-AI filters first and short-circuits, which reduces the number of billed Gemini calls (e.g. `WHERE AI.IF(...) AND category = 'tech'`).
+
+- **"I want a numeric quality/relevance/priority score."** → `AI.SCORE`. State the scale explicitly in the prompt (e.g. "rate 1–10") since there is no fixed default range. Commonly paired with `ORDER BY ... LIMIT` for ranking, and can be combined with `AI.IF` to pre-filter before scoring.
+
+- **"I want to bucket rows into known categories."** → `AI.CLASSIFY`. Pass `categories` as an `ARRAY<STRING>` (bare labels) or `ARRAY<STRUCT<STRING,STRING>>` (label + description, for more nuanced classification). Use `output_mode => 'multi'` for multi-label problems (returns 0..N categories, empty array if none apply); leave it unset or `'single'` for one best-fit label. Categories can also come from a table via a `DECLARE ... ARRAY<STRING> DEFAULT (SELECT ARRAY_AGG(...))` variable rather than a hardcoded literal.
+
+- **"I want to summarize/synthesize across many rows in a group."** → `AI.AGG(input, instruction [, connection_id =>][, endpoint =>])`, used with `GROUP BY` for per-group results or without it to aggregate the whole table into one STRING. Use `DISTINCT` to dedupe inputs and cut token usage. Prefer it over hand-rolling `STRING_AGG`/`ARRAY_AGG` into an `AI.GENERATE` prompt — `AI.AGG` batches automatically and can exceed the context window that a manual concatenation would blow past.
+
+- **Managed vs. raw `AI.GENERATE`:** all four managed functions are Preview, DSQ-only (no Provisioned Throughput), auto-select a cost/quality-optimized model when no `endpoint` is given, return `NULL` on error (no detailed status/full_response), and give up model-parameter control (temperature, top_p, etc.). Prefer them when the task fits their shape (boolean test, single score, fixed category set, group summary) and you want BigQuery's automatic prompt engineering rather than hand-writing prompts. Drop down to `AI.GENERATE` (with `output_schema` if needed) when you need: model parameter control, a non-default/pinned model via `endpoint` with fine-grained `model_params`, Provisioned Throughput, detailed `full_response`/`status` for debugging, or an output shape (multi-field structured record) that doesn't map onto BOOL/FLOAT64/STRING/ARRAY&lt;STRING&gt;. `AI.GENERATE_BOOL`/`AI.GENERATE_DOUBLE` are the direct general-purpose analogs of `AI.IF`/`AI.SCORE` respectively when you need that extra control.
+
+## Gotchas verified in this repo
+
+- All four functions are **Preview** status — expect the "contact bqml-feedback@google.com" support model, not GA stability guarantees.
+- None of the four support model-parameter control (temperature, top_p, thinking budget, etc.) — that control only exists on the general-purpose siblings (`AI.GENERATE`, `AI.GENERATE_BOOL/DOUBLE/INT`, `AI.GENERATE_TEXT`).
+- `AI.CLASSIFY` categories must be **string literals in the array** (or a `DECLARE`d variable) — you cannot pass a column reference directly as the categories argument; if categories are dynamic per-row, this is a hard modeling constraint to plan around.
+- `AI.CLASSIFY`'s `output_mode => 'multi'` is explicitly **not supported** when `optimization_mode` is set (the `MINIMIZE_COST`/`MAXIMIZE_QUALITY` optimized path) — multi-label classification and cost-optimized embeddings mode are mutually exclusive.
+- `optimization_mode` (on `AI.IF` and `AI.CLASSIFY`) requires roughly **3,000 rows minimum** to be effective, and in optimized mode only string columns are supported for multi-column prompts/inputs — this is a distilled local-model path (`MINIMIZE_COST`, up to ~230x token reduction) that only pays off at that scale.
+- `max_error_ratio` (available on `AI.IF`, `AI.SCORE`, `AI.CLASSIFY`) is **not supported** when `optimization_mode` is `MINIMIZE_COST` — you lose that safety valve in the cost-optimized path.
+- `AI.AGG` has documented **known issues**: rows with 10+ images in a single row may be silently skipped; rows containing arrays of `ObjectRefRuntime` values (via `OBJ.GET_ACCESS_URL`) may be skipped — **an argument for passing a bare `ObjectRef`, which sidesteps the issue entirely and is what the other eleven functions now document**, even though `AI.AGG`'s own page is the last one still specifying `ObjectRefRuntime`; Workforce Identity Federation without an explicit connection can fail long-running queries; Gemini 3.0/3.1 with connection-based auth may intermittently return unauthenticated errors.
+- `AI.AGG` output is capped at **10,000 tokens per group**, and the docs recommend staying under 20 million rows and 1,000 distinct groups per query to avoid timeouts — and it cannot use Gemini models that require a thinking budget.
+- `AI.AGG`'s actual processed-row count can diverge from expectations on complex queries (JOINs, `ORDER BY ... LIMIT`) — materialize the intended input to a separate table first if you need predictable, auditable row counts/costs.
+- `AI.SCORE` has **no fixed default score range** — omitting an explicit scale in the prompt (e.g. "1 to 10") produces inconsistent, hard-to-compare ranges across calls; always state the scale.
+- None of the four managed functions support `DISTINCT` or `GROUP BY` except `AI.AGG` (which supports both, being the one aggregate in the family) — attempting to use the scalar three (`AI.IF`, `AI.SCORE`, `AI.CLASSIFY`) as if they were aggregates will not work since they operate strictly row by row.
+- Row-count ceiling is **10,000,000 rows per job** for `AI.IF`/`AI.SCORE`/`AI.CLASSIFY`, versus a (recommended, not hard) **20,000,000** for `AI.AGG` — plan batch sizing accordingly, especially since none of these support Provisioned Throughput (DSQ only), so large jobs compete for shared quota.
+
+## Canonical snippet
+
+```sql
+SELECT
+  ticket_id,
+  ticket_text,
+  AI.CLASSIFY(
+    input => ticket_text,
+    categories => [
+      ('billing', 'questions or disputes about charges, invoices, or refunds'),
+      ('technical', 'product bugs, errors, or how-to questions'),
+      ('account', 'login, password, or account-access issues'),
+      ('other', 'anything that does not clearly fit the above')
+    ]
+  ) AS category
+FROM `my_project.my_dataset.support_tickets`;
+```
+
+This uses category descriptions (not just bare labels) for more nuanced classification and includes an explicit "other" bucket to catch inputs that don't closely match any defined category.
+
+## Go deeper
+
+Full extracted notebook walkthroughs live in this skill's `narrative/` folder — no need to be inside the source repo:
+
+- [`narrative/ai_if.md`](../narrative/ai_if.md) (source: `functions/ai_if/`)
+- [`narrative/ai_score.md`](../narrative/ai_score.md) (source: `functions/ai_score/`)
+- [`narrative/ai_classify.md`](../narrative/ai_classify.md) (source: `functions/ai_classify/`)
+- [`narrative/ai_agg.md`](../narrative/ai_agg.md) (source: `functions/ai_agg/`)
+
+Full syntax/options tables: see the source repo at `bigquery-ai/reference/managed-functions.md`, indexed from `bigquery-ai/RESOURCES.md`.
+
+**Lens 3 per-statement syntax cards (`syntax/`):** [`syntax/INDEX.md`](../syntax/INDEX.md) — [`syntax/aiif.md`](../syntax/aiif.md), [`syntax/aiscore.md`](../syntax/aiscore.md), [`syntax/aiclassify.md`](../syntax/aiclassify.md), [`syntax/aiagg.md`](../syntax/aiagg.md)

bigquery_data_science/_skills/bigquery-ai/reference/predictive-ai.md

@@ -0,0 +1,113 @@
+# Predictive AI in BigQuery AI Functions
+
+Contents: [Options](#options) · [Choosing among them](#choosing-among-them) · [Gotchas verified in this repo](#gotchas-verified-in-this-repo) · [Canonical snippets](#canonical-snippets) · [Go deeper](#go-deeper)
+
+Zero-training prediction from two pre-trained foundation models: **TimesFM** for time series (`AI.FORECAST`, `AI.DETECT_ANOMALIES`, and the forecast branch of `AI.EVALUATE`) and **TabFM** for tabular rows (`AI.PREDICT`, and the tabular branch of `AI.EVALUATE`). Neither requires `CREATE MODEL`, a connection, an endpoint, or any persisted model object. TimesFM comes in two versions — 2.5 (default) and 3.0 (Preview), the latter adding multivariate forecasting with covariates to `AI.FORECAST`.
+
+
+## Options
+
+| Function | What it does | Use this when |
+|----------|---------------|----------------|
+| `AI.FORECAST` | Zero-training time series forecasting on TimesFM. Returns forecasted values, confidence intervals, and optionally the historical series alongside the forecast. With `model => 'TimesFM 3.0'` (Preview): several targets at once via `target_cols`, plus `past_covariate_cols` / `future_covariate_cols`. | You want a quick forecast of future values with no `CREATE MODEL` step, no connection, no training — including one that uses related series or a known future calendar. |
+| `AI.DETECT_ANOMALIES` | Builds a TimesFM forecast baseline from historical data, then compares a separate target dataset against that baseline to flag anomalous points with a probability score. | You want to flag outliers/anomalies in a series (or across many series via `id_cols`) against expected behavior. |
+| `AI.PREDICT` | Zero-shot regression **or** classification on tabular rows using TabFM, a pre-trained tabular foundation model. Learns in-context from a training relation at query time; nothing is fitted or stored. | You want a prediction on structured rows immediately, with no training step — a baseline, a one-off, or a gap-fill on a column. |
+| `AI.EVALUATE` | **Two distinct branches.** Forecast branch: generates a TimesFM forecast from history and scores it against actuals (MAE, MSE, RMSE, MAPE, sMAPE, MASE). Tabular branch: scores an `AI.PREDICT`-shaped call by passing `label_col`. | You want to score a forecast, benchmark context-window configurations, or measure TabFM accuracy on held-out rows. |
+
+## Choosing among them
+
+- **"I want a quick forecast with no model training"** → `AI.FORECAST`. Real limits vs. BigQuery ML's `ARIMA_PLUS`: no custom holiday effects, no hierarchical reconciliation, no `ML.EXPLAIN_FORECAST`-style decomposition, and no direct control over forecast bounds beyond `confidence_level`. External regressors are **no longer** on that list — TimesFM 3.0's `past_covariate_cols` / `future_covariate_cols` cover them (Preview; horizon capped at 1,024, context at 2,048 points), so the `ARIMA_PLUS_XREG` vs `AI.FORECAST` call is now about control and explainability, not about whether covariates are possible. It's a fixed-architecture foundation model, not a trainable per-series statistical model.
+- **"I want to flag outliers/anomalies in a series"** → `AI.DETECT_ANOMALIES`. It requires two inputs (a history table to build the baseline forecast, and a target table to test) with matching schemas — a materially different shape than `AI.FORECAST`'s single-input call.
+- **"I want to know where the series *changed*, not which points are odd"** → neither of the above. `ML.DETECT_CHANGE_POINTS` (sibling `bigquery-ml` skill, Preview 2026-08-20) finds windows where the level shifted *and stayed shifted*, with no model and no forecast baseline; `ML.TREND`/`ML.SEASONALITY` decompose the series the same way. A spike that returns to normal the next day is an anomaly and not a change point; a permanent step up in volume is a change point that may never register as an anomaly. Measured on one shared series: 79 row-level `AI.DETECT_ANOMALIES` hits against 2 change windows, **zero overlap** — orthogonal detectors, not two sensitivities of one. Caveat before recommending it: all three of those TVFs gap-fill each series to its own span first, so on a series with a data outage the interpolation manufactures the structural break they then report. Profile the gaps first.
+- **"I want to predict a column on structured rows"** → `AI.PREDICT`. Task type is inferred from the label column's **type**, not from a parameter (see gotchas — this is the single easiest thing to get wrong).
+- **"I want to score how good a prediction was"** → `AI.EVALUATE`, but pick the right branch. Passing forecast-shaped and tabular-shaped arguments together fails.
+- **When to reach for `../../bigquery-ml/` instead:** you need training-time control, scheduled retraining, interpretable coefficients, or a model artifact you can version and serve. `ARIMA_PLUS` is BQML's trainable alternative to `AI.FORECAST`; `LINEAR_REG`/`LOGISTIC_REG`/`BOOSTED_TREE_*` are the trainable alternatives to `AI.PREDICT`.
+
+## Gotchas verified in this repo
+
+### TimesFM (forecasting)
+
+- **Two model versions, one default — pin it.** All three TimesFM functions default to `TimesFM 2.5` and accept `model => 'TimesFM 3.0'` (Preview). Measured with the cache off: an unpinned call and `'TimesFM 2.5'` agree to the last digit. The default has moved before without a release note, so **pin `model` explicitly**. Multivariate `AI.FORECAST` *requires* `model => 'TimesFM 3.0'` — omit it and the call fails with the generic `Invalid table-valued function AI.FORECAST`, not a message naming the cause. TimesFM 3.0 moves to token-based pricing on 2026-12-01.
+- **Pinning is not the same as reproducibility, and for two of the three functions it does not deliver it.** `AI.EVALUATE` and `AI.DETECT_ANOMALIES` return a different answer on a minority of runs even with `model` **and** `context_window` both pinned and the query cache off. Measured on a deterministic synthetic series: 2 of 16 `AI.EVALUATE` runs returned `mean_absolute_error` `0.5913808905590097` where the other 14 returned `0.8652198481135486` — a ~32% swing, roughly 1 in 8. The **majority value is the correct one**: it equals MAE computed by hand from `AI.FORECAST` output over the same history and horizon. `AI.FORECAST` is steadier but not bit-stable: 10 pinned `TimesFM 2.5` runs gave 2 distinct vectors (9 + 1) differing in the third significant figure, 10 `TimesFM 3.0` runs gave 2 (7 + 3) differing only from the seventh. Not explained by the context window (every legal value swept), horizon truncation, or version mixing — it occurs on every version tested. **The mechanism is unknown; do not offer one.** Advise materializing the metric once and reading the stored table rather than re-running the function, and recomputing from `AI.FORECAST` output with ordinary SQL when a number has to be defensible.
+- **When testing any of this yourself, the query cache will lie to you.** Re-running identical SQL returns the cached first result, so an unstable function looks perfectly stable; changing only the `SELECT` list creates a different cache entry, so two "identical" loops can disagree for reasons unrelated to the model. Several variants in one `UNION ALL` mislead the same way — three TimesFM variants in one query returned three distinct values where isolated jobs showed two of them bit-identical. Run **one variant per job with `--nouse_cache`**, compare *value sets* over ~10 runs rather than single draws, and cross-check against a function that is deterministic.
+- **BigFrames lags the SQL surface here.** `bigframes.bigquery.ai.forecast()` (2.39.0) still hardcodes `model="TimesFM 2.0"` — a version the SQL reference page no longer lists. The unpinned wrapper call still executes (measured 2026-10-09) but runs a different model than SQL's default, and the wrapper has no `target_cols` or covariate arguments. Pass `model` explicitly; use SQL for the multivariate form.
+- **Limits differ by version.** `horizon` range is `[1, 10000]` on 2.5 and `[1, 1024]` on 3.0; max context is 15,360 points on 2.5 and 2,048 on 3.0, with older points silently ignored rather than erroring. `context_window` on 2.5 must be one of 64/128/256/512/1024/2048/4096/8192/15360 (off-grid values rejected) and defaults to the smallest window covering the input; on 3.0 it is any multiple of 32 from 64 to 2048, **defaults to the maximum**, and an off-grid value such as `100` is *accepted* rather than rejected — stay on the grid if you want to know what the model saw. Every rejection is the same generic `Invalid table-valued function` error.
+- `AI.FORECAST` and `AI.EVALUATE` have wildly different default horizons — 10 vs. 1024 — despite sharing the same `horizon` parameter. `AI.DETECT_ANOMALIES` silently caps evaluation at the most recent 1,024 target points. All three require a **minimum of 3 data points**.
+- **Multivariate output is restructured** (TimesFM 3.0, `target_cols`): the timestamp column keeps its *input* name instead of becoming `forecast_timestamp`, each target becomes a `STRUCT<value, prediction_interval_lower_bound, prediction_interval_upper_bound>` under its own name, and `time_series_type` is always present. With `future_covariate_cols`, supply one input row per future timestamp — targets NULL, covariates populated — because **`horizon` is silently capped by the future rows present** (30 requested with 14 rows returns 14) and defaults to 10 even when more rows exist. A past covariate alone needs no future rows. `data_col` plus a covariate argument is rejected. `forecast_end_timestamp` works on this form although the documented syntax omits it. `AI.EVALUATE` has no `target_cols` — score a multivariate forecast one target at a time.
+- `AI.FORECAST`'s output schema changes shape entirely based on `output_historical_time_series`: `FALSE` gives `forecast_timestamp`/`forecast_value`/bounds; `TRUE` gives a unioned `time_series_type` ('history'/'forecast') + `time_series_data` column instead, with bounds NULL for historical rows.
+- Each TimesFM function emits a per-row status column (`ai_forecast_status`, `ai_detect_anomalies_status`, `ai_evaluate_status`) holding the error string on failure — check it per-row, since partial failures (one bad series among many `id_cols`) surface there rather than failing the query. **The three columns are not consistent about what "success" looks like** (verified live 2026-09-02): `ai_evaluate_status` and `ai_detect_anomalies_status` return **`NULL`** on success, while `ai_forecast_status` returns a genuine **zero-length empty string**. So `WHERE ai_evaluate_status = ''` matches nothing, and `WHERE ai_evaluate_status <> ''` silently drops **every successful row** — the `NULL` comparison is unknown, not true. **Filter with `IS NOT NULL` / `IS NULL`**, which is correct for all three.
+- `horizon` and `forecast_end_timestamp` are mutually exclusive on `AI.FORECAST`.
+
+### TabFM (`AI.PREDICT`)
+
+- **Task type comes from the label column's type, not a parameter.** A categorical label encoded as `INT64` silently trains a *regression* and returns a continuous number instead of a class. Cast coded categoricals to `STRING` before passing them.
+- **Output is not deterministic, and `FARM_FINGERPRINT` does not fix it.** A deterministic split pins *which rows* train and predict; it does nothing to the predictions. Repeating a byte-identical `AI.PREDICT` call returns slightly different numbers, consistent with the model averaging shuffled ensemble passes (`n_ensembles`, value undocumented). Measured: regression MAE 236.52 / 236.35 / 235.84 and classification accuracy 0.9468 / 0.9362 / 0.9468 across repeated identical runs — under 1% regression drift, about 1pp on classification. Consequence: `AI.EVALUATE` scores its *own* fresh predictions, not the rows you materialized, so its metrics will not match a recomputation from your stored prediction table. **Materialize predictions once and join to them; never re-run to reproduce a number.** Drift is input-dependent — a 6-row prediction relation reproduced exactly across two runs — so no observed drift on a tiny input is not evidence of determinism.
+- **Large training relations fail on query memory, not on a documented cap.** The error is `resourcesExceeded` — *"The query could not be executed in the allotted memory. Peak usage: 124% of limit"*, with 91% of that usage reported as `other/unattributed`. Measured on a two-feature relation whose *prediction* side was only 2 rows, so the training relation alone drives it: 1,000 rows OK (44s), 5,000 OK (82s), 8,000 OK (70s), 10,000 fails, 50,000 fails, 200,000 fails. Failures burn ~1.37M slot_ms before dying versus ~210K for a run that succeeds. All measured **on-demand with no reservation**; untested under an Enterprise reservation, where the memory limit may differ. Size the training relation with this ceiling in mind and expect it to move with pricing model and column count.
+- Hard limits: **50 feature columns** (raised from 20; measured 2026-10-12 — 50 runs, 51 fails naming the cap) and **10 classes** for classification. Exceeding either errors at analysis time, before any inference is billed.
+- **Only six column types are accepted** (`STRING`, `BOOL`, `INT64`, `FLOAT64`, `NUMERIC`, `BIGNUMERIC`). `DATE`, `TIMESTAMP`, `BYTES`, `JSON`, `GEOGRAPHY`, `ARRAY`, and `STRUCT` are all rejected — extract date parts to `INT64` (e.g. `EXTRACT(MONTH FROM d)`) before passing them.
+- `label_col` defaults to a column literally named `label`; pass it explicitly otherwise.
+- **The documentation has a bug here:** the reference page says passthrough columns come from the *training* table. Observed behavior returns the *prediction* rows. Trust the behavior.
+- Latency is 30–95s per call. A notebook with several `AI.PREDICT` examples is a multi-minute run, not an interactive one.
+- Pricing (effective 2026-10-30) is token-based on a formula, not per-row: `input_tokens = (train_rows * columns + predict_rows * (columns - 1)) * n_ensembles`, `output_tokens = predict_rows * n_ensembles`, at $0.05/mtok in and $0.20/mtok out. Training-relation size drives cost on *every* call, since there is no reusable fitted model.
+- There is **no BigFrames wrapper** — no `bbq.ai.predict`. Use `read_gbq_query` with the SQL.
+
+### `AI.EVALUATE` across both branches
+
+- **The two branches are mutually exclusive** — mixing forecast-shaped and tabular-shaped arguments in one call errors. Choose by which model family you are scoring.
+- The tabular branch is just `AI.PREDICT`'s two relations plus `label_col`; the correspondence is exact, which makes it easy to score a prediction you already wrote.
+- **Only the TimesFM branch returns `ai_evaluate_status`.** The tabular branch's output has no status column — don't write a per-row status check against it.
+- The tabular branch returns **no `log_loss`, no `roc_auc`, and no confusion matrix** for classification. If you need those, use BigQuery ML's `ML.EVALUATE` on a trained classifier instead.
+- **The label column's TYPE silently decides what `precision`/`recall`/`f1_score` mean, and nothing in the output says which convention you got.** A `BOOL` `label_col` is scored **binary** — the positive (`TRUE`) class alone. The `STRING` rendering of the identical values is scored **multiclass and macro-averaged**, even at two classes. `accuracy` is the same either way. Measured on one imbalanced problem (12 positives of 62 rows; TabFM confusion matrix TP 12 / FP 11 / FN 0 / TN 39): `BOOL` returned precision `0.5217` and recall `1.0`, `STRING` returned `0.7609` and `0.89` — the two-class means. Four isolated runs of each agreed. **This is a BigQuery-wide convention rather than an `AI.EVALUATE` behavior** — BigQuery ML's `ML.METRICS` follows the identical rule, measured side by side (see the sibling `bigquery-ml` skill's `narrative/evaluation.md`). On balanced data the two nearly coincide, which is how it hides until the classes are lopsided; fix the type deliberately and say which convention a published number uses.
+
+## Canonical snippets
+
+```sql
+-- TimesFM: forecast a daily series
+SELECT *
+FROM AI.FORECAST(
+  (SELECT date, daily_bike_trips FROM `project.dataset.bike_trips_daily`),
+  data_col => 'daily_bike_trips',
+  timestamp_col => 'date',
+  model => 'TimesFM 2.5',   -- pin it; the default has moved before
+  horizon => 14,
+  confidence_level => 0.90
+);
+
+-- TimesFM 3.0 (Preview): one target, a past covariate, two future covariates.
+-- The input holds 14 future rows with daily_sales NULL and the future covariates filled.
+SELECT date, daily_sales.value, daily_sales.prediction_interval_lower_bound, daily_sales.prediction_interval_upper_bound
+FROM AI.FORECAST(
+  TABLE `project.dataset.sales_with_covariates`,
+  target_cols => ['daily_sales'], timestamp_col => 'date', model => 'TimesFM 3.0', horizon => 14,
+  past_covariate_cols => ['web_visits'], future_covariate_cols => ['promo_flag', 'is_weekend']
+);
+```
+
+```sql
+-- TabFM: zero-shot prediction, no CREATE MODEL anywhere
+SELECT *
+FROM AI.PREDICT(
+  (SELECT species, island, culmen_length_mm, body_mass_g
+   FROM `bigquery-public-data.ml_datasets.penguins` WHERE body_mass_g IS NOT NULL),
+  (SELECT species, island, culmen_length_mm
+   FROM `bigquery-public-data.ml_datasets.penguins` WHERE body_mass_g IS NULL),
+  label_col => 'body_mass_g'
+);
+```
+
+## Go deeper
+
+Full extracted notebook walkthroughs live in this skill's `narrative/` folder:
+
+- [`narrative/ai_forecast.md`](../narrative/ai_forecast.md) (source: `functions/ai_forecast/`) — cross-linked bidirectionally with bigquery-ml's arima_plus, the trainable alternative, for a direct side-by-side comparison
+- [`narrative/ai_detect_anomalies.md`](../narrative/ai_detect_anomalies.md) (source: `functions/ai_detect_anomalies/`)
+- [`narrative/ai_predict.md`](../narrative/ai_predict.md) (source: `functions/ai_predict/`)
+- [`narrative/ai_evaluate.md`](../narrative/ai_evaluate.md) (source: `functions/ai_evaluate/`) — covers both branches
+- [`narrative/time_series_intelligence.md`](../narrative/time_series_intelligence.md) — the TimesFM functions composed end-to-end
+- [`narrative/tabular_prediction.md`](../narrative/tabular_prediction.md) — the TabFM functions composed end-to-end, with a head-to-head against trained BigQuery ML models
+
+The `bigquery-ml` skill's `narrative/arima_plus.md` has the other half of the `AI.FORECAST` vs. `ARIMA_PLUS` comparison.
+
+Full syntax/options tables: see the source repo at `bigquery-ai/reference/predictive-ai.md`, indexed from `bigquery-ai/RESOURCES.md`.
+
+**Lens 3 per-statement syntax cards (`syntax/`):** [`syntax/INDEX.md`](../syntax/INDEX.md) — [`syntax/aiforecast.md`](../syntax/aiforecast.md), [`syntax/aidetect_anomalies.md`](../syntax/aidetect_anomalies.md), [`syntax/aipredict.md`](../syntax/aipredict.md), [`syntax/aievaluate.md`](../syntax/aievaluate.md)

bigquery_data_science/_skills/bigquery-ai/reference/workflows.md

@@ -0,0 +1,56 @@
+# Composed Workflows in BigQuery AI Functions
+
+Contents: [Choosing a starting template](#choosing-a-starting-template) · [Go deeper](#go-deeper)
+
+Individual functions rarely answer a real business question alone. These 14 workflows compose several functions into a genuine end-to-end task — use them as worked patterns to copy from, not just a function catalog.
+
+| Workflow | Functions used | What it does |
+|---|---|---|
+| `data_enrichment` | `AI.GENERATE` (Google Search grounding + `output_schema`), `AI.PREDICT` | Fill gaps two ways: retrieve the answer from the web, or infer it from the rows you already have. A direct contrast between grounded generation and zero-shot tabular inference. |
+| `content_analysis` | `AI.GENERATE_TABLE`, `AI.CLASSIFY`, `AI.SCORE`, `AI.GENERATE`, `AI.AGG` | Generate sample data, classify it, score it, and summarize findings — a good template for any "classify then score then summarize" pipeline. |
+| `semantic_search` | `AI.EMBED`, `VECTOR_SEARCH`, `AI.SEARCH` | Build and query a semantic search index three ways — manual, simplified, and hybrid — to see what each layer of convenience costs. |
+| `catalog_search` | `AI.EMBED`, `VECTOR_SEARCH`, `AI.SEARCH` | Hybrid retrieval over a real ~7,275-row product catalog: what rank fusion buys for an exact-token lookup and what it does not, plus a corpus large enough to actually populate a vector index. The clearest demonstration that a bare SKU is not retrievable at the small `top_k` these demos use — its row sits at semantic rank 2,072, outside the `10 * top_k` lexical candidate pool until `top_k >= 208` — while the same SKU alongside descriptive words surfaces either way. |
+| `rag_pipeline` | `AI.GENERATE_TABLE`, `AI.EMBED`, `VECTOR_SEARCH`, `AI.GENERATE` | Generate a knowledge base, embed it, search it (semantic and hybrid), answer questions grounded in the retrieved context. |
+| `tabular_prediction` | `AI.PREDICT`, `AI.EVALUATE`, `AI.KEY_DRIVERS`, `AI.GENERATE` | Zero-shot regression and classification with TabFM — no training step — then scored, explained by driver, and written up as a model card. Includes a head-to-head against trained BigQuery ML models. |
+| `document_rag` | `AI.PARSE_DOCUMENT`, `AI.EMBED`, `VECTOR_SEARCH`, `AI.GENERATE` | Parse real documents, embed chunks, search, answer questions with grounded context. **Currently blocked** — depends on `AI.PARSE_DOCUMENT`, which is offline for revision (see `document-processing.md`). |
+| `time_series_intelligence` | `AI.FORECAST`, `AI.DETECT_ANOMALIES`, `AI.EVALUATE`, `AI.KEY_DRIVERS` | Forecast a series, detect anomalies in it, evaluate forecast accuracy, then explain movement by segment — the single most complete tour of the forecasting/anomaly/driver functions together. |
+| `metric_diagnostics` | `AI.KEY_DRIVERS`, `AI.PREDICT`, `AI.CAUSAL_EFFECT`, `AI.GENERATE` | Explain why a metric moved between two periods, project where it goes next, then narrate both in plain language — pairs structured analysis and zero-shot prediction with a generation function for a readable output. Ends by *refusing* the causal claim: a placebo test runs `AI.CAUSAL_EFFECT` at six dates where nothing happened and one of them comes back significant, which is why the earlier steps stay descriptive. |
+| `document_intelligence` | `AI.CLASSIFY`, `AI.GENERATE`, `AI.SCORE`, `AI.AGG` | Classify mixed documents, extract key fields, score quality, summarize findings. |
+| `content_moderation` | `AI.GENERATE_TABLE`, `AI.IF`, `AI.CLASSIFY`, `AI.SCORE`, `AI.GENERATE`, `AI.AGG` | Flag, categorize, and score user-generated content — the broadest single-workflow tour of the managed functions. |
+| `multimodal_analysis` | `AI.EMBED`, `AI.SIMILARITY`, `AI.GENERATE` | Embed document images, find similar documents, generate visual descriptions. |
+| `log_analysis` | `AI.GENERATE_TABLE`, `AI.CLASSIFY`, `AI.SCORE`, `AI.AGG`, `AI.EMBED`, `VECTOR_SEARCH` | Classify tickets, score priority, summarize patterns, then retrieve by error code with hybrid search — a support/ops-triage template, and the sub-project's clearest measurement of how far up the ranking a lexical hit can lift a row (vector rank 24 of 30 → fused position 11) and why `top_k` decides whether you ever see it. |
+| `image_deduplication` | `AI.EMBED`, `VECTOR_SEARCH` | Group near-duplicate images using embedding similarity, e.g. to protect train/test split integrity. |
+
+## Choosing a starting template
+
+- **Need to classify + score + summarize unstructured records at volume** (tickets, documents, user content) → `content_analysis`, `document_intelligence`, `content_moderation`, or `log_analysis` — pick by how many of the managed functions (`AI.CLASSIFY`/`AI.SCORE`/`AI.IF`/`AI.AGG`) you need together; `content_moderation` is the most complete template.
+- **Need to answer questions grounded in your own data** → `rag_pipeline` (text) or `document_rag` (real documents — currently blocked, see above).
+- **Need to find similar items** (text, images, documents) → `semantic_search` or `image_deduplication` depending on whether you need a queryable index or a one-off grouping pass.
+- **Need search that weighs exact tokens** (SKUs, error codes, part numbers) → `catalog_search` for the full hybrid treatment on a real corpus with a vector index, or `log_analysis`'s Step 5 for the compact version. `semantic_search` shows hybrid alongside the two semantic-only approaches for contrast. **Set expectations first:** hybrid re-ranks and modestly widens, and `top_k` decides how far — a lexical hit reaches 10 semantic ranks deep at `top_k => 5`, 110 at `top_k => 30`, and 640 at `top_k => 64`. Two gates set that reach and effective reach is `min( score gate, 10 * top_k )`: the fused score has to beat the last slot on the page, which is what binds at `top_k` below 51, and BigQuery only pools the top `10 * top_k` rows for the lexical leg, which is what binds from 51 up. So it pays off when the query carries descriptive words *alongside* the token, and exact-token retrieval for a target at semantic rank `R` needs `top_k` of **at least** `R/10` — a floor, not a recipe, since below a few hundred ranks deep the score gate demands considerably more (a target at rank 100 needs `top_k` near 29, not 10). Read the reach table for the number that actually applies. When the token is the entire query and the answer must be certain, point the user at a `WHERE` predicate instead — `catalog_search` demonstrates exactly that trade-off. See `reference/embeddings-and-search.md` for the arithmetic and the reach table.
+- **Need to predict a value or category from structured rows with no training** → `tabular_prediction` (the full TabFM tour: predict, score, explain, narrate) or `metric_diagnostics` (drivers + a lighter projection).
+- **Need to explain a metric or forecast something over time** → `time_series_intelligence` (the full forecast/anomaly/evaluate/drivers tour) or `metric_diagnostics` (drivers + narration, lighter weight).
+- **Need to enrich/clean records against external knowledge** → `data_enrichment`, which contrasts the Google Search grounding pattern against zero-shot tabular inference for the same gap.
+- **Working with images/visual content specifically** → `multimodal_analysis`.
+
+## Go deeper
+
+Full extracted notebook walkthroughs live in this skill's `narrative/` folder:
+
+- [`narrative/catalog_search.md`](../narrative/catalog_search.md) (source: `workflows/catalog_search/`)
+- [`narrative/content_analysis.md`](../narrative/content_analysis.md) (source: `workflows/content_analysis/`)
+- [`narrative/content_moderation.md`](../narrative/content_moderation.md) (source: `workflows/content_moderation/`)
+- [`narrative/data_enrichment.md`](../narrative/data_enrichment.md) (source: `workflows/data_enrichment/`)
+- [`narrative/document_intelligence.md`](../narrative/document_intelligence.md) (source: `workflows/document_intelligence/`)
+- [`narrative/document_rag.md`](../narrative/document_rag.md) (source: `workflows/document_rag/`) — currently blocked on the AI.PARSE_DOCUMENT outage
+- [`narrative/image_deduplication.md`](../narrative/image_deduplication.md) (source: `workflows/image_deduplication/`)
+- [`narrative/log_analysis.md`](../narrative/log_analysis.md) (source: `workflows/log_analysis/`)
+- [`narrative/metric_diagnostics.md`](../narrative/metric_diagnostics.md) (source: `workflows/metric_diagnostics/`)
+- [`narrative/multimodal_analysis.md`](../narrative/multimodal_analysis.md) (source: `workflows/multimodal_analysis/`)
+- [`narrative/rag_pipeline.md`](../narrative/rag_pipeline.md) (source: `workflows/rag_pipeline/`)
+- [`narrative/semantic_search.md`](../narrative/semantic_search.md) (source: `workflows/semantic_search/`)
+- [`narrative/tabular_prediction.md`](../narrative/tabular_prediction.md) (source: `workflows/tabular_prediction/`)
+- [`narrative/time_series_intelligence.md`](../narrative/time_series_intelligence.md) (source: `workflows/time_series_intelligence/`)
+
+Full function-to-workflow map and "How Functions Relate" diagram: see the README in the source repo (`bigquery-ai/README.md`).
+
+**Lens 3 per-statement syntax cards (`syntax/`):** [`syntax/INDEX.md`](../syntax/INDEX.md) — [`syntax/aigenerate.md`](../syntax/aigenerate.md), [`syntax/aigenerate_table.md`](../syntax/aigenerate_table.md), [`syntax/aiclassify.md`](../syntax/aiclassify.md), [`syntax/aiembed.md`](../syntax/aiembed.md), [`syntax/vector_search.md`](../syntax/vector_search.md), [`syntax/aiforecast.md`](../syntax/aiforecast.md), [`syntax/aipredict.md`](../syntax/aipredict.md), [`syntax/aikey_drivers.md`](../syntax/aikey_drivers.md)

bigquery_data_science/_skills/bigquery-ai/skill.manifest.json

@@ -0,0 +1,109 @@
+{
+  "name": "bigquery-ai",
+  "version": "0.7.0",
+  "source_project": "bigquery-ai",
+  "narrative_sources": {},
+  "generated_date": "2026-10-19",
+  "source_commit": null,
+  "reference_files": [
+    "augmented-analytics.md",
+    "cloud-ai-service-models.md",
+    "document-processing.md",
+    "embeddings-and-search.md",
+    "general-purpose-generation.md",
+    "managed-functions.md",
+    "predictive-ai.md",
+    "workflows.md"
+  ],
+  "syntax_files": [
+    "aiagg.md",
+    "aicausal_effect-covered-in-bigquery-ml.md",
+    "aiclassify.md",
+    "aicount_tokens.md",
+    "aidetect_anomalies.md",
+    "aiembed.md",
+    "aievaluate.md",
+    "aiforecast.md",
+    "aigenerate.md",
+    "aigenerate_bool.md",
+    "aigenerate_double.md",
+    "aigenerate_embedding.md",
+    "aigenerate_int.md",
+    "aigenerate_table.md",
+    "aigenerate_text.md",
+    "aiif.md",
+    "aikey_drivers.md",
+    "aiparse_document.md",
+    "aipredict.md",
+    "aiscore.md",
+    "aisearch.md",
+    "aisimilarity.md",
+    "choosing-between-these-and-the-ai-functions.md",
+    "hybrid-search-capability.md",
+    "mlannotate_image.md",
+    "mlgenerate_embedding.md",
+    "mlgenerate_text.md",
+    "mlprocess_document.md",
+    "mltranscribe.md",
+    "mltranslate.md",
+    "mlunderstand_text.md",
+    "multimodal-input-patterns-by-function.md",
+    "object-tables.md",
+    "objectref-and-objectrefruntime-schema-reference.md",
+    "objfetch_metadata.md",
+    "objget_access_url.md",
+    "objget_read_url.md",
+    "objlist.md",
+    "objmake_ref.md",
+    "per-row-failures-and-the-retry-pattern.md",
+    "setup-apis-connection-roles.md",
+    "two-input-shapes.md",
+    "vector_search.md",
+    "what-the-five-have-in-common.md"
+  ],
+  "narrative_files": [
+    "ai_agg.md",
+    "ai_classify.md",
+    "ai_count_tokens.md",
+    "ai_detect_anomalies.md",
+    "ai_embed.md",
+    "ai_evaluate.md",
+    "ai_forecast.md",
+    "ai_generate.md",
+    "ai_generate_bool.md",
+    "ai_generate_double.md",
+    "ai_generate_embedding.md",
+    "ai_generate_int.md",
+    "ai_generate_table.md",
+    "ai_generate_text.md",
+    "ai_if.md",
+    "ai_key_drivers.md",
+    "ai_parse_document.md",
+    "ai_predict.md",
+    "ai_score.md",
+    "ai_search.md",
+    "ai_similarity.md",
+    "catalog_search.md",
+    "content_analysis.md",
+    "content_moderation.md",
+    "data_enrichment.md",
+    "document_intelligence.md",
+    "document_rag.md",
+    "image_deduplication.md",
+    "log_analysis.md",
+    "metric_diagnostics.md",
+    "ml_annotate_image.md",
+    "ml_generate_embedding.md",
+    "ml_generate_text.md",
+    "ml_process_document.md",
+    "ml_transcribe.md",
+    "ml_translate.md",
+    "ml_understand_text.md",
+    "multimodal_analysis.md",
+    "rag_pipeline.md",
+    "semantic_search.md",
+    "tabular_prediction.md",
+    "time_series_intelligence.md",
+    "vector_search.md"
+  ]
+}

bigquery_data_science/_skills/bigquery-ai/syntax/INDEX.md

@@ -0,0 +1,71 @@
+# `bigquery-ai` Syntax Cards Index (Lens 3)
+
+Focused, per-statement syntax and option cards sliced from [`bigquery-ai/reference/`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference). Every card is strictly under 25 KB.
+
+## Augmented Analytics (`augmented-analytics.md`)
+
+- [`AI.CAUSAL_EFFECT` — covered in `bigquery-ml`](aicausal_effect-covered-in-bigquery-ml.md)
+- [`AI.KEY_DRIVERS`](aikey_drivers.md)
+
+## Cloud AI Service Models (`cloud-ai-service-models.md`)
+
+- [What the five have in common](what-the-five-have-in-common.md)
+- [Setup: APIs, connection, roles](setup-apis-connection-roles.md)
+- [Two input shapes](two-input-shapes.md)
+- [`ML.TRANSLATE`](mltranslate.md)
+- [`ML.UNDERSTAND_TEXT`](mlunderstand_text.md)
+- [`ML.ANNOTATE_IMAGE`](mlannotate_image.md)
+- [`ML.TRANSCRIBE`](mltranscribe.md)
+- [Per-row failures and the retry pattern](per-row-failures-and-the-retry-pattern.md)
+- [Choosing between these and the `AI.*` functions](choosing-between-these-and-the-ai-functions.md)
+
+## Document Processing (`document-processing.md`)
+
+- [`ML.PROCESS_DOCUMENT`](mlprocess_document.md)
+- [`AI.PARSE_DOCUMENT`](aiparse_document.md)
+
+## Embedding Generation and Semantic Search (`embedding-generation-and-semantic-search.md`)
+
+- [`AI.EMBED`](aiembed.md)
+- [`AI.GENERATE_EMBEDDING`](aigenerate_embedding.md)
+- [`ML.GENERATE_EMBEDDING`](mlgenerate_embedding.md)
+- [`AI.SIMILARITY`](aisimilarity.md)
+- [`VECTOR_SEARCH`](vector_search.md)
+- [`AI.SEARCH`](aisearch.md)
+- [Hybrid Search *(capability)*](hybrid-search-capability.md)
+
+## General Purpose Functions (`general-purpose-functions.md`)
+
+- [`AI.GENERATE`](aigenerate.md)
+- [`AI.GENERATE_TEXT`](aigenerate_text.md)
+- [`AI.GENERATE_TABLE`](aigenerate_table.md)
+- [`AI.GENERATE_BOOL`](aigenerate_bool.md)
+- [`AI.GENERATE_DOUBLE`](aigenerate_double.md)
+- [`AI.GENERATE_INT`](aigenerate_int.md)
+- [`ML.GENERATE_TEXT`](mlgenerate_text.md)
+- [`AI.COUNT_TOKENS`](aicount_tokens.md)
+
+## Managed Functions (`managed-functions.md`)
+
+- [`AI.IF`](aiif.md)
+- [`AI.SCORE`](aiscore.md)
+- [`AI.CLASSIFY`](aiclassify.md)
+- [`AI.AGG`](aiagg.md)
+
+## Predictive AI (`predictive-ai.md`)
+
+- [`AI.FORECAST`](aiforecast.md)
+- [`AI.DETECT_ANOMALIES`](aidetect_anomalies.md)
+- [`AI.PREDICT`](aipredict.md)
+- [`AI.EVALUATE`](aievaluate.md)
+
+## Unstructured Data Infrastructure (`unstructured-data-infrastructure.md`)
+
+- [Object Tables](object-tables.md)
+- [`OBJ.LIST`](objlist.md)
+- [`OBJ.MAKE_REF`](objmake_ref.md)
+- [`OBJ.FETCH_METADATA`](objfetch_metadata.md)
+- [`OBJ.GET_ACCESS_URL`](objget_access_url.md)
+- [`OBJ.GET_READ_URL`](objget_read_url.md)
+- [ObjectRef and ObjectRefRuntime Schema Reference](objectref-and-objectrefruntime-schema-reference.md)
+- [Multimodal Input Patterns by Function](multimodal-input-patterns-by-function.md)

bigquery_data_science/_skills/bigquery-ai/syntax/aiagg.md

@@ -0,0 +1,67 @@
+# `AI.AGG`
+
+> **Source:** [`bigquery-ai/reference/managed-functions.md#aiagg`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/managed-functions.md#aiagg)
+
+- **Description:** (Preview) Aggregate function that uses a Vertex AI Gemini model to aggregate data based on natural language instructions. Automatically performs multi-level aggregation through batching, so it can analyze data exceeding the Gemini context window. Returns a single STRING per group.
+- **Use cases:** Sentiment analysis across reviews, content summarization of text or images, log analysis and incident investigation, agent performance analysis, finding common categories or patterns across data, summarizing grouped data.
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-agg)
+- **Type:** Aggregate function -- returns a single STRING value per group (Preview). Use with GROUP BY for per-group results; without GROUP BY, aggregates all rows.
+
+**Syntax:**
+```sql
+AI.AGG(
+  [ DISTINCT ]
+  INPUT,
+  INSTRUCTION
+  [, connection_id => 'CONNECTION']
+  [, endpoint => 'ENDPOINT']
+)
+```
+
+**Inputs:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `DISTINCT` | keyword | Optional | When specified, deduplicates input values before aggregating. |
+| `INPUT` | STRING or STRUCT | Required (must be first argument) | The data to aggregate. STRING value, or STRUCT consisting of STRING values, object references, and arrays of either. **The reference page still specifies `ObjectRefRuntime` here — it is the only `AI.*` page that does.** A bare `ObjectRef` works anyway: `AI.AGG(STRUCT(ref), instruction)` over an object table returns real summaries (verified live 2026-09-15 over 50 PDFs), and the Known Issues below argue for it. |
+| `INSTRUCTION` | STRING | Required (must be second argument) | Natural language aggregation prompt describing what to extract or summarize from the input data. Can be a string literal or query parameter. |
+| `CONNECTION` | STRING | Optional | Connection to use, format: `[PROJECT_ID.]LOCATION.CONNECTION_ID`. If not specified, end-user credentials are used. |
+| `ENDPOINT` | STRING | Optional | Vertex AI endpoint. Any Gemini model that doesn't require thinking budget. If not specified, BigQuery chooses a model for you. |
+
+**Outputs:**
+
+| Return Type | Description |
+|-------------|-------------|
+| STRING | The aggregated result for the group. One result per GROUP BY group, or one result total without GROUP BY. |
+| STRING (partial) | If some Vertex AI calls fail, returns partial results from successful calls. |
+| NULL | Returned if all calls to Vertex AI fail, or if all input rows are invalid. |
+
+**Output cap:** 10,000 tokens per group.
+
+**Supported models:** Any Gemini model that doesn't require thinking budget. If no endpoint specified, BigQuery dynamically chooses a model. Uses various `gemini-2.5-*` models.
+
+**Best practices:**
+- Use `TO_JSON_STRING` to pass multiple columns as structured input -- the model sees each row's full context.
+- Use `DISTINCT` to remove duplicate inputs and reduce token usage.
+- Keep groups under 20 million rows per query and under 1,000 distinct groups to avoid timeouts.
+- Prefer AI.AGG over manual aggregation with AI.GENERATE (STRING_AGG/ARRAY_AGG into a prompt) -- AI.AGG handles batching automatically and scales beyond the context window.
+
+**Limitations:** Preview status. Returns NULL on total failure (no detailed error info). No model parameter control (temperature, top_p, etc.). Output capped at 10,000 tokens per group. Recommended limit of 20 million rows per query and 1,000 distinct groups. Cannot use Gemini models that require thinking budget.
+
+**Known issues:**
+- Input rows with 10 or more images in a single row might be skipped.
+- Input rows with arrays of ObjectRefRuntime objects that call `OBJ.GET_ACCESS_URL` might be skipped.
+- Some ObjectRefRuntime **image** objects created by `OBJ.GET_ACCESS_URL` might fail to process.
+- Both of the above are about `ObjectRefRuntime` specifically. Passing the `ObjectRef` sidesteps them — the practical argument for the unwrapped form even though this page's `INPUT` row still names the runtime type.
+- Individual rows larger than **10 MiB** might fail with an internal error.
+- Mixing `AI.AGG()` without `DISTINCT` and another aggregate function *with* `DISTINCT` in the same query might return an internal error.
+- Workforce Identity Federation without a specified Cloud resource connection may cause failures on long-running queries.
+- Queries using Gemini 3.0 or 3.1 with connection-based authentication might receive an unauthenticated error.
+
+**Cost caveat:** The actual number of rows processed by the model might differ from expectations, particularly with complex queries (JOIN, ORDER BY ... LIMIT). Materialize data to a separate table first to ensure predictable processing.
+
+**Locations:** All regions supporting Gemini models, plus US and EU multi-regions.
+
+**Provisioned throughput:** Not supported. Uses dynamic shared quota (DSQ) only.
+
+**BigFrames API:** No native BigFrames API. Use `bpd.read_gbq_query(sql)` to execute AI.AGG queries and get results as a BigFrames DataFrame.

bigquery_data_science/_skills/bigquery-ai/syntax/aicausal_effect-covered-in-bigquery-ml.md

@@ -0,0 +1,17 @@
+# `AI.CAUSAL_EFFECT` — covered in `bigquery-ml`
+
+> **Source:** [`bigquery-ai/reference/augmented-analytics.md#aicausal_effect--covered-in-bigquery-ml`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/augmented-analytics.md#aicausal_effect--covered-in-bigquery-ml)
+
+Full entry: **[`bigquery-ml/reference/model-free-functions.md#aicausal_effect`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/reference/model-free-functions.md#aicausal_effect)**
+Tested example: **[`bigquery-ml/workflows/causal_effect/`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/workflows/causal_effect)**
+
+This is a deliberate exception to the usual rule that every `AI.*` function is documented in this project. The dividing line between the two projects is *"does the reader manage a model artifact,"* not *"does the name start with `AI.`"* — and by that test this one belongs there:
+
+- Its subject is **causal inference**, which `bigquery-ml/workflows/` already covers five other ways ([DiD](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/workflows/difference_in_differences), [synthetic control](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/workflows/synthetic_control), [propensity score matching](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/workflows/propensity_score_matching), [uplift/CATE](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/workflows/uplift_cate), [double ML](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/workflows/price_elasticity_dml)). The only way to judge its estimate is against those, on the same data — which is exactly what the linked notebook does, and it finds a ~2.9× magnitude disagreement.
+- It is **not foundation-model backed**. Unlike every other function in this project, its counterfactual is measurably plain `ARIMA_PLUS` + `ML.FORECAST` on default options — identical to the last digit — and it rejects the `model` argument the TimesFM functions accept. It is also fully deterministic, which none of the TimesFM functions are: its p-value is a closed form, `2 · (1 − CDF(\|Z\|))` on the sum of the model's own pointwise standard errors, reconstructible bit for bit rather than sampled.
+
+It stays listed in this project's [function catalog](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/README.md#augmented-analytics--find-what-drives-metric-changes) so that catalog remains a complete inventory of the `AI.*` surface.
+
+**Where it appears in this project anyway:** [`workflows/metric_diagnostics/`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/workflows/metric_diagnostics) runs it as a **placebo test** at the end of an `AI.KEY_DRIVERS` → `AI.PREDICT` diagnostic. There is no intervention in that dataset, which is the point: six dates chosen only for spacing, and one of them returns a significant cumulative effect with nothing behind it, because a counterfactual fitted on a spring-and-summer ramp reads the autumn seasonal turn as an intervention. That is the discipline the shared heading makes easy to skip — the same trap as reading `AI.KEY_DRIVERS` on before/after periods as causal, arriving from the other direction.
+
+---

bigquery_data_science/_skills/bigquery-ai/syntax/aiclassify.md

@@ -0,0 +1,72 @@
+# `AI.CLASSIFY`
+
+> **Source:** [`bigquery-ai/reference/managed-functions.md#aiclassify`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ai/reference/managed-functions.md#aiclassify)
+
+- **Description:** (Preview) Scalar function that uses a Vertex AI Gemini model to classify inputs into categories that you provide. BigQuery automatically structures your input to improve the quality of the classification.
+- **Use cases:** Classify reviews by sentiment, classify products by categories, classify support tickets by topic, image classification by style or contents, multi-label classification.
+- [documentation](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-classify)
+- **Type:** Scalar function -- returns STRING or ARRAY\<STRING\> per row (Preview)
+
+**Syntax:**
+```sql
+AI.CLASSIFY(
+  [ input => ] 'INPUT',
+  [ categories => ] 'CATEGORIES'
+  [, examples => EXAMPLES ]
+  [, connection_id => 'CONNECTION' ]
+  [, endpoint => 'ENDPOINT' ]
+  [, output_mode => 'OUTPUT_MODE' ]
+  [, embeddings => EMBEDDINGS ]
+  [, optimization_mode => 'OPTIMIZATION_MODE' ]
+  [, max_error_ratio => MAX_ERROR_RATIO ]
+)
+```
+
+**Inputs:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `INPUT` | STRING or STRUCT | Required (must be first argument) | The input to classify. STRING, an `ObjectRef`, or a STRUCT with STRING/ARRAY\<STRING\>/ObjectRef/ARRAY\<ObjectRef\> fields. At most one video object. An object table's `ref` column goes in directly — `EXTERNAL_OBJECT_TRANSFORM` is not required (verified live 2026-09-15). |
+| `CATEGORIES` | ARRAY\<STRING\> or ARRAY\<STRUCT\<STRING, STRING\>\> | Required (must be second argument) | The categories to classify into. Without descriptions: `['positive', 'neutral', 'negative']`. With descriptions: `[('green', 'positive'), ('yellow', 'neutral'), ('red', 'negative')]`. Must be string literals (or use a DECLARE variable from a table column). |
+| `EXAMPLES` | ARRAY\<STRUCT\<STRING, STRING\>\> (single) or ARRAY\<STRUCT\<STRING, ARRAY\<STRING\>\>\> (multi) | Optional | Few-shot examples. For single mode: maps input to expected category. For multi mode: maps input to expected array of categories. Example: `[("This is a great phone", "tech"), ("The match was exciting", "sport")]`. |
+| `CONNECTION` | STRING | Optional | Connection to use, format: `[PROJECT_ID].LOCATION.CONNECTION_ID`. |
+| `ENDPOINT` | STRING | Optional | Vertex AI endpoint. If not specified, BigQuery dynamically chooses a model. |
+| `OUTPUT_MODE` | STRING | Optional | `'single'` or `'multi'`. Changes return type to ARRAY\<STRING\>. |
+| `EMBEDDINGS` | (Preview) | Optional | Embeddings for optimized mode, same as AI.IF. |
+| `OPTIMIZATION_MODE` | STRING | Optional (Preview) | `MINIMIZE_COST` or `MAXIMIZE_QUALITY`. Same as AI.IF. Note: `output_mode => 'multi'` is NOT supported in optimized mode. |
+| `MAX_ERROR_RATIO` | FLOAT64 | Optional | Range 0.0–1.0, default 1.0. Not supported when `optimization_mode` is `MINIMIZE_COST`. |
+
+**Categories from a table column (using DECLARE):**
+```sql
+DECLARE article_types ARRAY<STRING>
+  DEFAULT (SELECT ARRAY_AGG(category) FROM mydataset.categories);
+
+SELECT AI.CLASSIFY(body, categories => article_types) AS category
+FROM `bigquery-public-data.bbc_news.fulltext`;
+```
+
+**Outputs:**
+
+| Condition | Return Type | Description |
+|-----------|-------------|-------------|
+| No OUTPUT_MODE specified | STRING | The single category that best fits the input. |
+| OUTPUT_MODE = 'single' | ARRAY\<STRING\> | Array of length 1 containing the best-fit category. |
+| OUTPUT_MODE = 'multi' | ARRAY\<STRING\> | Array of 0 to N categories. Empty array if no category applies. |
+| Error | NULL | Returned if the call to Vertex AI is unsuccessful. |
+
+**Supported models:** Any GA or preview Gemini model. Uses various `gemini-2.5-*` models.
+
+**Best practices:**
+- Include an 'Other' category to handle input that doesn't closely match any category.
+- Use categories with descriptions for more nuanced classification.
+- Use `output_mode => 'multi'` when items may belong to multiple categories.
+
+**Limitations:** Preview status. Returns NULL on error. Categories must be string literals in the array (unless using a variable). No model parameter control. `output_mode => 'multi'` is not supported in optimized mode. Optimized mode requires ~3,000 rows minimum; only string columns supported for multi-column inputs.
+
+**Locations:** All regions supporting Gemini models, plus US and EU multi-regions.
+
+**Provisioned throughput:** Not supported. Uses dynamic shared quota (DSQ) only.
+
+**BigFrames API:** `bigframes.bigquery.ai.classify(input, categories)` — Returns a Series of STRING directly (not a struct). Takes `categories` as a list/tuple. Example: `bbq.ai.classify(df["text"], ["positive", "negative", "neutral"])`.
+
+---