@dataform/core 3.1.6-alpha.0 → 3.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md ADDED
@@ -0,0 +1,474 @@
1
+ # Dataform Core (`@dataform/core`)
2
+
3
+ `@dataform/core` is the open-source SQLX compiler, JavaScript workflow SDK, and Protocol Buffer schema package for [Dataform](https://cloud.google.com/dataform/docs/overview). It compiles SQLX (`.sqlx`), JavaScript (`.js`), action configuration (`actions.yaml`), BigQuery Property Graph (`graph.yaml`), and Jupyter Notebook (`.ipynb`) definitions into a dependency-ordered execution graph (`dataform.CompiledGraph`) for BigQuery.
4
+
5
+ For command-line execution (`dataform init`, `compile`, `test`, `run`, `format`), see the [Dataform CLI Reference](https://github.com/dataform-co/dataform/blob/main/cli/README.md) (`@dataform/cli`) and the [root repository overview](https://github.com/dataform-co/dataform/blob/main/readme.md).
6
+
7
+ ---
8
+
9
+ ## Contents
10
+
11
+ - [Installation and Version Pinning](#installation-and-version-pinning)
12
+ - [Project Structure](#project-structure)
13
+ - [Project Configuration (`workflow_settings.yaml`)](#project-configuration-workflow_settingsyaml)
14
+ - [SQLX Syntax Reference](#sqlx-syntax-reference)
15
+ - [SQLX Code Blocks](#sqlx-code-blocks)
16
+ - [SQLX Context Functions](#sqlx-context-functions)
17
+ - [Action Types Reference](#action-types-reference)
18
+ - [`table`](#table)
19
+ - [`incremental`](#incremental)
20
+ - [`view`](#view)
21
+ - [`operations`](#operations)
22
+ - [`assertion` and Built-in Assertions](#assertion-and-built-in-assertions)
23
+ - [`declaration`](#declaration)
24
+ - [`test` (Unit Tests)](#test-unit-tests)
25
+ - [`notebook`](#notebook)
26
+ - [`dataPreparation`](#datapreparation)
27
+ - [`propertyGraph` (`graph.yaml`)](#propertygraph-graphyaml)
28
+ - [`actions.yaml` Action Configs](#actionsyaml-action-configs)
29
+ - [JavaScript API (`Session` Globals)](#javascript-api-session-globals)
30
+ - [Package Exports and Bundled `.proto` Files](#package-exports-and-bundled-proto-files)
31
+ - [Documentation and Links](#documentation-and-links)
32
+
33
+ ---
34
+
35
+ ## Installation and Version Pinning
36
+
37
+ ### Via `workflow_settings.yaml` (Recommended)
38
+
39
+ In both **Dataform in Google Cloud** and **`@dataform/cli`**, specify `dataformCoreVersion` in your project's `workflow_settings.yaml`:
40
+
41
+ ```yaml
42
+ defaultProject: my-gcp-project
43
+ defaultLocation: US
44
+ defaultDataset: dataform
45
+ defaultAssertionDataset: dataform_assertions
46
+ dataformCoreVersion: <version>
47
+ ```
48
+
49
+ When `dataformCoreVersion` is set in `workflow_settings.yaml`, `@dataform/cli` automatically installs that version of `@dataform/core` in an isolated temporary directory at compile time (see [Stateless installs](https://github.com/dataform-co/dataform/blob/main/docs/stateless-install.md)).
50
+
51
+ ### Via `package.json` / `npm`
52
+
53
+ If you manage `@dataform/core` and external JavaScript packages with `package.json` (omitting `dataformCoreVersion` from `workflow_settings.yaml`), or if you consume `@dataform/core` programmatically:
54
+
55
+ ```bash
56
+ npm i @dataform/core
57
+ ```
58
+
59
+ ---
60
+
61
+ ## Project Structure
62
+
63
+ A standard Dataform project has the following layout:
64
+
65
+ ```text
66
+ my-dataform-project/
67
+ ├── workflow_settings.yaml # Project-level settings and defaults
68
+ ├── definitions/ # Action definitions (.sqlx, .js, actions.yaml, graph.yaml, .sql, .ipynb)
69
+ │ ├── sources.sqlx
70
+ │ ├── staging/
71
+ │ │ └── stg_orders.sqlx
72
+ │ └── marts/
73
+ │ └── fct_orders.sqlx
74
+ └── includes/ # Shared JavaScript modules and constants
75
+ └── constants.js
76
+ ```
77
+
78
+ - **`workflow_settings.yaml`**: Configures default project/dataset/location, custom variables (`vars`), target prefixes/suffixes, and runtime options.
79
+ - **`definitions/`**: Every `.sqlx` and `.js` file under `definitions/` (and any subdirectory) is evaluated to register actions in the compilation session. `definitions/**/actions.yaml` files register YAML-configured actions, and `definitions/**/graph.yaml` defines a BigQuery Property Graph.
80
+ - **`includes/`**: Top-level `includes/*.js` files are automatically attached to the global scope by their file basename (for example, `includes/constants.js` is available as `constants` inside any `.sqlx` or `.js` file in `definitions/`). Nested files under `includes/` can be imported via standard `require("includes/...")`.
81
+
82
+ > **Note on Project Execution & Security:** Dataform compiles SQLX and JavaScript project code using Node.js's native `node:vm` context. While module resolution is scoped to the project directory, `node:vm` is not a security sandbox against untrusted code. Only compile and run Dataform projects from trusted sources.
83
+
84
+ ---
85
+
86
+ ## Project Configuration (`workflow_settings.yaml`)
87
+
88
+ `workflow_settings.yaml` is validated against the [`dataform.WorkflowSettings`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-WorkflowSettings) schema.
89
+
90
+ | Field | Type | Default | Description |
91
+ |---|---|---|---|
92
+ | `dataformCoreVersion` | `string` | — | `@dataform/core` version to compile against (for example, `3.1.5`). |
93
+ | `defaultProject` | `string` | — | Default Google Cloud project ID (`database`) for compiled actions. |
94
+ | `defaultDataset` | `string` | `"dataform"` | Default BigQuery dataset (`schema`) for tables, views, and operations. |
95
+ | `defaultLocation` | `string` | — | Default BigQuery dataset location (for example, `US`, `EU`, `us-central1`). |
96
+ | `defaultAssertionDataset` | `string` | `"dataform_assertions"` | Default BigQuery dataset (`schema`) for assertion views. |
97
+ | `vars` | `map<string, string>` | `{}` | Custom string key-value variables accessible via `dataform.projectConfig.vars.<key>`. |
98
+ | `projectSuffix` | `string` | — | Suffix appended to target Google Cloud project IDs (`<project>_<projectSuffix>`). |
99
+ | `datasetSuffix` | `string` | — | Suffix appended to target dataset names (`<dataset>_<datasetSuffix>`). |
100
+ | `namePrefix` | `string` | — | Prefix prepended to target action names (`<namePrefix>_<name>`). |
101
+ | `builtinAssertionNamePrefix` | `string` | — | Prefix prepended to auto-generated built-in assertion names. |
102
+ | `disableAssertions` | `boolean` | `false` | Marks all manual and built-in assertions as `disabled: true` in the compiled graph. |
103
+ | `defaultReservation` | `string` | — | Default BigQuery reservation for query execution (`projects/<p>/locations/<l>/reservations/<r>` or `none`). |
104
+ | `defaultJobLabels` | `map<string, string>` | `{}` | Default labels applied to BigQuery jobs executed for actions in the project. |
105
+ | `preserveGovernanceControls` | `boolean` | `false` | Preserves existing BigQuery column governance controls (such as Data Policies) when recreating tables. |
106
+ | `lineage` | `object` | — | Knowledge Catalog Data Lineage settings (`lineage: { enabled: true }`). See [Data Lineage guide](https://github.com/dataform-co/dataform/blob/main/docs/lineage.md). |
107
+ | `defaultIcebergConfig` | `object` | — | Project-level defaults for BigQuery Apache Iceberg tables (`bucketName`, `tableFolderRoot`, `tableFolderSubpath`, `connection`). |
108
+ | `defaultLakehouseConfig` | `object` | — | Project-level defaults for 4-part Lakehouse targets (`catalog`, `namespace`). |
109
+ | `defaultNotebookRuntimeOptions` | `object` | — | Default runtime options for Notebook actions (`outputBucket`, `runtimeTemplateName`, `repositorySnapshotDestination`). |
110
+ | `includeTestsInCompiledGraph` | `boolean` | `false` | Includes unit test actions (`type: "test"`) in `CompiledGraph.tests`. |
111
+ | `extension` | `object` | — | External compilation extension package configuration (`name`, `compilationMode`). |
112
+
113
+ ---
114
+
115
+ ## SQLX Syntax Reference
116
+
117
+ A `.sqlx` file combines a `config {}` metadata block, optional JavaScript (`js {}`) and hook blocks, and a SQL body with `${...}` JavaScript template interpolation.
118
+
119
+ ### SQLX Code Blocks
120
+
121
+ | Block | Supported Action Types | Description |
122
+ |---|---|---|
123
+ | `config { ... }` | All `.sqlx` types | Declares the action `type` (defaults to `"operations"` if omitted), target overrides, dependencies, assertions, and BigQuery options. |
124
+ | `js { ... }` | All `.sqlx` types | Local JavaScript variables and helper functions scoped to the current `.sqlx` file. |
125
+ | `pre_operations { ... }` | `table`, `incremental`, `view` | SQL statements executed before the main table/view query runs (multiple statements can be terminated with `;` or split into separate compiled array entries with `---`). |
126
+ | `post_operations { ... }` | `table`, `incremental`, `view` | SQL statements executed after the main table/view query completes (multiple statements can be terminated with `;` or split into separate compiled array entries with `---`). |
127
+ | `incremental_where { ... }` | `incremental` | Optional SQL predicate that sets `Table.where` to filter rows during incremental updates (`SELECT * FROM (<query>) AS subquery WHERE <predicate>`). Not supported with `compilation_mode: "jit"`. |
128
+ | `input "<target>" { ... }` | `test` | Mock SQL query replacing a referenced upstream dependency (`ref("<target>")`) during a unit test. |
129
+ | `---` | `operations`, `pre_operations`, `post_operations` | Statement separator for multi-statement SQL blocks. |
130
+
131
+ ### SQLX Context Functions
132
+
133
+ Each action maps to an **action target** (`database.schema.name`) in BigQuery. By default, `name` is the action's filename (without extension), `schema` is `defaultDataset` (or `defaultAssertionDataset` for assertions), and `database` is `defaultProject` from `workflow_settings.yaml`.
134
+
135
+ Inside the SQL body and operation blocks of a `.sqlx` file (or via the `ctx` argument in JavaScript API callbacks), the following context methods are available:
136
+
137
+ | Function | Signature | Description |
138
+ |---|---|---|
139
+ | `ref` | `ref(name)` / `ref(schema, name)` / `ref(database, schema, name)` / `ref({ database?, schema?, name, includeDependentAssertions? })` | Resolves an action target to its backtick-quoted SQL identifier and registers it as an upstream dependency. |
140
+ | `resolve` | `resolve(name)` / `resolve(schema, name)` / `resolve(database, schema, name)` / `resolve(targetObj)` | Resolves an action target to its backtick-quoted SQL identifier **without** adding it as a dependency. |
141
+ | `self` | `self()` | Returns the fully qualified, backtick-quoted SQL identifier of the current action's target. |
142
+ | `name` | `name()` | Returns the finalized table/action name of the current action (including `namePrefix` if set). |
143
+ | `schema` | `schema()` | Returns the finalized dataset (`schema`) name of the current action (including `datasetSuffix` if set). |
144
+ | `database` | `database()` | Returns the finalized Google Cloud project (`database`) of the current action (including `projectSuffix` if set). |
145
+ | `when` | `when(condition, trueCase, falseCase = "")` | Returns `trueCase` when `condition` is truthy and `falseCase` otherwise (`table`, `incremental`, `view`, `operations`, `assertion`, `test`, `dataPreparation`). |
146
+ | `incremental` | `incremental()` | Returns `true` when compiling the incremental update query of an `incremental` table, or `false` when compiling its initial/full-refresh query (`table`, `incremental`, `view`). |
147
+
148
+ ---
149
+
150
+ ## Action Types Reference
151
+
152
+ Full field definitions for all action configurations are documented in the [Dataform Configs Reference](https://dataform-co.github.io/dataform/docs/reference/configs).
153
+
154
+ ### `table`
155
+
156
+ Creates or replaces a BigQuery table (`CREATE OR REPLACE TABLE`). See [`TableConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-TableConfig).
157
+
158
+ ```sql
159
+ -- definitions/marts/orders_by_day.sqlx
160
+ config {
161
+ type: "table",
162
+ schema: "marts",
163
+ description: "Daily aggregated order metrics",
164
+ columns: {
165
+ order_date: "UTC date of the order",
166
+ total_orders: "Count of completed orders"
167
+ },
168
+ bigquery: {
169
+ partitionBy: "order_date",
170
+ clusterBy: ["country_code"],
171
+ labels: {
172
+ domain: "commerce"
173
+ }
174
+ },
175
+ assertions: {
176
+ uniqueKey: ["order_date", "country_code"],
177
+ nonNull: ["order_date"]
178
+ },
179
+ tags: ["daily"]
180
+ }
181
+
182
+ SELECT
183
+ DATE(order_timestamp) AS order_date,
184
+ country_code,
185
+ COUNT(*) AS total_orders
186
+ FROM ${ref("stg_orders")}
187
+ GROUP BY 1, 2
188
+ ```
189
+
190
+ Key `table` config properties:
191
+ - **Target & metadata**: `name`, `schema` (or `dataset`), `database` (or `project`), `description`, `columns`, `tags`, `disabled`, `dependencies`, `dependOnDependencyAssertions`, `hermetic`, `metadata` (`overview`, `extraProperties`).
192
+ - **BigQuery options**: `partitionBy`, `partitionExpirationDays`, `requirePartitionFilter`, `clusterBy`, `labels`, `additionalOptions`, and `iceberg` (all accepted at top level or nested under `bigquery: {}`; for `view`, `bigquery: {}` accepts `partitionBy`, `clusterBy`, `labels`, and `additionalOptions`), plus top-level `preserveGovernanceControls`.
193
+ - **Execution & assertions**: `assertions` (`uniqueKey`, `uniqueKeys`, `nonNull`, `rowConditions`), `preOperations`, `postOperations`, `reservation`, `jobLabels`, `lakehouse`, `compilation_mode` (`"aot"` or `"jit"`).
194
+
195
+ ---
196
+
197
+ ### `incremental`
198
+
199
+ Builds a table from scratch on first run (or `--full-refresh`), and inserts or merges new rows on subsequent runs when the target table already exists. See [`IncrementalTableConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-IncrementalTableConfig).
200
+
201
+ ```sql
202
+ -- definitions/marts/events_incremental.sqlx
203
+ config {
204
+ type: "incremental",
205
+ uniqueKey: ["event_id"],
206
+ bigquery: {
207
+ partitionBy: "DATE(event_timestamp)",
208
+ updatePartitionFilter: "event_timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)"
209
+ },
210
+ protected: false
211
+ }
212
+
213
+ SELECT
214
+ event_id,
215
+ event_timestamp,
216
+ user_id
217
+ FROM ${ref("raw_events")}
218
+ ${when(incremental(), `WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM ${self()})`)}
219
+ ```
220
+
221
+ Incremental-specific config properties (in addition to `table` properties):
222
+ - `uniqueKey`: Column name(s) used to `MERGE` rows on incremental updates; if omitted, new rows are appended (`INSERT`).
223
+ - `incrementalStrategy`: `"merge"` (requires `uniqueKey`) or `"insert_overwrite"` (requires `partitionBy`) in SQLX and JavaScript (or `"INCREMENTAL_STRATEGY_MERGE"` / `"INCREMENTAL_STRATEGY_INSERT_OVERWRITE"` in `actions.yaml`).
224
+ - `updatePartitionFilter` / `incrementalPredicates`: Mutually exclusive SQL filter(s) restricting target partitions scanned during incremental updates (setting both is a compile error; prefer `incrementalPredicates`).
225
+ - `onSchemaChange`: `"IGNORE"` (default), `"FAIL"`, `"EXTEND"`, or `"SYNCHRONIZE"`.
226
+ - `protected` (default `true`): Prevents the table from being dropped and rebuilt from scratch on `--full-refresh`. Set `protected: false` to allow full refreshes.
227
+
228
+ ---
229
+
230
+ ### `view`
231
+
232
+ Creates a virtual view (`CREATE OR REPLACE VIEW`) or a BigQuery materialized view (`materialized: true`). See [`ViewConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-ViewConfig).
233
+
234
+ ```sql
235
+ -- definitions/staging/stg_orders.sqlx
236
+ config {
237
+ type: "view",
238
+ materialized: true,
239
+ bigquery: {
240
+ partitionBy: "DATE(order_timestamp)",
241
+ clusterBy: ["customer_id"]
242
+ }
243
+ }
244
+
245
+ SELECT
246
+ order_id,
247
+ customer_id,
248
+ order_timestamp
249
+ FROM ${ref("raw_orders")}
250
+ ```
251
+
252
+ ---
253
+
254
+ ### `operations`
255
+
256
+ Executes arbitrary SQL statements (DDL, DML, stored procedures, `GRANT`/`REVOKE`). Multiple statements are separated by `---`. See [`OperationConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-OperationConfig).
257
+
258
+ ```sql
259
+ -- definitions/ops/cleanup_temp_tables.sqlx
260
+ config {
261
+ type: "operations",
262
+ hasOutput: false,
263
+ dependencies: ["orders_by_day"]
264
+ }
265
+
266
+ DELETE FROM `my-gcp-project.staging.scratch_events`
267
+ WHERE created_at < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
268
+ ```
269
+
270
+ When `hasOutput: true` is set, the operation registers a referenceable target at `database.schema.name` that downstream actions can reference with `${ref("<name>")}` or `${self()}`.
271
+
272
+ ---
273
+
274
+ ### `assertion` and Built-in Assertions
275
+
276
+ An assertion is a data quality query that **fails if it returns one or more rows**. Assertions materialize as views in `defaultAssertionDataset` (`dataform_assertions` by default). See [`AssertionConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-AssertionConfig).
277
+
278
+ ```sql
279
+ -- definitions/assertions/assert_positive_order_amount.sqlx
280
+ config {
281
+ type: "assertion",
282
+ tags: ["critical"]
283
+ }
284
+
285
+ SELECT order_id, amount
286
+ FROM ${ref("stg_orders")}
287
+ WHERE amount <= 0
288
+ ```
289
+
290
+ You can also configure **built-in assertions** directly inside a `table`, `incremental`, or `view` `config` block:
291
+
292
+ ```sql
293
+ config {
294
+ type: "table",
295
+ assertions: {
296
+ uniqueKey: ["order_id"],
297
+ nonNull: ["order_id", "customer_id"],
298
+ rowConditions: [
299
+ "amount >= 0",
300
+ "order_timestamp <= CURRENT_TIMESTAMP()"
301
+ ]
302
+ }
303
+ }
304
+ ```
305
+
306
+ ---
307
+
308
+ ### `declaration`
309
+
310
+ Declares an external BigQuery table or view (not managed by Dataform) so it can be referenced via `${ref("<name>")}`, documented, and shown in the dependency graph. See [`DeclarationConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-DeclarationConfig).
311
+
312
+ ```sql
313
+ -- definitions/sources/raw_orders.sqlx
314
+ config {
315
+ type: "declaration",
316
+ database: "my-raw-data-project",
317
+ schema: "stripe",
318
+ name: "raw_orders",
319
+ description: "Raw Stripe orders landing table"
320
+ }
321
+ ```
322
+
323
+ ---
324
+
325
+ ### `test` (Unit Tests)
326
+
327
+ Validates the SQL logic of a target `dataset` (`table`, `view`, or `incremental` action) by replacing its upstream `ref()` inputs with static mock queries (`input "<ref_name>" { ... }`) and comparing the output against an expected query. Executed via `dataform test` or `dataform run --run-tests`.
328
+
329
+ ```sql
330
+ -- definitions/tests/orders_by_day_test.sqlx
331
+ config {
332
+ type: "test",
333
+ dataset: "orders_by_day"
334
+ }
335
+
336
+ input "stg_orders" {
337
+ SELECT TIMESTAMP("2026-01-01 10:00:00 UTC") AS order_timestamp, "PL" AS country_code
338
+ UNION ALL
339
+ SELECT TIMESTAMP("2026-01-01 12:00:00 UTC") AS order_timestamp, "PL" AS country_code
340
+ }
341
+
342
+ SELECT DATE("2026-01-01") AS order_date, "PL" AS country_code, 2 AS total_orders
343
+ ```
344
+
345
+ ---
346
+
347
+ ### `notebook`
348
+
349
+ Configures a Jupyter Notebook (`.ipynb`) action for execution in Google Cloud Dataform (via Vertex AI Colab Enterprise). See [`NotebookConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-NotebookConfig).
350
+
351
+ ```yaml
352
+ # definitions/actions.yaml
353
+ actions:
354
+ - notebook:
355
+ name: train_churn_model
356
+ filename: notebooks/train_churn_model.ipynb
357
+ dependencyTargets:
358
+ - name: orders_by_day
359
+ ```
360
+
361
+ ---
362
+
363
+ ### `dataPreparation`
364
+
365
+ Configures a BigQuery Data Preparation action from a `.dp.yaml` / `.yaml` definition (via `actions.yaml`) or a `.dp.sqlx` / `.sqlx` file (`type: "dataPreparation"`). See [`DataPreparationConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-DataPreparationConfig).
366
+
367
+ ---
368
+
369
+ ### `propertyGraph` (`graph.yaml`)
370
+
371
+ Defines a BigQuery Property Graph (`CREATE OR REPLACE PROPERTY GRAPH`) from a `definitions/**/graph.yaml` file (at most one `graph.yaml` per project) following the Google Semantic Ontology schema (`version: 0.2.0.dev0/google`). See the [BigQuery Property Graphs guide](https://github.com/dataform-co/dataform/blob/main/docs/property_graphs.md) for full syntax and examples.
372
+
373
+ ---
374
+
375
+ ### `actions.yaml` Action Configs
376
+
377
+ Plain `.sql` and `.ipynb` files can be registered as Dataform actions via `definitions/**/actions.yaml` files validated against [`dataform.ActionConfigs`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfigs):
378
+
379
+ ```yaml
380
+ # definitions/actions.yaml
381
+ actions:
382
+ - declaration:
383
+ name: raw_users
384
+ project: my-raw-project
385
+ dataset: raw
386
+ - view:
387
+ name: stg_users
388
+ filename: staging/stg_users.sql
389
+ dependencyTargets:
390
+ - name: raw_users
391
+ - table:
392
+ name: dim_users
393
+ filename: marts/dim_users.sql
394
+ dependencyTargets:
395
+ - name: stg_users
396
+ partitionBy: created_date
397
+ ```
398
+
399
+ ---
400
+
401
+ ## JavaScript API (`Session` Globals)
402
+
403
+ Inside `definitions/**/*.js` files (and `js {}` blocks in `.sqlx` files), Dataform attaches the active [`Session`](https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md) instance to `global.dataform` and binds its action-builder methods as global functions:
404
+
405
+ | Global Function / Object | Signature | Description |
406
+ |---|---|---|
407
+ | `publish` | `publish(name, queryOrConfig?)` | Creates a `table`, `view` (default if `type` is omitted), or `incremental` action. Returns a `Table`, `View`, or `IncrementalTable` builder (`.query(ctx => ...)`). |
408
+ | `operate` | `operate(name, queriesOrConfig?)` | Creates an `operations` action. Returns an `Operation` builder (`.queries(ctx => ...)`). |
409
+ | `assert` | `assert(name, queryOrConfig?)` | Creates an `assertion` action. Returns an `Assertion` builder (`.query(ctx => ...)`). |
410
+ | `declare` | `declare(config)` | Declares an external data source (`{ database?, schema?, name, description?, columns? }`). |
411
+ | `test` | `test(name)` | Creates a unit test action (`.dataset(target).input(refName, sql).expect(sql)`). |
412
+ | `notebook` | `notebook(config)` | Creates a `Notebook` action (`{ name?, filename, location?, project?, dependencyTargets?, tags?, disabled?, description? }`). |
413
+ | `getContents` | `getContents(relativeMdPath)` | Reads a `.md` file inside the project directory (resolved relative to the caller file) and returns its contents as a string for `description` fields. |
414
+ | `jitData` | `jitData(key, value)` | Registers structured data in `CompiledGraph.jitData` for Just-in-Time (JiT) compilation workflows. |
415
+ | `dataform.projectConfig` | `dataform.ProjectConfig` | Read-only access to the resolved project configuration, including custom variables (`dataform.projectConfig.vars`). |
416
+
417
+ Example (`definitions/generate_staging_views.js`):
418
+
419
+ ```js
420
+ const sourceTables = ["orders", "customers", "products"];
421
+
422
+ sourceTables.forEach((tableName) => {
423
+ declare({
424
+ schema: "raw",
425
+ name: `raw_${tableName}`,
426
+ });
427
+
428
+ publish(`stg_${tableName}`, {
429
+ type: "view",
430
+ schema: "staging",
431
+ description: getContents(`./docs/stg_${tableName}.md`),
432
+ }).query((ctx) => `SELECT * FROM ${ctx.ref("raw", `raw_${tableName}`)}`);
433
+ });
434
+ ```
435
+
436
+ ---
437
+
438
+ ## Package Exports and Bundled `.proto` Files
439
+
440
+ ### TypeScript / JavaScript Exports
441
+
442
+ When imported as an npm library (`import * as dataformCore from "@dataform/core"`), `@dataform/core` exports the following symbols (with TypeScript typings in `bundle.d.ts`):
443
+
444
+ - **`version`** (`string`): The installed `@dataform/core` semantic version string (for example, `"3.1.5"`).
445
+ - **`session`** (`Session`): The singleton compilation `Session` instance.
446
+ - **`compiler`** (`(code: string, path: string) => string`): Transpiles `.sqlx`, `.yaml`/`.yml`, `.sql`, `.ipynb`, and `.md` source strings into executable JavaScript modules.
447
+ - **`main`** (`(coreExecutionRequest: Uint8Array | string) => Uint8Array | string`): Entry point invoked by the compilation VM sandbox; accepts an encoded `dataform.CoreExecutionRequest` proto and returns an encoded `dataform.CoreExecutionResponse` proto containing `CompiledGraph`.
448
+ - **`jitCompiler`** (`IJitCompiler`): Just-in-Time (JiT) action compiler entry point.
449
+ - **`supportedFeatures`** (`dataform.SupportedFeatures[]`): Feature capability flags supported by this `@dataform/core` version (such as `ARRAY_BUFFER_IPC`).
450
+ - **`IDataformExtension`** / **`IJitCompiler`**: TypeScript interfaces for extensions and JiT compilation.
451
+
452
+ ### Bundled Protocol Buffer Schemas
453
+
454
+ The published `@dataform/core` package includes the canonical `.proto` definitions at the package root:
455
+
456
+ - `core.proto` — `CompiledGraph`, `Table`, `Operation`, `Assertion`, `Declaration`, `Test`, `Notebook`, `DataPreparation`, `PropertyGraph`, `Target`, `ProjectConfig`, `CoreExecutionRequest`, `CoreExecutionResponse`.
457
+ - `configs.proto` — `WorkflowSettings`, `ActionConfigs`, `ActionConfig` (`TableConfig`, `ViewConfig`, `IncrementalTableConfig`, `AssertionConfig`, `OperationConfig`, `DeclarationConfig`, `NotebookConfig`, `DataPreparationConfig`).
458
+ - `property_graph_config.proto` — `PropertyGraphConfig` (`SemanticModel`, `Entity`, `Relationship`, `Deployment`).
459
+ - `db_adapter.proto` & `jit.proto` — Database adapter and Just-in-Time (JiT) compilation RPC messages.
460
+ - `extension.proto` — `Extension` configuration schema.
461
+
462
+ ---
463
+
464
+ ## Documentation and Links
465
+
466
+ - [Dataform Documentation Home](https://cloud.google.com/dataform/docs/overview)
467
+ - [Dataform Core Reference (Google Cloud Docs)](https://cloud.google.com/dataform/docs/reference/dataform-core-reference)
468
+ - [Dataform Configs Reference (`configs.proto`)](https://dataform-co.github.io/dataform/docs/reference/configs)
469
+ - [Dataform JavaScript `Session` Reference](https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md)
470
+ - [Dataform CLI Reference (`@dataform/cli`)](https://github.com/dataform-co/dataform/blob/main/cli/README.md)
471
+ - [BigQuery Property Graphs Guide (`graph.yaml`)](https://github.com/dataform-co/dataform/blob/main/docs/property_graphs.md)
472
+ - [Knowledge Catalog Data Lineage Guide](https://github.com/dataform-co/dataform/blob/main/docs/lineage.md)
473
+ - [Stateless Installs Guide (`.dataformignore`)](https://github.com/dataform-co/dataform/blob/main/docs/stateless-install.md)
474
+ - [GitHub Repository & Issue Tracker](https://github.com/dataform-co/dataform)