@dataform/core 3.1.6-alpha.0 → 3.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +474 -0
- package/bundle.d.ts +159 -1
- package/bundle.js +1 -1
- package/core.proto +8 -0
- package/package.json +3 -30
- package/property_graph_config.proto +76 -17
package/README.md
ADDED
|
@@ -0,0 +1,474 @@
|
|
|
1
|
+
# Dataform Core (`@dataform/core`)
|
|
2
|
+
|
|
3
|
+
`@dataform/core` is the open-source SQLX compiler, JavaScript workflow SDK, and Protocol Buffer schema package for [Dataform](https://cloud.google.com/dataform/docs/overview). It compiles SQLX (`.sqlx`), JavaScript (`.js`), action configuration (`actions.yaml`), BigQuery Property Graph (`graph.yaml`), and Jupyter Notebook (`.ipynb`) definitions into a dependency-ordered execution graph (`dataform.CompiledGraph`) for BigQuery.
|
|
4
|
+
|
|
5
|
+
For command-line execution (`dataform init`, `compile`, `test`, `run`, `format`), see the [Dataform CLI Reference](https://github.com/dataform-co/dataform/blob/main/cli/README.md) (`@dataform/cli`) and the [root repository overview](https://github.com/dataform-co/dataform/blob/main/readme.md).
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Contents
|
|
10
|
+
|
|
11
|
+
- [Installation and Version Pinning](#installation-and-version-pinning)
|
|
12
|
+
- [Project Structure](#project-structure)
|
|
13
|
+
- [Project Configuration (`workflow_settings.yaml`)](#project-configuration-workflow_settingsyaml)
|
|
14
|
+
- [SQLX Syntax Reference](#sqlx-syntax-reference)
|
|
15
|
+
- [SQLX Code Blocks](#sqlx-code-blocks)
|
|
16
|
+
- [SQLX Context Functions](#sqlx-context-functions)
|
|
17
|
+
- [Action Types Reference](#action-types-reference)
|
|
18
|
+
- [`table`](#table)
|
|
19
|
+
- [`incremental`](#incremental)
|
|
20
|
+
- [`view`](#view)
|
|
21
|
+
- [`operations`](#operations)
|
|
22
|
+
- [`assertion` and Built-in Assertions](#assertion-and-built-in-assertions)
|
|
23
|
+
- [`declaration`](#declaration)
|
|
24
|
+
- [`test` (Unit Tests)](#test-unit-tests)
|
|
25
|
+
- [`notebook`](#notebook)
|
|
26
|
+
- [`dataPreparation`](#datapreparation)
|
|
27
|
+
- [`propertyGraph` (`graph.yaml`)](#propertygraph-graphyaml)
|
|
28
|
+
- [`actions.yaml` Action Configs](#actionsyaml-action-configs)
|
|
29
|
+
- [JavaScript API (`Session` Globals)](#javascript-api-session-globals)
|
|
30
|
+
- [Package Exports and Bundled `.proto` Files](#package-exports-and-bundled-proto-files)
|
|
31
|
+
- [Documentation and Links](#documentation-and-links)
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## Installation and Version Pinning
|
|
36
|
+
|
|
37
|
+
### Via `workflow_settings.yaml` (Recommended)
|
|
38
|
+
|
|
39
|
+
In both **Dataform in Google Cloud** and **`@dataform/cli`**, specify `dataformCoreVersion` in your project's `workflow_settings.yaml`:
|
|
40
|
+
|
|
41
|
+
```yaml
|
|
42
|
+
defaultProject: my-gcp-project
|
|
43
|
+
defaultLocation: US
|
|
44
|
+
defaultDataset: dataform
|
|
45
|
+
defaultAssertionDataset: dataform_assertions
|
|
46
|
+
dataformCoreVersion: <version>
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
When `dataformCoreVersion` is set in `workflow_settings.yaml`, `@dataform/cli` automatically installs that version of `@dataform/core` in an isolated temporary directory at compile time (see [Stateless installs](https://github.com/dataform-co/dataform/blob/main/docs/stateless-install.md)).
|
|
50
|
+
|
|
51
|
+
### Via `package.json` / `npm`
|
|
52
|
+
|
|
53
|
+
If you manage `@dataform/core` and external JavaScript packages with `package.json` (omitting `dataformCoreVersion` from `workflow_settings.yaml`), or if you consume `@dataform/core` programmatically:
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
npm i @dataform/core
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Project Structure
|
|
62
|
+
|
|
63
|
+
A standard Dataform project has the following layout:
|
|
64
|
+
|
|
65
|
+
```text
|
|
66
|
+
my-dataform-project/
|
|
67
|
+
├── workflow_settings.yaml # Project-level settings and defaults
|
|
68
|
+
├── definitions/ # Action definitions (.sqlx, .js, actions.yaml, graph.yaml, .sql, .ipynb)
|
|
69
|
+
│ ├── sources.sqlx
|
|
70
|
+
│ ├── staging/
|
|
71
|
+
│ │ └── stg_orders.sqlx
|
|
72
|
+
│ └── marts/
|
|
73
|
+
│ └── fct_orders.sqlx
|
|
74
|
+
└── includes/ # Shared JavaScript modules and constants
|
|
75
|
+
└── constants.js
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
- **`workflow_settings.yaml`**: Configures default project/dataset/location, custom variables (`vars`), target prefixes/suffixes, and runtime options.
|
|
79
|
+
- **`definitions/`**: Every `.sqlx` and `.js` file under `definitions/` (and any subdirectory) is evaluated to register actions in the compilation session. `definitions/**/actions.yaml` files register YAML-configured actions, and `definitions/**/graph.yaml` defines a BigQuery Property Graph.
|
|
80
|
+
- **`includes/`**: Top-level `includes/*.js` files are automatically attached to the global scope by their file basename (for example, `includes/constants.js` is available as `constants` inside any `.sqlx` or `.js` file in `definitions/`). Nested files under `includes/` can be imported via standard `require("includes/...")`.
|
|
81
|
+
|
|
82
|
+
> **Note on Project Execution & Security:** Dataform compiles SQLX and JavaScript project code using Node.js's native `node:vm` context. While module resolution is scoped to the project directory, `node:vm` is not a security sandbox against untrusted code. Only compile and run Dataform projects from trusted sources.
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## Project Configuration (`workflow_settings.yaml`)
|
|
87
|
+
|
|
88
|
+
`workflow_settings.yaml` is validated against the [`dataform.WorkflowSettings`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-WorkflowSettings) schema.
|
|
89
|
+
|
|
90
|
+
| Field | Type | Default | Description |
|
|
91
|
+
|---|---|---|---|
|
|
92
|
+
| `dataformCoreVersion` | `string` | — | `@dataform/core` version to compile against (for example, `3.1.5`). |
|
|
93
|
+
| `defaultProject` | `string` | — | Default Google Cloud project ID (`database`) for compiled actions. |
|
|
94
|
+
| `defaultDataset` | `string` | `"dataform"` | Default BigQuery dataset (`schema`) for tables, views, and operations. |
|
|
95
|
+
| `defaultLocation` | `string` | — | Default BigQuery dataset location (for example, `US`, `EU`, `us-central1`). |
|
|
96
|
+
| `defaultAssertionDataset` | `string` | `"dataform_assertions"` | Default BigQuery dataset (`schema`) for assertion views. |
|
|
97
|
+
| `vars` | `map<string, string>` | `{}` | Custom string key-value variables accessible via `dataform.projectConfig.vars.<key>`. |
|
|
98
|
+
| `projectSuffix` | `string` | — | Suffix appended to target Google Cloud project IDs (`<project>_<projectSuffix>`). |
|
|
99
|
+
| `datasetSuffix` | `string` | — | Suffix appended to target dataset names (`<dataset>_<datasetSuffix>`). |
|
|
100
|
+
| `namePrefix` | `string` | — | Prefix prepended to target action names (`<namePrefix>_<name>`). |
|
|
101
|
+
| `builtinAssertionNamePrefix` | `string` | — | Prefix prepended to auto-generated built-in assertion names. |
|
|
102
|
+
| `disableAssertions` | `boolean` | `false` | Marks all manual and built-in assertions as `disabled: true` in the compiled graph. |
|
|
103
|
+
| `defaultReservation` | `string` | — | Default BigQuery reservation for query execution (`projects/<p>/locations/<l>/reservations/<r>` or `none`). |
|
|
104
|
+
| `defaultJobLabels` | `map<string, string>` | `{}` | Default labels applied to BigQuery jobs executed for actions in the project. |
|
|
105
|
+
| `preserveGovernanceControls` | `boolean` | `false` | Preserves existing BigQuery column governance controls (such as Data Policies) when recreating tables. |
|
|
106
|
+
| `lineage` | `object` | — | Knowledge Catalog Data Lineage settings (`lineage: { enabled: true }`). See [Data Lineage guide](https://github.com/dataform-co/dataform/blob/main/docs/lineage.md). |
|
|
107
|
+
| `defaultIcebergConfig` | `object` | — | Project-level defaults for BigQuery Apache Iceberg tables (`bucketName`, `tableFolderRoot`, `tableFolderSubpath`, `connection`). |
|
|
108
|
+
| `defaultLakehouseConfig` | `object` | — | Project-level defaults for 4-part Lakehouse targets (`catalog`, `namespace`). |
|
|
109
|
+
| `defaultNotebookRuntimeOptions` | `object` | — | Default runtime options for Notebook actions (`outputBucket`, `runtimeTemplateName`, `repositorySnapshotDestination`). |
|
|
110
|
+
| `includeTestsInCompiledGraph` | `boolean` | `false` | Includes unit test actions (`type: "test"`) in `CompiledGraph.tests`. |
|
|
111
|
+
| `extension` | `object` | — | External compilation extension package configuration (`name`, `compilationMode`). |
|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
|
|
115
|
+
## SQLX Syntax Reference
|
|
116
|
+
|
|
117
|
+
A `.sqlx` file combines a `config {}` metadata block, optional JavaScript (`js {}`) and hook blocks, and a SQL body with `${...}` JavaScript template interpolation.
|
|
118
|
+
|
|
119
|
+
### SQLX Code Blocks
|
|
120
|
+
|
|
121
|
+
| Block | Supported Action Types | Description |
|
|
122
|
+
|---|---|---|
|
|
123
|
+
| `config { ... }` | All `.sqlx` types | Declares the action `type` (defaults to `"operations"` if omitted), target overrides, dependencies, assertions, and BigQuery options. |
|
|
124
|
+
| `js { ... }` | All `.sqlx` types | Local JavaScript variables and helper functions scoped to the current `.sqlx` file. |
|
|
125
|
+
| `pre_operations { ... }` | `table`, `incremental`, `view` | SQL statements executed before the main table/view query runs (multiple statements can be terminated with `;` or split into separate compiled array entries with `---`). |
|
|
126
|
+
| `post_operations { ... }` | `table`, `incremental`, `view` | SQL statements executed after the main table/view query completes (multiple statements can be terminated with `;` or split into separate compiled array entries with `---`). |
|
|
127
|
+
| `incremental_where { ... }` | `incremental` | Optional SQL predicate that sets `Table.where` to filter rows during incremental updates (`SELECT * FROM (<query>) AS subquery WHERE <predicate>`). Not supported with `compilation_mode: "jit"`. |
|
|
128
|
+
| `input "<target>" { ... }` | `test` | Mock SQL query replacing a referenced upstream dependency (`ref("<target>")`) during a unit test. |
|
|
129
|
+
| `---` | `operations`, `pre_operations`, `post_operations` | Statement separator for multi-statement SQL blocks. |
|
|
130
|
+
|
|
131
|
+
### SQLX Context Functions
|
|
132
|
+
|
|
133
|
+
Each action maps to an **action target** (`database.schema.name`) in BigQuery. By default, `name` is the action's filename (without extension), `schema` is `defaultDataset` (or `defaultAssertionDataset` for assertions), and `database` is `defaultProject` from `workflow_settings.yaml`.
|
|
134
|
+
|
|
135
|
+
Inside the SQL body and operation blocks of a `.sqlx` file (or via the `ctx` argument in JavaScript API callbacks), the following context methods are available:
|
|
136
|
+
|
|
137
|
+
| Function | Signature | Description |
|
|
138
|
+
|---|---|---|
|
|
139
|
+
| `ref` | `ref(name)` / `ref(schema, name)` / `ref(database, schema, name)` / `ref({ database?, schema?, name, includeDependentAssertions? })` | Resolves an action target to its backtick-quoted SQL identifier and registers it as an upstream dependency. |
|
|
140
|
+
| `resolve` | `resolve(name)` / `resolve(schema, name)` / `resolve(database, schema, name)` / `resolve(targetObj)` | Resolves an action target to its backtick-quoted SQL identifier **without** adding it as a dependency. |
|
|
141
|
+
| `self` | `self()` | Returns the fully qualified, backtick-quoted SQL identifier of the current action's target. |
|
|
142
|
+
| `name` | `name()` | Returns the finalized table/action name of the current action (including `namePrefix` if set). |
|
|
143
|
+
| `schema` | `schema()` | Returns the finalized dataset (`schema`) name of the current action (including `datasetSuffix` if set). |
|
|
144
|
+
| `database` | `database()` | Returns the finalized Google Cloud project (`database`) of the current action (including `projectSuffix` if set). |
|
|
145
|
+
| `when` | `when(condition, trueCase, falseCase = "")` | Returns `trueCase` when `condition` is truthy and `falseCase` otherwise (`table`, `incremental`, `view`, `operations`, `assertion`, `test`, `dataPreparation`). |
|
|
146
|
+
| `incremental` | `incremental()` | Returns `true` when compiling the incremental update query of an `incremental` table, or `false` when compiling its initial/full-refresh query (`table`, `incremental`, `view`). |
|
|
147
|
+
|
|
148
|
+
---
|
|
149
|
+
|
|
150
|
+
## Action Types Reference
|
|
151
|
+
|
|
152
|
+
Full field definitions for all action configurations are documented in the [Dataform Configs Reference](https://dataform-co.github.io/dataform/docs/reference/configs).
|
|
153
|
+
|
|
154
|
+
### `table`
|
|
155
|
+
|
|
156
|
+
Creates or replaces a BigQuery table (`CREATE OR REPLACE TABLE`). See [`TableConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-TableConfig).
|
|
157
|
+
|
|
158
|
+
```sql
|
|
159
|
+
-- definitions/marts/orders_by_day.sqlx
|
|
160
|
+
config {
|
|
161
|
+
type: "table",
|
|
162
|
+
schema: "marts",
|
|
163
|
+
description: "Daily aggregated order metrics",
|
|
164
|
+
columns: {
|
|
165
|
+
order_date: "UTC date of the order",
|
|
166
|
+
total_orders: "Count of completed orders"
|
|
167
|
+
},
|
|
168
|
+
bigquery: {
|
|
169
|
+
partitionBy: "order_date",
|
|
170
|
+
clusterBy: ["country_code"],
|
|
171
|
+
labels: {
|
|
172
|
+
domain: "commerce"
|
|
173
|
+
}
|
|
174
|
+
},
|
|
175
|
+
assertions: {
|
|
176
|
+
uniqueKey: ["order_date", "country_code"],
|
|
177
|
+
nonNull: ["order_date"]
|
|
178
|
+
},
|
|
179
|
+
tags: ["daily"]
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
SELECT
|
|
183
|
+
DATE(order_timestamp) AS order_date,
|
|
184
|
+
country_code,
|
|
185
|
+
COUNT(*) AS total_orders
|
|
186
|
+
FROM ${ref("stg_orders")}
|
|
187
|
+
GROUP BY 1, 2
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
Key `table` config properties:
|
|
191
|
+
- **Target & metadata**: `name`, `schema` (or `dataset`), `database` (or `project`), `description`, `columns`, `tags`, `disabled`, `dependencies`, `dependOnDependencyAssertions`, `hermetic`, `metadata` (`overview`, `extraProperties`).
|
|
192
|
+
- **BigQuery options**: `partitionBy`, `partitionExpirationDays`, `requirePartitionFilter`, `clusterBy`, `labels`, `additionalOptions`, and `iceberg` (all accepted at top level or nested under `bigquery: {}`; for `view`, `bigquery: {}` accepts `partitionBy`, `clusterBy`, `labels`, and `additionalOptions`), plus top-level `preserveGovernanceControls`.
|
|
193
|
+
- **Execution & assertions**: `assertions` (`uniqueKey`, `uniqueKeys`, `nonNull`, `rowConditions`), `preOperations`, `postOperations`, `reservation`, `jobLabels`, `lakehouse`, `compilation_mode` (`"aot"` or `"jit"`).
|
|
194
|
+
|
|
195
|
+
---
|
|
196
|
+
|
|
197
|
+
### `incremental`
|
|
198
|
+
|
|
199
|
+
Builds a table from scratch on first run (or `--full-refresh`), and inserts or merges new rows on subsequent runs when the target table already exists. See [`IncrementalTableConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-IncrementalTableConfig).
|
|
200
|
+
|
|
201
|
+
```sql
|
|
202
|
+
-- definitions/marts/events_incremental.sqlx
|
|
203
|
+
config {
|
|
204
|
+
type: "incremental",
|
|
205
|
+
uniqueKey: ["event_id"],
|
|
206
|
+
bigquery: {
|
|
207
|
+
partitionBy: "DATE(event_timestamp)",
|
|
208
|
+
updatePartitionFilter: "event_timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)"
|
|
209
|
+
},
|
|
210
|
+
protected: false
|
|
211
|
+
}
|
|
212
|
+
|
|
213
|
+
SELECT
|
|
214
|
+
event_id,
|
|
215
|
+
event_timestamp,
|
|
216
|
+
user_id
|
|
217
|
+
FROM ${ref("raw_events")}
|
|
218
|
+
${when(incremental(), `WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM ${self()})`)}
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
Incremental-specific config properties (in addition to `table` properties):
|
|
222
|
+
- `uniqueKey`: Column name(s) used to `MERGE` rows on incremental updates; if omitted, new rows are appended (`INSERT`).
|
|
223
|
+
- `incrementalStrategy`: `"merge"` (requires `uniqueKey`) or `"insert_overwrite"` (requires `partitionBy`) in SQLX and JavaScript (or `"INCREMENTAL_STRATEGY_MERGE"` / `"INCREMENTAL_STRATEGY_INSERT_OVERWRITE"` in `actions.yaml`).
|
|
224
|
+
- `updatePartitionFilter` / `incrementalPredicates`: Mutually exclusive SQL filter(s) restricting target partitions scanned during incremental updates (setting both is a compile error; prefer `incrementalPredicates`).
|
|
225
|
+
- `onSchemaChange`: `"IGNORE"` (default), `"FAIL"`, `"EXTEND"`, or `"SYNCHRONIZE"`.
|
|
226
|
+
- `protected` (default `true`): Prevents the table from being dropped and rebuilt from scratch on `--full-refresh`. Set `protected: false` to allow full refreshes.
|
|
227
|
+
|
|
228
|
+
---
|
|
229
|
+
|
|
230
|
+
### `view`
|
|
231
|
+
|
|
232
|
+
Creates a virtual view (`CREATE OR REPLACE VIEW`) or a BigQuery materialized view (`materialized: true`). See [`ViewConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-ViewConfig).
|
|
233
|
+
|
|
234
|
+
```sql
|
|
235
|
+
-- definitions/staging/stg_orders.sqlx
|
|
236
|
+
config {
|
|
237
|
+
type: "view",
|
|
238
|
+
materialized: true,
|
|
239
|
+
bigquery: {
|
|
240
|
+
partitionBy: "DATE(order_timestamp)",
|
|
241
|
+
clusterBy: ["customer_id"]
|
|
242
|
+
}
|
|
243
|
+
}
|
|
244
|
+
|
|
245
|
+
SELECT
|
|
246
|
+
order_id,
|
|
247
|
+
customer_id,
|
|
248
|
+
order_timestamp
|
|
249
|
+
FROM ${ref("raw_orders")}
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
---
|
|
253
|
+
|
|
254
|
+
### `operations`
|
|
255
|
+
|
|
256
|
+
Executes arbitrary SQL statements (DDL, DML, stored procedures, `GRANT`/`REVOKE`). Multiple statements are separated by `---`. See [`OperationConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-OperationConfig).
|
|
257
|
+
|
|
258
|
+
```sql
|
|
259
|
+
-- definitions/ops/cleanup_temp_tables.sqlx
|
|
260
|
+
config {
|
|
261
|
+
type: "operations",
|
|
262
|
+
hasOutput: false,
|
|
263
|
+
dependencies: ["orders_by_day"]
|
|
264
|
+
}
|
|
265
|
+
|
|
266
|
+
DELETE FROM `my-gcp-project.staging.scratch_events`
|
|
267
|
+
WHERE created_at < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
When `hasOutput: true` is set, the operation registers a referenceable target at `database.schema.name` that downstream actions can reference with `${ref("<name>")}` or `${self()}`.
|
|
271
|
+
|
|
272
|
+
---
|
|
273
|
+
|
|
274
|
+
### `assertion` and Built-in Assertions
|
|
275
|
+
|
|
276
|
+
An assertion is a data quality query that **fails if it returns one or more rows**. Assertions materialize as views in `defaultAssertionDataset` (`dataform_assertions` by default). See [`AssertionConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-AssertionConfig).
|
|
277
|
+
|
|
278
|
+
```sql
|
|
279
|
+
-- definitions/assertions/assert_positive_order_amount.sqlx
|
|
280
|
+
config {
|
|
281
|
+
type: "assertion",
|
|
282
|
+
tags: ["critical"]
|
|
283
|
+
}
|
|
284
|
+
|
|
285
|
+
SELECT order_id, amount
|
|
286
|
+
FROM ${ref("stg_orders")}
|
|
287
|
+
WHERE amount <= 0
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
You can also configure **built-in assertions** directly inside a `table`, `incremental`, or `view` `config` block:
|
|
291
|
+
|
|
292
|
+
```sql
|
|
293
|
+
config {
|
|
294
|
+
type: "table",
|
|
295
|
+
assertions: {
|
|
296
|
+
uniqueKey: ["order_id"],
|
|
297
|
+
nonNull: ["order_id", "customer_id"],
|
|
298
|
+
rowConditions: [
|
|
299
|
+
"amount >= 0",
|
|
300
|
+
"order_timestamp <= CURRENT_TIMESTAMP()"
|
|
301
|
+
]
|
|
302
|
+
}
|
|
303
|
+
}
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
---
|
|
307
|
+
|
|
308
|
+
### `declaration`
|
|
309
|
+
|
|
310
|
+
Declares an external BigQuery table or view (not managed by Dataform) so it can be referenced via `${ref("<name>")}`, documented, and shown in the dependency graph. See [`DeclarationConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-DeclarationConfig).
|
|
311
|
+
|
|
312
|
+
```sql
|
|
313
|
+
-- definitions/sources/raw_orders.sqlx
|
|
314
|
+
config {
|
|
315
|
+
type: "declaration",
|
|
316
|
+
database: "my-raw-data-project",
|
|
317
|
+
schema: "stripe",
|
|
318
|
+
name: "raw_orders",
|
|
319
|
+
description: "Raw Stripe orders landing table"
|
|
320
|
+
}
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
---
|
|
324
|
+
|
|
325
|
+
### `test` (Unit Tests)
|
|
326
|
+
|
|
327
|
+
Validates the SQL logic of a target `dataset` (`table`, `view`, or `incremental` action) by replacing its upstream `ref()` inputs with static mock queries (`input "<ref_name>" { ... }`) and comparing the output against an expected query. Executed via `dataform test` or `dataform run --run-tests`.
|
|
328
|
+
|
|
329
|
+
```sql
|
|
330
|
+
-- definitions/tests/orders_by_day_test.sqlx
|
|
331
|
+
config {
|
|
332
|
+
type: "test",
|
|
333
|
+
dataset: "orders_by_day"
|
|
334
|
+
}
|
|
335
|
+
|
|
336
|
+
input "stg_orders" {
|
|
337
|
+
SELECT TIMESTAMP("2026-01-01 10:00:00 UTC") AS order_timestamp, "PL" AS country_code
|
|
338
|
+
UNION ALL
|
|
339
|
+
SELECT TIMESTAMP("2026-01-01 12:00:00 UTC") AS order_timestamp, "PL" AS country_code
|
|
340
|
+
}
|
|
341
|
+
|
|
342
|
+
SELECT DATE("2026-01-01") AS order_date, "PL" AS country_code, 2 AS total_orders
|
|
343
|
+
```
|
|
344
|
+
|
|
345
|
+
---
|
|
346
|
+
|
|
347
|
+
### `notebook`
|
|
348
|
+
|
|
349
|
+
Configures a Jupyter Notebook (`.ipynb`) action for execution in Google Cloud Dataform (via Vertex AI Colab Enterprise). See [`NotebookConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-NotebookConfig).
|
|
350
|
+
|
|
351
|
+
```yaml
|
|
352
|
+
# definitions/actions.yaml
|
|
353
|
+
actions:
|
|
354
|
+
- notebook:
|
|
355
|
+
name: train_churn_model
|
|
356
|
+
filename: notebooks/train_churn_model.ipynb
|
|
357
|
+
dependencyTargets:
|
|
358
|
+
- name: orders_by_day
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
---
|
|
362
|
+
|
|
363
|
+
### `dataPreparation`
|
|
364
|
+
|
|
365
|
+
Configures a BigQuery Data Preparation action from a `.dp.yaml` / `.yaml` definition (via `actions.yaml`) or a `.dp.sqlx` / `.sqlx` file (`type: "dataPreparation"`). See [`DataPreparationConfig`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfig-DataPreparationConfig).
|
|
366
|
+
|
|
367
|
+
---
|
|
368
|
+
|
|
369
|
+
### `propertyGraph` (`graph.yaml`)
|
|
370
|
+
|
|
371
|
+
Defines a BigQuery Property Graph (`CREATE OR REPLACE PROPERTY GRAPH`) from a `definitions/**/graph.yaml` file (at most one `graph.yaml` per project) following the Google Semantic Ontology schema (`version: 0.2.0.dev0/google`). See the [BigQuery Property Graphs guide](https://github.com/dataform-co/dataform/blob/main/docs/property_graphs.md) for full syntax and examples.
|
|
372
|
+
|
|
373
|
+
---
|
|
374
|
+
|
|
375
|
+
### `actions.yaml` Action Configs
|
|
376
|
+
|
|
377
|
+
Plain `.sql` and `.ipynb` files can be registered as Dataform actions via `definitions/**/actions.yaml` files validated against [`dataform.ActionConfigs`](https://dataform-co.github.io/dataform/docs/reference/configs#dataform-ActionConfigs):
|
|
378
|
+
|
|
379
|
+
```yaml
|
|
380
|
+
# definitions/actions.yaml
|
|
381
|
+
actions:
|
|
382
|
+
- declaration:
|
|
383
|
+
name: raw_users
|
|
384
|
+
project: my-raw-project
|
|
385
|
+
dataset: raw
|
|
386
|
+
- view:
|
|
387
|
+
name: stg_users
|
|
388
|
+
filename: staging/stg_users.sql
|
|
389
|
+
dependencyTargets:
|
|
390
|
+
- name: raw_users
|
|
391
|
+
- table:
|
|
392
|
+
name: dim_users
|
|
393
|
+
filename: marts/dim_users.sql
|
|
394
|
+
dependencyTargets:
|
|
395
|
+
- name: stg_users
|
|
396
|
+
partitionBy: created_date
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
---
|
|
400
|
+
|
|
401
|
+
## JavaScript API (`Session` Globals)
|
|
402
|
+
|
|
403
|
+
Inside `definitions/**/*.js` files (and `js {}` blocks in `.sqlx` files), Dataform attaches the active [`Session`](https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md) instance to `global.dataform` and binds its action-builder methods as global functions:
|
|
404
|
+
|
|
405
|
+
| Global Function / Object | Signature | Description |
|
|
406
|
+
|---|---|---|
|
|
407
|
+
| `publish` | `publish(name, queryOrConfig?)` | Creates a `table`, `view` (default if `type` is omitted), or `incremental` action. Returns a `Table`, `View`, or `IncrementalTable` builder (`.query(ctx => ...)`). |
|
|
408
|
+
| `operate` | `operate(name, queriesOrConfig?)` | Creates an `operations` action. Returns an `Operation` builder (`.queries(ctx => ...)`). |
|
|
409
|
+
| `assert` | `assert(name, queryOrConfig?)` | Creates an `assertion` action. Returns an `Assertion` builder (`.query(ctx => ...)`). |
|
|
410
|
+
| `declare` | `declare(config)` | Declares an external data source (`{ database?, schema?, name, description?, columns? }`). |
|
|
411
|
+
| `test` | `test(name)` | Creates a unit test action (`.dataset(target).input(refName, sql).expect(sql)`). |
|
|
412
|
+
| `notebook` | `notebook(config)` | Creates a `Notebook` action (`{ name?, filename, location?, project?, dependencyTargets?, tags?, disabled?, description? }`). |
|
|
413
|
+
| `getContents` | `getContents(relativeMdPath)` | Reads a `.md` file inside the project directory (resolved relative to the caller file) and returns its contents as a string for `description` fields. |
|
|
414
|
+
| `jitData` | `jitData(key, value)` | Registers structured data in `CompiledGraph.jitData` for Just-in-Time (JiT) compilation workflows. |
|
|
415
|
+
| `dataform.projectConfig` | `dataform.ProjectConfig` | Read-only access to the resolved project configuration, including custom variables (`dataform.projectConfig.vars`). |
|
|
416
|
+
|
|
417
|
+
Example (`definitions/generate_staging_views.js`):
|
|
418
|
+
|
|
419
|
+
```js
|
|
420
|
+
const sourceTables = ["orders", "customers", "products"];
|
|
421
|
+
|
|
422
|
+
sourceTables.forEach((tableName) => {
|
|
423
|
+
declare({
|
|
424
|
+
schema: "raw",
|
|
425
|
+
name: `raw_${tableName}`,
|
|
426
|
+
});
|
|
427
|
+
|
|
428
|
+
publish(`stg_${tableName}`, {
|
|
429
|
+
type: "view",
|
|
430
|
+
schema: "staging",
|
|
431
|
+
description: getContents(`./docs/stg_${tableName}.md`),
|
|
432
|
+
}).query((ctx) => `SELECT * FROM ${ctx.ref("raw", `raw_${tableName}`)}`);
|
|
433
|
+
});
|
|
434
|
+
```
|
|
435
|
+
|
|
436
|
+
---
|
|
437
|
+
|
|
438
|
+
## Package Exports and Bundled `.proto` Files
|
|
439
|
+
|
|
440
|
+
### TypeScript / JavaScript Exports
|
|
441
|
+
|
|
442
|
+
When imported as an npm library (`import * as dataformCore from "@dataform/core"`), `@dataform/core` exports the following symbols (with TypeScript typings in `bundle.d.ts`):
|
|
443
|
+
|
|
444
|
+
- **`version`** (`string`): The installed `@dataform/core` semantic version string (for example, `"3.1.5"`).
|
|
445
|
+
- **`session`** (`Session`): The singleton compilation `Session` instance.
|
|
446
|
+
- **`compiler`** (`(code: string, path: string) => string`): Transpiles `.sqlx`, `.yaml`/`.yml`, `.sql`, `.ipynb`, and `.md` source strings into executable JavaScript modules.
|
|
447
|
+
- **`main`** (`(coreExecutionRequest: Uint8Array | string) => Uint8Array | string`): Entry point invoked by the compilation VM sandbox; accepts an encoded `dataform.CoreExecutionRequest` proto and returns an encoded `dataform.CoreExecutionResponse` proto containing `CompiledGraph`.
|
|
448
|
+
- **`jitCompiler`** (`IJitCompiler`): Just-in-Time (JiT) action compiler entry point.
|
|
449
|
+
- **`supportedFeatures`** (`dataform.SupportedFeatures[]`): Feature capability flags supported by this `@dataform/core` version (such as `ARRAY_BUFFER_IPC`).
|
|
450
|
+
- **`IDataformExtension`** / **`IJitCompiler`**: TypeScript interfaces for extensions and JiT compilation.
|
|
451
|
+
|
|
452
|
+
### Bundled Protocol Buffer Schemas
|
|
453
|
+
|
|
454
|
+
The published `@dataform/core` package includes the canonical `.proto` definitions at the package root:
|
|
455
|
+
|
|
456
|
+
- `core.proto` — `CompiledGraph`, `Table`, `Operation`, `Assertion`, `Declaration`, `Test`, `Notebook`, `DataPreparation`, `PropertyGraph`, `Target`, `ProjectConfig`, `CoreExecutionRequest`, `CoreExecutionResponse`.
|
|
457
|
+
- `configs.proto` — `WorkflowSettings`, `ActionConfigs`, `ActionConfig` (`TableConfig`, `ViewConfig`, `IncrementalTableConfig`, `AssertionConfig`, `OperationConfig`, `DeclarationConfig`, `NotebookConfig`, `DataPreparationConfig`).
|
|
458
|
+
- `property_graph_config.proto` — `PropertyGraphConfig` (`SemanticModel`, `Entity`, `Relationship`, `Deployment`).
|
|
459
|
+
- `db_adapter.proto` & `jit.proto` — Database adapter and Just-in-Time (JiT) compilation RPC messages.
|
|
460
|
+
- `extension.proto` — `Extension` configuration schema.
|
|
461
|
+
|
|
462
|
+
---
|
|
463
|
+
|
|
464
|
+
## Documentation and Links
|
|
465
|
+
|
|
466
|
+
- [Dataform Documentation Home](https://cloud.google.com/dataform/docs/overview)
|
|
467
|
+
- [Dataform Core Reference (Google Cloud Docs)](https://cloud.google.com/dataform/docs/reference/dataform-core-reference)
|
|
468
|
+
- [Dataform Configs Reference (`configs.proto`)](https://dataform-co.github.io/dataform/docs/reference/configs)
|
|
469
|
+
- [Dataform JavaScript `Session` Reference](https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md)
|
|
470
|
+
- [Dataform CLI Reference (`@dataform/cli`)](https://github.com/dataform-co/dataform/blob/main/cli/README.md)
|
|
471
|
+
- [BigQuery Property Graphs Guide (`graph.yaml`)](https://github.com/dataform-co/dataform/blob/main/docs/property_graphs.md)
|
|
472
|
+
- [Knowledge Catalog Data Lineage Guide](https://github.com/dataform-co/dataform/blob/main/docs/lineage.md)
|
|
473
|
+
- [Stateless Installs Guide (`.dataformignore`)](https://github.com/dataform-co/dataform/blob/main/docs/stateless-install.md)
|
|
474
|
+
- [GitHub Repository & Issue Tracker](https://github.com/dataform-co/dataform)
|