@revos/cli 0.2.1 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +271 -71
- package/dist/adapters/oclif/commands/action-runs/get.mjs +1 -1
- package/dist/adapters/oclif/commands/action-runs/list.mjs +8 -2
- package/dist/adapters/oclif/commands/actions/get-input-schema.mjs +2 -2
- package/dist/adapters/oclif/commands/actions/get-params-schema.mjs +2 -2
- package/dist/adapters/oclif/commands/actions/get.mjs +1 -1
- package/dist/adapters/oclif/commands/actions/list.mjs +8 -4
- package/dist/adapters/oclif/commands/ai-instructions/create.mjs +1 -1
- package/dist/adapters/oclif/commands/ai-instructions/delete.mjs +1 -1
- package/dist/adapters/oclif/commands/ai-instructions/get.mjs +1 -1
- package/dist/adapters/oclif/commands/ai-instructions/list.mjs +8 -2
- package/dist/adapters/oclif/commands/ai-instructions/update.mjs +1 -1
- package/dist/adapters/oclif/commands/api.d.mts +11 -0
- package/dist/adapters/oclif/commands/api.mjs +112 -0
- package/dist/adapters/oclif/commands/apply.d.mts +28 -0
- package/dist/adapters/oclif/commands/apply.mjs +77 -0
- package/dist/adapters/oclif/commands/auth/login.d.mts +5 -4
- package/dist/adapters/oclif/commands/auth/login.mjs +22 -11
- package/dist/adapters/oclif/commands/auth/logout.d.mts +1 -1
- package/dist/adapters/oclif/commands/auth/logout.mjs +2 -2
- package/dist/adapters/oclif/commands/auth/status.d.mts +2 -2
- package/dist/adapters/oclif/commands/auth/status.mjs +2 -2
- package/dist/adapters/oclif/commands/connections/create.d.mts +6 -0
- package/dist/adapters/oclif/commands/connections/create.mjs +8 -0
- package/dist/adapters/oclif/commands/connections/delete.d.mts +6 -0
- package/dist/adapters/oclif/commands/connections/delete.mjs +8 -0
- package/dist/adapters/oclif/commands/connections/get.d.mts +6 -0
- package/dist/adapters/oclif/commands/connections/get.mjs +8 -0
- package/dist/adapters/oclif/commands/connections/list.d.mts +6 -0
- package/dist/adapters/oclif/commands/connections/list.mjs +14 -0
- package/dist/adapters/oclif/commands/connections/update.d.mts +6 -0
- package/dist/adapters/oclif/commands/connections/update.mjs +8 -0
- package/dist/adapters/oclif/commands/cubes/create.d.mts +6 -0
- package/dist/adapters/oclif/commands/cubes/create.mjs +8 -0
- package/dist/adapters/oclif/commands/cubes/delete.d.mts +6 -0
- package/dist/adapters/oclif/commands/cubes/delete.mjs +8 -0
- package/dist/adapters/oclif/commands/cubes/get.d.mts +6 -0
- package/dist/adapters/oclif/commands/cubes/get.mjs +8 -0
- package/dist/adapters/oclif/commands/cubes/list.d.mts +6 -0
- package/dist/adapters/oclif/commands/cubes/list.mjs +13 -0
- package/dist/adapters/oclif/commands/cubes/update.d.mts +6 -0
- package/dist/adapters/oclif/commands/cubes/update.mjs +8 -0
- package/dist/adapters/oclif/commands/diff.d.mts +27 -0
- package/dist/adapters/oclif/commands/diff.mjs +66 -0
- package/dist/adapters/oclif/commands/gservice-account-keys/get.mjs +1 -1
- package/dist/adapters/oclif/commands/gservice-account-keys/reveal.mjs +2 -2
- package/dist/adapters/oclif/commands/gservice-accounts/create.mjs +1 -1
- package/dist/adapters/oclif/commands/gservice-accounts/delete.mjs +1 -1
- package/dist/adapters/oclif/commands/gservice-accounts/get.mjs +1 -1
- package/dist/adapters/oclif/commands/gservice-accounts/list.mjs +7 -2
- package/dist/adapters/oclif/commands/init.d.mts +2 -1
- package/dist/adapters/oclif/commands/init.mjs +26 -23
- package/dist/adapters/oclif/commands/org/create.mjs +1 -1
- package/dist/adapters/oclif/commands/org/current.d.mts +2 -2
- package/dist/adapters/oclif/commands/org/current.mjs +2 -2
- package/dist/adapters/oclif/commands/org/get.mjs +1 -1
- package/dist/adapters/oclif/commands/org/list.d.mts +3 -11
- package/dist/adapters/oclif/commands/org/list.mjs +26 -26
- package/dist/adapters/oclif/commands/org/switch.d.mts +3 -2
- package/dist/adapters/oclif/commands/org/switch.mjs +10 -3
- package/dist/adapters/oclif/commands/pull.d.mts +28 -0
- package/dist/adapters/oclif/commands/pull.mjs +88 -0
- package/dist/adapters/oclif/commands/score-groups/create.mjs +3 -2
- package/dist/adapters/oclif/commands/score-groups/delete.mjs +1 -1
- package/dist/adapters/oclif/commands/score-groups/get.mjs +1 -1
- package/dist/adapters/oclif/commands/score-groups/list.mjs +3 -2
- package/dist/adapters/oclif/commands/score-groups/update.mjs +1 -1
- package/dist/adapters/oclif/commands/scores/create.mjs +3 -2
- package/dist/adapters/oclif/commands/scores/delete.mjs +1 -1
- package/dist/adapters/oclif/commands/scores/list.mjs +3 -2
- package/dist/adapters/oclif/commands/scores/update.mjs +1 -1
- package/dist/adapters/oclif/commands/segments/create.mjs +1 -1
- package/dist/adapters/oclif/commands/segments/delete.mjs +1 -1
- package/dist/adapters/oclif/commands/segments/evaluate.mjs +2 -2
- package/dist/adapters/oclif/commands/segments/get-evaluation-history.mjs +2 -2
- package/dist/adapters/oclif/commands/segments/get-version.mjs +2 -2
- package/dist/adapters/oclif/commands/segments/get.mjs +1 -1
- package/dist/adapters/oclif/commands/segments/list-versions.mjs +16 -5
- package/dist/adapters/oclif/commands/segments/list.mjs +9 -2
- package/dist/adapters/oclif/commands/segments/restore-version.mjs +2 -2
- package/dist/adapters/oclif/commands/segments/update.mjs +1 -1
- package/dist/adapters/oclif/commands/sources/create.d.mts +11 -0
- package/dist/adapters/oclif/commands/sources/create.mjs +16 -0
- package/dist/adapters/oclif/commands/sources/delete.d.mts +6 -0
- package/dist/adapters/oclif/commands/sources/delete.mjs +8 -0
- package/dist/adapters/oclif/commands/sources/get.d.mts +6 -0
- package/dist/adapters/oclif/commands/sources/get.mjs +8 -0
- package/dist/adapters/oclif/commands/sources/list-streams.d.mts +6 -0
- package/dist/adapters/oclif/commands/sources/list-streams.mjs +31 -0
- package/dist/adapters/oclif/commands/sources/list.d.mts +6 -0
- package/dist/adapters/oclif/commands/sources/list.mjs +13 -0
- package/dist/adapters/oclif/commands/{integrations/get.d.mts → sources/update.d.mts} +4 -4
- package/dist/adapters/oclif/commands/sources/update.mjs +21 -0
- package/dist/adapters/oclif/commands/status.d.mts +26 -0
- package/dist/adapters/oclif/commands/status.mjs +77 -0
- package/dist/adapters/oclif/commands/table-views/create.mjs +3 -2
- package/dist/adapters/oclif/commands/table-views/delete.mjs +1 -1
- package/dist/adapters/oclif/commands/table-views/list.mjs +3 -2
- package/dist/adapters/oclif/commands/table-views/update.mjs +1 -1
- package/dist/adapters/oclif/commands/tables/create.mjs +1 -1
- package/dist/adapters/oclif/commands/tables/delete.mjs +1 -1
- package/dist/adapters/oclif/commands/tables/get.mjs +1 -1
- package/dist/adapters/oclif/commands/tables/list.mjs +3 -2
- package/dist/adapters/oclif/commands/tables/update.mjs +1 -1
- package/dist/{base.command-d7VW6WTp.d.mts → base.command-D7X3ZNtY.d.mts} +0 -1
- package/dist/{base.command-YiwlGlKs.mjs → base.command-cV5d65r8.mjs} +15 -12
- package/dist/chunk-CfYAbeIz.mjs +13 -0
- package/dist/core-CMrP5BQS.mjs +2378 -0
- package/dist/{factory-BrFKT8t-.mjs → factory-C6XLqhT9.mjs} +44 -10
- package/dist/iac-render-BSZZEP0n.mjs +17 -0
- package/dist/index-BqKwXXAo.d.mts +598 -0
- package/dist/index.d.mts +3 -4
- package/dist/index.mjs +2 -2
- package/dist/{presets-D9b6IWKy.mjs → presets-CJbFbHlw.mjs} +35 -8
- package/dist/templates/.claude/settings.json +39 -0
- package/dist/templates/.devcontainer/setup.sh +3 -0
- package/dist/templates/AGENTS.md +33 -20
- package/dist/templates/dbt/dbt_project.yml +2 -2
- package/dist/templates/skills/create-connections/SKILL.md +210 -0
- package/dist/templates/skills/create-connections/references/mappers.md +152 -0
- package/dist/templates/skills/{create-semantic-model → create-cubes}/SKILL.md +20 -18
- package/dist/templates/skills/create-cubes/references/bq-pk-fk-conventions.md +183 -0
- package/dist/templates/skills/{create-semantic-model → create-cubes}/references/cube-examples.md +2 -2
- package/dist/templates/skills/create-cubes/references/hubspot-entities.md +289 -0
- package/dist/templates/skills/create-cubes/references/jira-entities.md +201 -0
- package/dist/templates/skills/create-cubes/references/netsuite-entities.md +121 -0
- package/dist/templates/skills/create-cubes/references/stripe-entities.md +114 -0
- package/dist/templates/skills/create-dbt-transformations/SKILL.md +43 -22
- package/dist/templates/skills/create-dbt-transformations/references/edge-cases.md +20 -2
- package/dist/templates/skills/create-dbt-transformations/references/schema-conventions.md +21 -7
- package/dist/templates/skills/create-dbt-transformations/references/sql-templates.md +34 -20
- package/dist/templates/skills/explore-lakehouse/SKILL.md +3 -3
- package/dist/templates/skills/load-sample-data/SKILL.md +1 -1
- package/dist/templates/skills/visualize-semantic-model/SKILL.md +159 -0
- package/dist/templates/skills/visualize-semantic-model/scripts/render_graph.py +186 -0
- package/dist/{types-Y_ht_ja5.d.mts → types-CGjxcj4L.d.mts} +3 -0
- package/package.json +44 -7
- package/dist/adapters/oclif/commands/integrations/create.d.mts +0 -11
- package/dist/adapters/oclif/commands/integrations/create.mjs +0 -16
- package/dist/adapters/oclif/commands/integrations/get.mjs +0 -21
- package/dist/adapters/oclif/commands/integrations/list.d.mts +0 -11
- package/dist/adapters/oclif/commands/integrations/list.mjs +0 -16
- package/dist/adapters/oclif/commands/integrations/update.d.mts +0 -15
- package/dist/adapters/oclif/commands/integrations/update.mjs +0 -21
- package/dist/adapters/oclif/commands/overlays/diff.d.mts +0 -19
- package/dist/adapters/oclif/commands/overlays/diff.mjs +0 -80
- package/dist/adapters/oclif/commands/overlays/pull.d.mts +0 -15
- package/dist/adapters/oclif/commands/overlays/pull.mjs +0 -45
- package/dist/adapters/oclif/commands/overlays/push.d.mts +0 -18
- package/dist/adapters/oclif/commands/overlays/push.mjs +0 -59
- package/dist/adapters/oclif/commands/overlays/status.d.mts +0 -18
- package/dist/adapters/oclif/commands/overlays/status.mjs +0 -53
- package/dist/core-jpFPylBb.mjs +0 -997
- package/dist/index-DD2Vr-pu.d.mts +0 -193
- package/dist/types-C_p_6rkj.d.mts +0 -69
- /package/dist/templates/skills/{create-semantic-model → create-cubes}/references/key-patterns.md +0 -0
- /package/dist/templates/skills/{create-semantic-model → create-cubes}/references/validation-queries.md +0 -0
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
# Jira Entities Reference
|
|
2
|
+
|
|
3
|
+
## Table naming
|
|
4
|
+
|
|
5
|
+
Airbyte syncs Jira tables with a configurable prefix (default: `jira_`).
|
|
6
|
+
Inspect the BigQuery dataset to find the actual prefix:
|
|
7
|
+
|
|
8
|
+
```sql
|
|
9
|
+
SELECT table_name FROM `<dataset>.INFORMATION_SCHEMA.TABLES`
|
|
10
|
+
WHERE table_name LIKE '%issues%' OR table_name LIKE '%projects%'
|
|
11
|
+
ORDER BY table_name LIMIT 20;
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Throughout this document `<prefix>` is a placeholder for that prefix.
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Primary entities
|
|
19
|
+
|
|
20
|
+
| Cube name | BigQuery table | PK | Notes |
|
|
21
|
+
| ---------------------------- | ---------------------------- | ----------- | ------------------------------------------------ |
|
|
22
|
+
| `<prefix>issues` | `<prefix>issues` | `id` | `key` is display name; metadata in `fields` JSON |
|
|
23
|
+
| `<prefix>projects` | `<prefix>projects` | `id` | `name` |
|
|
24
|
+
| `<prefix>issue_types` | `<prefix>issue_types` | `id` | `name` |
|
|
25
|
+
| `<prefix>issue_priorities` | `<prefix>issue_priorities` | `id` | `name` |
|
|
26
|
+
| `<prefix>users` | `<prefix>users` | `accountId` | `displayName`; aliased 4× (see below) |
|
|
27
|
+
| `<prefix>sprints` | `<prefix>sprints` | `id` | `name`; FK `boardId` |
|
|
28
|
+
| `<prefix>boards` | `<prefix>boards` | `id` | `name`; FK `projectId` |
|
|
29
|
+
| `<prefix>issue_comments` | `<prefix>issue_comments` | — | FK `issueId`; `author` is JSON object |
|
|
30
|
+
| `<prefix>issue_worklogs` | `<prefix>issue_worklogs` | — | FK `issueId`; `author` is JSON object |
|
|
31
|
+
| `<prefix>project_components` | `<prefix>project_components` | — | FK `projectId` as INT64 |
|
|
32
|
+
| `<prefix>project_versions` | `<prefix>project_versions` | — | FK `projectId` as INT64 |
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Users table aliasing
|
|
37
|
+
|
|
38
|
+
The single `<prefix>users` table must be exposed as **separate cubes** for each
|
|
39
|
+
role because Cube.js does not support joining the same table twice with
|
|
40
|
+
different conditions. Each alias has its own cube name and `sql_table` pointing
|
|
41
|
+
to the same physical table.
|
|
42
|
+
|
|
43
|
+
| Cube name | Role | Join condition on `<prefix>issues` |
|
|
44
|
+
| ------------------------------ | -------------- | -------------------------------------------------- |
|
|
45
|
+
| `<prefix>users_assignee` | Assignee | `JSON_VALUE(fields, '$.assignee.accountId')` |
|
|
46
|
+
| `<prefix>users_reporter` | Reporter | `JSON_VALUE(fields, '$.reporter.accountId')` |
|
|
47
|
+
| `<prefix>users_creator` | Creator | `JSON_VALUE(fields, '$.creator.accountId')` |
|
|
48
|
+
| `<prefix>users_comment_author` | Comment author | `JSON_VALUE(issue_comments.author, '$.accountId')` |
|
|
49
|
+
| `<prefix>users_worklog_author` | Worklog author | `JSON_VALUE(issue_worklogs.author, '$.accountId')` |
|
|
50
|
+
| `<prefix>users` | Direct queries | (no joins defined) |
|
|
51
|
+
|
|
52
|
+
Template for each alias:
|
|
53
|
+
|
|
54
|
+
```yaml
|
|
55
|
+
name: <prefix>users_assignee
|
|
56
|
+
sql_table: "`<dataset>.<prefix>users`"
|
|
57
|
+
joins:
|
|
58
|
+
<prefix>issues:
|
|
59
|
+
relationship: one_to_many
|
|
60
|
+
sql: >
|
|
61
|
+
${CUBE}.accountId =
|
|
62
|
+
JSON_VALUE(${<prefix>issues.fields}, '$.assignee.accountId')
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Corresponding join on `<prefix>issues`:
|
|
66
|
+
|
|
67
|
+
```yaml
|
|
68
|
+
joins:
|
|
69
|
+
<prefix>users_assignee:
|
|
70
|
+
relationship: many_to_one
|
|
71
|
+
sql: >
|
|
72
|
+
JSON_VALUE(${CUBE}.fields, '$.assignee.accountId') =
|
|
73
|
+
${<prefix>users_assignee.accountId}
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## Issues: fields JSON column
|
|
79
|
+
|
|
80
|
+
Issue metadata lives in a single `fields` JSON column. Extract with `JSON_VALUE`:
|
|
81
|
+
|
|
82
|
+
| Field | JSON path |
|
|
83
|
+
| ------------------- | ----------------------------------------- |
|
|
84
|
+
| Issue type ID | `$.issuetype.id` |
|
|
85
|
+
| Priority ID | `$.priority.id` |
|
|
86
|
+
| Assignee account ID | `$.assignee.accountId` |
|
|
87
|
+
| Reporter account ID | `$.reporter.accountId` |
|
|
88
|
+
| Creator account ID | `$.creator.accountId` |
|
|
89
|
+
| Status | `$.status.name` |
|
|
90
|
+
| Summary | `$.summary` |
|
|
91
|
+
| Story points | `$.story_points` or `$.customfield_10016` |
|
|
92
|
+
|
|
93
|
+
Example dimension:
|
|
94
|
+
|
|
95
|
+
```yaml
|
|
96
|
+
dimensions:
|
|
97
|
+
status:
|
|
98
|
+
sql: "JSON_VALUE(${CUBE}.fields, '$.status.name')"
|
|
99
|
+
type: string
|
|
100
|
+
issue_type_name:
|
|
101
|
+
sql: "JSON_VALUE(${CUBE}.fields, '$.issuetype.name')"
|
|
102
|
+
type: string
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## Bridge / junction cubes (public: false)
|
|
108
|
+
|
|
109
|
+
### sprint_issues
|
|
110
|
+
|
|
111
|
+
Sprints and issues are many-to-many. The `sprint_issues` table has columns
|
|
112
|
+
`issueId` (STRING) and `sprintId` (INT64). Composite PK required.
|
|
113
|
+
|
|
114
|
+
```yaml
|
|
115
|
+
name: <prefix>sprint_issues
|
|
116
|
+
sql_table: "`<dataset>.<prefix>sprint_issues`"
|
|
117
|
+
public: false
|
|
118
|
+
dimensions:
|
|
119
|
+
id:
|
|
120
|
+
sql: "${CUBE}.issueId || '_' || CAST(${CUBE}.sprintId AS STRING)"
|
|
121
|
+
type: string
|
|
122
|
+
primary_key: true
|
|
123
|
+
joins:
|
|
124
|
+
<prefix>issues:
|
|
125
|
+
relationship: many_to_one
|
|
126
|
+
sql: "${CUBE}.issueId = ${<prefix>issues.id}"
|
|
127
|
+
<prefix>sprints:
|
|
128
|
+
relationship: many_to_one
|
|
129
|
+
sql: "${CUBE}.sprintId = ${<prefix>sprints.id}"
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
Issues and sprints join through this bridge:
|
|
133
|
+
|
|
134
|
+
```yaml
|
|
135
|
+
# On <prefix>issues:
|
|
136
|
+
joins:
|
|
137
|
+
<prefix>sprint_issues:
|
|
138
|
+
relationship: one_to_many
|
|
139
|
+
sql: "${CUBE}.id = ${<prefix>sprint_issues.issueId}"
|
|
140
|
+
|
|
141
|
+
# On <prefix>sprints:
|
|
142
|
+
joins:
|
|
143
|
+
<prefix>sprint_issues:
|
|
144
|
+
relationship: one_to_many
|
|
145
|
+
sql: "${CUBE}.id = ${<prefix>sprint_issues.sprintId}"
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
### board_issues
|
|
149
|
+
|
|
150
|
+
Board issues (`board_issues` table) link boards to issues. Composite PK uses `id` + `boardId`.
|
|
151
|
+
|
|
152
|
+
```yaml
|
|
153
|
+
name: <prefix>board_issues
|
|
154
|
+
sql_table: "`<dataset>.<prefix>board_issues`"
|
|
155
|
+
public: false
|
|
156
|
+
dimensions:
|
|
157
|
+
composite_id:
|
|
158
|
+
sql: "${CUBE}.id || '_' || CAST(${CUBE}.boardId AS STRING)"
|
|
159
|
+
type: string
|
|
160
|
+
primary_key: true
|
|
161
|
+
joins:
|
|
162
|
+
<prefix>issues:
|
|
163
|
+
relationship: many_to_one
|
|
164
|
+
sql: "${CUBE}.id = ${<prefix>issues.id}"
|
|
165
|
+
<prefix>boards:
|
|
166
|
+
relationship: many_to_one
|
|
167
|
+
sql: "${CUBE}.boardId = ${<prefix>boards.id}"
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
## Type casting pitfalls
|
|
173
|
+
|
|
174
|
+
### project_components / project_versions → projects
|
|
175
|
+
|
|
176
|
+
`project_components.projectId` and `project_versions.projectId` are INT64 but
|
|
177
|
+
`projects.id` is STRING. Always cast:
|
|
178
|
+
|
|
179
|
+
```yaml
|
|
180
|
+
# On <prefix>project_components:
|
|
181
|
+
joins:
|
|
182
|
+
<prefix>projects:
|
|
183
|
+
relationship: many_to_one
|
|
184
|
+
sql: "CAST(${CUBE}.projectId AS STRING) = ${<prefix>projects.id}"
|
|
185
|
+
|
|
186
|
+
# On <prefix>projects:
|
|
187
|
+
joins:
|
|
188
|
+
<prefix>project_components:
|
|
189
|
+
relationship: one_to_many
|
|
190
|
+
sql: "SAFE_CAST(${CUBE}.id AS INT64) = ${<prefix>project_components.projectId}"
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
---
|
|
194
|
+
|
|
195
|
+
## Common pitfalls
|
|
196
|
+
|
|
197
|
+
1. **`fields` JSON column** — most issue attributes live here, not as top-level columns. Always check `INFORMATION_SCHEMA` before assuming a column exists at the top level.
|
|
198
|
+
2. **User aliases must be separate cubes** — do not try to join `users` twice from `issues`; Cube.js requires distinct cube names per join target.
|
|
199
|
+
3. **`sprintId` is INT64** — cast to STRING in the composite PK to avoid type errors.
|
|
200
|
+
4. **`issue_comments.author` is a JSON object** — extract `accountId` with `JSON_VALUE`, not a direct column reference.
|
|
201
|
+
5. **`boards.projectId` vs `projects.id`** — both are strings here; no cast needed. But `project_components.projectId` is INT64 — always cast.
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
# NetSuite Entities Reference
|
|
2
|
+
|
|
3
|
+
## Table naming
|
|
4
|
+
|
|
5
|
+
Airbyte syncs NetSuite tables with a configurable prefix (default: `netsuite_`).
|
|
6
|
+
Inspect the BigQuery dataset to confirm:
|
|
7
|
+
|
|
8
|
+
```sql
|
|
9
|
+
SELECT table_name FROM `<dataset>.INFORMATION_SCHEMA.TABLES`
|
|
10
|
+
WHERE table_name LIKE '%customer%' OR table_name LIKE '%salesorder%'
|
|
11
|
+
ORDER BY table_name LIMIT 20;
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Throughout this document `<prefix>` is a placeholder for that prefix.
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Primary entities
|
|
19
|
+
|
|
20
|
+
| Cube name | BigQuery table | PK | Notes |
|
|
21
|
+
| --------------------- | --------------------- | ---- | ------------------------------ |
|
|
22
|
+
| `<prefix>customer` | `<prefix>customer` | `id` | FK `subsidiary` is JSON object |
|
|
23
|
+
| `<prefix>contact` | `<prefix>contact` | `id` | FK `company` is JSON object |
|
|
24
|
+
| `<prefix>opportunity` | `<prefix>opportunity` | `id` | FK `entity` is JSON object |
|
|
25
|
+
| `<prefix>salesorder` | `<prefix>salesorder` | `id` | FK `entity` is JSON object |
|
|
26
|
+
| `<prefix>employee` | `<prefix>employee` | `id` | — |
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## FK extraction pattern
|
|
31
|
+
|
|
32
|
+
NetSuite stores foreign keys as **JSON objects** with an `id` field rather than
|
|
33
|
+
as flat FK columns. Use `JSON_VALUE` to extract the ID and expose it as a
|
|
34
|
+
computed dimension.
|
|
35
|
+
|
|
36
|
+
### contact → customer
|
|
37
|
+
|
|
38
|
+
`contact.company` is a JSON object: `{"id": "123", "refName": "Acme Corp"}`.
|
|
39
|
+
|
|
40
|
+
```yaml
|
|
41
|
+
name: <prefix>contact
|
|
42
|
+
sql_table: "`<dataset>.<prefix>contact`"
|
|
43
|
+
dimensions:
|
|
44
|
+
id:
|
|
45
|
+
sql: "id"
|
|
46
|
+
type: string
|
|
47
|
+
primary_key: true
|
|
48
|
+
customer_id:
|
|
49
|
+
sql: "JSON_VALUE(${CUBE}.company, '$.id')"
|
|
50
|
+
type: string
|
|
51
|
+
joins:
|
|
52
|
+
<prefix>customer:
|
|
53
|
+
relationship: many_to_one
|
|
54
|
+
sql: "${CUBE.customer_id} = ${<prefix>customer.id}"
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### customer → subsidiary
|
|
58
|
+
|
|
59
|
+
`customer.subsidiary` is a JSON object: `{"id": "1", "refName": "Main Subsidiary"}`.
|
|
60
|
+
|
|
61
|
+
```yaml
|
|
62
|
+
name: <prefix>customer
|
|
63
|
+
sql_table: "`<dataset>.<prefix>customer`"
|
|
64
|
+
dimensions:
|
|
65
|
+
subsidiary_id:
|
|
66
|
+
sql: "JSON_VALUE(${CUBE}.subsidiary, '$.id')"
|
|
67
|
+
type: string
|
|
68
|
+
joins:
|
|
69
|
+
<prefix>contact:
|
|
70
|
+
relationship: one_to_many
|
|
71
|
+
sql: "${CUBE}.id = ${<prefix>contact.customer_id}"
|
|
72
|
+
<prefix>salesorder:
|
|
73
|
+
relationship: one_to_many
|
|
74
|
+
sql: "${CUBE}.id = ${<prefix>salesorder.customer_id}"
|
|
75
|
+
<prefix>opportunity:
|
|
76
|
+
relationship: one_to_many
|
|
77
|
+
sql: "${CUBE}.id = ${<prefix>opportunity.customer_id}"
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### opportunity / salesorder → customer
|
|
81
|
+
|
|
82
|
+
Both `opportunity` and `salesorder` use `entity` (not `company`) as the FK column:
|
|
83
|
+
|
|
84
|
+
```yaml
|
|
85
|
+
name: <prefix>opportunity
|
|
86
|
+
sql_table: "`<dataset>.<prefix>opportunity`"
|
|
87
|
+
dimensions:
|
|
88
|
+
id:
|
|
89
|
+
sql: "id"
|
|
90
|
+
type: string
|
|
91
|
+
primary_key: true
|
|
92
|
+
customer_id:
|
|
93
|
+
sql: "JSON_VALUE(${CUBE}.entity, '$.id')"
|
|
94
|
+
type: string
|
|
95
|
+
joins:
|
|
96
|
+
<prefix>customer:
|
|
97
|
+
relationship: many_to_one
|
|
98
|
+
sql: "${CUBE.customer_id} = ${<prefix>customer.id}"
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
Same pattern for `<prefix>salesorder`.
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
## Relationship graph
|
|
106
|
+
|
|
107
|
+
```
|
|
108
|
+
customer ──< contact
|
|
109
|
+
customer ──< opportunity
|
|
110
|
+
customer ──< salesorder
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
|
|
115
|
+
## Common pitfalls
|
|
116
|
+
|
|
117
|
+
1. **FK columns are JSON objects** — `contact.company`, `opportunity.entity`, `salesorder.entity`, `customer.subsidiary` are all JSON objects. Never join directly; always extract with `JSON_VALUE(..., '$.id')` and expose as a computed dimension.
|
|
118
|
+
2. **`entity` vs `company`** — contacts use `company`, but opportunities and sales orders use `entity` as the customer FK column.
|
|
119
|
+
3. **PK column name is `id` (lowercase)** — not `internalId` or similar. Verify in INFORMATION_SCHEMA.
|
|
120
|
+
4. **Subsidiary** — `customer.subsidiary` holds the subsidiary FK. If the schema has multiple subsidiaries, you may need a `subsidiary` cube joined via `subsidiary_id`.
|
|
121
|
+
5. **No `_airbyte_extracted_at` guarantee** — some NetSuite streams use `lastModifiedDate` instead. Check actual column names before writing `refresh_key`.
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# Stripe Entities Reference
|
|
2
|
+
|
|
3
|
+
## Table naming
|
|
4
|
+
|
|
5
|
+
Airbyte syncs Stripe tables with a configurable prefix (default: `stripe_`).
|
|
6
|
+
Inspect the BigQuery dataset to confirm:
|
|
7
|
+
|
|
8
|
+
```sql
|
|
9
|
+
SELECT table_name FROM `<dataset>.INFORMATION_SCHEMA.TABLES`
|
|
10
|
+
WHERE table_name LIKE '%customers%' OR table_name LIKE '%invoices%'
|
|
11
|
+
ORDER BY table_name LIMIT 20;
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Throughout this document `<prefix>` is a placeholder for that prefix.
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Primary entities
|
|
19
|
+
|
|
20
|
+
| Cube name | BigQuery table | PK | Notes |
|
|
21
|
+
| ----------------------- | ----------------------- | ---- | -------------------------------- |
|
|
22
|
+
| `<prefix>customers` | `<prefix>customers` | `id` | `name` is display name |
|
|
23
|
+
| `<prefix>subscriptions` | `<prefix>subscriptions` | `id` | FK `customer` (→ customers.id) |
|
|
24
|
+
| `<prefix>invoices` | `<prefix>invoices` | `id` | FK `customer`, FK `subscription` |
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## Relationship graph
|
|
29
|
+
|
|
30
|
+
```
|
|
31
|
+
customers ──< subscriptions ──< invoices
|
|
32
|
+
└────────────────────────< invoices
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
- customer → subscriptions: `one_to_many` via `subscriptions.customer = customers.id`
|
|
36
|
+
- customer → invoices: `one_to_many` via `invoices.customer = customers.id`
|
|
37
|
+
- subscription → invoices: `one_to_many` via `invoices.subscription = subscriptions.id`
|
|
38
|
+
- subscription → latest_invoice: `many_to_one` via `subscriptions.latest_invoice = latest_invoice.id`
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## Standard cube definitions
|
|
43
|
+
|
|
44
|
+
### customers
|
|
45
|
+
|
|
46
|
+
```yaml
|
|
47
|
+
name: <prefix>customers
|
|
48
|
+
sql_table: "`<dataset>.<prefix>customers`"
|
|
49
|
+
joins:
|
|
50
|
+
<prefix>subscriptions:
|
|
51
|
+
relationship: one_to_many
|
|
52
|
+
sql: "${CUBE}.id = ${<prefix>subscriptions.customer}"
|
|
53
|
+
<prefix>invoices:
|
|
54
|
+
relationship: one_to_many
|
|
55
|
+
sql: "${CUBE}.id = ${<prefix>invoices.customer}"
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
### subscriptions
|
|
59
|
+
|
|
60
|
+
```yaml
|
|
61
|
+
name: <prefix>subscriptions
|
|
62
|
+
sql_table: "`<dataset>.<prefix>subscriptions`"
|
|
63
|
+
joins:
|
|
64
|
+
<prefix>customers:
|
|
65
|
+
relationship: many_to_one
|
|
66
|
+
sql: "${CUBE}.customer = ${<prefix>customers.id}"
|
|
67
|
+
<prefix>invoices:
|
|
68
|
+
relationship: one_to_many
|
|
69
|
+
sql: "${CUBE}.id = ${<prefix>invoices.subscription}"
|
|
70
|
+
<prefix>latest_invoice:
|
|
71
|
+
relationship: many_to_one
|
|
72
|
+
sql: "${CUBE}.latest_invoice = ${<prefix>latest_invoice.id}"
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### invoices
|
|
76
|
+
|
|
77
|
+
```yaml
|
|
78
|
+
name: <prefix>invoices
|
|
79
|
+
sql_table: "`<dataset>.<prefix>invoices`"
|
|
80
|
+
joins:
|
|
81
|
+
<prefix>customers:
|
|
82
|
+
relationship: many_to_one
|
|
83
|
+
sql: "${CUBE}.customer = ${<prefix>customers.id}"
|
|
84
|
+
<prefix>subscriptions:
|
|
85
|
+
relationship: many_to_one
|
|
86
|
+
sql: "${CUBE}.subscription = ${<prefix>subscriptions.id}"
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## Special cube: latest_invoice
|
|
92
|
+
|
|
93
|
+
`latest_invoice` is an alias for the `invoices` table (public: false) used
|
|
94
|
+
exclusively for the `subscriptions.latest_invoice` FK join. Needed because
|
|
95
|
+
Cube.js does not support two joins to the same table under the same cube name.
|
|
96
|
+
|
|
97
|
+
```yaml
|
|
98
|
+
name: <prefix>latest_invoice
|
|
99
|
+
sql_table: "`<dataset>.<prefix>invoices`"
|
|
100
|
+
public: false
|
|
101
|
+
joins:
|
|
102
|
+
<prefix>subscriptions:
|
|
103
|
+
relationship: one_to_many
|
|
104
|
+
sql: "${CUBE}.id = ${<prefix>subscriptions.latest_invoice}"
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
## Common pitfalls
|
|
110
|
+
|
|
111
|
+
1. **`latest_invoice` must be a separate cube** — subscriptions needs both `invoices` (for all invoices) and `latest_invoice` (for the most recent one). Same physical table, different cube names.
|
|
112
|
+
2. **FK column names without suffix** — `subscriptions.customer` is the raw Stripe customer ID (not `customer_id`). Same for `invoices.subscription` and `invoices.customer`. Check actual column names in INFORMATION_SCHEMA.
|
|
113
|
+
3. **Stripe IDs are strings** — all IDs start with a prefix (`cus_`, `sub_`, `in_`, etc.). No casting needed.
|
|
114
|
+
4. **Timestamps** — Stripe tables from Airbyte use `_airbyte_extracted_at` as the sync timestamp. Use it for `refresh_key`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: create-dbt-transformations
|
|
3
|
-
description: Create new dbt transformations (
|
|
3
|
+
description: Create new dbt transformations (silver/gold models) in the RevOS dbt project. Use when asked to create a dbt model, build a transformation, add a new layer model, declare a raw source, or register a new raw table. Bronze is source-declarations only — no SQL files. Covers dbt project conventions, sources, materialization, schema.yml, and validation commands.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Create dbt Transformations
|
|
@@ -22,32 +22,35 @@ Warn the user: "The `explore-lakehouse` skill is not installed — using `bq sho
|
|
|
22
22
|
## Layer Conventions
|
|
23
23
|
|
|
24
24
|
- **gold** — business-ready models exposed for reporting or downstream consumption.
|
|
25
|
-
- **silver** — cleaned, deduplicated, type-conformed intermediates.
|
|
26
|
-
- **bronze** —
|
|
25
|
+
- **silver** — cleaned, deduplicated, type-conformed intermediates. Lowest SQL layer; reads raw data via `{{ source('bronze', '<table>') }}`.
|
|
26
|
+
- **bronze** — **not a SQL layer**. Holds only `dbt/models/bronze/schema.yml`, which declares raw tables as dbt sources. No `.sql` files belong under `dbt/models/bronze/`.
|
|
27
27
|
|
|
28
28
|
When layer is not obvious from context, ask (see Checkpoint 1).
|
|
29
29
|
|
|
30
30
|
## Sources (bronze layer)
|
|
31
31
|
|
|
32
|
-
Raw tables
|
|
32
|
+
Raw tables loaded into the warehouse by your ingestion pipeline are not dbt models. Declare them as dbt sources so silver models can reference them with `{{ source() }}`.
|
|
33
33
|
|
|
34
34
|
Sources are declared in `dbt/models/bronze/schema.yml` under a `sources:` block using `schema` (the BigQuery dataset):
|
|
35
35
|
|
|
36
36
|
```yaml
|
|
37
37
|
sources:
|
|
38
|
-
- name:
|
|
38
|
+
- name: bronze
|
|
39
39
|
schema: "{{ env_var('REVOS_BQ_DATASET') }}"
|
|
40
40
|
tables:
|
|
41
41
|
- name: hubspot_contacts
|
|
42
42
|
```
|
|
43
43
|
|
|
44
|
-
Reference in
|
|
44
|
+
Reference in silver SQL:
|
|
45
45
|
|
|
46
46
|
```sql
|
|
47
|
-
|
|
47
|
+
-- dbt/models/silver/silver_hubspot_contacts.sql
|
|
48
|
+
SELECT * FROM {{ source('bronze', 'hubspot_contacts') }}
|
|
48
49
|
```
|
|
49
50
|
|
|
50
|
-
|
|
51
|
+
`{{ source('bronze', 'hubspot_contacts') }}` resolves to `${REVOS_BQ_DATASET}.hubspot_contacts` — the same dataset where raw tables live — so silver has direct access without a bronze SQL view in between.
|
|
52
|
+
|
|
53
|
+
See [schema-conventions.md](references/schema-conventions.md) for the full declaration pattern.
|
|
51
54
|
|
|
52
55
|
## Materialization
|
|
53
56
|
|
|
@@ -63,10 +66,12 @@ Materialized table lives at: `$REVOS_BQ_DATASET.<model_name>`
|
|
|
63
66
|
|
|
64
67
|
**When to use `{{ ref() }}` vs. `{{ source() }}`:**
|
|
65
68
|
|
|
66
|
-
| Context
|
|
67
|
-
|
|
|
68
|
-
| dbt SQL → other dbt model
|
|
69
|
-
| dbt SQL → raw
|
|
69
|
+
| Context | Use |
|
|
70
|
+
| -------------------------------------------------- | ----------------------------------- |
|
|
71
|
+
| dbt SQL → other dbt model | `{{ ref('<model>') }}` |
|
|
72
|
+
| dbt SQL → raw table (silver reading from `bronze`) | `{{ source('bronze', '<table>') }}` |
|
|
73
|
+
|
|
74
|
+
Silver is the lowest SQL layer — `{{ source('bronze', ...) }}` is used in silver only. Gold reads from silver via `{{ ref() }}`. There are no SQL files in `dbt/models/bronze/`.
|
|
70
75
|
|
|
71
76
|
Always declare raw tables as sources before referencing them. Do not use bare fully qualified names — that bypasses dbt's dependency graph and source freshness tracking.
|
|
72
77
|
|
|
@@ -89,12 +94,12 @@ dbt build --select path:models/<layer> # entire layer
|
|
|
89
94
|
|
|
90
95
|
For each transformation (one at a time — do not batch):
|
|
91
96
|
|
|
92
|
-
1. Determine the target layer (Checkpoint 1 if unclear).
|
|
97
|
+
1. Determine the target layer — **silver** or **gold** only (Checkpoint 1 if unclear). Refuse bronze SQL models (see Checkpoint 4).
|
|
93
98
|
2. Determine the model name.
|
|
94
99
|
3. Check if that model already exists (Checkpoint 2 if yes).
|
|
95
100
|
4. Gather source data and transformation logic. For bridge models, apply the bridge template ([sql-templates.md](references/sql-templates.md)).
|
|
96
|
-
5.
|
|
97
|
-
6. Generate `dbt/models/<
|
|
101
|
+
5. If the model reads raw data, ensure each raw table is declared under the `bronze` source in `dbt/models/bronze/schema.yml`; add it if missing.
|
|
102
|
+
6. Generate `dbt/models/<silver|gold>/<model_name>.sql`. **Never** generate `.sql` files under `dbt/models/bronze/`.
|
|
98
103
|
7. Detect the primary key (Checkpoint 3 if ambiguous).
|
|
99
104
|
8. Add model entry to `dbt/models/<layer>/schema.yml` with PK and FK tests. See [schema-conventions.md](references/schema-conventions.md).
|
|
100
105
|
9. Run `dbt run --select <model_name>` and report result.
|
|
@@ -115,8 +120,9 @@ Ask if the layer is not obvious:
|
|
|
115
120
|
Which layer should this transformation live in?
|
|
116
121
|
|
|
117
122
|
- gold: business-ready, exposed for reporting or downstream consumption
|
|
118
|
-
- silver: cleaned/intermediate,
|
|
119
|
-
|
|
123
|
+
- silver: cleaned/intermediate, reads raw via `{{ source('bronze', ...) }}`
|
|
124
|
+
|
|
125
|
+
(bronze is not a SQL layer — it only holds `schema.yml` source declarations.)
|
|
120
126
|
```
|
|
121
127
|
|
|
122
128
|
Layer is obvious when the user explicitly names it.
|
|
@@ -148,6 +154,21 @@ I could not unambiguously detect the primary key. Candidates:
|
|
|
148
154
|
Which column(s) should be the primary key?
|
|
149
155
|
```
|
|
150
156
|
|
|
157
|
+
### Checkpoint 4: Bronze SQL Model Refused
|
|
158
|
+
|
|
159
|
+
If the user explicitly asks to create a bronze SQL model:
|
|
160
|
+
|
|
161
|
+
```text
|
|
162
|
+
Bronze is not a SQL layer in this project — it only holds source
|
|
163
|
+
declarations in `dbt/models/bronze/schema.yml`. Silver reads raw data
|
|
164
|
+
directly via `{{ source('bronze', '<raw_table>') }}`.
|
|
165
|
+
|
|
166
|
+
Would you like to create this as a silver model instead?
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Do not generate any file under `dbt/models/bronze/` other than
|
|
170
|
+
`schema.yml`.
|
|
171
|
+
|
|
151
172
|
---
|
|
152
173
|
|
|
153
174
|
## Primary Key Detection
|
|
@@ -169,9 +190,9 @@ A column is a FK candidate if it matches `<entity>_id` where `<entity>` ≠ mode
|
|
|
169
190
|
|
|
170
191
|
## Timestamp Column Propagation (Gold Models)
|
|
171
192
|
|
|
172
|
-
Every gold model **must** propagate at least one timestamp column so downstream
|
|
193
|
+
Every gold model **must** propagate at least one timestamp column so downstream cubes can use SQL-based `refresh_key` (see `create-cubes` skill). Priority:
|
|
173
194
|
|
|
174
|
-
1.
|
|
195
|
+
1. An ingestion-time column on the raw table (e.g. Airbyte writes `_airbyte_extracted_at`) — propagate when present.
|
|
175
196
|
2. `updated_at` / `modified_at` — CDC-friendly streams.
|
|
176
197
|
3. `created_at` — insert-only fact tables.
|
|
177
198
|
|
|
@@ -181,8 +202,8 @@ If the upstream source has none of these, document it in a SQL comment: `-- no t
|
|
|
181
202
|
|
|
182
203
|
See [sql-templates.md](references/sql-templates.md) for:
|
|
183
204
|
|
|
184
|
-
-
|
|
185
|
-
- Standard
|
|
205
|
+
- Standard silver model template (reads raw via `{{ source('bronze', ...) }}`)
|
|
206
|
+
- Standard gold model template (reads silver via `{{ ref() }}`)
|
|
186
207
|
- Bridge model (JSON array) template with concrete example
|
|
187
208
|
- Bridge model naming convention and SQL content rules
|
|
188
209
|
|
|
@@ -201,7 +222,7 @@ See [edge-cases.md](references/edge-cases.md) for: missing SQL details, missing
|
|
|
201
222
|
```text
|
|
202
223
|
Created dbt transformation: <model_name>
|
|
203
224
|
|
|
204
|
-
Layer: <
|
|
225
|
+
Layer: <silver | gold>
|
|
205
226
|
File: dbt/models/<layer>/<model_name>.sql
|
|
206
227
|
Materialization: <inherited: table | overridden: <type>>
|
|
207
228
|
Primary key: <pk_column> (or composite: <col_1>, <col_2>)
|
|
@@ -26,9 +26,27 @@ The transformation you described references `<missing_model>`, which does not
|
|
|
26
26
|
exist in dbt/models/. Should I create that model first?
|
|
27
27
|
```
|
|
28
28
|
|
|
29
|
-
## Source is a raw
|
|
29
|
+
## Source is a raw table not yet declared as a dbt source
|
|
30
30
|
|
|
31
|
-
Declare it
|
|
31
|
+
Declare it under `sources: - name: bronze` in `dbt/models/bronze/schema.yml`
|
|
32
|
+
first (see [schema-conventions.md](schema-conventions.md)), then reference it
|
|
33
|
+
with `{{ source('bronze', '<table>') }}` in the silver model SQL. Do not use
|
|
34
|
+
fully qualified BigQuery names directly — that bypasses dbt's dependency
|
|
35
|
+
graph and source freshness tracking.
|
|
36
|
+
|
|
37
|
+
## User asks to create a bronze SQL model
|
|
38
|
+
|
|
39
|
+
Refuse and redirect:
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
Bronze is not a SQL layer in this project — `dbt/models/bronze/` only
|
|
43
|
+
contains `schema.yml` declaring raw tables as sources. Silver reads raw
|
|
44
|
+
data directly via `{{ source('bronze', '<raw_table>') }}`.
|
|
45
|
+
|
|
46
|
+
Should I create this as a silver model instead?
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Do not generate any file under `dbt/models/bronze/` other than `schema.yml`.
|
|
32
50
|
|
|
33
51
|
## run fails
|
|
34
52
|
|
|
@@ -10,9 +10,14 @@
|
|
|
10
10
|
|
|
11
11
|
---
|
|
12
12
|
|
|
13
|
-
Each layer has one shared `schema.yml` at
|
|
13
|
+
Each SQL layer (silver, gold) has one shared `schema.yml` at
|
|
14
|
+
`dbt/models/<layer>/schema.yml`. Append new models; do not create per-model
|
|
15
|
+
files.
|
|
14
16
|
|
|
15
|
-
|
|
17
|
+
The bronze directory is **not** a SQL layer — its `schema.yml` contains only
|
|
18
|
+
source declarations, no `models:` block.
|
|
19
|
+
|
|
20
|
+
If a layer's `schema.yml` does not exist, create it with:
|
|
16
21
|
|
|
17
22
|
```yaml
|
|
18
23
|
version: 2
|
|
@@ -22,7 +27,9 @@ models:
|
|
|
22
27
|
|
|
23
28
|
## Declaring Sources (bronze layer)
|
|
24
29
|
|
|
25
|
-
|
|
30
|
+
`dbt/models/bronze/schema.yml` is the only file in `dbt/models/bronze/`. It
|
|
31
|
+
declares raw tables as dbt sources so that silver models can reference them
|
|
32
|
+
with `{{ source('bronze', '<table>') }}`.
|
|
26
33
|
|
|
27
34
|
`schema` maps to the BigQuery dataset (`REVOS_BQ_DATASET`):
|
|
28
35
|
|
|
@@ -30,23 +37,30 @@ Raw tables must be declared as dbt sources before they can be referenced with `{
|
|
|
30
37
|
version: 2
|
|
31
38
|
|
|
32
39
|
sources:
|
|
33
|
-
- name:
|
|
40
|
+
- name: bronze
|
|
34
41
|
schema: "{{ env_var('REVOS_BQ_DATASET') }}"
|
|
35
42
|
tables:
|
|
36
43
|
- name: hubspot_contacts
|
|
37
44
|
- name: hubspot_deals
|
|
38
45
|
- name: stripe_charges
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
The corresponding silver model entry lives in `dbt/models/silver/schema.yml`:
|
|
49
|
+
|
|
50
|
+
```yaml
|
|
51
|
+
version: 2
|
|
39
52
|
|
|
40
53
|
models:
|
|
41
|
-
- name:
|
|
54
|
+
- name: silver_hubspot_contacts
|
|
42
55
|
...
|
|
43
56
|
```
|
|
44
57
|
|
|
45
58
|
Rules:
|
|
46
59
|
|
|
47
|
-
- Use `
|
|
48
|
-
- Each raw table referenced in
|
|
60
|
+
- Use `bronze` as the source name for all raw tables.
|
|
61
|
+
- Each raw table referenced in silver SQL needs a corresponding entry under `tables:`.
|
|
49
62
|
- If the source block already exists, append to the `tables:` list only.
|
|
63
|
+
- Do **not** add a `models:` block to `dbt/models/bronze/schema.yml` — bronze contains source declarations only.
|
|
50
64
|
|
|
51
65
|
## Standard Model Entry
|
|
52
66
|
|