dremiojs 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.eslintrc.json +14 -0
- package/.prettierrc +7 -0
- package/README.md +59 -0
- package/dremiodocs/dremio-cloud/cloud-api-reference.md +748 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-about.md +225 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-admin.md +3754 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-bring-data.md +6098 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-changelog.md +32 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-developer.md +1147 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-explore-analyze.md +2522 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-get-started.md +300 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-help-support.md +869 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-manage-govern.md +800 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-overview.md +36 -0
- package/dremiodocs/dremio-cloud/dremio-cloud-security.md +1844 -0
- package/dremiodocs/dremio-cloud/sql-docs.md +7180 -0
- package/dremiodocs/dremio-software/dremio-software-acceleration.md +1575 -0
- package/dremiodocs/dremio-software/dremio-software-admin.md +884 -0
- package/dremiodocs/dremio-software/dremio-software-client-applications.md +3277 -0
- package/dremiodocs/dremio-software/dremio-software-data-products.md +560 -0
- package/dremiodocs/dremio-software/dremio-software-data-sources.md +8701 -0
- package/dremiodocs/dremio-software/dremio-software-deploy-dremio.md +3446 -0
- package/dremiodocs/dremio-software/dremio-software-get-started.md +848 -0
- package/dremiodocs/dremio-software/dremio-software-monitoring.md +422 -0
- package/dremiodocs/dremio-software/dremio-software-reference.md +677 -0
- package/dremiodocs/dremio-software/dremio-software-security.md +2074 -0
- package/dremiodocs/dremio-software/dremio-software-v25-api.md +32637 -0
- package/dremiodocs/dremio-software/dremio-software-v26-api.md +36757 -0
- package/jest.config.js +10 -0
- package/package.json +25 -0
- package/src/api/catalog.ts +74 -0
- package/src/api/jobs.ts +105 -0
- package/src/api/reflection.ts +77 -0
- package/src/api/source.ts +61 -0
- package/src/api/user.ts +32 -0
- package/src/client/base.ts +66 -0
- package/src/client/cloud.ts +37 -0
- package/src/client/software.ts +73 -0
- package/src/index.ts +16 -0
- package/src/types/catalog.ts +31 -0
- package/src/types/config.ts +18 -0
- package/src/types/job.ts +18 -0
- package/src/types/reflection.ts +29 -0
- package/tests/integration_manual.ts +95 -0
- package/tsconfig.json +19 -0
|
@@ -0,0 +1,560 @@
|
|
|
1
|
+
# Dremio Software - Data Products
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Source: https://docs.dremio.com/current/data-products/
|
|
8
|
+
|
|
9
|
+
Version: current [26.x]
|
|
10
|
+
|
|
11
|
+
On this page
|
|
12
|
+
|
|
13
|
+
# Build Data Products
|
|
14
|
+
|
|
15
|
+
A data product is a self-contained data asset that has been prepared, can be trusted, and has an SLA. It is meant to optimize for reuse, consistency, and sharing of data. Organizations have data products that are domain-specific and data products that are common at the organization level.
|
|
16
|
+
|
|
17
|
+
In Dremio, there are two levels of data products:
|
|
18
|
+
|
|
19
|
+
1. Data products that are created by curating and transforming one or more source tables into a view.
|
|
20
|
+
2. Data products that are created at the business or application layer and are used in reports, dashboards, and other applications.
|
|
21
|
+
|
|
22
|
+
Here are two examples of data products:
|
|
23
|
+
|
|
24
|
+
* Sam is a data analyst in the sales team, and he is asked to create a data product that provides a unified data asset that segments customers into different groups based on behavior, spending, and demographics. He creates a data product with a curated and transformed view of customer data from multiple source tables (e.g., customer demographics, purchase history, and engagement activity). The data in the data product is cleansed, transformed, enriched, certified, and structured for easy consumption by marketing and sales teams.
|
|
25
|
+
* Alice is an analyst in the finance team, and she needs to create an executive financial dashboard at the business or application layer for her company’s E-staff team. She creates a data product that powers a visualization report for financial performance metrics for her company, including revenue, expenses, and profitability over time. The data product integrates multiple data sources from various domains, including transactional and financial data, and presents the information in a format that is easily consumable by executives for decision-making.
|
|
26
|
+
|
|
27
|
+
## Data Product Lifecyle
|
|
28
|
+
|
|
29
|
+
Data products are created and managed through a lifecycle similar to the software development lifecycle. The data product lifecycle encompasses the end-to-end process of developing, deploying, and maintaining data products, ensuring they provide ongoing value to users. The image below depicts the phases of the lifecycle, which teams iterate through.
|
|
30
|
+
|
|
31
|
+

|
|
32
|
+
|
|
33
|
+
### Discover
|
|
34
|
+
|
|
35
|
+
To make data products reusable, they must be easily discoverable and explorable. Cross-functional or organization level data products must be accessible across various team to drive consistency. Users need to be able to quickly understand how to interpret the data in a data product and determine if it is relevant to the business problem at hand. For this to happen, data products must be published with adequate metadata such as domain, descriptions and definitions, tags, and usage information. To learn more about data discovery, see [Discover Data](/current/data-products/discover).
|
|
36
|
+
|
|
37
|
+
### Develop
|
|
38
|
+
|
|
39
|
+
Data products can be developed using SQL in Dremio's SQL Runner. You can just as easily use your IDE of choice for development. To learn more about developing data products, see [Develop Data Products](/current/data-products/develop).
|
|
40
|
+
|
|
41
|
+
### Deploy
|
|
42
|
+
|
|
43
|
+
To learn more about deploying your semantic layer with dbt, see [Deploy with dbt](/current/data-products/deploy-with-dbt).
|
|
44
|
+
|
|
45
|
+
### Govern
|
|
46
|
+
|
|
47
|
+
Effective data governance ensures secure, compliant, and transparent management of data by ensuring documentation and traceability, enforcing fine-grained access policies, and tracking dataset lineage to enhance data quality, minimize risks, and optimize value. To learn more about governance, see [Govern Data](/current/data-products/govern).
|
|
48
|
+
|
|
49
|
+
### Serve
|
|
50
|
+
|
|
51
|
+
Data products can be served in multiple forms. For example, data products are served in the Dremio console for adhoc analysis or they can be incorporated into a dashboard or report. This phase of the lifecycle is focused on delivering insights and data output to users ensuring accessibility and usability. For more information on the client applications that support connectivity to Dremio, see [Connecting Client Applications to Dremio](/current/client-applications).
|
|
52
|
+
|
|
53
|
+
### Observe
|
|
54
|
+
|
|
55
|
+
To fully empower users to manage their data products, they must be able to monitor them continuously to assess usage and performance over time. Observability enables data product owners to make iterative improvements based on their users' needs.
|
|
56
|
+
|
|
57
|
+
## Additional Resources
|
|
58
|
+
|
|
59
|
+
Find out more about data products by enrolling in the [Data Product Fundamentals course in Dremio University](https://university.dremio.com/course/data-product-fundamentals).
|
|
60
|
+
|
|
61
|
+
Was this page helpful?
|
|
62
|
+
|
|
63
|
+
[Previous
|
|
64
|
+
|
|
65
|
+
Clustering](/current/load-data/clustering)[Next
|
|
66
|
+
|
|
67
|
+
Discover Data](/current/data-products/discover/)
|
|
68
|
+
|
|
69
|
+
* Data Product Lifecyle
|
|
70
|
+
+ Discover
|
|
71
|
+
+ Develop
|
|
72
|
+
+ Deploy
|
|
73
|
+
+ Govern
|
|
74
|
+
+ Serve
|
|
75
|
+
+ Observe
|
|
76
|
+
* Additional Resources
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
# Source: https://docs.dremio.com/current/data-products/discover/
|
|
81
|
+
|
|
82
|
+
Version: current [26.x]
|
|
83
|
+
|
|
84
|
+
On this page
|
|
85
|
+
|
|
86
|
+
# Discover Data
|
|
87
|
+
|
|
88
|
+
Dremio simplifies the discovery of data objects and other Dremio entities.
|
|
89
|
+
|
|
90
|
+
Data discovery capabilities in Dremio include the following:
|
|
91
|
+
|
|
92
|
+
* Catalog exploration in the Dremio console
|
|
93
|
+
* AI-enabled semantic search for objects and entities
|
|
94
|
+
* Star objects
|
|
95
|
+
|
|
96
|
+
Data discovery capabilities are optimized for data that is governed by Dremio. Files and folders that have not been formatted as a table in Dremio may not be easily discoverable.
|
|
97
|
+
|
|
98
|
+
## Catalog exploration in the Dremio console
|
|
99
|
+
|
|
100
|
+
The Datasets page allows you to navigate through and explore objects that you have access to in Dremio. To learn more about the Datasets page, see [Quick Tour of the Datasets Page](/current/get-started/quick_tour#datasets-page). Once you have located the table of view that you are interested in, you can use the Details panel to learn more about it.
|
|
101
|
+
|
|
102
|
+
## AI-enabled semantic search for objects Enterprise
|
|
103
|
+
|
|
104
|
+
You can use the search bar in Dremio to find objects that are accessible through Dremio. This capability performs keyword and semantic search to find tables, views, and other objects that are related to your search criteria. Search only returns results that you have privileges to see. See [Searching for Dremio Objects](/current/data-products/discover/semantic-search).
|
|
105
|
+
|
|
106
|
+
## Object Metadata
|
|
107
|
+
|
|
108
|
+
Metadata such as the owner of the table or view, when it was created, and columns is easily accessible through metadata cards. Metadata cards can help you learn more about the object of interest and provide quick links to actions that you may want to take on the table or view. See [Metadata Cards](/current/data-products/discover/metadata) for more information.
|
|
109
|
+
|
|
110
|
+
## Star objects
|
|
111
|
+
|
|
112
|
+
You can star objects that you use frequently to provide easier discoverability and access. Starring objects such as sources, spaces, folders, tables, and views will surface them in the **Starred** tab of the SQL Runner. See [Star Objects](/current/data-products/discover/bookmarks) to learn more.
|
|
113
|
+
|
|
114
|
+
Was this page helpful?
|
|
115
|
+
|
|
116
|
+
[Previous
|
|
117
|
+
|
|
118
|
+
Build Data Products](/current/data-products/)[Next
|
|
119
|
+
|
|
120
|
+
Searching for Dremio Objects](/current/data-products/discover/semantic-search)
|
|
121
|
+
|
|
122
|
+
* Catalog exploration in the Dremio console
|
|
123
|
+
* AI-enabled semantic search for objects Enterprise
|
|
124
|
+
* Object Metadata
|
|
125
|
+
* Star objects
|
|
126
|
+
|
|
127
|
+
---
|
|
128
|
+
|
|
129
|
+
# Source: https://docs.dremio.com/current/data-products/develop
|
|
130
|
+
|
|
131
|
+
Version: current [26.x]
|
|
132
|
+
|
|
133
|
+
On this page
|
|
134
|
+
|
|
135
|
+
# Develop Data Products
|
|
136
|
+
|
|
137
|
+
You can curate and transform your data to create a data product by
|
|
138
|
+
|
|
139
|
+
* Writing SQL in the SQL Runner
|
|
140
|
+
* Using the low-code tranformation flows in the SQL Runner to help generate SQL
|
|
141
|
+
|
|
142
|
+
## Write SQL in the SQL Runner
|
|
143
|
+
|
|
144
|
+
You can use the SQL Runner to transform your data and create data products in Dremio.
|
|
145
|
+
|
|
146
|
+
* For a quick tour of the SQL Runner and the supported capabilities, see [Quick Tour of the SQL Runner](/current/get-started/quick_tour#sql-runner).
|
|
147
|
+
* See the [SQL Reference](/current/reference/sql) for functions and commands that you can use to transform and work with your data.
|
|
148
|
+
* You can also create data products using an IDE of your choice. Use [Arrow Flight JDBC](/current/client-applications/drivers/arrow-flight-sql-jdbc-driver) to establish a connection to Dremio from and IDE to do your development.
|
|
149
|
+
|
|
150
|
+
### Create a View
|
|
151
|
+
|
|
152
|
+
You can create a view from an existing table or view by transforming the data as required and performing the following steps:
|
|
153
|
+
|
|
154
|
+
1. Compose the query in the SQL Runner and click **Run** to validate the query. After running the query, click the arrow next to **Save Script As** in the top right of the SQL editor, and select **Save View as...** from the drop-down menu.
|
|
155
|
+
2. Name the new view and select where the view will be located. If the location is not updated, the view will get saved to your home space. Once the view is saved, you will be navigated to the Dataset page.
|
|
156
|
+
|
|
157
|
+
### Retrieve a View Definition
|
|
158
|
+
|
|
159
|
+
If you have the `SELECT` privilege for a view, you can see the underlying definition in the SQL editor but cannot edit the view directly. To see a view definition, on the Datasets page, hover over the line containing the view and click  on the right.
|
|
160
|
+
|
|
161
|
+
tip
|
|
162
|
+
|
|
163
|
+
If you have the required privileges, you can run `SHOW CREATE VIEW <view_name>` in the SQL editor to see the view definition. See [SHOW CREATE VIEW](/current/reference/sql/commands/show-create-view).
|
|
164
|
+
|
|
165
|
+
### Edit a View
|
|
166
|
+
|
|
167
|
+
If you have the privileges required to edit a view, you can see and edit the definition of the view.
|
|
168
|
+
|
|
169
|
+
To edit a view, perform the following steps:
|
|
170
|
+
|
|
171
|
+
1. On the Datasets page, hover over the line containing the view and click  on the right. The view definition will open.
|
|
172
|
+
2. Edit the view definition as needed and click **Run** to validate the query.
|
|
173
|
+
3. Click **Save View** in the top right corner of the SQL editor. This will overwrite the current definition of the view with the new definition.
|
|
174
|
+
|
|
175
|
+
### Delete a View
|
|
176
|
+
|
|
177
|
+
Perform the following steps to delete a view:
|
|
178
|
+
|
|
179
|
+
1. On the Datasets page, go to the folder or space where your view is located.
|
|
180
|
+
2. Hover over the line containing the view that you want to delete, click the ellipses (...) icon. From the list of actions, click **Delete**. Confirm that you want to delete the view.
|
|
181
|
+
|
|
182
|
+
caution
|
|
183
|
+
|
|
184
|
+
If you are deleting a table or view with children, you get a warning. Removing a table or view with children leaves you with disconnected views that you can no longer query.
|
|
185
|
+
|
|
186
|
+
## Use Low-Code to Transform Data in the SQL Runner
|
|
187
|
+
|
|
188
|
+
To begin a data transformation, via one of the following methods:
|
|
189
|
+
|
|
190
|
+
* Highlighting a portion or all of a field's value
|
|
191
|
+
* Using the dropdown menu for the transformation.
|
|
192
|
+
The dropdown is to the right of the field's name.
|
|
193
|
+
|
|
194
|
+
### Use Highlighting
|
|
195
|
+
|
|
196
|
+
The highlighting method is often the most intuitive method.
|
|
197
|
+
It provides enough context for Dremio to make some best guesses
|
|
198
|
+
about how to execute the transformation that you have in mind.
|
|
199
|
+
|
|
200
|
+
For instance, you could highlight a portion of a field that contains customer names to quickly perform an extract
|
|
201
|
+
that creates a new field with only last names.
|
|
202
|
+
|
|
203
|
+
**Suggestion Cards**
|
|
204
|
+
For transformations that are initiated by highlighting part or all of a field value,
|
|
205
|
+
Dremio uses a heuristic to determine a set of "suggestion cards" that represent its
|
|
206
|
+
best guesses as to your intended result.
|
|
207
|
+
|
|
208
|
+
You can click on one of these suggestion cards to inspect a preview of the new dataset and
|
|
209
|
+
confirm that it matches your expectations.
|
|
210
|
+
If no suggestion card is a perfect match, you can "flip" the card (by clicking the
|
|
211
|
+
pencil icon in the upper right corner) to tweak the
|
|
212
|
+
card's parameters before applying the transformation.
|
|
213
|
+
|
|
214
|
+
note
|
|
215
|
+
|
|
216
|
+
The highlight method is great for beginning an extract. However, in cases where other capabilities are required, the dropdown menu may be more useful.
|
|
217
|
+
|
|
218
|
+
### Use Dropdown Menus
|
|
219
|
+
|
|
220
|
+
The dropdown menu provides a more complete list of transformations that are applicable to the data type.
|
|
221
|
+
|
|
222
|
+
### Fix Inconsistent Data with a Join
|
|
223
|
+
|
|
224
|
+
In situations where the entries in a field are inconsistent
|
|
225
|
+
(for example, different spellings or abbreviations for the same name),
|
|
226
|
+
the following technique can be used to increase the quality of the dataset:
|
|
227
|
+
|
|
228
|
+
1. Identify the field with the problematic data. It may be useful to run this command in the SQL Editor:
|
|
229
|
+
|
|
230
|
+
Identify a field
|
|
231
|
+
|
|
232
|
+
```
|
|
233
|
+
SELECT DISTINCT myProblemFieldName FROM myDatasource.myTable
|
|
234
|
+
```
|
|
235
|
+
2. Download the results as CSV using the Download button.
|
|
236
|
+
3. Open the file in a text editor or Excel and create lookup values for the distinct values
|
|
237
|
+
from your table in a second column.
|
|
238
|
+
For example, standardizing variations in color names to a single canonical name.
|
|
239
|
+
4. Upload this file to your Home space on Dremio
|
|
240
|
+
5. Open this new dataset and hit the Join button located on the left above the field names
|
|
241
|
+
6. Select Custom Join and then the name of the inconsistent dataset you would like to fix, followed by Next
|
|
242
|
+
7. Drag over the name of the left column from your uploaded dataset, and match it with the name of the field you
|
|
243
|
+
would like to correct in the inconsistent dataset
|
|
244
|
+
8. Apply the Join then drop the old field, renaming the new one to take its place
|
|
245
|
+
9. Save the corrected dataset
|
|
246
|
+
|
|
247
|
+
### Clean Text
|
|
248
|
+
|
|
249
|
+
For text data, excess whitespace and changing capitalization schemes are two common data cleanliness issues.
|
|
250
|
+
Dremio provides two transformations for dealing with these possible inconsistencies:
|
|
251
|
+
**Trim Whitespace** and **Convert Case**.
|
|
252
|
+
|
|
253
|
+
### Handle Invalid, Empty and NULL Values
|
|
254
|
+
|
|
255
|
+
Empty or NULL text values are best eliminated by using Exclude.
|
|
256
|
+
You can initiate this transformation by:
|
|
257
|
+
|
|
258
|
+
1. Highlight a value from the field
|
|
259
|
+
that contains empty or NULL values, and
|
|
260
|
+
2. Select Exclude from the dropdown that appears.
|
|
261
|
+
This renders a list of the values in this field, and the frequency at which they occur.
|
|
262
|
+
3. Check the boxes next to the empty and/or NULL values you which to exclude from the dataset and click Apply.
|
|
263
|
+
|
|
264
|
+
### Work with Date Types
|
|
265
|
+
|
|
266
|
+
You can convert a text type field that contains date information into a proper date type field. This allows you to do more sophisticated analyses in external tools such as spotting by trends by month, year, or quarter.
|
|
267
|
+
|
|
268
|
+
You can begin this conversion by selecting 'Date & Time' from the type menu located to the left of the text field's name. In the subsequent dialog, select whether the output should be a time, date, date and time. It also gives a few default options for the format as well as a 'Custom' field for indicating a custom format. See [Data & Time Data Types](/current/reference/sql/data-types/#date--time-data-types) for more information on the conversions you can do.
|
|
269
|
+
|
|
270
|
+
Was this page helpful?
|
|
271
|
+
|
|
272
|
+
[Previous
|
|
273
|
+
|
|
274
|
+
Star Objects](/current/data-products/discover/bookmarks)[Next
|
|
275
|
+
|
|
276
|
+
Deploy with dbt](/current/data-products/deploy-with-dbt)
|
|
277
|
+
|
|
278
|
+
* Write SQL in the SQL Runner
|
|
279
|
+
+ Create a View
|
|
280
|
+
+ Retrieve a View Definition
|
|
281
|
+
+ Edit a View
|
|
282
|
+
+ Delete a View
|
|
283
|
+
* Use Low-Code to Transform Data in the SQL Runner
|
|
284
|
+
+ Use Highlighting
|
|
285
|
+
+ Use Dropdown Menus
|
|
286
|
+
+ Fix Inconsistent Data with a Join
|
|
287
|
+
+ Clean Text
|
|
288
|
+
+ Handle Invalid, Empty and NULL Values
|
|
289
|
+
+ Work with Date Types
|
|
290
|
+
|
|
291
|
+
---
|
|
292
|
+
|
|
293
|
+
# Source: https://docs.dremio.com/current/data-products/deploy-with-dbt
|
|
294
|
+
|
|
295
|
+
Version: current [26.x]
|
|
296
|
+
|
|
297
|
+
On this page
|
|
298
|
+
|
|
299
|
+
# Deploy with dbt
|
|
300
|
+
|
|
301
|
+
dbt enables analytics engineers to develop and manage semantic layers within dbt projects and deploy them to Dremio.
|
|
302
|
+
|
|
303
|
+
You can use Dremio's dbt connector `dbt-dremio` to transform data that is in data sources that are connected to a Dremio project.
|
|
304
|
+
|
|
305
|
+
## Prerequisites
|
|
306
|
+
|
|
307
|
+
* Download the `dbt-dremio` package from <https://github.com/dremio/dbt-dremio>.
|
|
308
|
+
* Ensure that Python 3.9.x or later is installed.
|
|
309
|
+
* Ensure that you are using Dremio Software version 22.0 or later.
|
|
310
|
+
* If you want to use TLS to secure the connection between dbt and Dremio Software, configure full wire encryption in your Dremio cluster. For more information, see the configuration of TLS for [Dremio on Kubernetes](/current/deploy-dremio/configuring-kubernetes/#transport-level-security) or [Dremio standalone clusters](/current/deploy-dremio/other-options/standalone/dremio-config/dremio-conf/wire-encryption-config/#full-wire-encryption-enterprise).
|
|
311
|
+
|
|
312
|
+
## Installing
|
|
313
|
+
|
|
314
|
+
Install this package from PyPi by running this command:
|
|
315
|
+
|
|
316
|
+
Install dbt-dremio package
|
|
317
|
+
|
|
318
|
+
```
|
|
319
|
+
pip install dbt-dremio
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
note
|
|
323
|
+
|
|
324
|
+
`dbt-dremio` works with dbt-core versions 1.8 and 1.9. Earlier versions of dbt-core are out of support from dbt.
|
|
325
|
+
|
|
326
|
+
## Initializing a dbt Project
|
|
327
|
+
|
|
328
|
+
1. Run the command `dbt init <project_name>`.
|
|
329
|
+
2. Select `dremio` as the database to use.
|
|
330
|
+
3. Select one of these options to generate a profile for your project:
|
|
331
|
+
* `software_with_username_password` for working with a Dremio Software cluster and authenticating to the cluster with a username and a password
|
|
332
|
+
* `software_with_pat` for working with a Dremio Software cluster and authenticating to the cluster with a personal access token
|
|
333
|
+
|
|
334
|
+
Next, configure the profile for your dbt project.
|
|
335
|
+
|
|
336
|
+
## Profiles
|
|
337
|
+
|
|
338
|
+
When you initialize a dbt project, you create one of these three profiles. You must configure it before trying to connect to Dremio Cloud or Dremio Software.
|
|
339
|
+
|
|
340
|
+
* Profile for Dremio Software with Username/Password Authentication
|
|
341
|
+
* Profile for Dremio Software with Authentication Through a Personal Access Token
|
|
342
|
+
|
|
343
|
+
For descriptions of the configurations in these profiles, see Configurations.
|
|
344
|
+
|
|
345
|
+
### Dremio Software Profile with Username & Password
|
|
346
|
+
|
|
347
|
+
Example Profile
|
|
348
|
+
|
|
349
|
+
```
|
|
350
|
+
[project name]:
|
|
351
|
+
outputs:
|
|
352
|
+
dev:
|
|
353
|
+
password: b9JtkIgI3uup9gGxxK
|
|
354
|
+
port: 9047
|
|
355
|
+
software_host: 192.0.2.0
|
|
356
|
+
object_storage_source: Samples
|
|
357
|
+
object_storage_path: "samples.dremio.com"."Dremio University"
|
|
358
|
+
dremio_space: Space1
|
|
359
|
+
dremio_space_folder: Folder1.Folder2
|
|
360
|
+
threads: 1
|
|
361
|
+
type: dremio
|
|
362
|
+
use_ssl: true
|
|
363
|
+
user: userName
|
|
364
|
+
target: dev
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
### Dremio Software Profile with Personal Access Token
|
|
368
|
+
|
|
369
|
+
Example Profile
|
|
370
|
+
|
|
371
|
+
```
|
|
372
|
+
[project name]:
|
|
373
|
+
outputs:
|
|
374
|
+
dev:
|
|
375
|
+
pat: A1BCDrE2FwgH3IJkLM4NoPqrsT5uV6WXyza7I8bcDEFgJ9hIj0Kl1MNOPq2Rstu
|
|
376
|
+
port: 9047
|
|
377
|
+
software_host: 192.0.2.0
|
|
378
|
+
object_storage_source: Samples
|
|
379
|
+
object_storage_path: "samples.dremio.com"."Dremio University"
|
|
380
|
+
dremio_space: Space1
|
|
381
|
+
dremio_space_folder: Folder1.Folder2
|
|
382
|
+
threads: 1
|
|
383
|
+
type: dremio
|
|
384
|
+
use_ssl: true
|
|
385
|
+
user: userName
|
|
386
|
+
target: dev
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
## Configurations
|
|
390
|
+
|
|
391
|
+
| Configuration | Required? | Default Value | Description |
|
|
392
|
+
| --- | --- | --- | --- |
|
|
393
|
+
| `password` | Yes, if you are not using the pat configuration. | None | The password of the account to use when logging into the Dremio cluster. |
|
|
394
|
+
| `pat` | Yes, if you are not using the user and password configurations. | None | The personal access token to use for authenticating to Dremio. See [Personal Access Tokens](/current/security/authentication/personal-access-tokens/) for instructions about obtaining a token. The use of a personal access token takes precedence if values for the three configurations user, password and pat are specified. |
|
|
395
|
+
| `port` | Yes | `9047` | Port for Dremio Software cluster API endpoints. |
|
|
396
|
+
| `software_host` | Yes | None | The hostname or IP address of the coordinator node of the Dremio cluster. |
|
|
397
|
+
| `enterprise_catalog_namespace` | No | None | The name of the catalog in which to create tables, materialized views, tests, and other objects, and views. The dbt aliases are `datalake` (for objects) and `database` (for views). This name corresponds to the name of a catalog in the **Open Catalogs** section of the Datasets page in Dremio. |
|
|
398
|
+
| `enterprise_catalog_folder` | No | None | The path in the catalog in which to create objects / views. The dbt aliases are `root_path` (for objects) and `schema` (for views). Nested folders in the path are separated with periods. This value corresponds to the path in this location in the Datasets page in Dremio. |
|
|
399
|
+
| `object_storage_source` | No | $scratch | The name of the filesystem in which to create tables, materialized views, tests, and other objects. The dbt alias is `datalake`. This name corresponds to the name of a source in the **Object Storage** section of the Datasets page in Dremio: |
|
|
400
|
+
| `object_storage_path` | No | `no_schema` | The path in the filesystem in which to create objects. The default is the root level of the filesystem. The dbt alias is `root_path`. Nested folders in the path are separated with periods. This value corresponds to the path in this location in the Datasets page in Dremio: 'samples.dremio.com'.'Dremio University' |
|
|
401
|
+
| `dremio_space` | No | `@<username>` | The value of the Dremio space in which to create views. The dbt alias is `database`. This value corresponds to the name in this location in the **Spaces** section of the Datasets page in Dremio: Spaces1 |
|
|
402
|
+
| `dremio_space_folder` | No | `no_schema` | The folder in the Dremio space in which to create views. The default is the top level in the space. The dbt alias is `schema`. Nested folders are separated with periods. This value corresponds to the path in this location in the Datasets page in Dremio: |
|
|
403
|
+
| `threads` | Yes | 1 | The number of threads the dbt project runs on. |
|
|
404
|
+
| `type` | Yes | dremio | Auto-populated when creating a Dremio project. Do not change this value. |
|
|
405
|
+
| `use_ssl` | Yes | `true` | Acceptable values are `true` and `false`. If the value is set to true, ensure that full wire encryption is configured in your Dremio cluster. See [Prerequisites](/current/data-products/deploy-with-dbt#prerequisites). |
|
|
406
|
+
| `verify_ssl` | No | `true` | Acceptable values are `true` and `false`. Set to `false` if using a self-signed certificate or if the root certificate authority (CA) is not included in Python’s CA certificates. |
|
|
407
|
+
| `user` | Yes | None | The username of the account to use when logging into the Dremio cluster. |
|
|
408
|
+
|
|
409
|
+
## Known Issues
|
|
410
|
+
|
|
411
|
+
[Model contracts](https://docs.getdbt.com/docs/collaborate/govern/model-contracts) are not supported.
|
|
412
|
+
|
|
413
|
+
## Additional Resources
|
|
414
|
+
|
|
415
|
+
Learn more about DataOps by enrolling in the [DataOps with Apache Iceberg course in Dremio University](https://university.dremio.com/course/dataops-with-apache-iceberg).
|
|
416
|
+
|
|
417
|
+
Was this page helpful?
|
|
418
|
+
|
|
419
|
+
[Previous
|
|
420
|
+
|
|
421
|
+
Develop Data Products](/current/data-products/develop)[Next
|
|
422
|
+
|
|
423
|
+
Govern Data](/current/data-products/govern/)
|
|
424
|
+
|
|
425
|
+
* Prerequisites
|
|
426
|
+
* Installing
|
|
427
|
+
* Initializing a dbt Project
|
|
428
|
+
* Profiles
|
|
429
|
+
+ Dremio Software Profile with Username & Password
|
|
430
|
+
+ Dremio Software Profile with Personal Access Token
|
|
431
|
+
* Configurations
|
|
432
|
+
* Known Issues
|
|
433
|
+
* Additional Resources
|
|
434
|
+
|
|
435
|
+
---
|
|
436
|
+
|
|
437
|
+
# Source: https://docs.dremio.com/current/data-products/govern/
|
|
438
|
+
|
|
439
|
+
Version: current [26.x]
|
|
440
|
+
|
|
441
|
+
On this page
|
|
442
|
+
|
|
443
|
+
# Govern Data
|
|
444
|
+
|
|
445
|
+
Effective data governance is essential for managing secure, compliant, transparent, and traceable data products. In addition to securing your data using role-based access control (RBAC), data governance for data products allows you to apply fine-grained access controls (row access & column-masking policies) on your data, as well as trace the lineage of any dataset in the lakehouse. Together, these governance tools help organizations optimize the value they derive from their data to enhance data quality and transparency, while minimizing the risks associated with data misuse and non-compliance.
|
|
446
|
+
Dremio also provides tools for describing, identifying, and displaying datasets using wikis and tags.
|
|
447
|
+
|
|
448
|
+
## Row-Access and Column-Masking Policies
|
|
449
|
+
|
|
450
|
+
[Row access & column-masking (or fine-grained access control) policies](/current/data-products/govern/row-column-policies-udf) provide mechanisms to enforce data privacy and security rules directly on your data.
|
|
451
|
+
|
|
452
|
+
## Lineage
|
|
453
|
+
|
|
454
|
+
[Data lineage](/current/data-products/govern/lineage) allows you to track and visualize data as it moves through the various stages of a data pipeline. It provides clarity on where the data comes from, how it is transformed, and where it is used.
|
|
455
|
+
|
|
456
|
+
## Wikis and Tags
|
|
457
|
+
|
|
458
|
+
[Wikis](/current/data-products/govern/wikis-tags) for datasets allow users to document and describe datasets in the Open Catalog using a rich text editor with [Github-flavored markdown](https://github.github.com/gfm/).
|
|
459
|
+
|
|
460
|
+
[Tags](/current/data-products/govern/wikis-tags) for datasets help organize and retrieve data efficiently by allowing users to search, filter, and locate datasets through specific tag associations.
|
|
461
|
+
|
|
462
|
+
Was this page helpful?
|
|
463
|
+
|
|
464
|
+
[Previous
|
|
465
|
+
|
|
466
|
+
Deploy with dbt](/current/data-products/deploy-with-dbt)[Next
|
|
467
|
+
|
|
468
|
+
Row-Access and Column-Masking Policies](/current/data-products/govern/row-column-policies-udf)
|
|
469
|
+
|
|
470
|
+
* Row-Access and Column-Masking Policies
|
|
471
|
+
* Lineage
|
|
472
|
+
* Wikis and Tags
|
|
473
|
+
|
|
474
|
+
---
|
|
475
|
+
|
|
476
|
+
# Source: https://docs.dremio.com/current/data-products/discover
|
|
477
|
+
|
|
478
|
+
Version: current [26.x]
|
|
479
|
+
|
|
480
|
+
On this page
|
|
481
|
+
|
|
482
|
+
# Discover Data
|
|
483
|
+
|
|
484
|
+
Dremio simplifies the discovery of data objects and other Dremio entities.
|
|
485
|
+
|
|
486
|
+
Data discovery capabilities in Dremio include the following:
|
|
487
|
+
|
|
488
|
+
* Catalog exploration in the Dremio console
|
|
489
|
+
* AI-enabled semantic search for objects and entities
|
|
490
|
+
* Star objects
|
|
491
|
+
|
|
492
|
+
Data discovery capabilities are optimized for data that is governed by Dremio. Files and folders that have not been formatted as a table in Dremio may not be easily discoverable.
|
|
493
|
+
|
|
494
|
+
## Catalog exploration in the Dremio console
|
|
495
|
+
|
|
496
|
+
The Datasets page allows you to navigate through and explore objects that you have access to in Dremio. To learn more about the Datasets page, see [Quick Tour of the Datasets Page](/current/get-started/quick_tour#datasets-page). Once you have located the table of view that you are interested in, you can use the Details panel to learn more about it.
|
|
497
|
+
|
|
498
|
+
## AI-enabled semantic search for objects Enterprise
|
|
499
|
+
|
|
500
|
+
You can use the search bar in Dremio to find objects that are accessible through Dremio. This capability performs keyword and semantic search to find tables, views, and other objects that are related to your search criteria. Search only returns results that you have privileges to see. See [Searching for Dremio Objects](/current/data-products/discover/semantic-search).
|
|
501
|
+
|
|
502
|
+
## Object Metadata
|
|
503
|
+
|
|
504
|
+
Metadata such as the owner of the table or view, when it was created, and columns is easily accessible through metadata cards. Metadata cards can help you learn more about the object of interest and provide quick links to actions that you may want to take on the table or view. See [Metadata Cards](/current/data-products/discover/metadata) for more information.
|
|
505
|
+
|
|
506
|
+
## Star objects
|
|
507
|
+
|
|
508
|
+
You can star objects that you use frequently to provide easier discoverability and access. Starring objects such as sources, spaces, folders, tables, and views will surface them in the **Starred** tab of the SQL Runner. See [Star Objects](/current/data-products/discover/bookmarks) to learn more.
|
|
509
|
+
|
|
510
|
+
Was this page helpful?
|
|
511
|
+
|
|
512
|
+
[Previous
|
|
513
|
+
|
|
514
|
+
Build Data Products](/current/data-products/)[Next
|
|
515
|
+
|
|
516
|
+
Searching for Dremio Objects](/current/data-products/discover/semantic-search)
|
|
517
|
+
|
|
518
|
+
* Catalog exploration in the Dremio console
|
|
519
|
+
* AI-enabled semantic search for objects Enterprise
|
|
520
|
+
* Object Metadata
|
|
521
|
+
* Star objects
|
|
522
|
+
|
|
523
|
+
---
|
|
524
|
+
|
|
525
|
+
# Source: https://docs.dremio.com/current/data-products/govern
|
|
526
|
+
|
|
527
|
+
Version: current [26.x]
|
|
528
|
+
|
|
529
|
+
On this page
|
|
530
|
+
|
|
531
|
+
# Govern Data
|
|
532
|
+
|
|
533
|
+
Effective data governance is essential for managing secure, compliant, transparent, and traceable data products. In addition to securing your data using role-based access control (RBAC), data governance for data products allows you to apply fine-grained access controls (row access & column-masking policies) on your data, as well as trace the lineage of any dataset in the lakehouse. Together, these governance tools help organizations optimize the value they derive from their data to enhance data quality and transparency, while minimizing the risks associated with data misuse and non-compliance.
|
|
534
|
+
Dremio also provides tools for describing, identifying, and displaying datasets using wikis and tags.
|
|
535
|
+
|
|
536
|
+
## Row-Access and Column-Masking Policies
|
|
537
|
+
|
|
538
|
+
[Row access & column-masking (or fine-grained access control) policies](/current/data-products/govern/row-column-policies-udf) provide mechanisms to enforce data privacy and security rules directly on your data.
|
|
539
|
+
|
|
540
|
+
## Lineage
|
|
541
|
+
|
|
542
|
+
[Data lineage](/current/data-products/govern/lineage) allows you to track and visualize data as it moves through the various stages of a data pipeline. It provides clarity on where the data comes from, how it is transformed, and where it is used.
|
|
543
|
+
|
|
544
|
+
## Wikis and Tags
|
|
545
|
+
|
|
546
|
+
[Wikis](/current/data-products/govern/wikis-tags) for datasets allow users to document and describe datasets in the Open Catalog using a rich text editor with [Github-flavored markdown](https://github.github.com/gfm/).
|
|
547
|
+
|
|
548
|
+
[Tags](/current/data-products/govern/wikis-tags) for datasets help organize and retrieve data efficiently by allowing users to search, filter, and locate datasets through specific tag associations.
|
|
549
|
+
|
|
550
|
+
Was this page helpful?
|
|
551
|
+
|
|
552
|
+
[Previous
|
|
553
|
+
|
|
554
|
+
Deploy with dbt](/current/data-products/deploy-with-dbt)[Next
|
|
555
|
+
|
|
556
|
+
Row-Access and Column-Masking Policies](/current/data-products/govern/row-column-policies-udf)
|
|
557
|
+
|
|
558
|
+
* Row-Access and Column-Masking Policies
|
|
559
|
+
* Lineage
|
|
560
|
+
* Wikis and Tags
|