dremiojs 1.0.0 → 1.2.0

package/README.md

@@ -10,6 +10,49 @@ npm install dremiojs
 
 ## Usage
 
+The recommended way to use `dremiojs` is by configuring your connection in `~/.dremio/profiles.yaml` (compatible with `dremio-cli`).
+
+### 1. Setup Profile
+Create `~/.dremio/profiles.yaml`:
+
+```yaml
+profiles:
+  cloud:
+    type: cloud
+    base_url: https://api.dremio.cloud
+    project_id: <PROJECT_ID>
+    auth:
+      type: pat
+      token: <PAT>
+  local:
+    type: software
+    base_url: http://localhost:9047
+    auth:
+      type: username_password
+      username: admin
+      password: password1
+default_profile: cloud
+```
+
+### 2. Connect & Query
+
+```typescript
+import { Dremio } from 'dremiojs';
+
+async function main() {
+    // Automatically loads 'default' profile from yaml
+    const client = await Dremio.fromProfile();
+    
+    // Execute SQL
+    const results = await client.jobs.executeQuery('SELECT 1');
+    console.log(results);
+}
+```
+
+## Advanced Usage (Explicit Configuration)
+
+You can also configure clients manually without a profile file.
+
 ### Dremio Cloud
 
 ```typescript
@@ -20,16 +63,6 @@ const client = new DremioCloudClient({
     authToken: 'YOUR_PAT',
     projectId: 'YOUR_PROJECT_ID'
 });
-
-async function main() {
-    // List catalog
-    const catalog = await client.catalog.getByPath([]);
-    console.log(catalog.children);
-    
-    // Execute SQL
-    const results = await client.jobs.executeQuery('SELECT * FROM sys.version');
-    console.log(results);
-}
 ```
 
 ### Dremio Software
@@ -37,23 +70,40 @@ async function main() {
 ```typescript
 import { DremioSoftwareClient } from 'dremiojs';
 
+// OAuth (Client Credentials)
 const client = new DremioSoftwareClient({
-    baseUrl: 'http://dremio:9047/api/v3',
-    authToken: 'YOUR_PAT'
-    // OR username/password:
-    // username: 'dremio',
-    // password: 'password123'
+    baseUrl: 'https://dremio.local/api/v3',
+    auth: {
+        type: 'oauth',
+        client_id: 'service-id',
+        client_secret: 'secret'
+    }
 });
-
-async function main() {
-    const results = await client.jobs.executeQuery('SELECT 1');
-}
 ```
 
 ## Features
 
+- **Profile Support**: Seamless `profiles.yaml` integration for Cloud/Software.
+- **Advanced Auth**: Support for PAT Exchange and OAuth Client Credentials.
 - **Universal Support**: Works with both Dremio Cloud and Dremio Software.
 - **Strict Typing**: Full TypeScript definitions for Catalog, Jobs, Reflections, etc.
 - **SQL Helper**: Easy `executeQuery` method that handles job submission and polling.
 - **Catalog Management**: Create, update, delete Spaces, Sources, Folders, and Datasets.
 - **Reflections**: Manage reflections programmatically.
+
+- [Configuration & Environment Variables](docs/configuration.md)
+- [Client Usage](docs/client.md)
+- [Catalog API](docs/catalog.md)
+## Documentation
+
+- [Configuration & Environment Variables](docs/configuration.md)
+- [Client Usage](docs/client.md)
+- [Catalog API](docs/catalog.md)
+- [SQL & Jobs API](docs/sql_and_jobs.md)
+- [Reflections API](docs/reflections.md)
+- [Sources API](docs/sources.md)
+- [Users API](docs/users.md)
+- [Governance & Collaboration](docs/governance_collaboration.md)
+- [Scripts, Lineage & Tokens](docs/scripts_lineage_tokens.md)
+- [Provisioning (Engines)](docs/provisioning.md)
+- [Service Users & Credentials](docs/service_users.md)

package/docs/catalog.md

@@ -0,0 +1,76 @@
+# Catalog API Guide
+
+The Catalog API is the primary way to interact with Dremio's semantic layer (Spaces, Folders, Virtual Datasets) and physical sources.
+
+## Accessing the Catalog
+
+### Listing the Root
+To list the top-level items (Spaces and Sources):
+
+```typescript
+const root = await client.catalog.getByPath([]);
+console.log(root.children);
+```
+
+### Getting Items by Path
+You can traverse the catalog hierarchy using an array of path segments.
+
+```typescript
+// Get a specific dataset: "MarketingSpace.Campaigns.Q1_Results"
+const dataset = await client.catalog.getByPath(['MarketingSpace', 'Campaigns', 'Q1_Results']);
+console.log(dataset.id, dataset.type);
+```
+
+### Getting Items by ID
+If you already have the immutable ID of an object (e.g., from a search or list operation):
+
+```typescript
+const item = await client.catalog.get('dremio-dataset-uuid-1234');
+```
+
+## Creating Items
+
+### Creating a Space
+```typescript
+await client.catalog.create({
+    type: 'SPACE',
+    name: 'NewMarketingSpace'
+});
+```
+
+### Creating a Folder
+```typescript
+await client.catalog.create({
+    type: 'FOLDER',
+    path: ['NewMarketingSpace', 'Folder A']
+});
+```
+
+### Creating a Virtual Dataset (View)
+To create a VDS, you must provide the SQL definition and the context (path).
+
+```typescript
+await client.catalog.create({
+    type: 'DATASET',
+    path: ['NewMarketingSpace', 'MyView'],
+    sql: 'SELECT * FROM "Samples"."samples.dremio.com"."SF_incidents"',
+    format: { type: 'VIRTUAL_DATASET' } // Required specific to API v3/Cloud
+});
+```
+
+## Updating Items
+
+```typescript
+// Rename or Move (check API specifics for move vs update)
+await client.catalog.update('item-id', {
+    ...item,
+    tag: item.tag, // Optimistic locking tag is often required
+    sql: 'SELECT * FROM new_table' // Updating SQL of a view
+});
+```
+
+## Deleting Items
+
+```typescript
+await client.catalog.delete('item-id');
+```

package/docs/client.md

@@ -0,0 +1,55 @@
+# Client Instance Guide
+
+The library provides two distinct clients to handle the subtle differences between Dremio Cloud and Dremio Software APIs. Both clients share a common interface for most resources, making it easy to switch between them or support hybrid deployments.
+
+## Dremio Cloud Client
+
+The `DremioCloudClient` is optimized for Dremio Cloud's multi-tenant architecture. It automatically handles the `projectId` context required for most API calls.
+
+```typescript
+import { DremioCloudClient } from 'dremiojs';
+
+const cloudClient = new DremioCloudClient({
+    baseUrl: 'https://api.dremio.cloud/v0',
+    authToken: process.env.DREMIO_CLOUD_TOKEN,
+    projectId: process.env.DREMIO_CLOUD_PROJECT_ID
+});
+
+// The client automatically appends /projects/{id} to relevant paths
+await cloudClient.catalog.getByPath(['space', 'dataset']);
+```
+
+## Dremio Software Client
+
+The `DremioSoftwareClient` supports standard Dremio Software deployments (Docker, K8s, RPM). It includes logic for:
+- API v2 Login (username/password) exchange for tokens.
+- Automatic token management (if using username/pass).
+- Standard v3 API usage.
+
+```typescript
+import { DremioSoftwareClient } from 'dremiojs';
+
+const softwareClient = new DremioSoftwareClient({
+    baseUrl: 'http://localhost:9047/api/v3',
+    authToken: process.env.DREMIO_PAT 
+});
+
+// Or using Login
+const legacyClient = new DremioSoftwareClient({
+    baseUrl: 'http://localhost:9047/api/v3',
+    username: 'admin',
+    password: 'password123'
+});
+
+// The client performs login on the first request if no token is present
+await legacyClient.jobs.executeQuery('SELECT 1'); 
+```
+
+## Shared Capabilities
+
+Both clients extend `BaseClient` and expose:
+- `catalog`: Manage datasets, spaces, folders.
+- `jobs`: Execute SQL and track jobs.
+- `reflections`: specific Reflection management.
+- `sources`: specific Source management.
+- `users`: User and Role lookups.

package/docs/configuration.md

@@ -0,0 +1,150 @@
+# Configuration & Authentication
+
+`dremiojs` offers a flexible configuration system that supports:
+1.  **Profiles (New):** Automatically load configuration from `~/.dremio/profiles.yaml`.
+2.  **Explicit Config:** Pass configuration objects directly to the client constructor.
+3.  **Environment Variables:** Fallback support for standard environment variables.
+
+## 1. Using Profiles (Recommended)
+
+The easiest way to specificy connection details is using the `Dremio` factory, which reads from your local `~/.dremio/profiles.yaml` file (compatible with `dremio-cli` and `dremio-simple-query`).
+
+### Installation
+Ensure you have a profiles file at `~/.dremio/profiles.yaml`.
+
+```typescript
+import { Dremio } from 'dremiojs';
+
+async function main() {
+    // 1. Load the 'default' profile
+    const client = await Dremio.fromProfile();
+    
+    // 2. Load a specific named profile (e.g. 'prod')
+    const prodClient = await Dremio.fromProfile('prod');
+    
+    const results = await client.jobs.executeQuery('SELECT 1');
+    console.log(results);
+}
+```
+
+### Profile Reference (`profiles.yaml`)
+
+Below is a complete reference of supported profile types.
+
+```yaml
+profiles:
+  # --- Dremio Cloud (PAT Auth) ---
+  cloud:
+    type: cloud
+    # Base URL (defaults to https://api.dremio.cloud)
+    base_url: https://api.dremio.cloud
+    project_id: <YOUR_PROJECT_ID>
+    auth:
+      type: pat
+      token: <YOUR_PAT_TOKEN>
+
+  # --- Dremio Software (PAT Auth) ---
+  software_pat:
+    type: software
+    base_url: https://dremio.company.com
+    auth:
+      type: pat
+      token: <YOUR_PAT_TOKEN>
+
+  # --- Dremio Software (Username/Password) ---
+  # Useful for local development or legacy auth
+  local:
+    type: software
+    base_url: http://localhost:9047
+    ssl: 'false' # Disable SSL verification for localhost
+    auth:
+      type: username_password
+      username: admin
+      password: password123
+
+  # --- Dremio Software (OAuth 2.0 / Service Account) ---
+  # Uses Client Credentials Grant
+  service_account:
+    type: software
+    base_url: https://dremio.company.com
+    auth:
+      type: oauth
+      client_id: <CLIENT_ID>
+      client_secret: <CLIENT_SECRET>
+      # Scope is optional, defaults to implicit or 'dremio.all' on retry
+      scope: dremio.all 
+
+default_profile: cloud
+```
+
+## 2. Explicit Configuration
+
+You can also instantiate clients manually, which is useful for web applications or when configuration is injected at runtime.
+
+### Dremio Cloud Client
+
+```typescript
+import { DremioCloudClient } from 'dremiojs';
+
+const client = new DremioCloudClient({
+    projectId: 'your-project-id',
+    authToken: 'your-pat', // Can be a PAT or a Session Token
+    baseUrl: 'https://api.dremio.cloud/v0', // Optional
+    // Or use the auth object
+    auth: {
+        type: 'pat',
+        token: 'your-pat'
+    }
+});
+```
+
+### Dremio Software Client
+
+```typescript
+import { DremioSoftwareClient } from 'dremiojs';
+
+// PAT Auth
+const client = new DremioSoftwareClient({
+    baseUrl: 'http://localhost:9047/api/v3',
+    authToken: 'your-pat'
+});
+
+// Username/Password
+const client = new DremioSoftwareClient({
+    baseUrl: 'http://localhost:9047/api/v3',
+    auth: {
+        type: 'username_password',
+        username: 'admin',
+        password: 'password123'
+    }
+});
+
+// OAuth (Client Credentials)
+const client = new DremioSoftwareClient({
+    baseUrl: 'https://dremio.example.com/api/v3',
+    auth: {
+        type: 'oauth',
+        client_id: '...',
+        client_secret: '...'
+    }
+});
+```
+
+## 3. Environment Variables (Fallback)
+
+If no profiles are found, `ConfigLoader` checks for these legacy environment variables:
+
+- **Cloud**: `DREMIO_CLOUD_TOKEN`, `DREMIO_CLOUD_PROJECTID`, `DREMIO_CLOUD_BASE_URL`
+- **Software**: `DREMIO_SOFTWARE_TOKEN`, `DREMIO_SOFTWARE_BASE_URL`, `DREMIO_USER`, `DREMIO_PASSWORD`
+
+## SSL/TLS Configuration
+
+For self-signed certificates (common in local software deployments), set `checkSSLCerts: false`. In `profiles.yaml`, set `ssl: 'false'`.
+
+```typescript
+const client = new DremioSoftwareClient({
+    baseUrl: 'https://localhost:9047',
+    checkSSLCerts: false 
+});
+```
+

package/docs/governance_collaboration.md

@@ -0,0 +1,41 @@
+# Governance & Collaboration API Guide
+
+## Governance (Grants)
+
+Manage access control lists (ACLs) and grants for Catalog entities.
+
+### Getting Grants
+```typescript
+const grants = await client.governance.getGrants('entity-id');
+// Output: [{ granteeType: 'USER', id: '...', privileges: ['SELECT'] }]
+```
+
+### Updating Grants
+```typescript
+await client.governance.updateGrants('entity-id', [
+    { granteeType: 'ROLE', id: 'role-id', privileges: ['SELECT', 'ALTER'] }
+]);
+```
+
+## Collaboration (Tags & Wiki)
+
+Manage metadata that helps users discover and understand data.
+
+### Tags
+```typescript
+// Get Tags
+const tags = await client.collaboration.getTags('dataset-id');
+
+// Set Tags
+await client.collaboration.setTags('dataset-id', ['marketing', 'q1-2024']);
+```
+
+### Wiki
+```typescript
+// Get Wiki Markdown
+const wiki = await client.collaboration.getWiki('dataset-id');
+console.log(wiki.text);
+
+// Set Wiki
+await client.collaboration.setWiki('dataset-id', '# Sales Data\n\nThis dataset contains...');
+```

package/docs/provisioning.md

@@ -0,0 +1,15 @@
+# Provisioning API Guide
+
+Manage Dremio Engines (Cloud) or Queues (Software WLM).
+
+## Engines (Dremio Cloud)
+
+```typescript
+// List available engines
+const engines = await client.provisioning.listEngines();
+engines.forEach(engine => {
+    console.log(`Engine: ${engine.name} (${engine.size}) - ${engine.status}`);
+});
+```
+
+*Note: Software WLM Queue management is structurally different and may be added to this namespace in future updates.*

package/docs/reflections.md

@@ -0,0 +1,54 @@
+# Reflections API Guide
+
+Reflections are Dremio's query acceleration technology. This API allows you to manage raw and aggregation reflections programmatically.
+
+## Listing Reflections
+
+You can list all reflections or filter by dataset.
+
+```typescript
+// List all accessible reflections
+const allReflections = await client.reflections.list();
+
+// List reflections for a specific dataset
+const datasetReflections = await client.reflections.list({
+    datasetId: 'dataset-uuid-123'
+});
+```
+
+## Getting a Reflection
+
+```typescript
+const reflection = await client.reflections.get('reflection-uuid');
+console.log(reflection.name, reflection.status);
+```
+
+## Creating a Reflection
+
+Creating a reflection implies defining which dataset it accelerates and its configuration (dimensions, measures, etc.).
+
+```typescript
+await client.reflections.create({
+    type: 'RAW',
+    name: 'Raw_Acceleration',
+    datasetId: 'dataset-uuid-123',
+    enabled: true,
+    displayFieldIds: ['col1', 'col2'] // Fields to include
+});
+```
+
+## Updating a Reflection
+
+Commonly used to enable/disable reflections or update their definition.
+
+```typescript
+await client.reflections.update('reflection-uuid', {
+    enabled: false // Disable reflection
+});
+```
+
+## Deleting a Reflection
+
+```typescript
+await client.reflections.delete('reflection-uuid');
+```

package/docs/scripts_lineage_tokens.md

@@ -0,0 +1,45 @@
+# Scripts, Lineage & Tokens Guide
+
+## Scripts API
+
+Manage saved SQL scripts (Dremio Cloud & Software).
+
+```typescript
+// List Scripts
+const scripts = await client.scripts.list({ maxResults: 10 });
+
+// Get a Script
+const myScript = await client.scripts.get('script-id');
+
+// Create a Script
+const newScript = await client.scripts.create({
+    name: 'Daily Report',
+    content: 'SELECT count(*) FROM sales',
+    context: ['Marketing']
+});
+
+// Update
+await client.scripts.update('script-id', { content: 'SELECT * FROM sales' });
+
+// Delete
+await client.scripts.delete('script-id');
+```
+
+## Lineage API
+
+Explore the data lineage graph for a dataset.
+
+```typescript
+const graph = await client.lineage.get('dataset-id');
+console.log(graph); 
+// Returns nodes and edges representing parent/child relationships
+```
+
+## Token Management
+
+Programmatically generate Personal Access Tokens (PATs) (where supported).
+
+```typescript
+const tokenResponse = await client.tokens.createToken('user-id', 'My New Token');
+console.log('New PAT:', tokenResponse.token);
+```

package/docs/service_users.md

@@ -0,0 +1,77 @@
+# Service Users & Credentials Guide
+
+Managing non-human accounts (Service Users) is essential for CI/CD and automated workflows. dremiojs provides full lifecycle management for Service Users and their OAuth credentials.
+
+> **Note**: Service Users are primarily a **Dremio Enterprise** feature. Calls to these endpoints may fail on Community Edition.
+
+## Service User Lifecycle
+
+### 1. Create a Service User
+
+```typescript
+const serviceUser = await client.users.create({
+    name: 'ci_pipeline_bot',
+    identityType: 'SERVICE_USER',
+    active: true
+});
+console.log('Service User Created:', serviceUser.id);
+```
+
+### 2. Create Client Credentials (Secret)
+
+Generate a Client ID and Secret for this user. This secret is only shown ONCE.
+
+```typescript
+const credential = await client.credentials.create(
+    serviceUser.id,
+    'gitlab-access-token', 
+    180 // Lifetime in days
+);
+
+console.log('Client ID:', credential.clientSecretConfig.clientId);
+console.log('Client Secret:', credential.clientSecretConfig.clientSecret); 
+// SAVE THE SECRET NOW! It cannot be retrieved again.
+```
+
+### 3. List Credentials
+
+View active credentials for a user.
+
+```typescript
+const creds = await client.credentials.list(serviceUser.id);
+creds.forEach(c => console.log(c.name, c.credentialType));
+```
+
+### 4. Delete Credentials
+
+Revoke a specific credential.
+
+```typescript
+await client.credentials.delete(serviceUser.id, credential.id);
+```
+
+### 5. Delete Service User
+
+```typescript
+await client.users.delete(serviceUser.id, serviceUser.tag);
+```
+
+## Dremio Cloud Service Accounts
+
+In Dremio Cloud, "Service Users" are typically handled as regular users with **Personal Access Tokens (PATs)**.
+
+### Creating a Token Programmatically
+
+You can generate tokens for users (if you have permissions) using the `TokensResource`.
+
+```typescript
+// Create a token for a specific user ID
+const { token } = await client.tokens.createToken(
+    'user-uuid-1234', 
+    'ci-pipeline-token', 
+    90 // Lifetime in days (default 30)
+);
+
+console.log('Generated PAT:', token);
+```
+

package/docs/sources.md

@@ -0,0 +1,53 @@
+# Sources API Guide
+
+Sources are the physical connections to your data lakes and databases (e.g., S3, HDFS, PostgreSQL, Snowflake).
+
+## Listing Sources
+
+```typescript
+const sources = await client.sources.list();
+sources.forEach(source => {
+    console.log(source.name, source.type); // e.g. "MyS3", "S3"
+});
+```
+
+## Getting a Source
+
+```typescript
+const source = await client.sources.get('source-id');
+```
+
+## Creating a Source
+
+Creating a source requires a specific configuration object that varies by source type (S3, Nessie, etc.). *Note: The specific config payload structure depends on the Dremio version and source type.*
+
+```typescript
+// Example: Creating an S3 Source
+await client.sources.create({
+    name: 'MyS3Source',
+    type: 'S3',
+    config: {
+        bucketName: 'my-bucket',
+        accessKey: '...',
+        accessSecret: '...'
+        // ... other S3 specific props
+    }
+});
+```
+
+## Updating a Source
+
+```typescript
+await client.sources.update('source-id', {
+    config: {
+        ...oldConfig,
+        accessSecret: 'NEW_SECRET'
+    }
+});
+```
+
+## Deleting a Source
+
+```typescript
+await client.sources.delete('source-id');
+```