dataiku-sdk 3.6.1 → 3.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -6
- package/bin/bun-version.js +14 -0
- package/bin/dss.js +11 -15
- package/dist/build-metadata.json +1 -1
- package/dist/packages/types/src/index.d.ts +74 -1
- package/dist/packages/types/src/index.js +53 -1
- package/dist/src/cli/coerce.d.ts +1 -0
- package/dist/src/cli/coerce.js +19 -15
- package/dist/src/cli/command-syntax.json +3521 -416
- package/dist/src/cli/commands/analysis.js +19 -1
- package/dist/src/cli/commands/bundle.js +22 -0
- package/dist/src/cli/commands/connection.js +26 -2
- package/dist/src/cli/commands/data-quality.js +33 -1
- package/dist/src/cli/commands/dataset.js +35 -4
- package/dist/src/cli/commands/discussion.js +15 -0
- package/dist/src/cli/commands/doctor.js +1 -1
- package/dist/src/cli/commands/flow.d.ts +5 -0
- package/dist/src/cli/commands/flow.js +57 -0
- package/dist/src/cli/commands/future.js +8 -0
- package/dist/src/cli/commands/index.js +6 -0
- package/dist/src/cli/commands/job.js +29 -6
- package/dist/src/cli/commands/metrics.js +21 -1
- package/dist/src/cli/commands/ml-task.d.ts +5 -0
- package/dist/src/cli/commands/ml-task.js +86 -16
- package/dist/src/cli/commands/mlflow.d.ts +11 -0
- package/dist/src/cli/commands/mlflow.js +121 -0
- package/dist/src/cli/commands/plugin.js +5 -11
- package/dist/src/cli/commands/project-deployer.js +70 -0
- package/dist/src/cli/commands/project-library.js +2 -5
- package/dist/src/cli/commands/project.js +8 -0
- package/dist/src/cli/commands/recipe.d.ts +9 -0
- package/dist/src/cli/commands/recipe.js +93 -7
- package/dist/src/cli/commands/saved-model.d.ts +1 -1
- package/dist/src/cli/commands/saved-model.js +9 -76
- package/dist/src/cli/commands/streaming-endpoint.js +30 -1
- package/dist/src/cli/commands/webapp.js +8 -0
- package/dist/src/cli/commands/wiki.js +22 -0
- package/dist/src/cli/commands/workspace.js +8 -0
- package/dist/src/cli/contract.d.ts +16 -15
- package/dist/src/cli/contract.js +139 -3404
- package/dist/src/cli/doctor.d.ts +13 -0
- package/dist/src/cli/doctor.js +41 -1
- package/dist/src/cli/flags.d.ts +7 -1
- package/dist/src/cli/flags.js +78 -6
- package/dist/src/cli/helpers/app-successor.js +1 -2
- package/dist/src/cli/helpers/documentation.d.ts +21 -0
- package/dist/src/cli/helpers/documentation.js +53 -0
- package/dist/src/cli/helpers/trained-model.d.ts +26 -0
- package/dist/src/cli/helpers/trained-model.js +144 -0
- package/dist/src/cli/list-output.d.ts +7 -0
- package/dist/src/cli/list-output.js +77 -0
- package/dist/src/cli/list-shapes.d.ts +21 -0
- package/dist/src/cli/list-shapes.js +111 -0
- package/dist/src/cli/output-schemas.d.ts +0 -14
- package/dist/src/cli/output-schemas.js +4 -7
- package/dist/src/cli/output.d.ts +0 -1
- package/dist/src/cli/output.js +27 -19
- package/dist/src/cli/plans.d.ts +14 -0
- package/dist/src/cli/plans.js +3684 -0
- package/dist/src/cli/usage.d.ts +7 -0
- package/dist/src/cli/usage.js +20 -1
- package/dist/src/cli/version.d.ts +2 -2
- package/dist/src/cli/version.js +2 -2
- package/dist/src/cli.js +127 -113
- package/dist/src/client.d.ts +52 -43
- package/dist/src/client.js +267 -96
- package/dist/src/errors.d.ts +11 -1
- package/dist/src/errors.js +31 -47
- package/dist/src/generated/action-output-schemas.json +345 -89
- package/dist/src/index.d.ts +5 -1
- package/dist/src/index.js +3 -0
- package/dist/src/resources/analyses.d.ts +8 -0
- package/dist/src/resources/analyses.js +13 -1
- package/dist/src/resources/bundles.d.ts +34 -0
- package/dist/src/resources/bundles.js +67 -0
- package/dist/src/resources/connections.d.ts +9 -0
- package/dist/src/resources/connections.js +15 -0
- package/dist/src/resources/data-quality.d.ts +20 -0
- package/dist/src/resources/data-quality.js +30 -0
- package/dist/src/resources/dataset-create.d.ts +19 -0
- package/dist/src/resources/dataset-create.js +66 -0
- package/dist/src/resources/datasets.d.ts +25 -15
- package/dist/src/resources/datasets.js +59 -111
- package/dist/src/resources/discussions.d.ts +5 -0
- package/dist/src/resources/discussions.js +7 -0
- package/dist/src/resources/documentation.d.ts +25 -0
- package/dist/src/resources/documentation.js +16 -0
- package/dist/src/resources/flow.d.ts +48 -0
- package/dist/src/resources/flow.js +65 -0
- package/dist/src/resources/futures.d.ts +23 -0
- package/dist/src/resources/futures.js +32 -0
- package/dist/src/resources/jobs.js +102 -2
- package/dist/src/resources/metrics.d.ts +10 -0
- package/dist/src/resources/metrics.js +10 -0
- package/dist/src/resources/ml-tasks.d.ts +65 -2
- package/dist/src/resources/ml-tasks.js +102 -28
- package/dist/src/resources/mlflow-extension.d.ts +73 -0
- package/dist/src/resources/mlflow-extension.js +105 -0
- package/dist/src/resources/projects.d.ts +6 -0
- package/dist/src/resources/projects.js +8 -0
- package/dist/src/resources/recipe-create.d.ts +65 -0
- package/dist/src/resources/recipe-create.js +338 -0
- package/dist/src/resources/recipes.d.ts +28 -29
- package/dist/src/resources/recipes.js +126 -245
- package/dist/src/resources/saved-models.d.ts +10 -6
- package/dist/src/resources/saved-models.js +16 -14
- package/dist/src/resources/scenarios.js +8 -14
- package/dist/src/resources/streaming-endpoints.d.ts +12 -0
- package/dist/src/resources/streaming-endpoints.js +15 -0
- package/dist/src/resources/trained-model.d.ts +60 -0
- package/dist/src/resources/trained-model.js +98 -0
- package/dist/src/resources/webapps.d.ts +8 -0
- package/dist/src/resources/webapps.js +10 -0
- package/dist/src/resources/wiki.d.ts +5 -0
- package/dist/src/resources/wiki.js +7 -0
- package/dist/src/resources/workspaces.d.ts +2 -0
- package/dist/src/resources/workspaces.js +4 -0
- package/dist/src/utils/response-file.js +20 -9
- package/dist/src/utils/secret-sanitize.d.ts +7 -0
- package/dist/src/utils/secret-sanitize.js +46 -0
- package/package.json +25 -26
- package/plugin.json +1 -1
- package/skills/dataiku-dss/SKILL.md +1 -1
- package/skills/dataiku-dss/references/authentication.md +3 -1
- package/skills/dataiku-dss/references/coding.md +1 -0
- package/skills/dataiku-dss/references/discovery.md +4 -6
- package/skills/dataiku-dss/references/troubleshooting.md +3 -2
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
import type { DataikuClient } from "../client.js";
|
|
2
|
+
import { type DocumentationTemplate } from "./documentation.js";
|
|
3
|
+
import { type DssTask } from "./futures.js";
|
|
4
|
+
/** Sampling and execution settings of a diagnostics computation (dataikuapi `computationParams`). */
|
|
5
|
+
export interface ModelDiagnosticsComputation {
|
|
6
|
+
/** Records sampled from the test set (DSS default 10000). */
|
|
7
|
+
sampleSize?: number;
|
|
8
|
+
randomState?: number;
|
|
9
|
+
nJobs?: number;
|
|
10
|
+
debugMode?: boolean;
|
|
11
|
+
}
|
|
12
|
+
/** Diagnostics a trained model computes on demand, then serves as results. */
|
|
13
|
+
export type ComputedModelDiagnostic = "subpopulation-analyses" | "partial-dependencies" | "timeseries-residuals";
|
|
14
|
+
/** Every diagnostics result a trained model serves. */
|
|
15
|
+
export type ModelDiagnostic = ComputedModelDiagnostic | "per-timeseries-metrics" | "per-timeseries-evaluation-forecasts";
|
|
16
|
+
/** Diagnostics that are started with POST and return a DSS future. */
|
|
17
|
+
export declare const COMPUTED_MODEL_DIAGNOSTICS: Record<string, true>;
|
|
18
|
+
/** Diagnostics results served by GET (computed ones plus per-series results of forecasting models). */
|
|
19
|
+
export declare const MODEL_DIAGNOSTICS: Record<string, true>;
|
|
20
|
+
/**
|
|
21
|
+
* Request body that starts a diagnostics computation. Subpopulation analyses
|
|
22
|
+
* and partial dependencies take the features to analyse; timeseries
|
|
23
|
+
* residuals take no body.
|
|
24
|
+
*/
|
|
25
|
+
export declare function modelDiagnosticsBody(kind: ComputedModelDiagnostic, features: string[], computation?: ModelDiagnosticsComputation): Record<string, unknown> | undefined;
|
|
26
|
+
/** Query options for the scoring JAR export (GET .../scoring-jar). */
|
|
27
|
+
export interface ScoringJarOptions {
|
|
28
|
+
fullClassName?: string;
|
|
29
|
+
includeLibs?: boolean;
|
|
30
|
+
}
|
|
31
|
+
/**
|
|
32
|
+
* One trained model: a visual-ML lab model (`mlTasks.model(...)`) or a
|
|
33
|
+
* saved-model version (`savedModels.version(...)`). Both expose the same
|
|
34
|
+
* routes (user meta, scoring exports, diagnostics, documentation) under
|
|
35
|
+
* different bases. Generated documents are downloaded from the owning
|
|
36
|
+
* resource (`mlTasks.downloadModelDocumentation`, `savedModels.downloadDocumentation`).
|
|
37
|
+
*/
|
|
38
|
+
export declare class TrainedModel {
|
|
39
|
+
private readonly client;
|
|
40
|
+
/** Route of the trained model, e.g. `.../models/lab/{a}/{t}/models/{id}`. */
|
|
41
|
+
private readonly modelPath;
|
|
42
|
+
constructor(client: DataikuClient,
|
|
43
|
+
/** Route of the trained model, e.g. `.../models/lab/{a}/{t}/models/{id}`. */
|
|
44
|
+
modelPath: string);
|
|
45
|
+
/** Replace the user metadata: only the `userMeta` field of the model details, edited. */
|
|
46
|
+
setUserMeta(userMeta: Record<string, unknown>): Promise<void>;
|
|
47
|
+
/** GET the Java scoring JAR (license-gated, optimized-scoring models only) as a binary stream. */
|
|
48
|
+
downloadScoringJar(options?: ScoringJarOptions): Promise<Response>;
|
|
49
|
+
/** GET the PMML export (license-gated, PMML-compatible models only) as a stream. */
|
|
50
|
+
downloadScoringPmml(): Promise<Response>;
|
|
51
|
+
/**
|
|
52
|
+
* Start computing subpopulation analyses or partial dependencies over
|
|
53
|
+
* `features`, or timeseries residuals. Returns the DSS future.
|
|
54
|
+
*/
|
|
55
|
+
compute(kind: ComputedModelDiagnostic, features?: string[], computation?: ModelDiagnosticsComputation): Promise<DssTask>;
|
|
56
|
+
/** Results computed so far (subpopulation analyses, partial dependencies, residuals, per-series metrics or forecasts). */
|
|
57
|
+
get(kind: ModelDiagnostic): Promise<unknown>;
|
|
58
|
+
/** Start generating the model documentation (docx); the finished result carries `exportId`. */
|
|
59
|
+
generateDocumentation(template?: DocumentationTemplate): Promise<DssTask>;
|
|
60
|
+
}
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
import { ClientValidationError, } from "../errors.js";
|
|
2
|
+
import { requireObject, } from "./base.js";
|
|
3
|
+
import { startDocumentation, } from "./documentation.js";
|
|
4
|
+
import { dssTask, } from "./futures.js";
|
|
5
|
+
/** Diagnostics that are started with POST and return a DSS future. */
|
|
6
|
+
export const COMPUTED_MODEL_DIAGNOSTICS = {
|
|
7
|
+
"subpopulation-analyses": true,
|
|
8
|
+
"partial-dependencies": true,
|
|
9
|
+
"timeseries-residuals": true,
|
|
10
|
+
};
|
|
11
|
+
/** Diagnostics results served by GET (computed ones plus per-series results of forecasting models). */
|
|
12
|
+
export const MODEL_DIAGNOSTICS = {
|
|
13
|
+
...COMPUTED_MODEL_DIAGNOSTICS,
|
|
14
|
+
"per-timeseries-metrics": true,
|
|
15
|
+
"per-timeseries-evaluation-forecasts": true,
|
|
16
|
+
};
|
|
17
|
+
/**
|
|
18
|
+
* Request body that starts a diagnostics computation. Subpopulation analyses
|
|
19
|
+
* and partial dependencies take the features to analyse; timeseries
|
|
20
|
+
* residuals take no body.
|
|
21
|
+
*/
|
|
22
|
+
export function modelDiagnosticsBody(kind, features, computation = {}) {
|
|
23
|
+
if (kind === "timeseries-residuals")
|
|
24
|
+
return undefined;
|
|
25
|
+
if (features.length === 0) {
|
|
26
|
+
throw new ClientValidationError(`${kind} needs at least one feature.`, "missing_required_arg");
|
|
27
|
+
}
|
|
28
|
+
return {
|
|
29
|
+
features,
|
|
30
|
+
computationParams: {
|
|
31
|
+
...(computation.sampleSize !== undefined ? { sample_size: computation.sampleSize, } : {}),
|
|
32
|
+
...(computation.randomState !== undefined ? { random_state: computation.randomState, } : {}),
|
|
33
|
+
...(computation.nJobs !== undefined ? { n_jobs: computation.nJobs, } : {}),
|
|
34
|
+
...(computation.debugMode !== undefined ? { debug_mode: computation.debugMode, } : {}),
|
|
35
|
+
},
|
|
36
|
+
};
|
|
37
|
+
}
|
|
38
|
+
/**
|
|
39
|
+
* One trained model: a visual-ML lab model (`mlTasks.model(...)`) or a
|
|
40
|
+
* saved-model version (`savedModels.version(...)`). Both expose the same
|
|
41
|
+
* routes (user meta, scoring exports, diagnostics, documentation) under
|
|
42
|
+
* different bases. Generated documents are downloaded from the owning
|
|
43
|
+
* resource (`mlTasks.downloadModelDocumentation`, `savedModels.downloadDocumentation`).
|
|
44
|
+
*/
|
|
45
|
+
export class TrainedModel {
|
|
46
|
+
client;
|
|
47
|
+
modelPath;
|
|
48
|
+
constructor(client,
|
|
49
|
+
/** Route of the trained model, e.g. `.../models/lab/{a}/{t}/models/{id}`. */
|
|
50
|
+
modelPath) {
|
|
51
|
+
this.client = client;
|
|
52
|
+
this.modelPath = modelPath;
|
|
53
|
+
}
|
|
54
|
+
/** Replace the user metadata: only the `userMeta` field of the model details, edited. */
|
|
55
|
+
async setUserMeta(userMeta) {
|
|
56
|
+
await this.client.putVoid(`${this.modelPath}/user-meta`, requireObject(userMeta, "userMeta"));
|
|
57
|
+
}
|
|
58
|
+
/** GET the Java scoring JAR (license-gated, optimized-scoring models only) as a binary stream. */
|
|
59
|
+
async downloadScoringJar(options = {}) {
|
|
60
|
+
const query = new URLSearchParams();
|
|
61
|
+
if (options.fullClassName !== undefined)
|
|
62
|
+
query.set("fullClassName", options.fullClassName);
|
|
63
|
+
if (options.includeLibs !== undefined)
|
|
64
|
+
query.set("includeLibs", String(options.includeLibs));
|
|
65
|
+
return this.client.stream(`${this.modelPath}/scoring-jar${query.size > 0 ? `?${query}` : ""}`);
|
|
66
|
+
}
|
|
67
|
+
/** GET the PMML export (license-gated, PMML-compatible models only) as a stream. */
|
|
68
|
+
async downloadScoringPmml() {
|
|
69
|
+
return this.client.stream(`${this.modelPath}/scoring-pmml`);
|
|
70
|
+
}
|
|
71
|
+
/**
|
|
72
|
+
* Start computing subpopulation analyses or partial dependencies over
|
|
73
|
+
* `features`, or timeseries residuals. Returns the DSS future.
|
|
74
|
+
*/
|
|
75
|
+
async compute(kind, features = [], computation = {}) {
|
|
76
|
+
if (COMPUTED_MODEL_DIAGNOSTICS[kind] !== true) {
|
|
77
|
+
throw new ClientValidationError(`Computed diagnostic must be one of ${Object.keys(COMPUTED_MODEL_DIAGNOSTICS).join(", ")}.`, "invalid_enum");
|
|
78
|
+
}
|
|
79
|
+
const raw = await this.client.post(`${this.modelPath}/${kind}`, modelDiagnosticsBody(kind, features, computation));
|
|
80
|
+
return dssTask(raw, `Computing ${kind}`);
|
|
81
|
+
}
|
|
82
|
+
/** Results computed so far (subpopulation analyses, partial dependencies, residuals, per-series metrics or forecasts). */
|
|
83
|
+
async get(kind) {
|
|
84
|
+
if (MODEL_DIAGNOSTICS[kind] !== true) {
|
|
85
|
+
throw new ClientValidationError(`Diagnostic must be one of ${Object.keys(MODEL_DIAGNOSTICS).join(", ")}.`, "invalid_enum");
|
|
86
|
+
}
|
|
87
|
+
return this.client.get(`${this.modelPath}/${kind}`);
|
|
88
|
+
}
|
|
89
|
+
/** Start generating the model documentation (docx); the finished result carries `exportId`. */
|
|
90
|
+
async generateDocumentation(template = { kind: "default", }) {
|
|
91
|
+
const base = `${this.modelPath}/generate-documentation-from`;
|
|
92
|
+
return startDocumentation(this.client, {
|
|
93
|
+
default: `${base}-default-template`,
|
|
94
|
+
file: `${base}-custom-template`,
|
|
95
|
+
folder: `${base}-template-in-folder`,
|
|
96
|
+
}, template, "Model documentation generation");
|
|
97
|
+
}
|
|
98
|
+
}
|
|
@@ -68,4 +68,12 @@ export declare class WebappsResource extends BaseResource {
|
|
|
68
68
|
restartBackendAndWait(webappId: string, projectKey?: string, options?: WebappWaitOptions): Promise<FutureWaitResult>;
|
|
69
69
|
/** Get a webapp backend's state. */
|
|
70
70
|
getBackendState(webappId: string, projectKey?: string): Promise<Record<string, unknown>>;
|
|
71
|
+
/**
|
|
72
|
+
* Trust a webapp's code (POST /actions/trust): for the caller, or for every
|
|
73
|
+
* user with `trustForEverybody` (admin). Returns `{canAccess, trustedCodeStatus}`.
|
|
74
|
+
*/
|
|
75
|
+
trust(webappId: string, opts?: {
|
|
76
|
+
projectKey?: string;
|
|
77
|
+
trustForEverybody?: boolean;
|
|
78
|
+
}): Promise<Record<string, unknown>>;
|
|
71
79
|
}
|
|
@@ -95,4 +95,14 @@ export class WebappsResource extends BaseResource {
|
|
|
95
95
|
const id = encodeURIComponent(webappId);
|
|
96
96
|
return this.client.get(`/public/api/projects/${this.enc(projectKey)}/webapps/${id}/backend/state`);
|
|
97
97
|
}
|
|
98
|
+
/**
|
|
99
|
+
* Trust a webapp's code (POST /actions/trust): for the caller, or for every
|
|
100
|
+
* user with `trustForEverybody` (admin). Returns `{canAccess, trustedCodeStatus}`.
|
|
101
|
+
*/
|
|
102
|
+
async trust(webappId, opts = {}) {
|
|
103
|
+
const query = opts.trustForEverybody === undefined
|
|
104
|
+
? ""
|
|
105
|
+
: `?trustForEverybody=${opts.trustForEverybody}`;
|
|
106
|
+
return this.client.post(`/public/api/projects/${this.enc(opts.projectKey)}/webapps/${encodeURIComponent(webappId)}/actions/trust${query}`);
|
|
107
|
+
}
|
|
98
108
|
}
|
|
@@ -14,6 +14,11 @@ export interface WikiArticleUpdateOptions {
|
|
|
14
14
|
}
|
|
15
15
|
export declare class WikiResource extends BaseResource {
|
|
16
16
|
settings(projectKey?: string): Promise<WikiSettings>;
|
|
17
|
+
/**
|
|
18
|
+
* Replace the wiki properties (home article and taxonomy, PUT /wiki/) with
|
|
19
|
+
* an object obtained from `settings`, edited.
|
|
20
|
+
*/
|
|
21
|
+
updateSettings(settings: Record<string, unknown>, projectKey?: string): Promise<void>;
|
|
17
22
|
list(projectKey?: string): Promise<WikiArticleData[]>;
|
|
18
23
|
get(articleIdOrName: string, projectKey?: string): Promise<WikiArticleData>;
|
|
19
24
|
create(opts: WikiArticleCreateOptions): Promise<WikiArticleData>;
|
|
@@ -33,6 +33,13 @@ export class WikiResource extends BaseResource {
|
|
|
33
33
|
const raw = await this.client.get(`/public/api/projects/${this.enc(projectKey)}/wiki/`);
|
|
34
34
|
return this.client.safeParse(WikiSettingsSchema, raw, "wiki.settings");
|
|
35
35
|
}
|
|
36
|
+
/**
|
|
37
|
+
* Replace the wiki properties (home article and taxonomy, PUT /wiki/) with
|
|
38
|
+
* an object obtained from `settings`, edited.
|
|
39
|
+
*/
|
|
40
|
+
async updateSettings(settings, projectKey) {
|
|
41
|
+
await this.client.putVoid(`/public/api/projects/${this.enc(projectKey)}/wiki/`, settings);
|
|
42
|
+
}
|
|
36
43
|
async list(projectKey) {
|
|
37
44
|
const settings = await this.settings(projectKey);
|
|
38
45
|
const ids = taxonomyIds(settings.taxonomy);
|
|
@@ -56,4 +56,6 @@ export declare class WorkspacesResource extends BaseResource {
|
|
|
56
56
|
listObjects(workspaceKey: string): Promise<WorkspaceObject[]>;
|
|
57
57
|
/** Add an object to a workspace. */
|
|
58
58
|
addObject(workspaceKey: string, object: WorkspaceObjectInput): Promise<WorkspaceObject>;
|
|
59
|
+
/** Remove an object from a workspace by its workspace object id (from `listObjects`). */
|
|
60
|
+
removeObject(workspaceKey: string, workspaceObjectId: string): Promise<void>;
|
|
59
61
|
}
|
|
@@ -28,4 +28,8 @@ export class WorkspacesResource extends BaseResource {
|
|
|
28
28
|
async addObject(workspaceKey, object) {
|
|
29
29
|
return this.client.post(`/public/api/workspaces/${encodeURIComponent(workspaceKey)}/objects`, object);
|
|
30
30
|
}
|
|
31
|
+
/** Remove an object from a workspace by its workspace object id (from `listObjects`). */
|
|
32
|
+
async removeObject(workspaceKey, workspaceObjectId) {
|
|
33
|
+
await this.client.del(`/public/api/workspaces/${encodeURIComponent(workspaceKey)}/objects/${encodeURIComponent(workspaceObjectId)}`);
|
|
34
|
+
}
|
|
31
35
|
}
|
|
@@ -1,9 +1,6 @@
|
|
|
1
|
-
import {
|
|
2
|
-
import { createWriteStream, } from "node:fs";
|
|
1
|
+
import { closeSync, openSync, } from "node:fs";
|
|
3
2
|
import { chmod, rename, rm, stat, } from "node:fs/promises";
|
|
4
3
|
import { basename, dirname, join, } from "node:path";
|
|
5
|
-
import { Readable, } from "node:stream";
|
|
6
|
-
import { pipeline, } from "node:stream/promises";
|
|
7
4
|
/** Stream with backpressure; publish only a complete download, preserving existing output on failure. */
|
|
8
5
|
export async function writeResponseToFile(path, response, permissions = "private") {
|
|
9
6
|
let mode;
|
|
@@ -17,16 +14,30 @@ export async function writeResponseToFile(path, response, permissions = "private
|
|
|
17
14
|
mode = 0o666 & ~process.umask();
|
|
18
15
|
}
|
|
19
16
|
}
|
|
20
|
-
const
|
|
21
|
-
|
|
22
|
-
|
|
17
|
+
const temporaryPath = join(dirname(path), `.${basename(path)}.tmp-${crypto.randomUUID()}`);
|
|
18
|
+
// Exclusive, private temp file; Bun.write streams the body into it natively.
|
|
19
|
+
let fd;
|
|
23
20
|
try {
|
|
24
|
-
|
|
21
|
+
fd = openSync(temporaryPath, "wx", 0o600);
|
|
22
|
+
}
|
|
23
|
+
catch (error) {
|
|
24
|
+
// Nothing will read the body; release the connection before surfacing the error.
|
|
25
|
+
await response.body?.cancel().catch(() => { });
|
|
26
|
+
throw error;
|
|
27
|
+
}
|
|
28
|
+
try {
|
|
29
|
+
let bytes;
|
|
30
|
+
try {
|
|
31
|
+
bytes = await Bun.write(Bun.file(fd), response);
|
|
32
|
+
}
|
|
33
|
+
finally {
|
|
34
|
+
closeSync(fd);
|
|
35
|
+
}
|
|
25
36
|
// Keep partial contents private; restore publication permissions only after completion.
|
|
26
37
|
if (mode !== undefined)
|
|
27
38
|
await chmod(temporaryPath, mode);
|
|
28
39
|
await rename(temporaryPath, path);
|
|
29
|
-
return
|
|
40
|
+
return bytes;
|
|
30
41
|
}
|
|
31
42
|
finally {
|
|
32
43
|
await rm(temporaryPath, { force: true, });
|
|
@@ -46,6 +46,13 @@ export declare function redactUrlUserinfo(text: string): string;
|
|
|
46
46
|
* recurse, arrays recurse. Non-string scalars pass through unchanged.
|
|
47
47
|
*/
|
|
48
48
|
export declare function sanitizeSecrets<T>(value: T, options: SecretSanitizeOptions): T;
|
|
49
|
+
/**
|
|
50
|
+
* Every string value under a credential-like key (any depth), plus bare
|
|
51
|
+
* `key` fields such as `authRealm.key`. Collected from a command's own JSON
|
|
52
|
+
* input so that DSS error text echoing the payload can be scrubbed; error
|
|
53
|
+
* redaction only, so the broad key match errs toward hiding.
|
|
54
|
+
*/
|
|
55
|
+
export declare function credentialValues(value: unknown): string[];
|
|
49
56
|
/**
|
|
50
57
|
* Scrub secret material from an error's message, stack, and string body.
|
|
51
58
|
* `DataikuError.message` is built at construction, so all three carriers need
|
|
@@ -77,6 +77,52 @@ export function sanitizeSecrets(value, options) {
|
|
|
77
77
|
}
|
|
78
78
|
return value;
|
|
79
79
|
}
|
|
80
|
+
/** Credential-like key fragments (normalized); values under them are secrets. */
|
|
81
|
+
const CREDENTIAL_KEY_FRAGMENTS = [
|
|
82
|
+
"password",
|
|
83
|
+
"passwd",
|
|
84
|
+
"passphrase",
|
|
85
|
+
"secret",
|
|
86
|
+
"token",
|
|
87
|
+
"apikey",
|
|
88
|
+
"accesskey",
|
|
89
|
+
"privatekey",
|
|
90
|
+
"credential",
|
|
91
|
+
"authorization",
|
|
92
|
+
];
|
|
93
|
+
/** Keys naming how credentials work (`credentialsMode`, `passwordType`), not secrets. */
|
|
94
|
+
const CREDENTIAL_SETTING_SUFFIXES = ["mode", "type", "policy",];
|
|
95
|
+
/**
|
|
96
|
+
* Every string value under a credential-like key (any depth), plus bare
|
|
97
|
+
* `key` fields such as `authRealm.key`. Collected from a command's own JSON
|
|
98
|
+
* input so that DSS error text echoing the payload can be scrubbed; error
|
|
99
|
+
* redaction only, so the broad key match errs toward hiding.
|
|
100
|
+
*/
|
|
101
|
+
export function credentialValues(value) {
|
|
102
|
+
const found = [];
|
|
103
|
+
const visit = (node, sensitive) => {
|
|
104
|
+
if (typeof node === "string") {
|
|
105
|
+
if (sensitive && node.length > 0)
|
|
106
|
+
found.push(node);
|
|
107
|
+
return;
|
|
108
|
+
}
|
|
109
|
+
if (Array.isArray(node)) {
|
|
110
|
+
for (const item of node)
|
|
111
|
+
visit(item, sensitive);
|
|
112
|
+
return;
|
|
113
|
+
}
|
|
114
|
+
if (node === null || typeof node !== "object")
|
|
115
|
+
return;
|
|
116
|
+
for (const [key, item,] of Object.entries(node)) {
|
|
117
|
+
const normalized = normalizeSecretKey(key);
|
|
118
|
+
const setting = CREDENTIAL_SETTING_SUFFIXES.some((suffix) => normalized.endsWith(suffix));
|
|
119
|
+
visit(item, sensitive || (!setting && (normalized === "key"
|
|
120
|
+
|| CREDENTIAL_KEY_FRAGMENTS.some((fragment) => normalized.includes(fragment)))));
|
|
121
|
+
}
|
|
122
|
+
};
|
|
123
|
+
visit(value, false);
|
|
124
|
+
return found;
|
|
125
|
+
}
|
|
80
126
|
/**
|
|
81
127
|
* Scrub secret material from an error's message, stack, and string body.
|
|
82
128
|
* `DataikuError.message` is built at construction, so all three carriers need
|
package/package.json
CHANGED
|
@@ -1,11 +1,13 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dataiku-sdk",
|
|
3
|
-
"version": "3.
|
|
3
|
+
"version": "3.7.0",
|
|
4
4
|
"description": "Dataiku DSS SDK and CLI for programmatic access to DSS REST APIs",
|
|
5
|
+
"repository": {
|
|
6
|
+
"type": "git",
|
|
7
|
+
"url": "https://github.com/clssck/dataiku-sdk.git"
|
|
8
|
+
},
|
|
9
|
+
"license": "SEE LICENSE IN LICENSE",
|
|
5
10
|
"type": "module",
|
|
6
|
-
"workspaces": [
|
|
7
|
-
"packages/*"
|
|
8
|
-
],
|
|
9
11
|
"main": "dist/src/index.js",
|
|
10
12
|
"types": "dist/src/index.d.ts",
|
|
11
13
|
"bin": {
|
|
@@ -15,11 +17,12 @@
|
|
|
15
17
|
"dist/",
|
|
16
18
|
"plugin.json",
|
|
17
19
|
"skills/",
|
|
18
|
-
"bin/dss.js"
|
|
20
|
+
"bin/dss.js",
|
|
21
|
+
"bin/bun-version.js"
|
|
22
|
+
],
|
|
23
|
+
"workspaces": [
|
|
24
|
+
"packages/*"
|
|
19
25
|
],
|
|
20
|
-
"engines": {
|
|
21
|
-
"bun": ">=1.4.0"
|
|
22
|
-
},
|
|
23
26
|
"scripts": {
|
|
24
27
|
"build": "bun scripts/generate-action-output-schemas.mjs && tsc && bun scripts/generate-build-metadata.mjs",
|
|
25
28
|
"prepack": "bun scripts/sync-plugin-version.mjs --check && bun scripts/generate-action-output-schemas.mjs --check && tsc && bun scripts/generate-build-metadata.mjs",
|
|
@@ -40,30 +43,26 @@
|
|
|
40
43
|
"test:integration:rigorous": "cross-env RUN_DATAIKU_INTEGRATION=1 bun test tests/integration-rigorous.test.ts",
|
|
41
44
|
"test:integration:rigorous:mutating": "cross-env RUN_DATAIKU_INTEGRATION=1 RUN_DATAIKU_INTEGRATION_MUTATING=1 bun test tests/integration-rigorous.test.ts"
|
|
42
45
|
},
|
|
46
|
+
"dependencies": {
|
|
47
|
+
"@sinclair/typebox": "^0.34.52",
|
|
48
|
+
"node-stream-zip": "^1.16.0"
|
|
49
|
+
},
|
|
43
50
|
"devDependencies": {
|
|
44
|
-
"@oxlint/plugins": "1.
|
|
51
|
+
"@oxlint/plugins": "1.86.0",
|
|
45
52
|
"@types/bun": "^1.4.2",
|
|
46
53
|
"cross-env": "^10.1.0",
|
|
47
|
-
"dprint": "^0.
|
|
54
|
+
"dprint": "^0.60.1",
|
|
48
55
|
"js-tiktoken": "1.0.21",
|
|
49
|
-
"oxlint": "1.
|
|
50
|
-
"typescript": "^7.0.2"
|
|
51
|
-
"typescript-language-server": "6.0.0",
|
|
52
|
-
"typescript-language-service": "npm:typescript@5.9.3"
|
|
53
|
-
},
|
|
54
|
-
"dependencies": {
|
|
55
|
-
"@sinclair/typebox": "^0.34.52",
|
|
56
|
-
"node-stream-zip": "^1.16.0"
|
|
56
|
+
"oxlint": "1.86.0",
|
|
57
|
+
"typescript": "^7.0.2"
|
|
57
58
|
},
|
|
58
|
-
"
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
59
|
+
"bundleDependencies": [
|
|
60
|
+
"@sinclair/typebox"
|
|
61
|
+
],
|
|
62
|
+
"engines": {
|
|
63
|
+
"bun": ">=1.4.2"
|
|
62
64
|
},
|
|
63
65
|
"publishConfig": {
|
|
64
66
|
"access": "public"
|
|
65
|
-
}
|
|
66
|
-
"bundleDependencies": [
|
|
67
|
-
"@sinclair/typebox"
|
|
68
|
-
]
|
|
67
|
+
}
|
|
69
68
|
}
|
package/plugin.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
|
|
3
3
|
"name": "dataiku-sdk",
|
|
4
|
-
"version": "3.
|
|
4
|
+
"version": "3.7.0",
|
|
5
5
|
"description": "Dataiku DSS SDK and agent-oriented CLI for programmatic access to DSS REST APIs.",
|
|
6
6
|
"homepage": "https://github.com/clssck/dataiku-sdk",
|
|
7
7
|
"repository": "https://github.com/clssck/dataiku-sdk",
|
|
@@ -6,7 +6,7 @@ description: >-
|
|
|
6
6
|
Discover syntax: dss commands run --fields RESOURCE.ACTION.
|
|
7
7
|
license: LicenseRef-Dataiku-SDK-Limited-Use-1.0
|
|
8
8
|
compatibility: >-
|
|
9
|
-
Requires Bun >= 1.4.
|
|
9
|
+
Requires Bun >= 1.4.2, the CLI, DSS network access, and credentials.
|
|
10
10
|
---
|
|
11
11
|
|
|
12
12
|
# Dataiku DSS
|
|
@@ -36,4 +36,6 @@ dss auth login
|
|
|
36
36
|
```
|
|
37
37
|
`auth login` lists projects before saving: the key needs project-list permission. Returns `{"saved":true,"path":"..."}`. Storage: `credentials.json` in `DSS_CONFIG_DIR`, else `XDG_CONFIG_HOME/dataiku`, Windows `APPDATA/dataiku`, or `~/.config/dataiku`.
|
|
38
38
|
|
|
39
|
-
TLS: `--insecure` disables verification; `--ca-cert PATH` adds a PEM CA bundle. Environment equivalents: `NODE_TLS_REJECT_UNAUTHORIZED`, `NODE_EXTRA_CA_CERTS`.
|
|
39
|
+
TLS: `--insecure` disables verification; `--ca-cert PATH` adds a PEM CA bundle. Environment equivalents: `NODE_TLS_REJECT_UNAUTHORIZED`, `NODE_EXTRA_CA_CERTS`. A subprocess sees only its own env and cwd `.env`; `dss auth login --ca-cert PATH` saves the CA for all processes.
|
|
40
|
+
|
|
41
|
+
Connection `403` the UI allows: check the key's identity (`dss doctor`: `authSource`, login, groups). `connection get` needs admin; `connection info NAME` covers usable connections.
|
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
- Git: UAT=`create-branch --duplicate-project`; pull=rebase; checkout=switch. Remotes use DSS Git rules and SSH/server credentials. `--password-env`; `future-wait`.
|
|
4
4
|
- Internal `lib/`: `project-library`. External Git: `project-git list-libraries`. Safe edit: `get-bytes` → `diff` → `put --expect-sha256 HASH`; create with `--if-not-exists`. Paths reject traversal.
|
|
5
5
|
- Python: `code run --file`. Inspect/diff/backup recipes before `recipe run --dry-run`.
|
|
6
|
+
- Grouping/prepare: build with `recipe create` flags (`--group-by`, `--aggregate`; `--rename`, `--formula`, `--filter`, ...). Other visual payloads (join): no published schema; template = `recipe get-payload` after create. Join `MANUAL` selection uses top-level `selectedColumns: [{name, table, alias?}]` (`table` = input index). DSS-engine computed columns run as H2 SQL: no GREL `+` concat, use `SQL` mode `||`. Grouping `first` needs an ordering column.
|
|
6
7
|
- Notebook saves return hashes; guard with `--expect-hash`. Clear outputs via DSS DELETE; `unload-jupyter --all` composes session deletes.
|
|
7
8
|
- Inspect code-env definitions/logs before updates; guard webapp/API settings with `--expect-hash`. `exact:false`: live state not guessed; use command `--dry-run`.
|
|
8
9
|
- Plugin authoring: `dss plugin` create-dev, move-to-dev, contents-put, push/pull. No notebook execute/checkpoints or webapp/API delete; use the DSS UI.
|
|
@@ -8,11 +8,13 @@
|
|
|
8
8
|
- Exits: 0 success, 1 usage/configuration, 2 DSS/internal/permission-or-environment, 3 transient/retryable DSS error, 4 failed long-running result/assertion.
|
|
9
9
|
- Recipe payload stdout: JSON string. With `--output PATH`: exact bytes to file, JSON string equal to `PATH` on stdout.
|
|
10
10
|
- General `--fields a,b,c`: object projection, element-wise for object arrays. Dotted paths traverse nested objects; missing fields become `null`; strings/scalars pass through.
|
|
11
|
+
- `*.list`: compact items (ids, kind, next-step fields such as recipe `inputs`/`outputs`); `--full`/`--fields` use DSS objects. `--contains TEXT` filters ids; `--limit N` caps (`list_truncated` warning).
|
|
11
12
|
|
|
12
13
|
## Discovery
|
|
13
14
|
|
|
14
15
|
```text
|
|
15
16
|
dss commands run
|
|
17
|
+
dss commands run --contains "propagate schema"
|
|
16
18
|
dss agent contract --fields commands.actions.dataset
|
|
17
19
|
dss commands run --fields dataset.create
|
|
18
20
|
dss commands run --fields dataset.preview.usage,dataset.preview.description,dataset.preview.flags,dataset.preview.examples
|
|
@@ -21,10 +23,6 @@ dss agent contract --fields protocol,agentContractVersion,cli,stdio,planning,com
|
|
|
21
23
|
dss agent contract --fields commands.actions
|
|
22
24
|
```
|
|
23
25
|
|
|
24
|
-
`commands run
|
|
26
|
+
`commands run`: resource→action summary (~1.6k tokens). Unknown name: `--contains TEXT` gives the top 5 actions with usage. Known resource: `agent contract --fields commands.actions.RESOURCE`. Bootstrap once with the six-field projection (~250 tokens). Global flags (`--fields`, `--verbose`, connection/TLS) appear once in `commands.globalFlags`, not per entry.
|
|
25
27
|
|
|
26
|
-
Look up syntax before invoking
|
|
27
|
-
|
|
28
|
-
Comma-separate selectors to batch lookups. Keys echo selectors: `--fields dataset.create` → `{"dataset.create":{...}}`. Empty `--fields` fails usage validation, never dumps everything. Unknown resources/actions exit 1 with compact JSON errors listing valid options.
|
|
29
|
-
|
|
30
|
-
Full registry: `commands run --output PATH` only. With `--fields`, export that subset instead. Files contain compact JSON; stdout is `{"path":"PATH"}`, never registry content.
|
|
28
|
+
Look up syntax before invoking: `RESOURCE.ACTION` is one entry, `.FIELD` nests, commas batch, keys echo selectors. Reads: `usage,description,flags,examples`. Writes: the full entry once (flags, side effects, idempotency, dry-run, payload schemas, cleanup, exits). Unknown flags, resources, and actions exit 1 with the usage line or valid options. Full registry: `commands run --output PATH` only; stdout is `{"path":"PATH"}`.
|
|
@@ -4,7 +4,8 @@
|
|
|
4
4
|
|
|
5
5
|
- Code/SQL: use `--file`/`--sql-file`; shells, especially PowerShell, mangle quotes, `$`, and multiline text. Non-UTF-8 consoles (e.g. Windows cp1252): write non-ASCII results to UTF-8 files or `--output PATH`, not the console.
|
|
6
6
|
- Build errors: `dss job log <id> --errors-only` selects errors/tracebacks. For full logs, use `--output PATH` without `--errors-only`. Logs may arrive as one JVM-noisy line; `Error in Python process: At line <N>` identifies the payload source line.
|
|
7
|
-
-
|
|
7
|
+
- Schemas: code recipes set output schemas at run time. Visual/SQL query recipes: `recipe create` fills created or empty outputs (`outputSchemaUpdated`); `recipe set-payload` updates empty or DSS-derived ones, warns `recipe_output_schema_outdated` for hand-edited ones. Apply with `dss recipe update-schema NAME` (one recipe), `dss flow propagate-schema DATASET --wait` (every downstream recipe), or build `--auto-update-schema`. `built_dataset_has_no_columns`: the DONE build wrote nothing. Manual columns: `dss dataset refresh-schema NAME --data-file columns.json`. `dss dataset validate-build` catches file-backed misconfiguration.
|
|
8
|
+
- `Inline` (editable) datasets: the public API cannot write rows, so `dataset create` rejects them; use `UploadedFiles` + `dataset upload-file`, or a python recipe output. `code run` is a scenario step; some DSS versions refuse dataset writes there.
|
|
8
9
|
- `dss dataset download`: default 100k rows; result `{ path, rows, truncated, limit }`. If truncated, raise `--limit`. Specify `--output` or read the `dataset_download_default_location` path warning. CSV is spreadsheet-safe; `--raw-data` retains formula prefixes. For large tables, aggregate in SQL or use a recipe.
|
|
9
10
|
- `dss dashboard create`/`update`: before POST/PUT, every `INSIGHT` tile needs agreeing/resolvable `insightId` and `targetInsightId`; any `datasetSmartName` must resolve. Missing/stale references: `validation_failed`. Cross-project `PROJECT.DATASET` needs dataset-project read permission; `403` blocks saving an unverifiable dashboard.
|
|
10
11
|
- `dss api-service list-packages SERVICE` checks parent settings first. `not_found`: missing service; verify service ID/project key. Empty array: existing service, no deployable packages. Never convert `not_found` to an empty list or retry the lower-level route.
|
|
@@ -26,4 +27,4 @@ Dispatch/runtime failure: one compact stdout error:
|
|
|
26
27
|
|
|
27
28
|
`doctor`/`batch`/`cleanup` command failures return direct stdout result objects; check nonzero exit before interpreting. Recover from structured `code`, `category`, `exitCode`, `retryable`, `status`, `details`, never message scraping.
|
|
28
29
|
|
|
29
|
-
`
|
|
30
|
+
DSS failures: `error` is `STATUS Text: DSS message` (credentials redacted, bounded); `hint` names the next check (resource, project). `details` appears only with content: `elapsedMs`/`target`, `dssErrorType`, `bodyTruncated`, and `retry` when a retry or timeout happened. Refused connections, unknown hosts, bad URLs, and untrusted certificates fail at once as `validation_failed` (exit 2); `transient` (exit 3) means automatic retries already ran.
|