dataiku-sdk 3.1.0 → 3.3.0

package/README.md

@@ -2,6 +2,18 @@
 
 Agent-only TypeScript SDK and `dss` CLI for Dataiku DSS automation.
 
+## API coverage
+
+The SDK and CLI include scenario run management; project, recipe, and dataset metadata; managed datasets, lineage, and table imports; external saved models and MLflow imports; LLM completions and embeddings; knowledge banks; project folders and data collections; users, groups, connections, macros, and plugin administration. Discover exact inputs with `dss commands run` and scoped command contracts rather than assuming flags from API field names. MLflow archive/folder imports and evaluations resolve the external container config explicitly: DSS 15 requires the `containerExecConfigName` parameter, so the SDK/CLI send `NONE` rather than omitting it (the plan surface matches the executed request).
+
+The [DSS 15 coverage matrix](docs/API-COVERAGE-DSS-15.md) maps documented operations to SDK methods and CLI actions, records documentation discrepancies, and identifies remaining gaps. A [machine-readable matrix](docs/API-COVERAGE-DSS-15.json) is also available. Coverage is not full DSS parity or live compatibility certification: availability depends on DSS version, permissions, licensing, and installed capabilities. One known documentation discrepancy: `plugin git-branches` is served by GET on DSS 15 (the same route rejects POST with 405 `rawMethodPOSTnotsupported`, contrary to the public docs).
+
+Metadata setters replace the complete metadata object: fetch it, edit it, then submit the replacement. Metadata previews expose that replacement under `next`. New mutation previews (`--plan` and `--dry-run`) make no DSS requests; sensitive payloads may be represented by redacted summaries rather than literal request bodies. LLM completions and embeddings are cost-bearing operations, and macros execute plugin code. Review their contracts before execution.
+
+ML training and macro waits bound status requests by the remaining wait deadline, including response bodies and retries. The client `requestTimeoutMs` still caps each request: an overall wait budget can shorten that cap, never extend it. An individual request timeout propagates as an error until the overall wait budget expires. Future and Project Git waits use adaptive polling by default; explicit polling intervals remain fixed. Early CLI usage errors load at most the selected resource definitions, not the full command runtime.
+
+Malformed user/group listings and scenario run histories fail explicitly instead of appearing empty. Scenario and macro run responses must contain usable run identifiers; missing identifiers are server-response errors, not caller-validation errors.
+
 ## Platform support
 
 The published `dss` CLI requires Bun >= 1.4.0 and supports Linux, macOS, and Windows. The release gate runs Bun 1.4.0 on all three operating systems. The runtime dependency is pure JavaScript, so the same package also runs on every Bun-supported x64 and ARM64 system.
@@ -23,12 +35,168 @@ Bun is the only supported runtime and the package manager. Examples below assume
 `--no-env-file` disables Bun's automatic preloading only; the CLI still applies its documented `.env` handling unless `DATAIKU_DISABLE_ENV=1` is set.
 `dss version` reports whether the running code came from `source` or `dist`, the runtime (always `bun`), the packaged build revision when available, and `staleBuild` when that revision differs from the checkout. Release builds generate `dist/build-metadata.json`; source snapshots without Git metadata still build but report no build revision.
 
+## Self-provisioning live tests
+
+From a checkout, `test:live` is a separate Bun script, not a `dss` command. It reads
+the existing `.env` without changing it and needs project-creation access plus a
+writable managed storage connection. No imported tutorials are required.
+
+```sh
+bun run test:live setup
+bun run test:live run --case core.dataset.baseline
+bun run test:live run --case "core.recipe.*"
+bun run test:live run --profile all          # every profile's run cases in one lab
+bun run test:live run --profile ml --case "ml.*"
+bun run test:live run --profile all --case "applications.*,infrastructure.sql-select"
+bun run test:live status
+bun run test:live clean               # delete only this lab, including its root project
+# CI: setup + run + cleanup, even when cases fail
+bun run test:live all
+```
+
+`setup` is idempotent for a ready lab; `run` reuses its datasets, graph and collaboration
+fixtures. Cases use temporary resources or restore baseline values. Cases accept exact
+IDs, comma-separated lists, repeated `--case`, and trailing-prefix `*` matching; quote
+wildcards. Selected runs omit the legacy suites. Unknown, setup-only or inactive-profile
+or empty selections fail before credentials, state creation or provisioning; `--case` is valid only
+for `run`/`all`. `run` refuses
+an incomplete or cleaned lab rather than silently rebuilding it. Clean failed setup
+before provisioning a replacement. Authentication and transport failures retain their
+original errors; they are not evidence that the lab is missing.
+
+`--profile all` is a selection operator, not a lifecycle command: it widens run-case
+availability to every profile (core, ml, applications, infrastructure) in the same
+persistent lab, and setup-only cases stay unselectable. The selector itself neither
+provisions beyond the selected cases nor cleans — combined with the `all` VERB the
+run still ends in the usual destructive cleanup.
+
+Core fixtures include deterministic CSVs with nulls, duplicates, Unicode and quoting;
+server-provisioned managed datasets; sync, Prepare, join, fuzzy-join, grouping and Python
+recipes; folders, scenarios, variables, wiki, notebooks, dashboards, insights, libraries,
+metrics and quality rules. Project export/import/duplicate cases use separate owned
+projects. Managed dataset types, paths and recipe output schemas come from DSS rather
+than tutorial-specific connection guesses.
+
+Profiles add to core: `ml` trains and deploys a small decision tree and checks clustering.
+`applications` Designer cases provision disposable owned templates when
+`DATAIKU_LIVE_APP_TEMPLATE_ID` is absent. That variable optionally selects an external
+read-only template; version writes always use an owned template. The owned workflow
+exercises form variables, a Python scenario, and a managed-folder download. Explicit-key
+instance and successor creation still blocks before POST when target absence is unprovable.
+Packaged Business Apps are separate: their cases use `DATAIKU_LIVE_BUSINESS_APP_ID`,
+`DATAIKU_LIVE_BAPP_INSTANCE_PROJECT`, `DATAIKU_LIVE_BAPP_USER`, or
+`DATAIKU_LIVE_BAPP_ARCHIVE_PATH`, and report missing prerequisites as blockers.
+Other application and infrastructure read/CRUD cases run without these. The SQL probe
+case needs `DATAIKU_SQL_CONNECTION` or `DATAIKU_SQL_DATASET_FULL_NAME` (other
+infrastructure read-only cases run without either).
+The SQL case permits only the fixed, table-free `SELECT 1 AS one` probe with an
+explicit target and an owned project whose incarnation is checked before execution.
+It does not authorize arbitrary SQL, alternate query inputs or global mutations. Missing
+prerequisites are recorded as **blocked**, never passed. Required blocked cases exit
+nonzero; optional blocked cases remain visible in reports. Global administration,
+external Git mutations and unsupported capabilities are not silently exercised.
+
+SQL catalog/import cases can provision an owned file-backed SQLite connection and
+seed a real table. Project and plugin Git cases use owned bare repositories served on loopback
+through the server's `git-http-backend`; they never change global Git rules. These
+fixtures require server-side Python execution and the relevant admin/Git permissions.
+Host paths and process identities are recorded in the manifest. Cleanup verifies
+ownership before stopping services and deleting directories; failed cleanup retains
+the runner project for recovery.
+
+State is ignored by Git under `.live-tests/<server-hash>/`. Use `--state-dir PATH` to keep
+a persistent demo or verification lab outside the default root — the directory is
+created on demand, isolated per server hash, and never cleaned implicitly. For `run`,
+`clean` or `status` you can alternatively select an existing lab with `--manifest PATH`
+(the two flags overlap only there; `setup`/`all` require `--state-dir` instead). Keep the manifest and
+`cleanup.jsonl`: cleanup requires exact project identities and creation-incarnation
+hashes, never a prefix sweep. A changed server/project identity or unconfirmed creation
+fails closed; investigate its journal before manual recovery. Locks prevent concurrent
+lab runs; after an uncatchable process kill, confirm its recorded process is gone before
+removing a stale lock. Interrupting a run preserves the lab; `all` attempts cleanup.
+
+New globally-scoped case families create only isolated, disposable resources: users,
+groups, meanings, workspaces, data collections, API-deployer infra/services/deployments,
+and plugins require an explicit reservation in the lab manifest plus an exact
+ownership marker written into a lab-controlled field and re-read on every GET
+(`ctx.markerFor`; code-envs instead bind on the server's `desc.creationTag`
+incarnation snapshot, and project folders have no nonce field — they bind on the
+server-generated id plus owner binding). The lifecycle is explicit: reserve checks the
+pending reservation against the manifest; create runs the command and binds the entry
+only after a GET proves the marker/incarnation identity; bound-target mutations
+re-verify the live object before every request. A GET that cannot prove identity
+leaves the entry unconfirmed rather than bound. A failed creation is never deleted
+blind and a conflicting pre-existing resource is recorded, not removed. Pre-existing
+accounts, connections and configuration are never modified or deleted.
+
+Trained-ML demonstration state is retained across iterations in the lab: the `ml`
+profile keeps a real trained decision-tree model and its fixtures, so subsequent
+iterations read model metadata, settings and versions without retraining.
+
+Core fixtures include deterministic CSVs with nulls, duplicates, Unicode and quoting;
+server-provisioned managed datasets; sync, Prepare, join, fuzzy-join, grouping and Python
+recipes; folders, scenarios, variables, wiki, notebooks, dashboards, insights, libraries,
+metrics and quality rules. Project export/import/duplicate cases use separate owned
+projects. Managed dataset types, paths and recipe output schemas come from DSS rather
+than tutorial-specific connection guesses. Expanded case families beyond the original
+core set cover bundles (export/publish lifecycles), Project Git (inspect, commit,
+remote and external-library lifecycles against owned loopback repositories), scenario statistics and
+payload round-trips, application template and instance manifests, webapp backend state,
+infrastructure reads (connections, users, groups, macros, code-envs, project folders,
+data collections), disposable global lifecycles under the ownership-marker contract,
+and disposable folders/notebooks with plugin.json receipt flows. Plugin cases that
+generate a plugin-managed code environment delete that derived environment explicitly
+before deleting the parent plugin (DSS does not cascade the deletion), and plugin updates
+require the `settings.codeEnvName` association created after plugin creation. MLflow
+import fixtures use canonical packaging (`code:'code'`, `data:'model'`, `env:'conda.yaml'`)
+inside owned runtime environments with pinned scientific versions, and both archive and
+folder imports/evaluations pass explicit `containerExecConfigName=NONE` (DSS 15 rejects
+requests that omit it; folder imports previously inherited a default); noncanonical
+model-path layouts fail with READ_META `ModuleNotFoundError`.
+
+Some capabilities have external prerequisites the lab cannot self-provision; each
+affected action records an explicit per-action blocker instead of passing or being
+silently omitted: an installed application template (`DATAIKU_LIVE_APP_TEMPLATE_ID`),
+external SQL connectivity for non-fixture workloads (`DATAIKU_SQL_CONNECTION`/`DATAIKU_SQL_DATASET_FULL_NAME`),
+an external auth backend (LDAP/SAML) for user external-directory reads and resyncs, plugin-store
+access, and any cost-bearing surface (LLM completions/embeddings bill per call; macros
+execute plugin code). Missing credentials, templates, external infrastructure or billing
+eligibility are reported as **blocked**, never as success.
+
+Each iteration retains logs and `report.json` with cases, timing, executed actions,
+capability reasons, coverage and external-project metadata integrity. Unavailable project
+metadata is reported separately in `integrity.unreadable`, not as confirmed drift;
+integrity remains unverified and the run fails until those reads succeed. `clean` retains
+`cleanup-report.json`, including deletion failures; starting teardown invalidates lab
+readiness even if cleanup only partially succeeds. If cleanup and integrity verification
+both fail, the cleanup error remains primary and the report preserves both results.
+Planned lifecycle commands retain ownership checks; local outputs stay inside the lab
+for both `--output` and `--output-file`. Treat local reports and exported archives as private project
+data. Coverage is an explicit inventory of every registered action, not a claim that
+every action has a live test: unexercised actions remain **uncovered**, and the
+expanded demonstration keeps each registered action visible as demonstrated, blocked
+with a recorded prerequisite, or uncovered — never silently dropped. Add cases through
+the typed catalogue in `tests/live-cases.ts`, `LiveContext.check` and guarded helpers,
+then update `tests/live-coverage.ts` when the
+registry changes; catalogue drift fails the offline gate. Plan/dry-run calls do not count
+as executed live coverage.
+
+Regular `bun run check` and `bun run lint` include the live runner and fixtures.
+Lint includes project-owned rules in `tools/oxlint/index.ts`: accumulator-copy, widen-then-assert, and chained-assertion errors, alongside the native accumulating-spread check.
+For focused verification:
+
+```sh
+bun run check:live
+bun test tests/live-context.test.ts tests/live-runner.test.ts tests/live-coverage.test.ts
+```
+
 ## CLI contract
 
 The complete agent-facing contract — stdout JSON discipline, stderr JSONL diagnostics, the error
 envelope, exit codes, `--fields` projection, command discovery, and planning/safety rules — is the
-canonical skill at [`skills/dataiku-dss/SKILL.md`](skills/dataiku-dss/SKILL.md); keep that file as
-the single source of truth for agent integrations and for the `dss install-skill` payload. Human
+skill bundle rooted at [`skills/dataiku-dss/SKILL.md`](skills/dataiku-dss/SKILL.md). The short
+entrypoint routes agents to focused references; the installed command registry remains the source
+of truth for command syntax and schemas. Human
 essentials: exactly one compact JSON value on stdout per command (void success is `{ok:true}`),
 failures as one structured JSON error object on stdout with a nonzero exit code, stderr reserved
 for JSONL diagnostics only, and exit codes `0` success / `1` usage error / `2` DSS or internal
@@ -46,8 +214,7 @@ dss commands run --output commands.json
 dss agent contract --fields protocol,agentContractVersion,cli,stdio,planning,compatibility
 ```
 
-`dss commands run` prints the compact resource/action summary; registry entries (~220k tokens
-exported) never dump to stdout — they travel only via `--output PATH`. See the canonical skill for
+`dss commands run` prints the compact resource/action summary; registry entries never dump to stdout — they travel only via `--output PATH`. See the canonical skill for
 the full `--fields` projection rules and planning workflow.
 
 ## Agent skill installation
@@ -61,7 +228,7 @@ dss install-skill --global --agent omp
 
 `--list-agents` only reports targetable agents; it does not write files. Auto-detection checks supported agent binaries/config directories (`claude`, `codex`, `cursor`, `pi`, `omp`). Passing `--agent NAME` forces one entry and reports `via:"flag"`.
 
-Project installs write `SKILL.md` under the target workspace:
+Project installs write `SKILL.md` and its sibling `references/` directory under the target workspace:
 
 - Claude: `.claude/skills/dataiku-dss/SKILL.md`
 - Codex: `.codex/skills/dataiku-dss/SKILL.md`
@@ -69,16 +236,44 @@ Project installs write `SKILL.md` under the target workspace:
 - Pi: `.pi/skills/dataiku-dss/SKILL.md`
 - OMP: `.omp/skills/dataiku-dss/SKILL.md`
 
-Global installs write under the agent's home config path, for example OMP: `~/.omp/agent/skills/dataiku-dss/SKILL.md`.
+Global installs write under the agent's home config path, for example OMP: `~/.omp/agent/skills/dataiku-dss/SKILL.md`. The reported `target` is the home directory for global installs and the resolved workspace directory for project installs; `installed[].path` names each entrypoint.
+
+Each entry in the `installed` array reports pre-install bundle state:
 
-Each entry in the `installed` array reports deterministic file state:
+- `status`: `missing` when the entrypoint is absent; `stale` when any bundled file is missing or differs; `current` when every bundled file matches.
+- `changed`: whether any bundled file would (or did) change. Byte-identical files are not rewritten.
+- `expectedSha256` and `actualSha256`: the canonical and existing **entrypoint** hashes. The latter is absent when the entrypoint is missing.
+- `files`: each bundled file's portable `relativePath`, destination `path`, `status`, `changed`, and hashes. Missing or stale references are detected even when `SKILL.md` itself is current.
 
-- `status`: `missing` (no file at the destination), `stale` (file present but bytes differ from the canonical skill), or `current` (byte-identical to the canonical skill).
-- `changed`: whether the install writes the file — `true` for `missing` and `stale`, `false` for `current`; installs skip byte-identical files and replace missing/stale files through a same-directory temporary file followed by rename.
-- `expectedSha256`: SHA-256 of the canonical skill bytes this SDK ships.
-- `actualSha256`: SHA-256 of the file currently at the destination; present only when the file exists.
+`--dry-run` and `--plan` never write. Real installs replace changed files through same-directory temporary files and rename, writing references before the entrypoint. Atomicity is per file, not a bundle-wide transaction. Unrelated files in the destination are preserved.
 
-`--dry-run` and `--plan` never write; they report the same `status`/`changed`/hash fields so you can see which copies a real install would refresh — a `stale` copy is detected and would be refreshed, while a `current` copy is already byte-identical and needs no refresh.
+### Token-efficient skill structure
+
+The entrypoint contains bootstrap/discovery, output interpretation, and universal safety rules. Seven focused files under [references/](skills/dataiku-dss/references/) cover authentication, discovery, mutations, application releases, flow maps, coding, and troubleshooting. All are linked directly from `SKILL.md` with task-specific read conditions; agents should not preload the whole directory.
+
+The entrypoint measures **774 o200k_base tokens**, within its **825-token** budget. Current Claude V5 reconstruction and GLM5 counts are **1,297** and **776**. Safety rules and all seven reference links remain. Reference content consumes context only when read; all five agent installers ship the complete bundle.
+
+All seven references are token-budgeted (each measured `o200k_base` baseline plus 5% headroom). Per-file baselines live in [the token-budget tests](tests/cli/agent-token-budget.test.ts); run `bun run test:tokens` to check them.
+
+References total **3,797 o200k_base tokens**; including the entrypoint, the bundle totals **4,571**. This is a sum of file counts, not a recommendation to preload them. Runnable examples, sharing safeguards, and cleanup conditions are retained.
+
+This uses progressive disclosure from the [Agent Skills specification](https://agentskills.io/specification) and [Anthropic's authoring guidance](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices): concise metadata/instructions, direct relative reference links, and domain-focused documents. No scripts or assets are needed here; generated command metadata is queried from the CLI rather than duplicated in the skill.
+
+`bun run test:tokens` enforces the pinned OpenAI `o200k_base` budgets. To also check ten native encodings (OpenAI, Claude, GLM, Qwen, DeepSeek, and Kimi), point the optional development-only gate at an OMP native module:
+
+```bash
+DSS_TOKENIZER_MODULE=/absolute/path/to/oh-my-pi/packages/natives/native/index.js bun run test:tokens
+```
+
+When configured, the native module also accelerates ordinary OpenAI token measurements after parity checks against the pinned JavaScript model. The JS fingerprint and independent cross-model reference checks remain authoritative; reserved special-token validation is preserved. Without this variable, measurements use JavaScript and only the native cross-model case is skipped. A configured missing/incompatible module fails instead of silently falling back. The cross-model gate permits 10% growth over recorded text baselines. Claude counts use ctok reconstructions, not an official tokenizer; counts exclude provider framing. No native dependency is added to the CLI.
+
+For a known resource, `dss agent contract --fields commands.actions.dataset` lists action names in **64 tokens**, versus **1,575** for the global index. For known writes, fetch the full action entry directly once: `dataset.create` costs **1,059 tokens**, versus **1,415** for compact discovery followed by the required full entry. Use compact `usage,description,flags,examples` projections for reads when those fields suffice. This is selective disclosure, not lossless compression. Scoped discovery builds only selected resource metadata; action-name indexes avoid constructing payload schemas, while full registry exports and recovery hints retain their existing content.
+
+Boolean flags accept bare flags or explicit `=true`/`=false` values, including aliases. Invalid boolean values fail before dispatch. Direct, batch, and local commands share execution-mode resolution; `--plan --dry-run` returns a plan with `plannedAndDryRun:true` when both modes are supported. Unreadable JSON/text payload files produce `validation_failed` usage errors (exit 1) with the flag, path, and filesystem cause.
+
+For raw streaming downloads, `--request-timeout MS` bounds each pending body read, not the total transfer duration: large progressing transfers remain supported. Buffered text/JSON consumers retain their bounded body-read budget. Request retries do not replay partially consumed streams. Project, bundle, and API-service archives and managed-folder downloads are written to private temporary files beside the destination, then renamed on success. Failures leave no partial output and preserve any existing destination.
+
+The SDK configuration `maxResponseBodyBytes` (default **50 MiB**) also bounds retained non-2xx response bodies. Oversized errors preserve HTTP status, header request IDs, and a UTF-8-safe prefix, with `DataikuError.bodyTruncated: true`; CLI errors expose `details.bodyTruncated: true`. Retry eligibility is unchanged, and error-body reads share the transport attempt deadline.
 
 ## Credentials
 
@@ -142,11 +337,14 @@ dss recipe get-payload compute_orders --output code.py --project-key MYPROJ
 dss install-skill --dry-run
 dss app validate-manifest --project-key MYAPP_INSTANCE
 dss app compare-manifest my-app --project-key MYAPP_INSTANCE
-dss app create-instance my-app --data '{"targetProjectKey":"MYAPP_INSTANCE"}' --wait --record-cleanup cleanup.jsonl
+dss app create-instance my-app --data '{"targetProjectName":"My app instance"}' --wait --record-cleanup cleanup.jsonl
+# Substitute the returned projectKey for MYAPP_INSTANCE in later commands.
 dss app manifest-version --project-key MYAPP_TEMPLATE
 dss app successor-preflight my-app --from MYAPP_INSTANCE --to MYAPP_INSTANCE_V2 --copy-permissions
 dss app set-manifest-version --manifest-version 1.4.0 --expect-hash PREFLIGHT_TEMPLATE_MANIFEST_HASH --project-key MYAPP_TEMPLATE
 dss app create-successor-instance my-app --from MYAPP_INSTANCE --to MYAPP_INSTANCE_V2 --copy-permissions --record-cleanup cleanup.jsonl
+# Or omit --to: the target key is generated during apply (preflight/plan allocate nothing).
+dss app create-successor-instance my-app --from MYAPP_INSTANCE --copy-permissions --record-cleanup cleanup.jsonl
 dss app verify-instance my-app --project-key MYAPP_INSTANCE_V2 --expect-version 1.4.0
 dss app permissions-snapshot --project-key MYAPP_INSTANCE --output permissions.json
 dss app permissions-diff --project-key MYAPP_INSTANCE --file permissions.json
@@ -193,8 +391,10 @@ dss app compare-manifest APP_ID --project-key RELEASE_INSTANCE
 dss app manifest-version --project-key APP_TEMPLATE
 
 # Run every read-only successor gate before changing the template version. This validates the
-# template references, verifies the predecessor, proves target absence, and optionally snapshots
-# the predecessor ACL. Keep template.manifestHash from the result for the next command.
+# template references, verifies the predecessor, proves target absence for an explicit --to, and
+# optionally snapshots the predecessor ACL. Keep template.manifestHash from the result for the
+# next command. Omit --to to leave the target to apply: preflight then reports
+# targetProjectKeyGeneratedDuringApply with no absence probe and no allocation.
 dss app successor-preflight APP_ID \
   --from RELEASE_INSTANCE --to RELEASE_INSTANCE_V2 --copy-permissions
 
@@ -208,20 +408,24 @@ dss app successor-preflight APP_ID \
 dss app set-manifest-version --manifest-version 1.4.0 \
   --expect-hash PREFLIGHT_TEMPLATE_MANIFEST_HASH --project-key APP_TEMPLATE
 
-# Create an instance only after its target key is confirmed absent, wait on the DSS future, and
-# record deterministic cleanup.
+# Create a fresh instance without global project visibility; the CLI generates the key.
+# Use the returned projectKey for verification and later commands; cleanup binds that instance.
 dss app create-instance APP_ID \
-  --data '{"targetProjectKey":"RELEASE_INSTANCE","targetProjectName":"Release instance"}' \
+  --data '{"targetProjectName":"Release instance"}' \
   --wait --record-cleanup cleanup.jsonl
 
 # Roll out a new template version to an existing instance as an additive successor: the old
 # instance is never modified or deleted, and the command always waits on the DSS future
 # (there is no --wait flag). The recorded cleanup entry targets only the new project key. If DSS
 # may have accepted creation but no future ID is returned, the outcome is indeterminate and no
-# unbound cleanup entry is written.
+# unbound cleanup entry is written. Omit --to to generate the target key during apply: preflight
+# and plans stay allocation-free, the key is generated once before the single POST, and cleanup
+# binds it through the creation future plus creationTag.
 dss app create-successor-instance APP_ID \
   --from RELEASE_INSTANCE --to RELEASE_INSTANCE_V2 \
   --name "Release instance v2" --copy-permissions --record-cleanup cleanup.jsonl
+dss app create-successor-instance APP_ID \
+  --from RELEASE_INSTANCE --name "Release instance v2" --copy-permissions --record-cleanup cleanup.jsonl
 
 # API readiness gate for the successor. apiReady:true with status API_VERIFIED_UI_PENDING is an
 # API-verified state only — never visual verification. The API key authenticates public REST only;
@@ -246,24 +450,33 @@ restore reject snapshots from another DSS server, project key, or observed incar
 client-side, non-atomic stale-identity checks: DSS exposes no conditional permission PUT, so the
 checks narrow and detect key-reuse races but cannot serialize the final check with the write.
 Snapshots contain access-control identities; commit them only when repository policy permits.
-`validate-manifest` checks `SCENARIO_RUN` scenario IDs, `DOWNLOAD_FILE`
-managed-folder IDs, and runtime-form parameter names against supported public project APIs.
-
-Some DSS deployments hide unknown project keys behind `403` instead of returning `404`.
-Instance creation requires confirmed target absence before POST: an inaccessible target not present
-in the visible project and app-instance lists is still unconfirmed and is rejected with
-`target_absence_unverifiable` / `permission_or_environment` rather than risking cleanup against a
-pre-existing project. Choosing another key does not solve deployments that mask every unknown key.
-DSS exposes no permission-independent public availability endpoint, and its published app-instance
-API does not guarantee duplicate-key rejection before any write or non-overwrite behavior, so
-there is no unsafe force or unconfirmed-target bypass. Use an identity with global project
-visibility. A definitive create rejection never produces a cleanup entry; an ambiguous POST
-without a future ID or verified
-incarnation also produces no cleanup entry.
-
-Static create plans describe, but do not execute, these reads:
-`preflightExecuted:false` and `preflightWillRunDuringApply:true`. Run `successor-preflight` when
-live read evidence is required before changing the template version.
+`validate-manifest` checks `SCENARIO_RUN` scenario IDs, `DOWNLOAD_MANAGED_FOLDER_FILE`
+folder IDs, and `PROJECT_VARIABLES_EDIT` parameter names against supported public project APIs.
+Unknown tile types and custom content remain opaque; reference validation is not UI-rendering proof.
+
+Fresh `app create-instance` requests can omit `targetProjectKey`: the CLI generates a cryptographic
+`APP_` + UUIDv4 key locally (122 random bits), without global visibility probes. It preserves
+`targetProjectName`; if omitted, the name defaults to the key. Normal app-instantiation permissions
+still apply. Use the returned `projectKey` for verification and later commands. Only omission
+selects generation; explicit blank, null, or non-string keys are rejected. Collision risk is
+negligible, not zero; DSS does not document atomic duplicate-key rejection. This follows the
+[official client's random-key pattern](https://github.com/dataiku/dataiku-api-client-python/blob/master/dataikuapi/dss/app.py).
+
+Caller-chosen keys, including successor `--to`, still require confirmed absence before POST.
+A masked `403` remains `target_absence_unverifiable` / `permission_or_environment`, even if absent from
+visible lists. DSS has no public availability endpoint or force bypass; exact keys need global
+visibility. Otherwise omit the key for a fresh instance, or omit successor `--to` to generate the
+successor target during apply. Definitive rejections produce no cleanup entry; ambiguous POSTs
+without a future ID or verified incarnation produce none either.
+
+Plans allocate and reserve nothing. Direct `create-instance` generated-key plans report
+`preflightExecuted:false` and `preflightWillRunDuringApply:false`; explicit-key plans report the
+latter as `true`. `create-successor-instance` plans always report
+`preflightWillRunDuringApply:true` — with `--to` the read-only gates re-run during apply, and
+without `--to` the target key is generated only then
+(`targetProjectKeyGeneratedDuringApply:true`, `preflightExecuted:false`). Use `--plan`
+for preview (`create-instance` does not support `--dry-run`). Run `successor-preflight` for named
+releases when live read evidence is required before changing the template version.
 
 Every new cleanup entry records the canonical DSS URL. App-instance cleanup records a
 `creationTag` hash observed after the DSS future identifies the target key; a future-addressable
@@ -432,7 +645,9 @@ dss notebook unload-jupyter --all --dry-run --project-key MYPROJ
 Code-environment reads expose definitions, deployment mode, installed/requested packages, project
 versions, usages, and bounded build logs. Fetch the definition/hash before `set-definition` or
 `set-packages --expect-hash`; package, image, Jupyter, create, and delete operations can return DSS
-futures with `--no-wait`.
+futures with `--no-wait`. `set-packages` replaces the requested package list wholesale: an
+explicitly empty `--packages ''` value (or an empty `--file`) clears the requested specs, while a
+call with no package source flag at all is a usage error.
 
 Webapp updates use GET–deep-merge–PUT; API-service `save-settings` is explicit full replacement.
 Run either with `--dry-run` to inspect `currentHash`/`nextHash`, then apply with

package/dist/build-metadata.json

@@ -1 +1 @@
-{"format":1,"buildRevision":"ceeb399ea1f0d7ea078db09da940b79fb732e2ad"}
+{"format":1,"buildRevision":"944ec2a6e0780142399032675854ef6b4ac957a7"}

package/dist/packages/types/src/index.d.ts

@@ -59,6 +59,12 @@ export declare const ProjectMetadataSchema: import("@sinclair/typebox").TObject<
     }>>;
 }>;
 export type ProjectMetadata = Static<typeof ProjectMetadataSchema>;
+export declare const ProjectTagsSchema: import("@sinclair/typebox").TObject<{
+    tags: import("@sinclair/typebox").TRecord<import("@sinclair/typebox").TString, import("@sinclair/typebox").TObject<{
+        color: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    }>>;
+}>;
+export type ProjectTags = Static<typeof ProjectTagsSchema>;
 export declare const DatasetSummarySchema: import("@sinclair/typebox").TObject<{
     name: import("@sinclair/typebox").TString;
     type: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
@@ -308,6 +314,85 @@ export declare const ScenarioWaitResultSchema: import("@sinclair/typebox").TObje
     }>>>;
 }>;
 export type ScenarioWaitResult = Static<typeof ScenarioWaitResultSchema>;
+/** A macro definition as returned by GET /projects/{pk}/runnables/{runnableType}. */
+export declare const MacroDefinitionSchema: import("@sinclair/typebox").TObject<{
+    runnableType: import("@sinclair/typebox").TString;
+    ownerPluginId: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    meta: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TObject<{
+        label: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    }>>;
+    longDescription: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    resultType: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    extension: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    mimeType: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    resultLabel: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    params: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TArray<import("@sinclair/typebox").TObject<{
+        name: import("@sinclair/typebox").TString;
+        type: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    }>>>;
+    adminParams: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TArray<import("@sinclair/typebox").TUnknown>>;
+}>;
+export type MacroDefinition = Static<typeof MacroDefinitionSchema>;
+/** One entry of GET /projects/{pk}/runnables (array[Macro]). */
+export declare const MacroSummarySchema: import("@sinclair/typebox").TObject<{
+    runnableType: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    /** Owning plugin id as the server returns it (live identifiers are server-authored). */
+    ownerPluginId: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    meta: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TObject<{
+        label: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    }>>;
+}>;
+export type MacroSummary = Static<typeof MacroSummarySchema>;
+export declare const MacroSummaryArraySchema: import("@sinclair/typebox").TArray<import("@sinclair/typebox").TObject<{
+    runnableType: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    /** Owning plugin id as the server returns it (live identifiers are server-authored). */
+    ownerPluginId: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    meta: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TObject<{
+        label: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    }>>;
+}>>;
+export type MacroSummaryArray = Static<typeof MacroSummaryArraySchema>;
+/**
+ * Poll-state response of GET /projects/{pk}/runnables/{runnableType}/state/{run}.
+ * While running: `running: true` and optionally a stack of `progress` entries.
+ * When finished: `running: false`; on failure `resultError`/`storedError` carry
+ * details, otherwise `type` is filled.
+ */
+export declare const MacroStateSchema: import("@sinclair/typebox").TObject<{
+    exists: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TBoolean>;
+    running: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TBoolean>;
+    empty: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TBoolean>;
+    type: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    progress: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TUnknown>;
+    resultError: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TUnknown>;
+    storedError: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TUnknown>;
+}>;
+export type MacroState = Static<typeof MacroStateSchema>;
+/** Run trigger response of POST /projects/{pk}/runnables/{runnableType}. */
+export declare const MacroRunStartSchema: import("@sinclair/typebox").TObject<{
+    runId: import("@sinclair/typebox").TString;
+}>;
+export type MacroRunStart = Static<typeof MacroRunStartSchema>;
+/** Result of {@link MacrosResource.runAndWait}: bounded client-side polling. */
+export declare const MacroWaitResultSchema: import("@sinclair/typebox").TObject<{
+    runnableType: import("@sinclair/typebox").TString;
+    runId: import("@sinclair/typebox").TString;
+    running: import("@sinclair/typebox").TBoolean;
+    success: import("@sinclair/typebox").TBoolean;
+    elapsedMs: import("@sinclair/typebox").TNumber;
+    pollCount: import("@sinclair/typebox").TNumber;
+    timedOut: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TBoolean>;
+    state: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TObject<{
+        exists: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TBoolean>;
+        running: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TBoolean>;
+        empty: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TBoolean>;
+        type: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+        progress: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TUnknown>;
+        resultError: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TUnknown>;
+        storedError: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TUnknown>;
+    }>>;
+}>;
+export type MacroWaitResult = Static<typeof MacroWaitResultSchema>;
 export declare const FlowZoneObjectTypeSchema: import("@sinclair/typebox").TUnion<[import("@sinclair/typebox").TLiteral<"DATASET">, import("@sinclair/typebox").TLiteral<"MANAGED_FOLDER">, import("@sinclair/typebox").TLiteral<"SAVED_MODEL">, import("@sinclair/typebox").TLiteral<"RECIPE">, import("@sinclair/typebox").TLiteral<"MODEL_EVALUATION_STORE">, import("@sinclair/typebox").TLiteral<"STREAMING_ENDPOINT">, import("@sinclair/typebox").TLiteral<"LABELING_TASK">, import("@sinclair/typebox").TLiteral<"RETRIEVABLE_KNOWLEDGE">]>;
 export type FlowZoneObjectType = Static<typeof FlowZoneObjectTypeSchema>;
 export declare const FlowZoneItemSchema: import("@sinclair/typebox").TObject<{
@@ -658,6 +743,11 @@ export declare const SqlNotebookCellSchema: import("@sinclair/typebox").TObject<
     id: import("@sinclair/typebox").TString;
     type: import("@sinclair/typebox").TString;
     name: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+    /**
+     * Cell source. DSS enforces a JSON string here: an array fails to parse
+     * server-side (`Expected a string but was BEGIN_ARRAY ... $.cells[0].code`),
+     * so the official-client example's line-array form does not apply to SQL.
+     */
     code: import("@sinclair/typebox").TString;
 }>;
 export type SqlNotebookCell = Static<typeof SqlNotebookCellSchema>;
@@ -668,12 +758,24 @@ export declare const SqlNotebookSummarySchema: import("@sinclair/typebox").TObje
     connection: import("@sinclair/typebox").TString;
 }>;
 export type SqlNotebookSummary = Static<typeof SqlNotebookSummarySchema>;
+/**
+ * A SQL notebook's content as returned by GET. The notebook record carries
+ * the server-allocated `id` (a request-supplied `id` is rejected on create)
+ * plus `name`/`projectKey`/`isRunning` and similar receipt fields via
+ * `additionalProperties`; requests supply `connection` + `cells` (and
+ * optionally `name` as the display handle).
+ */
 export declare const SqlNotebookContentSchema: import("@sinclair/typebox").TObject<{
     connection: import("@sinclair/typebox").TString;
     cells: import("@sinclair/typebox").TArray<import("@sinclair/typebox").TObject<{
         id: import("@sinclair/typebox").TString;
         type: import("@sinclair/typebox").TString;
         name: import("@sinclair/typebox").TOptional<import("@sinclair/typebox").TString>;
+        /**
+         * Cell source. DSS enforces a JSON string here: an array fails to parse
+         * server-side (`Expected a string but was BEGIN_ARRAY ... $.cells[0].code`),
+         * so the official-client example's line-array form does not apply to SQL.
+         */
         code: import("@sinclair/typebox").TString;
     }>>;
 }>;