@datarails-ai/dr-cli 1.6.152 → 1.6.154

package/dist/scopes-map.json

@@ -1,6 +1,6 @@
 {
   "version": 1,
-  "cliVersion": "1.6.152",
+  "cliVersion": "1.6.154",
   "_contract": "Resolve a command path through `aliases` (canonical prefix + alias segment -> canonical path) before any lookup. A path listed in `pendingCommands`, or whose first segment is a key of `pending`, is UNDECIDED, not unrestricted: fail closed or skip it, and never read a path's absence from `commands` as 'no requirement'. An entry carrying `denied` is UNAVAILABLE to any AGENT-SCOPED credential — one whose `scopes` claim carries at least one `resource:action` entry; a claim that is absent, empty, capability-only or wholly unparseable is UNRESTRICTED, and this mark does not apply to it — whatever else that entry says: it is not a ceiling and it is not always-available, its `denied.reason` says why and `denied.until` names the ticket that lifts it, and the absence of `requires` on it must NEVER be read as 'no requirement'. Only an entry in `commands` with an empty `requires` means always-available.",
   "commands": {
     "login": {
@@ -2207,6 +2207,24 @@
         "until": "DR-56164"
       }
     },
+    "connectors aws set-role": {
+      "denied": {
+        "reason": "The connectors control plane runs no dr-auth-sdk guard and checks no scope, so the `integrations` ceiling this command declares would be enforced by nobody. Closed to scoped credentials until that control plane adopts the SDK.",
+        "until": "DR-56164"
+      }
+    },
+    "connectors aws rotate-external-id": {
+      "denied": {
+        "reason": "The connectors control plane runs no dr-auth-sdk guard and checks no scope, so the `integrations` ceiling this command declares would be enforced by nobody. Closed to scoped credentials until that control plane adopts the SDK.",
+        "until": "DR-56164"
+      }
+    },
+    "connectors aws delete": {
+      "denied": {
+        "reason": "The connectors control plane runs no dr-auth-sdk guard and checks no scope, so the `integrations` ceiling this command declares would be enforced by nobody. Closed to scoped credentials until that control plane adopts the SDK.",
+        "until": "DR-56164"
+      }
+    },
     "filebox list": {
       "requires": [
         {

package/dist/skills/schedule-drip-task/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: schedule-drip-task
-description: Use when a customer needs files or data pushed OUT of Datarails on a schedule through DRIP (the connectors platform) - an SFTP export, "send this CSV to the customer's SFTP every hour", "push the file to their server", "schedule an export", "scheduled task", "dr connectors tasks", "sftp_export", "lib/sftp_export", or when operating an existing schedule - "is the export running", "why did the export fail", "the task is paused", "run the export now", "pause the export", "task runs", "consecutive failures", "paused (failures)", "paused (schedule)", "no active version", "newest source file is older than max_age_seconds", "the target must not be a customer org-*-sftp container", a 403 "scheduled tasks need a support user or an org admin". Not for pulling data INTO Datarails (drip-implement-connector) or for polled SFTP ingestion (dr connectors poll).
+description: Use when a customer needs files or data pushed OUT of Datarails on a schedule through DRIP (the connectors platform) - an SFTP export, "send this CSV to the customer's SFTP every hour", "push the file to their server", "schedule an export", "scheduled task", "dr connectors tasks", "sftp_export", "lib/sftp_export", or when operating an existing schedule - "is the export running", "why did the export fail", "the task is paused", "run the export now", "pause the export", "task runs", "consecutive failures", "paused (failures)", "paused (schedule)", "no active version", "newest source file is older than max_age_seconds", "the target must not be a customer org-*-sftp container", a 403 "scheduled tasks need a support user or an org admin", or when writing a new task type - "write a scheduled task", "new DRIP task", "process() TaskResult", "run hook", "task state". Not for pulling data INTO Datarails (drip-implement-connector) or for polled SFTP ingestion (dr connectors poll).
 ---
 
 # Schedule a DRIP task
@@ -10,6 +10,9 @@ datasources, and keeps a run history. The control plane only schedules; the runn
 artifact's `process()`. The first task type is **SFTP Export** (`lib/sftp_export`): copy files from
 our hosted SFTP to a customer's SFTP under a name the customer wants.
 
+To write a task type other than SFTP Export (the `process()` hook, `TaskResult`, state, retries),
+see [reference/writing-a-task.md](reference/writing-a-task.md).
+
 Commands: `dr connectors tasks create | list | get | update | delete | run | pause | resume | runs`.
 Every one takes `--server`, `--account` and `--json`; a `<task>` is its id or its name.
 

package/dist/skills/schedule-drip-task/reference/sftp-export.md

@@ -16,10 +16,11 @@ datasources (the control plane refuses secret-looking keys with a 422).
 | `settle_seconds` | no | 120 | Skip files modified more recently (the producer may still be writing). 0 to 7 days |
 | `max_age_seconds` | no | none | `newest` only: fail the run if the newest file is older than this before the scheduled time, or if no file matches (producer late or stopped) |
 | `max_files_per_run` | no | 50 | `all_new`: at most this many files per run (1 to 1000); the rest go next run |
+| `upload` | no | `atomic` | `atomic`: write a `.part`, then rename it, so no reader sees a half-written file. `direct`: write the final name itself, for a target folder that takes files away as they arrive (the `.part` would be gone before the rename). `newest` only |
 
 Behaviour worth knowing:
 
-- Upload is atomic: a random `.<name>.<hex>.part` is written, then renamed. The customer's login must
+- Upload is atomic by default: a random `.<name>.<hex>.part` is written, then renamed. A run that fails with "a file vanished while publishing the upload" means the target takes files as they arrive: set `"upload": "direct"`. The customer's login must
   be allowed to create and rename files in `target_dir`. Stale parts of ours older than 2 h are swept.
 - The source is never written, moved or deleted. Cleaning it is the producer's job; with `all_new`,
   a folder that keeps every sent file eventually fails the run with "archive or delete them".

package/dist/skills/schedule-drip-task/reference/writing-a-task.md

@@ -0,0 +1,92 @@
+# Writing a new task type
+
+A scheduled task is an ordinary DRIP artifact in dr-connectors (a connection plus a dataquery,
+published and made active like any connector, see `drip-implement-connector`), whose dataquery
+implements `process()`. The control plane only schedules it. On each run it sends the flavor's
+runner a job with `hook: "run"`, and the runner calls `connect()`, then `process()`, in one job.
+The full contract is in dr-connectors-runner `docs/connector-kit.md`, "Scheduled tasks: the `run`
+hook". `lib/sftp_export` in dr-connectors is the reference implementation.
+
+## The code
+
+```python
+# lib/<task>/dataqueries/<name>.py
+from dr_integrations_sdk import DataQuery, PermanentError, SyncRequest, TaskResult, kit
+
+OBJECT_TYPES = ("<objectType>",)
+
+
+class MyTask(DataQuery):
+    async def process(self, session, req: SyncRequest) -> TaskResult:
+        config = req.config                                   # the task's --config, validated here
+        state = (req.metadata or {}).get("state") or {}       # what the last ok run returned
+        scheduled_for = req.metadata["scheduled_for"]         # ISO UTC time of the slot this run is for
+        other = await kit.auth.get_credentials_for(req, "source")  # a non-primary datasource, by role
+        sent = await do_the_work(session, other, config, state)
+        return TaskResult(state={"sent": sent}, summary={"sent": len(sent)})
+
+
+def get_query_class(object_type: str):
+    if object_type == "<objectType>":
+        return MyTask
+    raise PermanentError(f"unknown objectType {object_type!r}")
+```
+
+| Piece | Rule |
+|---|---|
+| `connect()` | Gets the **primary** datasource's credentials as for any connector. Log in here, so a wrong password or host key fails before any work |
+| `req.config` | The task's `--config` JSON. Must carry `objectType` (it picks the dataquery class). Validate it before any login. Never put credentials in it: the control plane refuses secret-looking keys with a 422 |
+| Other datasources | `kit.auth.get_credentials_for(req, "<role>")`, one per `--datasource <role>=<id>`. An unbound role is a `PermanentError` naming it |
+| `req.metadata` | `state`, `scheduled_for`, `run_id`, `attempt` (1, or 2 on the retry), `trigger` (`schedule` or `manual`) |
+| Imports | stdlib, `dr_integrations_sdk`, `lib` and the flavor's allowlist only (dr-connector-images). Set `RUNTIME_FLAVOR` in the connection |
+| Logs | `kit.logging.get_logger("<task>")`. They reach Groundcover under the runner flavor's workload |
+
+## The result
+
+`TaskResult(state=…, summary=…, verdict=…)`. Every field is optional.
+
+- **`state`**: the task's own bookkeeping (ids, cursors, names already sent), never the data
+  processed. It must be a JSON object or `None`, at most 64 KiB. The control plane keeps it only
+  from an `ok` run; after a failure the next run gets the last good state.
+- **`summary`**: what the run did, shown by `dr connectors tasks runs`.
+- **`verdict`**: `ok` (default), `retry` (the run is sent once more after 60 s) or `block` (the run
+  fails).
+- The whole result must be JSON (no `datetime`, `set`, `Decimal`, NaN) and at most 128 KiB.
+  Anything else fails the run as a connector error.
+
+## Errors
+
+| Raised | Run |
+|---|---|
+| `PermanentError`, `AuthError` | Fails, no retry. Use for bad config, refused credentials, missing folders |
+| `TransientError`, `httpx` 429/5xx, `httpx.TransportError` | Retried once after 60 s, then fails |
+| Anything else | Fails as a connector error |
+
+- After 5 failed runs in a row the task pauses itself (`paused (failures)`).
+- The control plane stores only the first few hundred characters of a run's error. Keep the message
+  short. Log anything longer, such as a key or a listing, and read it in Groundcover.
+- Never put a secret in an error or a log line.
+
+## Make it safe to run twice
+
+A retry runs `process()` again, a manual run can land next to a scheduled one, and a run that dies
+leaves no state. So:
+
+- Record what was done in `state`, and skip it next time.
+- Write the output so that a half-finished run leaves nothing a reader would take as done. For
+  example, upload to a temporary name and rename it.
+- Use `scheduled_for`, not the clock, for anything named after the run's time: a retry then makes
+  the same name.
+
+## Ship it
+
+1. **Shared code goes in `lib/<task>/`**, with tests. A thin per-org artifact re-exports it. The
+   dataquery must set its own `CONNECTION` (the thin connection file's name), as in
+   [SKILL.md](../SKILL.md) step 3.
+2. **Merge to dr-connectors.** CI publishes the artifact to JFrog.
+3. **Register and activate it on the region:** `dr connectors publish <dataquery file>`, then
+   `dr connectors promote <artifact> --artifact-version v<N>`, as the customer's support user.
+4. **Schedule it:** `dr connectors tasks create` (see [SKILL.md](../SKILL.md)), then `tasks run` once
+   and check `tasks runs`.
+5. **Test on dev-1 first.** The heartbeat artifact in dr-connectors
+   (`dev-1/69_(test-connectors.com)/heartbeat/`) is a minimal task that carries a counter in `state`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@datarails-ai/dr-cli",
-  "version": "1.6.152",
+  "version": "1.6.154",
   "description": "Datarails CLI",
   "type": "module",
   "engines": {