opencode-skills-collection 4.0.68 → 4.0.70

package/bundled-skills/apify-integration-development/references/sdk-integration.md

@@ -0,0 +1,236 @@
+# SDK integrations
+
+Design guide for integrating Apify into an existing application by calling Actors directly with `apify-client` (JS/TS or Python) or the REST API. This is the lightest integration shape: no host platform, no tool registry, no plugin lifecycle - just your code calling Apify as a backend service. Apply the cross-cutting rules from `SKILL.md` on top.
+
+## 1. Wrap apify-client, never the Actor SDK
+
+- **`apify-client`** is the API client for **calling** Actors from your app.
+- **`apify`** is the SDK for **building** Actors (wrong package for this use case).
+
+Always install `apify-client`. Never install `apify` for integration work. Keep the dependency footprint small to minimize version conflicts and keep install time short.
+
+Stamp a custom `user-agent` suffix or the attribution header (`x-apify-integration-platform: <your-app>`) on the client so Apify can attribute traffic. If the integration was built using the Apify integration development skill, also set `x-apify-integration-origin: apify-integration-development-skill`.
+
+## 2. Token handling
+
+Get an `APIFY_TOKEN` from **Console > Settings > Integrations** at `https://console.apify.com/settings/integrations`. Account sign-up: `https://console.apify.com/sign-up` (free, no credit card). Store the token in an environment variable or a secrets manager - never hardcoded, never in chat logs or command output, never in URLs (query-string tokens leak through browser history and server logs).
+
+The token is a normal Bearer credential:
+
+```
+Authorization: Bearer <APIFY_TOKEN>
+```
+
+Use scoped tokens where possible and rotate them periodically.
+
+## 3. Find the right Actor before writing code
+
+Before writing integration code, find the Actor that fits the need. Use the MCP tools if available in your environment:
+
+- `search-actors` - search the Apify Store by keyword (search by platform/product name, not end goal).
+- `fetch-actor-details` - get the Actor's input schema, output format, and pricing.
+
+Alternatively, browse `https://apify.com/store`. Append `.md` to any Actor's Store URL to get its docs in markdown (e.g. `https://apify.com/apify/web-scraper.md`). Build input from the schema rather than guessing field names.
+
+## 4. JavaScript / TypeScript
+
+### Install
+
+```bash
+npm install apify-client
+```
+
+### Synchronous execution (wait for results)
+
+```typescript
+import { ApifyClient } from 'apify-client';
+
+const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
+
+const run = await client.actor('apify/web-scraper').call({
+    startUrls: [{ url: 'https://example.com' }],
+    maxPagesPerCrawl: 10,
+}, { maxTotalChargeUsd: 5 });
+
+const { items } = await client.dataset(run.defaultDatasetId).listItems();
+```
+
+`.call()` blocks until the Actor finishes. Use for short-running Actors (under a few minutes). Pass `maxTotalChargeUsd` in the **options** argument - never in the input.
+
+### Asynchronous execution (start and poll)
+
+```typescript
+const run = await client.actor('apify/web-scraper').start({
+    startUrls: [{ url: 'https://example.com' }],
+});
+
+// Poll for completion
+// Derive waitSecs from the run's timeoutSecs + a grace buffer, never unbounded.
+const finishedRun = await client.run(run.id).waitForFinish({ waitSecs: 120 });
+
+// Retrieve results
+const { items } = await client.dataset(finishedRun.defaultDatasetId).listItems();
+```
+
+Use `.start()` + `.waitForFinish()` for long-running Actors or when you need the run ID immediately.
+
+### Retrieving results
+
+```typescript
+// Dataset items (structured data from pushData)
+const { items } = await client.dataset(run.defaultDatasetId).listItems({
+    limit: 100,
+    offset: 0,
+});
+
+// Key-value store (files, screenshots, etc.)
+const record = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT');
+```
+
+### Error handling
+
+```typescript
+try {
+    const run = await client.actor('apify/web-scraper').call(input);
+
+    if (run.status !== 'SUCCEEDED') {
+        const log = await client.log(run.id).get();
+        throw new Error(`Actor failed with status ${run.status}: ${log}`);
+    }
+
+    const { items } = await client.dataset(run.defaultDatasetId).listItems();
+} catch (error) {
+    if (error.type === 'record-not-found') {
+        // Actor ID is wrong or Actor was deleted
+    } else if (error.statusCode === 401) {
+        // Invalid or missing APIFY_TOKEN
+    }
+    throw error;
+}
+```
+
+## 5. Python
+
+### Install
+
+```bash
+pip install apify-client
+```
+
+### Synchronous execution
+
+```python
+from decimal import Decimal
+from apify_client import ApifyClient
+import os
+
+client = ApifyClient(token=os.environ['APIFY_TOKEN'])
+
+run = client.actor('apify/web-scraper').call(
+    run_input={
+        'startUrls': [{'url': 'https://example.com'}],
+        'maxPagesPerCrawl': 10,
+    },
+    max_total_charge_usd=Decimal('5'),
+)
+
+items = client.dataset(run['defaultDatasetId']).list_items().items
+```
+
+### Asynchronous execution
+
+```python
+run = client.actor('apify/web-scraper').start(run_input={
+    'startUrls': [{'url': 'https://example.com'}],
+})
+
+# Poll for completion
+finished_run = client.run(run['id']).wait_for_finish()
+
+items = client.dataset(finished_run['defaultDatasetId']).list_items().items
+```
+
+### Async client (asyncio)
+
+```python
+from apify_client import ApifyClientAsync
+
+client = ApifyClientAsync(token=os.environ['APIFY_TOKEN'])
+
+run = await client.actor('apify/web-scraper').call(run_input={
+    'startUrls': [{'url': 'https://example.com'}],
+})
+
+items = (await client.dataset(run['defaultDatasetId']).list_items()).items
+```
+
+## 6. REST API (any language)
+
+For languages without an official client, use the REST API directly.
+
+### Start a run
+
+```
+POST https://api.apify.com/v2/actors/{actorId}/runs
+Authorization: Bearer <APIFY_TOKEN>
+Content-Type: application/json
+
+{ "startUrls": [{ "url": "https://example.com" }] }
+```
+
+### Get run status
+
+```
+GET https://api.apify.com/v2/actor-runs/{runId}
+Authorization: Bearer <APIFY_TOKEN>
+```
+
+### Get dataset items
+
+```
+GET https://api.apify.com/v2/datasets/{datasetId}/items?format=json
+Authorization: Bearer <APIFY_TOKEN>
+```
+
+The cost rule applies on the HTTP paths too: `maxItems` (pay-per-result) and `maxTotalChargeUsd` (other pricing models) go as **query parameters**, never in the JSON body where they are read as Actor input.
+
+For runs expected to finish within 300 seconds, the synchronous endpoint returns dataset items directly:
+
+```
+POST https://api.apify.com/v2/actors/{username}~{actor-name}/run-sync-get-dataset-items
+```
+
+Longer work must use the asynchronous flow: POST /runs -> poll GET /actor-runs/{id} -> GET /datasets/{id}/items.
+
+REST reference: `https://docs.apify.com/api/v2`. OpenAPI spec: `https://apify.com/openapi.json`.
+
+## 7. Best practices
+
+- **Set timeouts:** pass `timeoutSecs` as a run option / query parameter on `.call()` or `.start()`, or use `waitSecs` on `.call()`. Never put `timeoutSecs` in Actor input — it is a run option and an Actor whose schema rejects unknown fields will fail on it.
+- **Paginate large datasets:** use `limit` and `offset` when retrieving dataset items.
+- **Reuse clients:** create one `ApifyClient` instance and reuse it across calls.
+- **Handle Actor-specific input:** every Actor has its own input schema. Use `fetch-actor-details` MCP tool or append `.md` to the Actor's Store URL to get the schema before constructing input.
+- **Bound polling:** never `while (true)` - use the run's own timeout + a grace buffer, with an absolute ceiling.
+
+## 8. Documentation pointers
+
+- API client for JS: `https://docs.apify.com/api/client/js`
+- API client for Python: `https://docs.apify.com/api/client/python`
+- REST API reference: `https://docs.apify.com/api/v2`
+- Apify docs (LLM-friendly): `https://docs.apify.com/llms.txt`
+- Apify docs (full): `https://docs.apify.com/llms-full.txt`
+
+If the Apify MCP server is available, use `search-apify-docs` and `fetch-apify-docs` tools for contextual documentation lookups during development.
+
+## Definition-of-done checklist
+
+- [ ] Depends on `apify-client`, never `apify`.
+- [ ] Token stored in env var / secret manager; never hardcoded or logged.
+- [ ] Actor input built from the schema (MCP `fetch-actor-details` or `.md` URL), not guessed.
+- [ ] Sync `.call()` used for short runs; async `.start()` + `.waitForFinish()` for long ones.
+- [ ] Attribution header / user-agent suffix set on the client; skill-origin header included if built from this skill.
+- [ ] Cost cap (`maxTotalChargeUsd` / `max_total_charge_usd`) passed in options, never input.
+- [ ] Run status checked before consuming dataset; failed runs raise, not return empty.
+- [ ] Dataset and KV store retrieval covered; pagination on large datasets.
+- [ ] Errors mapped to actionable app-level messages.
+- [ ] REST API fallback documented for languages without a client.

package/bundled-skills/apify-integration-development/references/workflow-automation.md

@@ -0,0 +1,163 @@
+# Workflow automation integrations
+
+Design guide for integrating Apify into a workflow-automation platform (Zapier, n8n, Make, Pipedream, Activepieces, or a similar trigger/action/search host). This file is **UX-design focused and platform-agnostic**: it describes the capability surface, the resource model, and the user-facing behavior, not the host platform's internal build mechanics, release pipeline, or secrets. Apply the cross-cutting rules from `SKILL.md` on top.
+
+The host platform's model is rigid: every Apify capability must be expressed as one of **trigger**, **action (create)**, or **search (read)**. Decide the mapping before writing any code.
+
+## 1. Map Apify to the host's trigger / action / search model
+
+| Host type | Apify capability | Purpose |
+|---|---|---|
+| **Trigger** (hook) | Actor / Task run finished | Event-driven: start a workflow when a run reaches a terminal status |
+| **Trigger** (hidden, polling) | List Actors / Tasks | Back dynamic dropdowns - not user-facing steps |
+| **Action (create)** | Run Actor; Run Task; Run Actor + get dataset; Run Task + get dataset; Scrape single URL; Set KV record | Synchronous operations that produce or store data |
+| **Search (read)** | Get last run; Get run; List runs; Get dataset items; Get KV record | Find existing records, optionally branch on them |
+
+Reads of the last run or stored data are **searches**, because that is the host's mechanism for "find an existing record." Run-finished is a **webhook trigger**. Actor/Task *selection* is a **hidden trigger** that feeds dropdowns - reusing one hidden trigger across multiple actions keeps the surface DRY.
+
+## 2. Canonical capability matrix
+
+The complete set of operations a workflow integration should surface. Treat this as the floor, not a menu to trim without reason.
+
+**Triggers**
+- Actor run finished (webhook; user picks terminal statuses: SUCCEEDED / FAILED / TIMED-OUT / ABORTED)
+- Task run finished (same shape, scoped to a Task)
+
+**Actions (creates)**
+- Run Actor - fire an Actor run
+- Run Actor and get dataset items - fire and return results inline
+- Run Task - fire a saved Task configuration
+- Run Task and get dataset items - fire and return results inline
+- Scrape single URL - curated, 2-field wrapper over a content scraper (see section 10)
+- Set key-value store record - write a file/string to a KV store
+
+**Searches (reads)**
+- Get last Actor/Task run - the most recent matching run
+- Get run - by run ID
+- List runs - filterable by status
+- Get dataset items - by dataset ID, with offset/limit and optional fields/omit
+- Get key-value store record - by store ID + record key
+
+**Hidden selectors (back dropdowns, not user steps)**
+- List Actors (recently used source + Store source)
+- List Tasks
+
+If the host platform cannot represent every operation, prioritize in this order: run-finished trigger, run Actor + get dataset, get dataset items, scrape single URL, run Task + get dataset, get KV record, set KV record.
+
+## 3. Resource -> operation organization
+
+Organize the action node's surface as **resource -> operation**, mirroring both the host platform's UX convention and Apify's domain model. A two-level router (resource, then operation) keeps adding an operation a copy-and-adapt task and co-locates each operation's parameters and I/O. Users navigate Actor -> "Run actor", Dataset -> "Get items", etc. Avoid a flat list of dozens of operations.
+
+## 4. Actor / Task selection UX
+
+IDs (Actor, Task, run, dataset, KV store, record key) should use the host's resource-locator property type with multiple modes: **From list**, **By URL**, **By ID**. Extract the ID from a pasted Console URL (e.g. `https://console.apify.com/actors/<id>/input` -> `<id>`).
+
+Offer **two selection sources** in one selector:
+- **Recently used** (default) - the user's own Actors/Tasks, sorted by last run started. This is what most users want.
+- **Store** - browse the public marketplace by search term, sorted by popularity.
+
+A toggle with "recompute fields when changed" lets users switch the dropdown source without leaving the node. Choose a pagination limit that balances completeness against load time (the host UI must support searching within dropdown results).
+
+## 5. Dynamic input translation
+
+The highest-leverage UX piece. When a user selects an Actor, fetch the Actor's build and translate its `inputSchema` into the host platform's form fields dynamically. Hardcoding fields per Actor is unmaintainable across thousands of Actors; a translator supports *any* Actor's input as a host form.
+
+Handle the type/editor mappings that have no direct host equivalent, and degrade gracefully rather than crashing:
+
+| Apify type/editor | Host field |
+|---|---|
+| `string` + `javascript`/`python` editor | code field |
+| `string` + `textarea` | multiline text |
+| `string` + `datepicker` | datetime |
+| `string` + `select` / `enum` / `enumSuggestedValues` | choices dropdown |
+| `isSecret: true` | password |
+| `array` + `json`/`keyValue` editor | multiline text with a JSON.stringify default |
+| `array` + `requestListSources`/`pseudoUrls`/`globs`/`stringList` | flat string list (re-expand at run time) |
+| `object` + `proxy` editor | info box telling users to set it in Apify Console (host has no proxy UI) |
+| `object` + `schemaBased` | nested children fields |
+| `sectionCaption` | helpText/info box (host has no stackable sections) |
+
+Two gotchas:
+- On list fields, a `default` would pre-populate a non-removable first item and duplicate entries - move prefill values to `placeholder`.
+- Mark the build selector so a build-tag change **recomputes the generated fields**, since input schemas can differ across builds.
+
+## 6. Run UX: sync vs async, cost caps, bounded polling
+
+Offer a **sync/async toggle** on run-start actions:
+- Sync: block until the run finishes (document the host's hard timeout so users know when to choose async).
+- Async: return immediately with the run ID for long-running jobs.
+
+Use the asynchronous REST flow under the hood (POST /runs with `waitForFinish=0`, then poll). Polling must be **bounded** - use the run's own `timeoutSecs` + a grace buffer, with an absolute ceiling fallback (e.g. 24h). Poll at a fixed interval (e.g. 1s). "Run actor and get dataset" must additionally require `status === 'SUCCEEDED'` before fetching dataset items.
+
+Every run-start action exposes a **Maximum Cost per Run** field (`maxTotalChargeUsd`), `min: 0`, default `null`. Send it as a query parameter only when non-null and `> 0`; `0`/empty means *no limit*. Never let this be an Actor input field.
+
+## 7. Output normalization into one enriched shape
+
+Bare Actor run objects contain only storage IDs, not results - useless downstream. Enrich every run once, centrally, into a single shape returned by all run-producing and run-finding actions:
+
+- The run's `OUTPUT` from the default key-value store (and any user-selected store keys).
+- `datasetItems` - up to a sensible cap (e.g. 100 items) from the default dataset.
+- `datasetItemsFileUrls` - pre-built download URLs for JSON/CSV/XML/XLSX/HTML/RSS exports.
+- `detailsPageUrl` - a deep link to the run in Apify Console.
+- Strip fields useless to host users (`meta`, `stats`, `options`, `userId`, raw `output`, `standby`).
+
+Downstream steps map fields by name regardless of which action produced the run, because every action yields the same shape. Describe this shape to the host UI with sampled output fields (sample ~10 items, merge their keys).
+
+## 8. Guard against large datasets
+
+Estimate the full download size before inlining: fetch one item, multiply by the requested item count with a safety margin (e.g. 1.2x), compare to a payload cap (e.g. 10 MB). If it exceeds the cap, **do not attempt the inline download** - return a warning item plus the dataset file URLs. When a trigger fetches more items than the cap allows, push a warning pointing users to the "Get Dataset Items" action with `run.defaultDatasetId` for full results. The host platform has payload/time limits the Apify API does not; always offer a file-download fallback.
+
+## 9. Files and binary records
+
+Map the host's file-handling primitives (dehydration / stashing / signed URLs) to Apify's binary KV records. For "Get KV record":
+- JSON/text under a size threshold (e.g. 20 MB) -> parsed and returned as fields the user can map directly.
+- Binary or larger records -> a dehydrated pointer the host lazily fetches when a downstream step needs it.
+- Records above a hard limit (e.g. 120 MB, with margin below the host's 150 MB ceiling) -> rejected outright with an actionable message.
+
+## 10. Run-finished trigger UX
+
+Use the host's hook type backed by Apify webhooks:
+- **Subscribe**: create an Apify webhook scoped to an `actorId` or `actorTaskId`, with `eventTypes` from the user-selected terminal statuses. The webhook's `requestUrl` is the host's target URL. Make registration **idempotent** - derive a key from every field that distinguishes one registration from another (`resource:id:sortedEvents:requestUrl`) so re-activating a workflow does not create duplicate webhooks. Persist the created webhook ID so deactivation can delete it.
+- **Unsubscribe**: delete the webhook by its stored ID.
+- **Perform**: read the webhook payload and enrich it (section 7).
+- **Fallback list**: fetch the 3 most recent matching runs so users see realistic test data when configuring the trigger without waiting for a real event.
+
+Offer an event multi-select (SUCCEEDED / FAILED / TIMED-OUT / ABORTED) plus an "any" shortcut that expands to all four. Build the `condition: { actorId }` or `{ actorTaskId }` from the watched resource. Pass the webhook payload through as workflow data - it already carries the run metadata.
+
+## 11. Error UX
+
+Centralize API error mapping in one place, not per action. Match each Apify error to the host's error category:
+- 5xx and 429 -> retryable (so backoff applies).
+- `token-not-found` / auth errors -> an authentication error the user must fix.
+- `full-permission-actor-not-approved` -> a non-retryable error with the **validated** approval URL in the message (approval is a manual Console action; retrying cannot help).
+- 404 on KV record GET/HEAD -> return empty rather than throw (a missing key is a valid "no data" result).
+- Other 4xx -> a plain error with the API's actual message, or a generic fallback.
+
+Use retryable errors only when retrying can actually succeed. Keep error *codes* (like `EPERM`) out of any message field the host replaces, so the real text survives.
+
+## 12. Auth UX
+
+For consumer-facing automation platforms, prefer **OAuth2 with PKCE** over an API-token field. Users authorize through a browser; no raw token typing. Disable auto-refresh only if the host cannot surface a refresh failure gracefully - otherwise leave it on. Validate the token with a `GET /v2/users/me` test call and populate the connection label with the username/email.
+
+If the host is headless-only, fall back to an API-token credential with the same verify call on a "Verify" button.
+
+## 13. Convenience operations: "Scrape single URL"
+
+Beyond generic "run Actor", ship a curated **Scrape single URL** action: a 2-field form (`url`, `outputFormat`) wrapping a content scraper with sensible defaults (`maxCrawlDepth: 0`, `maxResults: 1`). Validate the URL first (`new URL()` + protocol + hostname check) with an actionable error - bad input must never start a paid run. Return a **lean, single-object** output: `{ ...pageMetadata, [outputFormat]: content }` - strip all content variants and re-add only the chosen one. This is ideal for LLM flows (a single object beats a dataset array) and lowers the barrier for non-power users. Point power users to the underlying Actor for advanced options.
+
+## Definition-of-done checklist
+
+- [ ] Capability matrix from section 2 is fully surfaced (or trimmed with documented reason).
+- [ ] Resource -> operation organization; no flat operation soup.
+- [ ] Actor/Task selection offers recently-used + Store sources with URL/ID extraction.
+- [ ] Actor input is translated dynamically from the build schema; degradation is graceful.
+- [ ] Run actions expose sync/async toggle and a `maxTotalChargeUsd` field (null = unlimited).
+- [ ] Polling is bounded; never `while (true)`.
+- [ ] All run-producing and run-finding actions return the one enriched run shape.
+- [ ] Large-dataset guard with file-URL fallback is in place.
+- [ ] Run-finished trigger is webhook-backed, idempotent, and has fallback sample data.
+- [ ] Error mapping is centralized; approval URLs are validated; codes don't clobber messages.
+- [ ] OAuth2 PKCE is the default consumer auth path; token fallback has a verify call.
+- [ ] A "Scrape single URL"-style convenience action exists with pre-run URL validation.
+- [ ] `x-apify-integration-platform` header is sent on every outbound request; `x-apify-integration-origin: apify-integration-development-skill` included if built from this skill.
+- [ ] Two test modes (mocked + live E2E) pass.