opencode-skills-collection 4.0.68 → 4.0.70

package/bundled-skills/apify-integration-development/SKILL.md

@@ -0,0 +1,168 @@
+---
+description: Curated upstream guidance for Apify Integration Development; use when the workflow matches the user goal.
+name: apify-integration-development
+source_repo: apify/agent-skills
+source_type: official
+source: apify
+date_added: '2026-09-21'
+risk: unknown
+---
+## When to Use
+- Use when this upstream workflow matches the user's stated goal.
+- Use when the task requires the procedures documented in this skill.
+
+# Apify Integration Development
+
+Design and build an **official Apify integration** for a company's product, with minimal help from Apify. This skill covers every integration shape Apify supports - workflow-automation apps, AI agent plugins (coding agents and harnesses), AI framework packages, and direct application clients - so a partner team can ship a first-class Apify integration end to end. The cross-cutting rules below apply to all of them, and one category-specific reference file carries the rest.
+
+> **Building an official integration?** Once you publish it, contact **integrations@apify.com** so the Apify team can review, test, and validate your integration before it reaches users. We'll check the capability surface, cost controls, error handling, and attribution headers, and help you close any gaps.
+
+## Step 0 - Learn the Apify model first (required)
+
+Before designing anything, fetch and read `https://apify.com/agents.md`. It is the canonical quickstart for AI agents and the single source of truth for vocabulary, the run flow, and the cost rule. If the fetch fails, the mini-glossary below keeps the skill usable.
+
+Apify vocabulary (always written with a capital A on the platform):
+
+- **Actor** - a serverless cloud program that takes JSON input, performs a task, and produces structured output. Not an AI agent.
+- **Actor Run** - one execution of an Actor. Each run has its own dataset, key-value store, and request queue, and ends in a terminal status (`SUCCEEDED`, `FAILED`, `TIMED-OUT`, `ABORTED`).
+- **Dataset** - append-only structured storage for a run's results. An Actor call returns the dataset ID, not its contents.
+- **Key-Value Store** - unstructured/file storage (screenshots, HTML, OUTPUT).
+- **Actor Task** - a saved, parameterized configuration for running an Actor.
+- **Apify Store** - the marketplace of Actors at `https://apify.com/store.md`.
+- **Apify Console** - the web UI at `https://console.apify.com`.
+- **Compute Unit (CU)** - billing unit: memory (MB) x duration (hours).
+
+Further terms (build, standby, request queue, proxy, pricing models): `https://docs.apify.com/llms.txt`.
+
+## Use Apify MCP for live context while planning
+
+The Apify MCP server is the fastest way to research Actors, schemas, pricing, and docs during integration design. See `https://docs.apify.com/integrations/mcp` (append `.md` for a markdown version).
+
+If Apify MCP tools are already available in this environment, use them:
+
+- `search-actors` - find Actors by platform/product keyword (search by product name, not end goal).
+- `fetch-actor-details` - read an Actor's input schema, output format, README, and pricing before you encode its shape into the integration.
+- `search-apify-docs` / `fetch-apify-docs` - pull contextual documentation pages.
+
+The anonymous discovery subset (`search-actors`, `fetch-actor-details`, `search-apify-docs`, `fetch-apify-docs`) works without an account, so you can research even before the developer has connected their token.
+
+## Pick your integration shape
+
+Read exactly one reference file based on the product you are integrating into. Each reference carries the category-specific UX design, a canonical capability matrix, and a definition-of-done checklist.
+
+| Product shape | Examples | Read |
+|---|---|---|
+| Workflow automation platform | Zapier, n8n, Make, Pipedream, Activepieces | `references/workflow-automation.md` |
+| AI agent plugin (coding agent or harness) | Cursor, Claude Code, Codex, GitHub Copilot (coding agents); OpenClaw-style runtimes, Hermes-style harnesses (harnesses) | `references/ai-harness-plugin.md` |
+| AI framework package (PyPI/npm for LLM frameworks) | LangChain, LlamaIndex, Haystack, Vercel AI SDK | `references/ai-framework-package.md` |
+| Application integration (direct client) | A backend service, scheduled job, product feature calling Actors via `apify-client` or REST | `references/sdk-integration.md` |
+
+Paths are relative to this skill folder. If your product spans two shapes (e.g. an AI harness built on top of a framework package), read both - the rules compose. The AI agent plugin reference covers **two approaches with different trade-offs**: a lightweight skills + MCP bundle for skills/MCP-aware coding agents, and a custom tool-registry plugin for OpenClaw/Hermes-style harnesses.
+
+## Cross-cutting design rules (true for every integration type)
+
+These invariants were extracted from every existing Apify integration. Apply them regardless of shape.
+
+### Vocabulary mirroring
+Model the integration's resources on Apify's domain (Actor / Run / Dataset / KV Store / Task). Users coming from Apify Console should find the same concepts under the same names.
+
+### Asynchronous run flow with bounded polling
+Actors can run for seconds to hours. Use the asynchronous flow, never the 300-second synchronous endpoint for anything but short jobs:
+
+```
+POST /v2/actors/{actorId}/runs            -> start, return runId
+GET  /v2/actor-runs/{runId}               -> poll until terminal status
+GET  /v2/datasets/{datasetId}/items       -> fetch results on SUCCEEDED
+```
+
+Polling must be **bounded**: use the run's own `timeoutSecs` plus a grace buffer, with an absolute ceiling fallback. Never `while (true)`. On a non-terminal status, surface the run ID so the user/agent can poll again or inspect the failure.
+
+### Cost is first-class
+Every path that starts a run must expose a cost control. The canonical control is `maxTotalChargeUsd` (caps the run's total charge on most pricing models) and `maxItems` (caps billed items on pay-per-result Actors). Send them as **options / query parameters**, never as Actor input - inside input they are either an Actor-declared field or simply invalid. `0` / empty / null means *no limit*. For LLM-facing integrations, the ceilings are **developer-controlled**; an LLM cannot widen them.
+
+### Attribution headers
+Stamp an integration header on every outbound request so Apify can attribute traffic: `x-apify-integration-platform: <your-platform>`. When a request is driven by an AI tool (not a human in a UI), also send `x-apify-integration-ai-tool: true`. If the integration was built using this skill, add `x-apify-integration-origin: apify-integration-development-skill` so Apify can distinguish skill-generated integrations from custom ones. One line, big telemetry payoff.
+
+### Authentication
+- Browser / consumer-facing (a human completes a sign-in): OAuth2 with PKCE. Do not ask for raw tokens.
+- Headless / server / CI (no human present): API token as `Authorization: Bearer <APIFY_TOKEN>`, stored in an env var or secret manager, never hardcoded or logged.
+
+Both paths are real - pick by who is present at auth time, not by which is easier.
+
+### Centralized HTTP layer
+One base-URL constant, shared between credentials and the HTTP layer. Retries with exponential backoff on 429 and 5xx. **Never retry non-idempotent `POST /runs` on network errors** - a duplicate Actor run is a real, billed, side-effecting operation. This is the single most important correctness invariant in the HTTP layer.
+
+### Error taxonomy
+Map Apify errors to the host platform's error categories (retryable vs auth vs permanent). Surface the API's actual error text, not a generic HTTP message. For permission-approval failures (a full-permission Actor needs explicit approval), include the approval URL after validating it is an absolute `http(s)` URL. For LLM consumers, return errors **as data** (JSON error objects), never as raised exceptions - the model needs something to read and reason about.
+
+### Webhooks over polling for run-finished events
+When the host supports inbound webhooks, register an Apify webhook scoped to `actorId` or `actorTaskId` with the terminal statuses the user picked. Make registration **idempotent** (a re-activated workflow should not create duplicate webhooks), persist the webhook ID so deactivation can clean it up, and always provide sample/fallback data so users can test the trigger without waiting for a real run.
+
+### Generate from OpenAPI where the host allows it
+If the host platform can generate UI fields from an OpenAPI spec, use Apify's spec (`https://apify.com/openapi.json`) and a tag allowlist. Hand-write only what the spec cannot express: convenience wrappers, bill-cap fields, lean AI-tool output contracts.
+
+### High-level convenience operations alongside generic runs
+Generic "run Actor" serves power users. Add a few opinionated, high-level actions for the common case (e.g. "Scrape single URL" wrapping a content scraper with `maxCrawlDepth: 0`, `maxResults: 1`) so non-power users get a 2-field form instead of a full Actor configuration. Validate the URL *before* starting a paid run.
+
+### Testing and release
+Keep two test modes: mocked (hermetic, no credentials) and live E2E (real API, CI-gated). Automate releases through the host platform's CI on Git tags / GitHub Releases. Never hand-edit versions or changelogs if a release workflow manages them.
+
+## Top anti-patterns to refuse on review
+
+1. Retrying `POST /runs` on a network error - duplicates a billed run.
+2. Unbounded `while (true)` polling - ties up the host with no ceiling.
+3. Putting `maxTotalChargeUsd` / `maxItems` inside Actor input instead of options - silently not a cap.
+4. Dumping a full dataset into an LLM context without size caps or untrusted-content fencing - prompt-injection and context blowout.
+5. One monolithic tool list for an LLM agent - routing accuracy degrades past ~8 tools; curate subsets.
+6. Surfacing a raw HTTP status/message instead of Apify's actual error text - users can't act on "400".
+
+## Minimal API surface every integration needs
+
+| Purpose | Method + path |
+|---|---|
+| Start an Actor run | `POST /v2/actors/{actorId}/runs` |
+| Start a Task run | `POST /v2/actor-tasks/{taskId}/runs` |
+| Poll a run | `GET /v2/actor-runs/{runId}` |
+| List runs | `GET /v2/actor-runs` |
+| Dataset items | `GET /v2/datasets/{datasetId}/items` |
+| KV record | `GET /v2/key-value-stores/{storeId}/records/{key}` |
+| Set KV record | `PUT /v2/key-value-stores/{storeId}/records/{key}` |
+| Store search | `GET /v2/store` |
+| Webhook CRUD | `POST/GET/DELETE /v2/webhooks` |
+| Validate token / current user | `GET /v2/users/me` |
+
+REST reference: `https://docs.apify.com/api/v2`. OpenAPI spec: `https://apify.com/openapi.json`.
+
+## Working workflow
+
+1. Fetch `https://apify.com/agents.md` and internalize the model.
+2. Pick the integration shape above and read the matching reference file.
+3. Use Apify MCP (if available) to research the concrete Actors, schemas, and pricing the integration will expose.
+4. Draft the **capability matrix** for the chosen category (each reference has one) and the UX spec (resource -> operation -> fields -> errors).
+5. Scaffold the integration following the category-specific rules in the reference.
+6. Verify against the **definition-of-done checklist** at the end of that reference.
+
+## Reference implementations to study
+
+Real, public integrations per category - read their source when in doubt:
+
+- Workflow automation: `@apify/n8n-nodes-apify` (npm), the Apify Zapier app.
+- AI agent plugins (coding agents): the Apify plugin bundle (MCP server + skills + router + slash commands) shipped for Cursor, Claude Code, Copilot, and similar tools.
+- AI agent plugins (harnesses): `apify-hermes-agent-plugin` (PyPI), `@apify/apify-openclaw-plugin`.
+- AI framework packages: `langchain-apify` (PyPI).
+- Application integration: see `references/sdk-integration.md` for the canonical `apify-client` usage in JS/TS, Python, and over REST.
+
+Support for integration questions: `integrations@apify.com`. Contact us both for design guidance while you build and for review/testing once you publish - we validate the capability surface, cost controls, error handling, and attribution before the integration reaches users.
+
+
+## Examples
+
+```text
+User: Apply this skill to my current task.
+Assistant: Follow the workflow in this skill, cite limitations, and ask before risky steps.
+```
+
+## Limitations
+
+- Imported upstream skill; verify credentials, permissions, and safety boundaries before execution.
+- Does not replace environment-specific validation, testing, or maintainer review.

package/bundled-skills/apify-integration-development/references/ai-framework-package.md

@@ -0,0 +1,158 @@
+# AI framework package integrations
+
+Design guide for building a PyPI/npm package that exposes Apify to an AI/LLM framework - LangChain, LlamaIndex, Haystack, Vercel AI SDK, or similar. These are *client-side* integrations: code that calls Apify Actors from outside the Apify runtime, for applications, agents, and RAG pipelines. Apply the cross-cutting rules from `SKILL.md` on top.
+
+## 1. Scope: client-side only, wrap apify-client, never the Actor SDK
+
+This package is for applications that call Apify Actors from outside the Apify runtime. It is **not** for code running *inside* an Actor. Apify Actors should run with limited permissions and use scoped tokens via the Actor SDK's `Actor.open_dataset()`; importing a framework client that reconstructs its own `ApifyClient` from an env-var token would bypass that scoping and pull an unnecessary dependency into Actor images.
+
+Dependency philosophy: wrap the official `apify-client` library, never the `apify` SDK. `apify` is for *building* Actors; `apify-client` is for *calling* them. Keep the runtime dependency surface minimal (`langchain-core`, `apify-client`, and a backport if needed) to minimize version conflicts and keep install time short in agent environments.
+
+Stamp a custom `user-agent` suffix (e.g. `; Origin/langchain`) or the attribution header on the client so Apify can attribute traffic.
+
+## 2. Layered architecture: client -> framework adapters -> public API
+
+```
+Public API       curated exports
+                        |
+   +--------------------+--------------------+
+   |                    |                    |
+Tools             Document loaders        Retriever
+(agents)          (RAG ingestion)         (RAG retrieval)
+   |                    |                    |
+              ApifyToolsClient  (sync)
+                        |
+              apify-client (sync + async)
+                        |
+              Apify REST API
+```
+
+| Layer | Role |
+|---|---|
+| **Client** | Thin, synchronous wrapper over `apify-client`. One method per Actor operation. No framework types here. |
+| **Tools** | Framework `BaseTool` subclasses for agent tool-calling. |
+| **Document loaders** | `BaseLoader` implementations for RAG ingestion. |
+| **Retriever** | `BaseRetriever` for RAG query-time retrieval. |
+
+Framework types live only above the client layer. The client layer speaks pure Python/JS dicts and `apify-client` objects. This lets the client be unit-tested with no framework dependency, and lets the framework-facing layers focus exclusively on schema, tool semantics, and envelope formatting.
+
+## 3. ApifyToolsClient: one sync gateway, typed method per Actor
+
+All Actor interaction goes through a single synchronous client class with one convenience method per supported Actor (e.g. `google_search`, `instagram_scrape`, `crawl_website`). Each method:
+
+- Builds the Actor-specific `run_input` dict, translating from the integration's normalized parameter names to the Actor's raw input schema. (Actor schemas are idiosyncratic - `searchStringsArray`, `directUrls`, `detailsUrls` vs `listingUrls`; the client absorbs that so the tool exposes clean names like `query`, `url`, `url_type`.)
+- Calls `client.actor(id).call(...)` which blocks until the run finishes.
+- Checks run status and raises if the run did not reach `SUCCEEDED` (a failed run must never silently return empty results).
+- Returns a `(run_details, items)` tuple (or just one where appropriate).
+
+**Why blocking?** Callers don't manage polling loops; the API stays simple. The async surface is handled at the framework layer (`asyncio.to_thread` / `Promise.resolve`) rather than duplicating every method in async form.
+
+Adding a new Actor tool means adding one client method (input translation + status check) and one tool class (schema + `_run`), not wiring up polling, retries, or async variants.
+
+## 4. Uniform JSON output envelope
+
+All tools return a JSON string of one shape:
+
+```json
+{"run": {"run_id": "...", "status": "...", "dataset_id": "...",
+         "started_at": "...", "finished_at": "..."},
+ "items": [...]}
+```
+
+`run` is `null` for dataset-only tools. An optional `notice` key surfaces out-of-band hints (e.g. an Actor returned demo placeholder data on the free plan). Serialize with `default=str` so non-JSON-native types (datetimes from a `clean=True` deserialiser) never throw mid-tool-call.
+
+A single predictable envelope lets agents parse results with one code path. The `run` metadata gives the agent enough to chain calls - run an Actor with one tool, then fetch the dataset with another using the returned `dataset_id`. End every tool description with "Use only the data returned; do not hallucinate missing fields."
+
+## 5. Safety clamps - defense against LLM-requested extremes
+
+An LLM invoking a tool can request absurd values: 10,000 results, 32 GB of memory, a 1-hour timeout. Clamp every request to **developer-controlled ceilings**:
+
+| Clamp | Default ceiling | Developer max |
+|---|---|
+| `timeout_secs` | 600 s |
+| `memory_mbytes` | 4,096 MB (snapped to nearest valid power-of-2) | 8,192 MB |
+| `items` / `limit` | 1,000 |
+| `max_crawl_depth` | 5 |
+
+Memory is notable: Apify accepts memory only as a power-of-2 (128, 256, 512, ..., 32768). Snap an arbitrary LLM value to the nearest valid step at or below the developer's cap. The default ceiling of 4,096 MB (4 GB) is generous for most Actors but well below the platform max, so LLM-requested extremes are clamped. The developer can raise the ceiling up to 8,192 MB, but an LLM cannot widen it beyond the developer-set value.
+
+Some Actors have runtime limits not declared in their input schema (e.g. a RAG web browser rejects `maxResults > 100` at runtime). These can't be derived by schema introspection - track them by hand as overrides on the specific tool so the clamp enforces the Actor's real ceiling.
+
+The ceilings are *developer-controlled fields* on the tool instance - an application can tighten them further, but the LLM cannot widen them. This makes the integration safe to hand to an autonomous agent without risking runaway compute costs.
+
+## 6. Curated tool subsets, not one monolithic list
+
+Tools are grouped into convenience lists:
+
+| List | Tools | Use case |
+|---|---|---|
+| Core | Run Actor, get dataset, run+get, scrape URL, run task, run task+get | Generic platform primitives |
+| Search | Google search, web crawler, RAG web browser, Google Maps, YouTube, e-commerce | Web search & content crawling |
+| Social | Instagram, LinkedIn, Twitter/X, TikTok, Facebook | Social media scraping |
+
+Warn explicitly: **don't bind all tools at once.** Most LLMs lose routing accuracy past ~8 tools, so pick the family the agent actually needs. Curated subsets let an agent built for social-media analysis avoid distinguishing among 19 tool descriptions.
+
+## 7. Hand-written tools + dynamic schema for the long tail
+
+Alongside hand-written tools (which get clean schemas and descriptions), ship one dynamic tool that takes an `actor_id` at construction, fetches the Actor's latest default build, and **generates an input model dynamically** from the build's input schema. Prune descriptions to a fixed length; limit properties to `type`, `default`, `prefill`, `enum`.
+
+This covers the long tail of Actors without a dedicated wrapper - you don't need a hand-written tool for every one of Apify's thousands of Actors. The trade-off is a looser schema (the LLM sees the raw Actor input shape) and a network call at construction time.
+
+## 8. Map to the framework's idiomatic surfaces
+
+Implement the framework's actual extension points, all backed by the same client:
+
+| Surface | Base class | Use case |
+|---|---|---|
+| **Tools** | `BaseTool` | Agent tool-calling (ReAct, LangGraph) |
+| **Document loaders** | `BaseLoader` | Batch RAG ingestion (load -> split -> embed -> vector store) |
+| **Retriever** | `BaseRetriever` | Query-time web retrieval for RAG chains |
+
+- A **dataset loader** loads an existing dataset by ID and maps each item to a `Document` via a user-supplied mapping function (every Actor's output schema is different, so give the user full control). Implement both eager `load()` and streaming `lazy_load()`.
+- A **crawl loader** is an *active* loader: it runs a content crawler on construction, then yields `Document`s with `page_content` (markdown) and `metadata` (`source`, `title`, `crawl_depth`).
+- A **search retriever** wraps a search-and-crawl Actor for low-latency interactive RAG; the async path runs the synchronous client off the event loop via `to_thread`.
+
+## 9. Token hygiene
+
+One canonical token parameter/env var (e.g. `apify_token` / `APIFY_TOKEN`). If a legacy name exists (`APIFY_API_TOKEN`), honor it with a `DeprecationWarning` but reject new code that declares it. Centralize the policy in two helpers: one for explicit `__init__` signatures, one for Pydantic `model_validator(mode='before')` hooks. Store the token as a `SecretStr` (excluded from repr and serialization) and never log it.
+
+## 10. Content extraction: markdown-first with defensive fallbacks
+
+When extracting page content from crawling Actors, prefer `markdown` over `text`, with a trailing `or ''` to guarantee a string even when a key is present but null. Follow a fixed fallback order for the source URL: nested `metadata.url` -> `crawledUrl` -> top-level `url`. Tolerate a `metadata` field that is missing or not a dict (some Actor responses surface `null`). Actor output shapes are inconsistent across versions and configurations; centralize one canonical fallback order so the retriever, loaders, and tools all agree on what "the content", "the source URL", and "the title" mean.
+
+## 11. Error mapping
+
+- **Client layer** raises `RuntimeError` for failed/empty runs and `ValueError` for invalid input. Wrap transport errors in `RuntimeError`.
+- **Tool layer** catches both and re-raises as the framework's tool-error type (e.g. `ToolException`) with `handle_tool_error = True`, which surfaces to the agent as a recoverable error message.
+
+An agent that gets a `ToolException` can read the message and retry with corrected input. An unhandled `RuntimeError` would crash the agent loop. The boundary is clean: the client raises domain errors; the tool adapts them to the framework's tool-error protocol.
+
+## 12. Packaging, release, and quality bar
+
+- Minimal runtime deps; an explicit sdist allowlist so local-only paths (`dist/`, `.venv/`, `docs/`, test fixtures) never reach the registry.
+- Release via conventional-commits-driven automation that reads commit-message prefixes to auto-generate the changelog and compute the version bump; a `BREAKING CHANGE:` footer triggers a major bump. Never hand-edit `version =` or `CHANGELOG.md` if the workflow manages them.
+- Strict linting (`select = ["ALL"]` with a curated ignore list), strict typing (`disallow_untyped_defs`), and **socket-disabled unit tests** so the unit suite is truly unit - no hidden integration dependencies. Integration tests need a real token (CI only).
+
+## 13. Position vs the Apify MCP server
+
+The README's top banner should direct users to Apify's MCP server (`https://mcp.apify.com`) as a richer, more featureful alternative for interactive agent workflows that need dynamic Actor discovery. The package is not deprecated, but the MCP path is recommended for new interactive agent sessions.
+
+The positioning: the package is the **programmatic, typed, registry-installable** option for code that outlives a single agent session (servers, scheduled jobs, pipelines); the MCP server is the **interactive, dynamic** option. Rather than compete, position them for their respective audiences.
+
+## Definition-of-done checklist
+
+- [ ] Package is client-side only; depends on `apify-client`, never `apify`.
+- [ ] Layered: thin client (no framework types) -> framework adapters -> curated public API.
+- [ ] One synchronous client with a typed method per supported Actor; input normalization centralized.
+- [ ] All tools return the uniform JSON envelope; serialization never throws on non-native types.
+- [ ] Developer-controlled safety clamps (timeout, memory power-of-2, items, depth) are in place; hand-tracked runtime limits override specific tools.
+- [ ] Tools grouped into curated subsets; documentation warns against binding all at once.
+- [ ] A dynamic-schema tool covers the long tail of Actors.
+- [ ] Framework surfaces (tools / loaders / retriever) all backed by the same client.
+- [ ] One canonical token name; legacy alias emits a deprecation warning; token is `SecretStr`, never logged.
+- [ ] Content extraction is markdown-first with documented fallback order.
+- [ ] Client raises domain errors; tools adapt them to the framework's tool-error protocol.
+- [ ] sdist allowlist excludes local paths; release automation drives versioning.
+- [ ] Unit tests are socket-disabled; lint/typing are strict.
+- [ ] README cross-references the MCP server for interactive/dynamic use.
+- [ ] Attribution header / user-agent suffix is set on the client; skill-origin header included if built from this skill.

package/bundled-skills/apify-integration-development/references/ai-harness-plugin.md

@@ -0,0 +1,192 @@
+# AI agent plugin integrations
+
+Design guide for building an Apify plugin that gives an AI agent access to Actors. There are **two plugin shapes**, and which one you build depends on the host:
+
+- **Approach A - Coding agent plugin (skills + MCP bundle):** for skills/MCP-aware coding assistants like Cursor, Claude Code, Codex, and GitHub Copilot. You assemble a small set of runtime artifacts the host already knows how to load, and the hosted Apify MCP server (`https://mcp.apify.com`) provides the tool surface. Minimal code.
+- **Approach B - Harness / assistant plugin (custom tool registry):** for agent runtimes like OpenClaw-style runtimes and Hermes-style harnesses that have their own tool registry and config file. You build a small custom toolset (`discover` / `start` / `collect`) backed by the `apify-client` SDK, using a stored credential rather than per-session OAuth.
+
+Apply the cross-cutting rules from `SKILL.md` on top of either approach.
+
+## Which approach? Trade-offs
+
+| Dimension | A - Coding agent plugin (skills + MCP) | B - Harness / assistant plugin (custom registry) |
+|---|---|---|
+| Target hosts | Cursor, Claude Code, Codex, GitHub Copilot | OpenClaw-style runtimes, Hermes-style harnesses, custom tool-calling agents |
+| Tool surface | Hosted Apify MCP server - no tool code to write | You implement `discover` / `start` / `collect` yourself |
+| Auth | OAuth (MCP) / `apify login` (CLI) / `APIFY_TOKEN` (SDK), per route | Stored API key resolved by the plugin, passed to `apify-client` |
+| Build effort | Low - assemble artifacts, no HTTP/retry/registry code | Higher - tools, schema, error taxonomy, host gotchas |
+| Capabilities | Run existing Actors **and** build/test/deploy new Actors **and** integrate an app | Broker the Store to the agent (run existing Actors) |
+| Maintenance | The MCP server owns the runtime surface | You own the tool code plus host-SDK compatibility |
+| Best when | The host supports skills/MCP and you want the fastest path | The host has its own registry and needs bespoke tools |
+
+If the host is a skills/MCP-aware coding tool, prefer Approach A. If the host is a custom harness with its own registry (no MCP), use Approach B. A product can ship both over time - start with whichever matches the primary host.
+
+---
+
+# Approach A - Coding agent plugin (skills + MCP bundle)
+
+For skills/MCP-aware coding assistants (Cursor, Claude Code, Codex, GitHub Copilot). This section describes the **installed plugin** - what the user gets and how the pieces interact at runtime - not how the bundle is produced.
+
+## A.1 The four runtime artifacts
+
+| Artifact | Role at runtime |
+|---|---|
+| **MCP server** | Registers `https://mcp.apify.com` with the host and exposes the callable tool surface (below). OAuth: the user signs in via browser on the first tool call that needs auth - no token in config. |
+| **Skills** | On-demand `SKILL.md` instruction documents. The host matches each skill's `description` against user intent and loads the body into context only when relevant, keeping baseline context small. |
+| **Subagent / router** | The entry point. Classifies the request into one of three routes, selects the transport (MCP vs CLI), and invokes the matching skill or tools. |
+| **Slash commands** | User-invoked entry points (e.g. `/create-actor <description>`) that drive a guided end-to-end workflow. |
+
+**MCP tool surface** once connected: `search-actors` (search the Store), `fetch-actor-details` (input schema, output format, pricing), `call-actor` (run with input JSON), `get-actor-run` (poll status), `get-dataset-items` (fetch results), `search-apify-docs` / `fetch-apify-docs` (docs). The discovery subset (`search-actors`, `fetch-actor-details`, `search-apify-docs`, `fetch-apify-docs`) works without an account.
+
+## A.2 The three routes the plugin serves
+
+The router classifies every request and routes it:
+
+| Signal | Route | How it runs |
+|---|---|---|
+| Use existing Actors (search, run, get data) | 1 | MCP tools directly; the CLI is the fallback when MCP is unavailable |
+| Build / test / deploy a custom Actor | 2 | Apify CLI (`apify create` / `run` / `push`) - local filesystem, no MCP equivalent |
+| Add Apify to an existing app | 3 | `apify-client` over HTTPS - neither MCP nor CLI |
+
+**MCP-vs-CLI selection (Route 1 only).** Detect transports once: MCP is available if a tool named `search-actors` is in the tool list; CLI is available if `apify --help` exits 0. Prefer MCP when present (no shell/install friction, OAuth auth); fall back to the CLI otherwise. Routes 2 and 3 are unaffected.
+
+**Naming trap.** The `apify` npm package is the **SDK for building** Actors (Route 2). The `apify-client` package is the **API client for calling** Actors (Route 3). Never confuse them.
+
+**Auth per route:** Route 1 (MCP) OAuth via browser prompt, never ask for a token; Route 1 CLI fallback + Route 2 `apify login --token <TOKEN>` once (the CLI ignores `APIFY_TOKEN`); Route 3 the `APIFY_TOKEN` env var.
+
+## A.3 Definition of done (Approach A)
+
+- [ ] MCP declared and reachable - `https://mcp.apify.com` registered and its tools appear in the tool list.
+- [ ] OAuth works - the first auth-requiring MCP call prompts a browser sign-in; no token in config.
+- [ ] Skills load by intent - each skill's `description` matches its requests; bodies load only when relevant.
+- [ ] Router classifies correctly - requests land on Route 1 / 2 / 3; ambiguous ones ask the user to choose.
+- [ ] Transport selection correct - Route 1 prefers MCP and falls back to CLI cleanly; Routes 2/3 use CLI/SDK.
+- [ ] Auth wired per route; the `apify` vs `apify-client` distinction is never confused.
+- [ ] Cost caps honored (`maxTotalChargeUsd` / `maxItems`) and attribution headers set (see `SKILL.md`).
+- [ ] Slash command (e.g. `/create-actor`) runs end to end.
+- [ ] Verified inside the actual target tool, not just in isolation.
+
+---
+
+# Approach B - Harness / assistant plugin (custom tool registry)
+
+For agent runtimes with their own tool registry (OpenClaw-style runtimes, Hermes-style harnesses, or any custom tool-calling agent). The plugin brokers the entire Apify Store to the agent - it does not bundle scrapers.
+
+The harness runs locally/persistently, has its own tool registry and config file, and calls Apify with a stored credential rather than per-session OAuth. So this shape borrows from the "API token + apify-client" path, not the MCP path.
+
+## 1. Shape decision: few composable tools vs a dynamic tool list
+
+Decide based on what the harness supports:
+
+- **Static tool registry** (tools registered once at plugin load, no per-Actor materialization): register a **small, fixed set of composable tools** and let the LLM compose them. This keeps the prompt budget small and the call graph legible.
+- **Dynamic tool registration** (the harness can materialize tools at runtime): you *can* expose a dynamic per-Actor tool list, but a fixed trio is still simpler and usually enough.
+
+The MCP server's surface (search / inspect / call / poll / fetch as separate dynamic tools) is one shape. A harness plugin is a different shape - do not copy it blindly.
+
+## 2. Canonical action set: discover / start / collect
+
+Three tools cover the entire workflow and map cleanly to the asynchronous REST flow (`POST /runs` -> poll `GET /actor-runs/{id}` -> `GET /datasets/{id}/items`):
+
+| Tool | Purpose | Why |
+|---|---|---|
+| **discover** | Search Apify Store by keyword, OR fetch a single Actor's input schema + README by `actorId` | Two modes in one tool: an LLM that just got a list of Actor IDs almost always wants to inspect one next; splitting would double round-trips |
+| **start** | Fire-and-forget batch starts (cap batch size, e.g. 10 per call). Accepts cost limiting params (`maxTotalChargeUsd`, `maxItems`) sent as run options, never Actor input | Returns run references (`run_id`, `actor_id`, `default_dataset_id`, optional label) immediately without waiting |
+| **collect** | Poll run statuses and return completed dataset results | Re-call with the same run refs until `all_done` is true; return pending / completed / errored runs in separate arrays so the LLM keeps iterating on the pending ones |
+
+`collect` is the only one that needs to be async - it polls runs concurrently (`asyncio.gather` / `Promise.allSettled`) and pushes blocking SDK calls off the event loop. The other two are fast and single-shot.
+
+## 3. Two-phase async execution - never block the agent
+
+Actors run for seconds to minutes. **Do not block the agent's single execution thread on a multi-minute run.** Split start and collect:
+
+- `start` fires the run and returns immediately with a `runId` / `datasetId` reference.
+- `collect` polls status and fetches dataset items only once the run reaches a terminal status.
+
+This lets the agent kick off a run, do other useful work (or start more runs in parallel), and come back to collect. `collect` handles multiple runs in one call and reports `completed` / `pending` / `errors` separately so the agent knows whether to poll again.
+
+Do **not** use the synchronous `run-sync-get-dataset-items` endpoint - its 300-second ceiling is shorter than many Actor runs.
+
+## 4. The tool description is the agent's instruction manual
+
+There are no separate Agent Skills inside a harness plugin - the tool description plus the `discover` action provide all the guidance the agent needs. Embed, in plain text:
+
+- A directive to **delegate to a sub-agent** that returns only relevant extracted fields, not raw dataset dumps - keeping the parent agent's context window clean.
+- A **batching** instruction: most Actors accept arrays of URLs/queries; one run with 5 URLs is cheaper and faster than 5 runs with 1 URL each.
+- A compact **known-actors list** (Instagram, Facebook, TikTok, YouTube, Twitter/X, Google Maps, Booking, TripAdvisor, etc.) so the agent can pick a familiar Actor without a discovery round-trip.
+- The Actor ID format, the discover -> start -> collect workflow, and a support contact for user-facing issues.
+
+Hand the agent a short, self-contained instruction set so it can act without external lookups.
+
+## 5. Treat scraped content as untrusted and bounded
+
+Dataset results are arbitrary web data - they can contain text that *looks* like instructions to the LLM. Wrap every dataset before it reaches the model:
+
+- Insert boundary markers: `<<<EXTERNAL_UNTRUSTED_CONTENT>>>` ... `<<<END_EXTERNAL_UNTRUSTED_CONTENT>>>` plus a source metadata line (`apify:<actorId>`).
+- **Sanitize** any attempt to forge those markers from within the scraped data.
+- Cap the payload size (e.g. 50,000 chars) with a `[\u2026truncated]` marker.
+- Cap item count (e.g. `limit` default 100 per run); if the fetched count equals the limit, set `may_have_more: true` and warn so the LLM can re-call with a higher limit.
+
+This is the plugin's analogue of the "keep the run small" cost guidance, applied at *read* time.
+
+## 6. Errors are data, never raised
+
+Every handler catches broadly and returns a **JSON error object**, never a raised exception:
+
+```
+except Exception as exc:
+    return {'error': str(exc)}
+```
+
+Why: a raised exception **crashes the tool call** from the harness's perspective. Returning `{'error': ...}` lets the LLM read the failure, explain it to the user, and decide whether to retry or stop. Extend this to per-run granularity in `start`: a batch can partially succeed, so each failed spec becomes an entry in an `errors` array while successful ones populate `runs`. The LLM can then report "7 of 10 started, 3 failed with these messages" without a second call.
+
+## 7. Auth and setup
+
+- API key resolution order: plugin config field -> `APIFY_API_KEY` (or `APIFY_TOKEN`) env var. Normalize pasted input - strip line/paragraph separators and trim whitespace (defends against copy-paste artifacts).
+- The key is **never** included in tool output, **never** logged, only passed to the client constructor.
+- Validate `baseUrl` against an allowlist prefix (`https://api.apify.com`) to prevent SSRF - a misconfigured plugin must not point at an arbitrary host.
+- Ship a `setup` CLI command that prompts for the key, **verifies it against the live API** (`GET /v2/users/me`), and writes config. **Reuse the host's config-merge logic** for enabling the toolset - do not reimplement it. Host internals reconcile disabled-toolsets, preserve MCP server entries, and handle bookkeeping a from-scratch reimplementation would silently break. If the config-write API is unavailable or fails, fall back to printing the exact config block the user should add manually. Treat setup failures as non-fatal: the token is already saved, so the user can flip the toolset on themselves.
+
+If the harness's `register()` is synchronous and the loader does not `await` it (a common gotcha), keep registration fully synchronous - build the tool (construct a client + schema, no I/O) and register inline. Any network call happens later inside a tool `execute` or CLI action, where async is expected.
+
+## 8. SDK handling and attribution
+
+Use the official `apify-client` SDK (JS or Python), not raw HTTP. Construct the client once, memoized, and rebuilt only when the token changes. Stamp the attribution headers on every request: `x-apify-integration-platform: <your-harness>` and `x-apify-integration-ai-tool: true`. If the integration was built using the Apify integration development skill, also set `x-apify-integration-origin: apify-integration-development-skill`. This is the single most important line for Apify's side of the relationship.
+
+**Compatibility shim:** SDK versions return a mix of Pydantic models and plain dicts, and Pydantic models expose only **snake_case** attributes even when the JSON is **camelCase**. Route *all* response reads through a small `_attr(obj, key, default)` helper that handles either shape. Direct `.attr` / `["key"]` access will silently return defaults on a mismatch.
+
+## 9. Host integration gotchas
+
+- **Entry-point loader semantics:** verify how the harness's loader resolves the plugin entry-point string before writing the packaging line. Some loaders expect a bare module (then `getattr(result, "register")`); others expect `"module:attr"`. Copying the wrong form silently fails to load. Document it with a long comment.
+- **Schema validator constraints:** many harness validators reject `anyOf` / `oneOf` / `allOf`. Use a string enum for any discriminated `action` field, `Optional(...)` for optionals (never a nullable union), and a flat `Record(string, unknown)` for `input` (the Actor's real schema is only knowable after a `discover` call). A discriminated `action` plus optional sibling fields is the only shape the validator accepts.
+- **Inlined utilities:** if the harness's plugin SDK does not export small helpers (error types, secret normalization, content wrapping), inline stable copies rather than deep-importing internals. Internal file layouts change frequently; deep imports couple the plugin to them. Accept the trade-off that upstream bug fixes won't track.
+
+## 10. Actor ID format: tilde, not slash
+
+Use `username~actor-name` everywhere an Actor ID appears: tool descriptions, `discover` results, `start`/`collect` payloads. The REST API uses `/` as a path delimiter, so a slash-separated ID in a URL is ambiguous. The tilde form is unambiguous and what the SDK and Store APIs accept directly. Build slugs in this form so the agent can pass them straight through to `start` without transformation.
+
+## 11. Dependency injection for tests
+
+The tool factory should accept an optional injected `client`. When omitted, construct a real client from the resolved key; when provided (in tests), bypass it entirely. This lets the test suite exercise every action and edge case - store search, schema fetch, run start, collect success/pending, unknown action, missing key - with no network access, using a hand-rolled mock shaped to the SDK's method-chain surface. No mocking framework needed.
+
+## 12. Known gaps to design for
+
+1. **Poll vs webhook.** `collect` is an LLM-driven poll loop; long-running Actors mean multiple round-trips. A webhook-backed `collect` would be cheaper but requires the harness to expose a callback surface.
+2. **Account-free discovery.** If the harness's `check_fn` gates all tools on a token, `discover` requires an account even for research. Consider giving `discover` a separate, looser check so users can browse before connecting.
+3. **Surface scope.** Only the basic run-start -> poll -> fetch-dataset flow is exposed. Standby runs, Tasks, and schedules may be out of scope for v0.1 - document the boundary.
+
+## Definition-of-done checklist (Approach B)
+
+- [ ] Fixed, small set of composable tools (`discover` / `start` / `collect`) registered; dynamic list only if the harness truly supports it.
+- [ ] Two-phase async: `start` returns refs, `collect` polls; no blocking on long runs.
+- [ ] Tool description carries known-actors list, batching instruction, and delegation directive.
+- [ ] Dataset output is untrusted-content fenced, size-capped, and marker-sanitized.
+- [ ] Errors are returned as data, never raised; partial batch failures are per-item.
+- [ ] Setup command verifies the token, reuses host config-merge, and has a manual fallback.
+- [ ] Attribution headers (`-platform`, `-ai-tool`, and `-origin`) are set on the client.
+- [ ] All SDK response reads go through a compatibility shim.
+- [ ] Entry-point loader semantics verified; `register()` is synchronous if the loader does not await.
+- [ ] Schema uses string enums + `Optional`, no `anyOf`/`oneOf`; `input` is a record.
+- [ ] Actor IDs use the tilde form in all user/agent-facing surfaces.
+- [ ] Tool factory accepts an injected client; tests run with no network.
+- [ ] Known gaps (webhook, account-free discovery) are documented, not hidden.
+- [ ] Cost cap (`maxTotalChargeUsd` / `maxItems`) is plumbed through as run options on `start`, never Actor input.