opencode-skills-collection 4.0.69 → 4.0.70

package/bundled-skills/apify-integration-development/references/ai-framework-package.md

@@ -0,0 +1,158 @@
+# AI framework package integrations
+
+Design guide for building a PyPI/npm package that exposes Apify to an AI/LLM framework - LangChain, LlamaIndex, Haystack, Vercel AI SDK, or similar. These are *client-side* integrations: code that calls Apify Actors from outside the Apify runtime, for applications, agents, and RAG pipelines. Apply the cross-cutting rules from `SKILL.md` on top.
+
+## 1. Scope: client-side only, wrap apify-client, never the Actor SDK
+
+This package is for applications that call Apify Actors from outside the Apify runtime. It is **not** for code running *inside* an Actor. Apify Actors should run with limited permissions and use scoped tokens via the Actor SDK's `Actor.open_dataset()`; importing a framework client that reconstructs its own `ApifyClient` from an env-var token would bypass that scoping and pull an unnecessary dependency into Actor images.
+
+Dependency philosophy: wrap the official `apify-client` library, never the `apify` SDK. `apify` is for *building* Actors; `apify-client` is for *calling* them. Keep the runtime dependency surface minimal (`langchain-core`, `apify-client`, and a backport if needed) to minimize version conflicts and keep install time short in agent environments.
+
+Stamp a custom `user-agent` suffix (e.g. `; Origin/langchain`) or the attribution header on the client so Apify can attribute traffic.
+
+## 2. Layered architecture: client -> framework adapters -> public API
+
+```
+Public API       curated exports
+                        |
+   +--------------------+--------------------+
+   |                    |                    |
+Tools             Document loaders        Retriever
+(agents)          (RAG ingestion)         (RAG retrieval)
+   |                    |                    |
+              ApifyToolsClient  (sync)
+                        |
+              apify-client (sync + async)
+                        |
+              Apify REST API
+```
+
+| Layer | Role |
+|---|---|
+| **Client** | Thin, synchronous wrapper over `apify-client`. One method per Actor operation. No framework types here. |
+| **Tools** | Framework `BaseTool` subclasses for agent tool-calling. |
+| **Document loaders** | `BaseLoader` implementations for RAG ingestion. |
+| **Retriever** | `BaseRetriever` for RAG query-time retrieval. |
+
+Framework types live only above the client layer. The client layer speaks pure Python/JS dicts and `apify-client` objects. This lets the client be unit-tested with no framework dependency, and lets the framework-facing layers focus exclusively on schema, tool semantics, and envelope formatting.
+
+## 3. ApifyToolsClient: one sync gateway, typed method per Actor
+
+All Actor interaction goes through a single synchronous client class with one convenience method per supported Actor (e.g. `google_search`, `instagram_scrape`, `crawl_website`). Each method:
+
+- Builds the Actor-specific `run_input` dict, translating from the integration's normalized parameter names to the Actor's raw input schema. (Actor schemas are idiosyncratic - `searchStringsArray`, `directUrls`, `detailsUrls` vs `listingUrls`; the client absorbs that so the tool exposes clean names like `query`, `url`, `url_type`.)
+- Calls `client.actor(id).call(...)` which blocks until the run finishes.
+- Checks run status and raises if the run did not reach `SUCCEEDED` (a failed run must never silently return empty results).
+- Returns a `(run_details, items)` tuple (or just one where appropriate).
+
+**Why blocking?** Callers don't manage polling loops; the API stays simple. The async surface is handled at the framework layer (`asyncio.to_thread` / `Promise.resolve`) rather than duplicating every method in async form.
+
+Adding a new Actor tool means adding one client method (input translation + status check) and one tool class (schema + `_run`), not wiring up polling, retries, or async variants.
+
+## 4. Uniform JSON output envelope
+
+All tools return a JSON string of one shape:
+
+```json
+{"run": {"run_id": "...", "status": "...", "dataset_id": "...",
+         "started_at": "...", "finished_at": "..."},
+ "items": [...]}
+```
+
+`run` is `null` for dataset-only tools. An optional `notice` key surfaces out-of-band hints (e.g. an Actor returned demo placeholder data on the free plan). Serialize with `default=str` so non-JSON-native types (datetimes from a `clean=True` deserialiser) never throw mid-tool-call.
+
+A single predictable envelope lets agents parse results with one code path. The `run` metadata gives the agent enough to chain calls - run an Actor with one tool, then fetch the dataset with another using the returned `dataset_id`. End every tool description with "Use only the data returned; do not hallucinate missing fields."
+
+## 5. Safety clamps - defense against LLM-requested extremes
+
+An LLM invoking a tool can request absurd values: 10,000 results, 32 GB of memory, a 1-hour timeout. Clamp every request to **developer-controlled ceilings**:
+
+| Clamp | Default ceiling | Developer max |
+|---|---|
+| `timeout_secs` | 600 s |
+| `memory_mbytes` | 4,096 MB (snapped to nearest valid power-of-2) | 8,192 MB |
+| `items` / `limit` | 1,000 |
+| `max_crawl_depth` | 5 |
+
+Memory is notable: Apify accepts memory only as a power-of-2 (128, 256, 512, ..., 32768). Snap an arbitrary LLM value to the nearest valid step at or below the developer's cap. The default ceiling of 4,096 MB (4 GB) is generous for most Actors but well below the platform max, so LLM-requested extremes are clamped. The developer can raise the ceiling up to 8,192 MB, but an LLM cannot widen it beyond the developer-set value.
+
+Some Actors have runtime limits not declared in their input schema (e.g. a RAG web browser rejects `maxResults > 100` at runtime). These can't be derived by schema introspection - track them by hand as overrides on the specific tool so the clamp enforces the Actor's real ceiling.
+
+The ceilings are *developer-controlled fields* on the tool instance - an application can tighten them further, but the LLM cannot widen them. This makes the integration safe to hand to an autonomous agent without risking runaway compute costs.
+
+## 6. Curated tool subsets, not one monolithic list
+
+Tools are grouped into convenience lists:
+
+| List | Tools | Use case |
+|---|---|---|
+| Core | Run Actor, get dataset, run+get, scrape URL, run task, run task+get | Generic platform primitives |
+| Search | Google search, web crawler, RAG web browser, Google Maps, YouTube, e-commerce | Web search & content crawling |
+| Social | Instagram, LinkedIn, Twitter/X, TikTok, Facebook | Social media scraping |
+
+Warn explicitly: **don't bind all tools at once.** Most LLMs lose routing accuracy past ~8 tools, so pick the family the agent actually needs. Curated subsets let an agent built for social-media analysis avoid distinguishing among 19 tool descriptions.
+
+## 7. Hand-written tools + dynamic schema for the long tail
+
+Alongside hand-written tools (which get clean schemas and descriptions), ship one dynamic tool that takes an `actor_id` at construction, fetches the Actor's latest default build, and **generates an input model dynamically** from the build's input schema. Prune descriptions to a fixed length; limit properties to `type`, `default`, `prefill`, `enum`.
+
+This covers the long tail of Actors without a dedicated wrapper - you don't need a hand-written tool for every one of Apify's thousands of Actors. The trade-off is a looser schema (the LLM sees the raw Actor input shape) and a network call at construction time.
+
+## 8. Map to the framework's idiomatic surfaces
+
+Implement the framework's actual extension points, all backed by the same client:
+
+| Surface | Base class | Use case |
+|---|---|---|
+| **Tools** | `BaseTool` | Agent tool-calling (ReAct, LangGraph) |
+| **Document loaders** | `BaseLoader` | Batch RAG ingestion (load -> split -> embed -> vector store) |
+| **Retriever** | `BaseRetriever` | Query-time web retrieval for RAG chains |
+
+- A **dataset loader** loads an existing dataset by ID and maps each item to a `Document` via a user-supplied mapping function (every Actor's output schema is different, so give the user full control). Implement both eager `load()` and streaming `lazy_load()`.
+- A **crawl loader** is an *active* loader: it runs a content crawler on construction, then yields `Document`s with `page_content` (markdown) and `metadata` (`source`, `title`, `crawl_depth`).
+- A **search retriever** wraps a search-and-crawl Actor for low-latency interactive RAG; the async path runs the synchronous client off the event loop via `to_thread`.
+
+## 9. Token hygiene
+
+One canonical token parameter/env var (e.g. `apify_token` / `APIFY_TOKEN`). If a legacy name exists (`APIFY_API_TOKEN`), honor it with a `DeprecationWarning` but reject new code that declares it. Centralize the policy in two helpers: one for explicit `__init__` signatures, one for Pydantic `model_validator(mode='before')` hooks. Store the token as a `SecretStr` (excluded from repr and serialization) and never log it.
+
+## 10. Content extraction: markdown-first with defensive fallbacks
+
+When extracting page content from crawling Actors, prefer `markdown` over `text`, with a trailing `or ''` to guarantee a string even when a key is present but null. Follow a fixed fallback order for the source URL: nested `metadata.url` -> `crawledUrl` -> top-level `url`. Tolerate a `metadata` field that is missing or not a dict (some Actor responses surface `null`). Actor output shapes are inconsistent across versions and configurations; centralize one canonical fallback order so the retriever, loaders, and tools all agree on what "the content", "the source URL", and "the title" mean.
+
+## 11. Error mapping
+
+- **Client layer** raises `RuntimeError` for failed/empty runs and `ValueError` for invalid input. Wrap transport errors in `RuntimeError`.
+- **Tool layer** catches both and re-raises as the framework's tool-error type (e.g. `ToolException`) with `handle_tool_error = True`, which surfaces to the agent as a recoverable error message.
+
+An agent that gets a `ToolException` can read the message and retry with corrected input. An unhandled `RuntimeError` would crash the agent loop. The boundary is clean: the client raises domain errors; the tool adapts them to the framework's tool-error protocol.
+
+## 12. Packaging, release, and quality bar
+
+- Minimal runtime deps; an explicit sdist allowlist so local-only paths (`dist/`, `.venv/`, `docs/`, test fixtures) never reach the registry.
+- Release via conventional-commits-driven automation that reads commit-message prefixes to auto-generate the changelog and compute the version bump; a `BREAKING CHANGE:` footer triggers a major bump. Never hand-edit `version =` or `CHANGELOG.md` if the workflow manages them.
+- Strict linting (`select = ["ALL"]` with a curated ignore list), strict typing (`disallow_untyped_defs`), and **socket-disabled unit tests** so the unit suite is truly unit - no hidden integration dependencies. Integration tests need a real token (CI only).
+
+## 13. Position vs the Apify MCP server
+
+The README's top banner should direct users to Apify's MCP server (`https://mcp.apify.com`) as a richer, more featureful alternative for interactive agent workflows that need dynamic Actor discovery. The package is not deprecated, but the MCP path is recommended for new interactive agent sessions.
+
+The positioning: the package is the **programmatic, typed, registry-installable** option for code that outlives a single agent session (servers, scheduled jobs, pipelines); the MCP server is the **interactive, dynamic** option. Rather than compete, position them for their respective audiences.
+
+## Definition-of-done checklist
+
+- [ ] Package is client-side only; depends on `apify-client`, never `apify`.
+- [ ] Layered: thin client (no framework types) -> framework adapters -> curated public API.
+- [ ] One synchronous client with a typed method per supported Actor; input normalization centralized.
+- [ ] All tools return the uniform JSON envelope; serialization never throws on non-native types.
+- [ ] Developer-controlled safety clamps (timeout, memory power-of-2, items, depth) are in place; hand-tracked runtime limits override specific tools.
+- [ ] Tools grouped into curated subsets; documentation warns against binding all at once.
+- [ ] A dynamic-schema tool covers the long tail of Actors.
+- [ ] Framework surfaces (tools / loaders / retriever) all backed by the same client.
+- [ ] One canonical token name; legacy alias emits a deprecation warning; token is `SecretStr`, never logged.
+- [ ] Content extraction is markdown-first with documented fallback order.
+- [ ] Client raises domain errors; tools adapt them to the framework's tool-error protocol.
+- [ ] sdist allowlist excludes local paths; release automation drives versioning.
+- [ ] Unit tests are socket-disabled; lint/typing are strict.
+- [ ] README cross-references the MCP server for interactive/dynamic use.
+- [ ] Attribution header / user-agent suffix is set on the client; skill-origin header included if built from this skill.

package/bundled-skills/apify-integration-development/references/ai-harness-plugin.md

@@ -0,0 +1,192 @@
+# AI agent plugin integrations
+
+Design guide for building an Apify plugin that gives an AI agent access to Actors. There are **two plugin shapes**, and which one you build depends on the host:
+
+- **Approach A - Coding agent plugin (skills + MCP bundle):** for skills/MCP-aware coding assistants like Cursor, Claude Code, Codex, and GitHub Copilot. You assemble a small set of runtime artifacts the host already knows how to load, and the hosted Apify MCP server (`https://mcp.apify.com`) provides the tool surface. Minimal code.
+- **Approach B - Harness / assistant plugin (custom tool registry):** for agent runtimes like OpenClaw-style runtimes and Hermes-style harnesses that have their own tool registry and config file. You build a small custom toolset (`discover` / `start` / `collect`) backed by the `apify-client` SDK, using a stored credential rather than per-session OAuth.
+
+Apply the cross-cutting rules from `SKILL.md` on top of either approach.
+
+## Which approach? Trade-offs
+
+| Dimension | A - Coding agent plugin (skills + MCP) | B - Harness / assistant plugin (custom registry) |
+|---|---|---|
+| Target hosts | Cursor, Claude Code, Codex, GitHub Copilot | OpenClaw-style runtimes, Hermes-style harnesses, custom tool-calling agents |
+| Tool surface | Hosted Apify MCP server - no tool code to write | You implement `discover` / `start` / `collect` yourself |
+| Auth | OAuth (MCP) / `apify login` (CLI) / `APIFY_TOKEN` (SDK), per route | Stored API key resolved by the plugin, passed to `apify-client` |
+| Build effort | Low - assemble artifacts, no HTTP/retry/registry code | Higher - tools, schema, error taxonomy, host gotchas |
+| Capabilities | Run existing Actors **and** build/test/deploy new Actors **and** integrate an app | Broker the Store to the agent (run existing Actors) |
+| Maintenance | The MCP server owns the runtime surface | You own the tool code plus host-SDK compatibility |
+| Best when | The host supports skills/MCP and you want the fastest path | The host has its own registry and needs bespoke tools |
+
+If the host is a skills/MCP-aware coding tool, prefer Approach A. If the host is a custom harness with its own registry (no MCP), use Approach B. A product can ship both over time - start with whichever matches the primary host.
+
+---
+
+# Approach A - Coding agent plugin (skills + MCP bundle)
+
+For skills/MCP-aware coding assistants (Cursor, Claude Code, Codex, GitHub Copilot). This section describes the **installed plugin** - what the user gets and how the pieces interact at runtime - not how the bundle is produced.
+
+## A.1 The four runtime artifacts
+
+| Artifact | Role at runtime |
+|---|---|
+| **MCP server** | Registers `https://mcp.apify.com` with the host and exposes the callable tool surface (below). OAuth: the user signs in via browser on the first tool call that needs auth - no token in config. |
+| **Skills** | On-demand `SKILL.md` instruction documents. The host matches each skill's `description` against user intent and loads the body into context only when relevant, keeping baseline context small. |
+| **Subagent / router** | The entry point. Classifies the request into one of three routes, selects the transport (MCP vs CLI), and invokes the matching skill or tools. |
+| **Slash commands** | User-invoked entry points (e.g. `/create-actor <description>`) that drive a guided end-to-end workflow. |
+
+**MCP tool surface** once connected: `search-actors` (search the Store), `fetch-actor-details` (input schema, output format, pricing), `call-actor` (run with input JSON), `get-actor-run` (poll status), `get-dataset-items` (fetch results), `search-apify-docs` / `fetch-apify-docs` (docs). The discovery subset (`search-actors`, `fetch-actor-details`, `search-apify-docs`, `fetch-apify-docs`) works without an account.
+
+## A.2 The three routes the plugin serves
+
+The router classifies every request and routes it:
+
+| Signal | Route | How it runs |
+|---|---|---|
+| Use existing Actors (search, run, get data) | 1 | MCP tools directly; the CLI is the fallback when MCP is unavailable |
+| Build / test / deploy a custom Actor | 2 | Apify CLI (`apify create` / `run` / `push`) - local filesystem, no MCP equivalent |
+| Add Apify to an existing app | 3 | `apify-client` over HTTPS - neither MCP nor CLI |
+
+**MCP-vs-CLI selection (Route 1 only).** Detect transports once: MCP is available if a tool named `search-actors` is in the tool list; CLI is available if `apify --help` exits 0. Prefer MCP when present (no shell/install friction, OAuth auth); fall back to the CLI otherwise. Routes 2 and 3 are unaffected.
+
+**Naming trap.** The `apify` npm package is the **SDK for building** Actors (Route 2). The `apify-client` package is the **API client for calling** Actors (Route 3). Never confuse them.
+
+**Auth per route:** Route 1 (MCP) OAuth via browser prompt, never ask for a token; Route 1 CLI fallback + Route 2 `apify login --token <TOKEN>` once (the CLI ignores `APIFY_TOKEN`); Route 3 the `APIFY_TOKEN` env var.
+
+## A.3 Definition of done (Approach A)
+
+- [ ] MCP declared and reachable - `https://mcp.apify.com` registered and its tools appear in the tool list.
+- [ ] OAuth works - the first auth-requiring MCP call prompts a browser sign-in; no token in config.
+- [ ] Skills load by intent - each skill's `description` matches its requests; bodies load only when relevant.
+- [ ] Router classifies correctly - requests land on Route 1 / 2 / 3; ambiguous ones ask the user to choose.
+- [ ] Transport selection correct - Route 1 prefers MCP and falls back to CLI cleanly; Routes 2/3 use CLI/SDK.
+- [ ] Auth wired per route; the `apify` vs `apify-client` distinction is never confused.
+- [ ] Cost caps honored (`maxTotalChargeUsd` / `maxItems`) and attribution headers set (see `SKILL.md`).
+- [ ] Slash command (e.g. `/create-actor`) runs end to end.
+- [ ] Verified inside the actual target tool, not just in isolation.
+
+---
+
+# Approach B - Harness / assistant plugin (custom tool registry)
+
+For agent runtimes with their own tool registry (OpenClaw-style runtimes, Hermes-style harnesses, or any custom tool-calling agent). The plugin brokers the entire Apify Store to the agent - it does not bundle scrapers.
+
+The harness runs locally/persistently, has its own tool registry and config file, and calls Apify with a stored credential rather than per-session OAuth. So this shape borrows from the "API token + apify-client" path, not the MCP path.
+
+## 1. Shape decision: few composable tools vs a dynamic tool list
+
+Decide based on what the harness supports:
+
+- **Static tool registry** (tools registered once at plugin load, no per-Actor materialization): register a **small, fixed set of composable tools** and let the LLM compose them. This keeps the prompt budget small and the call graph legible.
+- **Dynamic tool registration** (the harness can materialize tools at runtime): you *can* expose a dynamic per-Actor tool list, but a fixed trio is still simpler and usually enough.
+
+The MCP server's surface (search / inspect / call / poll / fetch as separate dynamic tools) is one shape. A harness plugin is a different shape - do not copy it blindly.
+
+## 2. Canonical action set: discover / start / collect
+
+Three tools cover the entire workflow and map cleanly to the asynchronous REST flow (`POST /runs` -> poll `GET /actor-runs/{id}` -> `GET /datasets/{id}/items`):
+
+| Tool | Purpose | Why |
+|---|---|---|
+| **discover** | Search Apify Store by keyword, OR fetch a single Actor's input schema + README by `actorId` | Two modes in one tool: an LLM that just got a list of Actor IDs almost always wants to inspect one next; splitting would double round-trips |
+| **start** | Fire-and-forget batch starts (cap batch size, e.g. 10 per call). Accepts cost limiting params (`maxTotalChargeUsd`, `maxItems`) sent as run options, never Actor input | Returns run references (`run_id`, `actor_id`, `default_dataset_id`, optional label) immediately without waiting |
+| **collect** | Poll run statuses and return completed dataset results | Re-call with the same run refs until `all_done` is true; return pending / completed / errored runs in separate arrays so the LLM keeps iterating on the pending ones |
+
+`collect` is the only one that needs to be async - it polls runs concurrently (`asyncio.gather` / `Promise.allSettled`) and pushes blocking SDK calls off the event loop. The other two are fast and single-shot.
+
+## 3. Two-phase async execution - never block the agent
+
+Actors run for seconds to minutes. **Do not block the agent's single execution thread on a multi-minute run.** Split start and collect:
+
+- `start` fires the run and returns immediately with a `runId` / `datasetId` reference.
+- `collect` polls status and fetches dataset items only once the run reaches a terminal status.
+
+This lets the agent kick off a run, do other useful work (or start more runs in parallel), and come back to collect. `collect` handles multiple runs in one call and reports `completed` / `pending` / `errors` separately so the agent knows whether to poll again.
+
+Do **not** use the synchronous `run-sync-get-dataset-items` endpoint - its 300-second ceiling is shorter than many Actor runs.
+
+## 4. The tool description is the agent's instruction manual
+
+There are no separate Agent Skills inside a harness plugin - the tool description plus the `discover` action provide all the guidance the agent needs. Embed, in plain text:
+
+- A directive to **delegate to a sub-agent** that returns only relevant extracted fields, not raw dataset dumps - keeping the parent agent's context window clean.
+- A **batching** instruction: most Actors accept arrays of URLs/queries; one run with 5 URLs is cheaper and faster than 5 runs with 1 URL each.
+- A compact **known-actors list** (Instagram, Facebook, TikTok, YouTube, Twitter/X, Google Maps, Booking, TripAdvisor, etc.) so the agent can pick a familiar Actor without a discovery round-trip.
+- The Actor ID format, the discover -> start -> collect workflow, and a support contact for user-facing issues.
+
+Hand the agent a short, self-contained instruction set so it can act without external lookups.
+
+## 5. Treat scraped content as untrusted and bounded
+
+Dataset results are arbitrary web data - they can contain text that *looks* like instructions to the LLM. Wrap every dataset before it reaches the model:
+
+- Insert boundary markers: `<<<EXTERNAL_UNTRUSTED_CONTENT>>>` ... `<<<END_EXTERNAL_UNTRUSTED_CONTENT>>>` plus a source metadata line (`apify:<actorId>`).
+- **Sanitize** any attempt to forge those markers from within the scraped data.
+- Cap the payload size (e.g. 50,000 chars) with a `[\u2026truncated]` marker.
+- Cap item count (e.g. `limit` default 100 per run); if the fetched count equals the limit, set `may_have_more: true` and warn so the LLM can re-call with a higher limit.
+
+This is the plugin's analogue of the "keep the run small" cost guidance, applied at *read* time.
+
+## 6. Errors are data, never raised
+
+Every handler catches broadly and returns a **JSON error object**, never a raised exception:
+
+```
+except Exception as exc:
+    return {'error': str(exc)}
+```
+
+Why: a raised exception **crashes the tool call** from the harness's perspective. Returning `{'error': ...}` lets the LLM read the failure, explain it to the user, and decide whether to retry or stop. Extend this to per-run granularity in `start`: a batch can partially succeed, so each failed spec becomes an entry in an `errors` array while successful ones populate `runs`. The LLM can then report "7 of 10 started, 3 failed with these messages" without a second call.
+
+## 7. Auth and setup
+
+- API key resolution order: plugin config field -> `APIFY_API_KEY` (or `APIFY_TOKEN`) env var. Normalize pasted input - strip line/paragraph separators and trim whitespace (defends against copy-paste artifacts).
+- The key is **never** included in tool output, **never** logged, only passed to the client constructor.
+- Validate `baseUrl` against an allowlist prefix (`https://api.apify.com`) to prevent SSRF - a misconfigured plugin must not point at an arbitrary host.
+- Ship a `setup` CLI command that prompts for the key, **verifies it against the live API** (`GET /v2/users/me`), and writes config. **Reuse the host's config-merge logic** for enabling the toolset - do not reimplement it. Host internals reconcile disabled-toolsets, preserve MCP server entries, and handle bookkeeping a from-scratch reimplementation would silently break. If the config-write API is unavailable or fails, fall back to printing the exact config block the user should add manually. Treat setup failures as non-fatal: the token is already saved, so the user can flip the toolset on themselves.
+
+If the harness's `register()` is synchronous and the loader does not `await` it (a common gotcha), keep registration fully synchronous - build the tool (construct a client + schema, no I/O) and register inline. Any network call happens later inside a tool `execute` or CLI action, where async is expected.
+
+## 8. SDK handling and attribution
+
+Use the official `apify-client` SDK (JS or Python), not raw HTTP. Construct the client once, memoized, and rebuilt only when the token changes. Stamp the attribution headers on every request: `x-apify-integration-platform: <your-harness>` and `x-apify-integration-ai-tool: true`. If the integration was built using the Apify integration development skill, also set `x-apify-integration-origin: apify-integration-development-skill`. This is the single most important line for Apify's side of the relationship.
+
+**Compatibility shim:** SDK versions return a mix of Pydantic models and plain dicts, and Pydantic models expose only **snake_case** attributes even when the JSON is **camelCase**. Route *all* response reads through a small `_attr(obj, key, default)` helper that handles either shape. Direct `.attr` / `["key"]` access will silently return defaults on a mismatch.
+
+## 9. Host integration gotchas
+
+- **Entry-point loader semantics:** verify how the harness's loader resolves the plugin entry-point string before writing the packaging line. Some loaders expect a bare module (then `getattr(result, "register")`); others expect `"module:attr"`. Copying the wrong form silently fails to load. Document it with a long comment.
+- **Schema validator constraints:** many harness validators reject `anyOf` / `oneOf` / `allOf`. Use a string enum for any discriminated `action` field, `Optional(...)` for optionals (never a nullable union), and a flat `Record(string, unknown)` for `input` (the Actor's real schema is only knowable after a `discover` call). A discriminated `action` plus optional sibling fields is the only shape the validator accepts.
+- **Inlined utilities:** if the harness's plugin SDK does not export small helpers (error types, secret normalization, content wrapping), inline stable copies rather than deep-importing internals. Internal file layouts change frequently; deep imports couple the plugin to them. Accept the trade-off that upstream bug fixes won't track.
+
+## 10. Actor ID format: tilde, not slash
+
+Use `username~actor-name` everywhere an Actor ID appears: tool descriptions, `discover` results, `start`/`collect` payloads. The REST API uses `/` as a path delimiter, so a slash-separated ID in a URL is ambiguous. The tilde form is unambiguous and what the SDK and Store APIs accept directly. Build slugs in this form so the agent can pass them straight through to `start` without transformation.
+
+## 11. Dependency injection for tests
+
+The tool factory should accept an optional injected `client`. When omitted, construct a real client from the resolved key; when provided (in tests), bypass it entirely. This lets the test suite exercise every action and edge case - store search, schema fetch, run start, collect success/pending, unknown action, missing key - with no network access, using a hand-rolled mock shaped to the SDK's method-chain surface. No mocking framework needed.
+
+## 12. Known gaps to design for
+
+1. **Poll vs webhook.** `collect` is an LLM-driven poll loop; long-running Actors mean multiple round-trips. A webhook-backed `collect` would be cheaper but requires the harness to expose a callback surface.
+2. **Account-free discovery.** If the harness's `check_fn` gates all tools on a token, `discover` requires an account even for research. Consider giving `discover` a separate, looser check so users can browse before connecting.
+3. **Surface scope.** Only the basic run-start -> poll -> fetch-dataset flow is exposed. Standby runs, Tasks, and schedules may be out of scope for v0.1 - document the boundary.
+
+## Definition-of-done checklist (Approach B)
+
+- [ ] Fixed, small set of composable tools (`discover` / `start` / `collect`) registered; dynamic list only if the harness truly supports it.
+- [ ] Two-phase async: `start` returns refs, `collect` polls; no blocking on long runs.
+- [ ] Tool description carries known-actors list, batching instruction, and delegation directive.
+- [ ] Dataset output is untrusted-content fenced, size-capped, and marker-sanitized.
+- [ ] Errors are returned as data, never raised; partial batch failures are per-item.
+- [ ] Setup command verifies the token, reuses host config-merge, and has a manual fallback.
+- [ ] Attribution headers (`-platform`, `-ai-tool`, and `-origin`) are set on the client.
+- [ ] All SDK response reads go through a compatibility shim.
+- [ ] Entry-point loader semantics verified; `register()` is synchronous if the loader does not await.
+- [ ] Schema uses string enums + `Optional`, no `anyOf`/`oneOf`; `input` is a record.
+- [ ] Actor IDs use the tilde form in all user/agent-facing surfaces.
+- [ ] Tool factory accepts an injected client; tests run with no network.
+- [ ] Known gaps (webhook, account-free discovery) are documented, not hidden.
+- [ ] Cost cap (`maxTotalChargeUsd` / `maxItems`) is plumbed through as run options on `start`, never Actor input.

package/bundled-skills/apify-integration-development/references/sdk-integration.md

@@ -0,0 +1,236 @@
+# SDK integrations
+
+Design guide for integrating Apify into an existing application by calling Actors directly with `apify-client` (JS/TS or Python) or the REST API. This is the lightest integration shape: no host platform, no tool registry, no plugin lifecycle - just your code calling Apify as a backend service. Apply the cross-cutting rules from `SKILL.md` on top.
+
+## 1. Wrap apify-client, never the Actor SDK
+
+- **`apify-client`** is the API client for **calling** Actors from your app.
+- **`apify`** is the SDK for **building** Actors (wrong package for this use case).
+
+Always install `apify-client`. Never install `apify` for integration work. Keep the dependency footprint small to minimize version conflicts and keep install time short.
+
+Stamp a custom `user-agent` suffix or the attribution header (`x-apify-integration-platform: <your-app>`) on the client so Apify can attribute traffic. If the integration was built using the Apify integration development skill, also set `x-apify-integration-origin: apify-integration-development-skill`.
+
+## 2. Token handling
+
+Get an `APIFY_TOKEN` from **Console > Settings > Integrations** at `https://console.apify.com/settings/integrations`. Account sign-up: `https://console.apify.com/sign-up` (free, no credit card). Store the token in an environment variable or a secrets manager - never hardcoded, never in chat logs or command output, never in URLs (query-string tokens leak through browser history and server logs).
+
+The token is a normal Bearer credential:
+
+```
+Authorization: Bearer <APIFY_TOKEN>
+```
+
+Use scoped tokens where possible and rotate them periodically.
+
+## 3. Find the right Actor before writing code
+
+Before writing integration code, find the Actor that fits the need. Use the MCP tools if available in your environment:
+
+- `search-actors` - search the Apify Store by keyword (search by platform/product name, not end goal).
+- `fetch-actor-details` - get the Actor's input schema, output format, and pricing.
+
+Alternatively, browse `https://apify.com/store`. Append `.md` to any Actor's Store URL to get its docs in markdown (e.g. `https://apify.com/apify/web-scraper.md`). Build input from the schema rather than guessing field names.
+
+## 4. JavaScript / TypeScript
+
+### Install
+
+```bash
+npm install apify-client
+```
+
+### Synchronous execution (wait for results)
+
+```typescript
+import { ApifyClient } from 'apify-client';
+
+const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
+
+const run = await client.actor('apify/web-scraper').call({
+    startUrls: [{ url: 'https://example.com' }],
+    maxPagesPerCrawl: 10,
+}, { maxTotalChargeUsd: 5 });
+
+const { items } = await client.dataset(run.defaultDatasetId).listItems();
+```
+
+`.call()` blocks until the Actor finishes. Use for short-running Actors (under a few minutes). Pass `maxTotalChargeUsd` in the **options** argument - never in the input.
+
+### Asynchronous execution (start and poll)
+
+```typescript
+const run = await client.actor('apify/web-scraper').start({
+    startUrls: [{ url: 'https://example.com' }],
+});
+
+// Poll for completion
+// Derive waitSecs from the run's timeoutSecs + a grace buffer, never unbounded.
+const finishedRun = await client.run(run.id).waitForFinish({ waitSecs: 120 });
+
+// Retrieve results
+const { items } = await client.dataset(finishedRun.defaultDatasetId).listItems();
+```
+
+Use `.start()` + `.waitForFinish()` for long-running Actors or when you need the run ID immediately.
+
+### Retrieving results
+
+```typescript
+// Dataset items (structured data from pushData)
+const { items } = await client.dataset(run.defaultDatasetId).listItems({
+    limit: 100,
+    offset: 0,
+});
+
+// Key-value store (files, screenshots, etc.)
+const record = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT');
+```
+
+### Error handling
+
+```typescript
+try {
+    const run = await client.actor('apify/web-scraper').call(input);
+
+    if (run.status !== 'SUCCEEDED') {
+        const log = await client.log(run.id).get();
+        throw new Error(`Actor failed with status ${run.status}: ${log}`);
+    }
+
+    const { items } = await client.dataset(run.defaultDatasetId).listItems();
+} catch (error) {
+    if (error.type === 'record-not-found') {
+        // Actor ID is wrong or Actor was deleted
+    } else if (error.statusCode === 401) {
+        // Invalid or missing APIFY_TOKEN
+    }
+    throw error;
+}
+```
+
+## 5. Python
+
+### Install
+
+```bash
+pip install apify-client
+```
+
+### Synchronous execution
+
+```python
+from decimal import Decimal
+from apify_client import ApifyClient
+import os
+
+client = ApifyClient(token=os.environ['APIFY_TOKEN'])
+
+run = client.actor('apify/web-scraper').call(
+    run_input={
+        'startUrls': [{'url': 'https://example.com'}],
+        'maxPagesPerCrawl': 10,
+    },
+    max_total_charge_usd=Decimal('5'),
+)
+
+items = client.dataset(run['defaultDatasetId']).list_items().items
+```
+
+### Asynchronous execution
+
+```python
+run = client.actor('apify/web-scraper').start(run_input={
+    'startUrls': [{'url': 'https://example.com'}],
+})
+
+# Poll for completion
+finished_run = client.run(run['id']).wait_for_finish()
+
+items = client.dataset(finished_run['defaultDatasetId']).list_items().items
+```
+
+### Async client (asyncio)
+
+```python
+from apify_client import ApifyClientAsync
+
+client = ApifyClientAsync(token=os.environ['APIFY_TOKEN'])
+
+run = await client.actor('apify/web-scraper').call(run_input={
+    'startUrls': [{'url': 'https://example.com'}],
+})
+
+items = (await client.dataset(run['defaultDatasetId']).list_items()).items
+```
+
+## 6. REST API (any language)
+
+For languages without an official client, use the REST API directly.
+
+### Start a run
+
+```
+POST https://api.apify.com/v2/actors/{actorId}/runs
+Authorization: Bearer <APIFY_TOKEN>
+Content-Type: application/json
+
+{ "startUrls": [{ "url": "https://example.com" }] }
+```
+
+### Get run status
+
+```
+GET https://api.apify.com/v2/actor-runs/{runId}
+Authorization: Bearer <APIFY_TOKEN>
+```
+
+### Get dataset items
+
+```
+GET https://api.apify.com/v2/datasets/{datasetId}/items?format=json
+Authorization: Bearer <APIFY_TOKEN>
+```
+
+The cost rule applies on the HTTP paths too: `maxItems` (pay-per-result) and `maxTotalChargeUsd` (other pricing models) go as **query parameters**, never in the JSON body where they are read as Actor input.
+
+For runs expected to finish within 300 seconds, the synchronous endpoint returns dataset items directly:
+
+```
+POST https://api.apify.com/v2/actors/{username}~{actor-name}/run-sync-get-dataset-items
+```
+
+Longer work must use the asynchronous flow: POST /runs -> poll GET /actor-runs/{id} -> GET /datasets/{id}/items.
+
+REST reference: `https://docs.apify.com/api/v2`. OpenAPI spec: `https://apify.com/openapi.json`.
+
+## 7. Best practices
+
+- **Set timeouts:** pass `timeoutSecs` as a run option / query parameter on `.call()` or `.start()`, or use `waitSecs` on `.call()`. Never put `timeoutSecs` in Actor input — it is a run option and an Actor whose schema rejects unknown fields will fail on it.
+- **Paginate large datasets:** use `limit` and `offset` when retrieving dataset items.
+- **Reuse clients:** create one `ApifyClient` instance and reuse it across calls.
+- **Handle Actor-specific input:** every Actor has its own input schema. Use `fetch-actor-details` MCP tool or append `.md` to the Actor's Store URL to get the schema before constructing input.
+- **Bound polling:** never `while (true)` - use the run's own timeout + a grace buffer, with an absolute ceiling.
+
+## 8. Documentation pointers
+
+- API client for JS: `https://docs.apify.com/api/client/js`
+- API client for Python: `https://docs.apify.com/api/client/python`
+- REST API reference: `https://docs.apify.com/api/v2`
+- Apify docs (LLM-friendly): `https://docs.apify.com/llms.txt`
+- Apify docs (full): `https://docs.apify.com/llms-full.txt`
+
+If the Apify MCP server is available, use `search-apify-docs` and `fetch-apify-docs` tools for contextual documentation lookups during development.
+
+## Definition-of-done checklist
+
+- [ ] Depends on `apify-client`, never `apify`.
+- [ ] Token stored in env var / secret manager; never hardcoded or logged.
+- [ ] Actor input built from the schema (MCP `fetch-actor-details` or `.md` URL), not guessed.
+- [ ] Sync `.call()` used for short runs; async `.start()` + `.waitForFinish()` for long ones.
+- [ ] Attribution header / user-agent suffix set on the client; skill-origin header included if built from this skill.
+- [ ] Cost cap (`maxTotalChargeUsd` / `max_total_charge_usd`) passed in options, never input.
+- [ ] Run status checked before consuming dataset; failed runs raise, not return empty.
+- [ ] Dataset and KV store retrieval covered; pagination on large datasets.
+- [ ] Errors mapped to actionable app-level messages.
+- [ ] REST API fallback documented for languages without a client.