@getopencrawl/cli 0.1.1 → 0.2.1

package/README.md

@@ -1,323 +1,100 @@
-# OpenCrawl CLI
+# opencrawl
 
-Node.js 20 or later. Core CLI commands have no required package dependencies; browser capture uses optional Playwright.
+The OpenCrawl CLI. This initial release installs an automatically updating
+command and signs you in. **Index queries are not available yet.**
 
-The canonical deployment is live at `https://app.getopencrawl.com`; [STATUS.md](../STATUS.md) records its verification. Copy the exact immutable `npm install -g` command from **Connect an agent**, or use `packages.cli.url` from the [current manifest](https://app.getopencrawl.com/downloads/manifest.json) and use `opencrawl login --server https://app.getopencrawl.com`. Existing installations keep their original origin and account pins. Follow the [domain migration steps](../docs/cli.md#moving-to-the-canonical-domain), including flushing old capture and starting a new native chat after reinstalling.
-
-## Start here: new to OpenCrawl
-
-No account, card or browser is needed. These commands create a free organization, describe the data you need to the OpenCrawl agent, and build a project that crawls and refreshes in the background. The same steps over HTTP or MCP are in [the agent guide](https://app.getopencrawl.com/start.md).
-
-```sh
-npm install -g "$(curl -fsS https://app.getopencrawl.com/downloads/manifest.json | node -pe 'JSON.parse(require("fs").readFileSync(0)).packages.cli.url')"
-opencrawl agent signup --organization "Acme" --email you@acme.com --server https://app.getopencrawl.com
-opencrawl agent "Watch the planning commission agendas at https://example.gov/agendas daily, including linked PDFs, for data-center rezonings"
-opencrawl agent "answers to its questions" --session SESSION_ID
-opencrawl agent --session SESSION_ID --build
-```
-
-- `signup` saves the organization key as the default CLI account and in an owner-only key file, and prints where, never the key. `--build` does the same for the project key.
-- The OpenCrawl agent checks every page with OpenCrawl's crawler before a plan is ready. A page it cannot read (blocked by robots.txt or by the site) is listed in `questions`; replace it or drop it.
-- Read a project's watches, jobs and research with its project account: `opencrawl --account PROJECT_ACCOUNT research list`, `research job FIRST_CHECK_JOB --wait`, `research search "question" --collection WATCH_ID`, `search "question"`. The organization key cannot read inside projects (404).
-- Research questions run after their watches' first check, then daily. To change a built project, send the change on the same session, then `--build` again: new watches and research are added and existing ones keep running.
-- For MCP or server code, use the project key file: `OPENCRAWL_API_KEY="$(cat KEY_FILE)"`. Never paste keys into chat or commit them.
-
-From this repository:
-
-```sh
-npm install -g ./cli
-opencrawl login --server http://localhost:8000
-opencrawl search "What did we decide about onboarding?"
-opencrawl context "Write the launch plan" --budget 3000
-```
-
-Create a workspace API key in **Connect an agent → API keys and developer tools → Create key** and paste it into the hidden login prompt. Production servers require HTTPS. The key is saved in `~/.config/opencrawl/config.json` with owner-only file permissions, bound to the verified server, user, and workspace. Server URLs must be origins, such as `https://crawl.example.com`, without a path, credentials, query, or fragment. Standard ports and trailing slashes normalize to the same origin. Do not put API keys in command-line arguments.
-
-## Run hosted research
-
-With a matching server/client release, save the customer, goal, optional existing collection IDs and `schedule_minutes: 0` in `brief.json`. Save a random request key before sending:
-
-```sh
-opencrawl research agent create --file brief.json --request-key SAVED_RANDOM_REQUEST_KEY --json
-opencrawl research agent-run AGENT_ID RUN_ID --wait --timeout 300 --json
-opencrawl research agent receipt --operation create --request-key SAVED_RANDOM_REQUEST_KEY --json
-```
-
-Saved findings JSON preserves fetched-page provenance under `source_kind: "web_page"` and `page`. The page record contains the requested/final URLs, full extracted-page hash and supplied window bounds. Citation hashes/offsets still describe that window; add `page.start` to a citation offset for its full-page position. The CLI validates metadata and bounds, not full hashes from a partial quote. Older `web_search`/`indexed` findings remain readable without `page`, and these reads do not fetch the source URL.
-
-Replace placeholders with your saved key and returned IDs. The first command starts bounded external research; the second only reads its persisted run; the third recovers the original admission after uncertainty. Never create a new key because a response was lost. Findings retain source quotes, qualifications and gaps. Older relevance text remains visible; new runs omit the redundant selection label in text output and preserve it in JSON. Use the intended workspace account or customer child key. See the [brief format, SDK/MCP equivalents and recovery contract](../docs/research-agents.md).
-
-### Explicit bounded completion policy
-
-Where the operator has enabled it, add `"research_policy":"completion_v1"` to an agent creation JSON file, or explicitly request it for one manual run:
-
-```sh
-opencrawl research agent run AGENT_ID --revision REVISION --request-key SAVED_RANDOM_REQUEST_KEY --research-policy completion_v1 --json
-```
-
-The setting applies to that admission, not the saved agent or daily schedule. Omit it to preserve the ordinary request. The server refuses an unavailable policy; the CLI never enables it, changes the key or submits a fallback request. Recover an uncertain admission with the original operation and request key, as above.
-
-Valid optional completion metadata adds fixed notices alongside all saved gaps. It describes bounded work and cannot certify whole-question completeness; malformed or absent optional metadata adds no notice and leaves findings, execution status and JSON unchanged. A recorded lower bound on completed model usage is not a total for an interrupted run. This CLI implementation does not establish that the policy is enabled on the selected server.
-
-
-For an operator-enabled final presentation run, explicitly select `completion_v2` in the same `research_policy` field (or CLI manual-run `--research-policy completion_v2`). Omitting the field keeps the existing request unchanged. The optional `usage.presentation` record reports bounded presentation status and known/uncertain usage; it does not certify completeness or replace findings, gaps, quotes, or `usage.answer_view` validation. SDK/shared readers expose it through `readResearchPresentation(run)` and ignore malformed optional metadata.
-
-For a validated `completion_v2` receipt, the default research-agent read wait ends 540 seconds after that wait began, allowing 480 seconds of server work plus settlement time. Repeated polls never renew it. An explicit SDK `maxWaitMs` or CLI `--timeout` still takes precedence; ordinary/v1 waits remain 300 seconds and individual default GET requests remain capped at 30 seconds. Durable customer-research poll batches and operation leases are unchanged. A timeout only stops local waiting: continue reading the same run, without another admission.
-
-Daily research is optional and uses a separate schedule revision. After the initial run finishes:
-
-```sh
-opencrawl research agent schedule AGENT_ID --json
-opencrawl research agent schedule AGENT_ID daily --revision SCHEDULE_REVISION --json
-opencrawl research agent schedule AGENT_ID pause --revision SCHEDULE_REVISION --json
-opencrawl research agent schedule AGENT_ID resume --revision SCHEDULE_REVISION --json
-opencrawl research agent repeats AGENT_ID RUN_ID --json
-```
-
-Enable/resume sets the first due time 24 hours later; it does not start a run immediately. Schedule pause stops future admissions and pending undispatched daily work, preserving manual and already dispatched daily runs. Ordinary agent pause also cancels unfinished work. Read current schedule state after an uncertain write; never replace its revision automatically. Inspect blocked and deferred reasons. Repeat references preserve every saved finding and distinguish `not_assessed` from `first_seen`; neither means a verified new business event. Updated clients are required for scheduled-agent reads and mixed agent lists because truthful daily cadence is `1440`, rejected by older strict Agent validators. Direct saved runs and historical manual receipts keep their original DTOs.
-
-Use admission history for recurring consumption rather than only the latest run:
-
-```sh
-opencrawl research agent runs AGENT_ID --limit 25 --json
-opencrawl research agent runs AGENT_ID --limit 25 --cursor RETURNED_CURSOR --json
-opencrawl research agent-run AGENT_ID PENDING_RUN_ID --json
-```
-
-Each history command reads exactly one page `{items,next_cursor,has_more}`. The nonnull opaque cursor persists even on an empty tail page; poll it again for new admissions. Default 25/max 100 items and a 1 MiB envelope cap may return fewer. Save run rows, pending IDs and cursor atomically with one serialized consumer per agent, then read pending IDs until terminal. This is an admission feed, not a completion-update feed; never rerun work to recover a read failure. Whole-page denial on inaccessible historical evidence leaves your checkpoint unchanged and needs operator attention. Preserve all terminal statuses, including partial with no findings and unknown failures. The SDK [durable polling example](../sdk/README.md#poll-saved-daily-findings-from-your-server) describes the persistence contract.
-
-
-## Choose a workspace account
-
-Each successful login selects a default account and retains previously saved accounts. List their non-secret identifiers with `profiles`, then use `--account` to select one explicitly:
-
-```sh
-opencrawl profiles
-opencrawl search "What did we decide?" --account PROFILE_ID
-opencrawl install claude-code --account PROFILE_ID
-opencrawl logout --account PROFILE_ID
-```
-
-Replace `PROFILE_ID` with an identifier returned by `profiles`. Replacing a key for the same server, user, and workspace keeps that identifier, so installed clients can continue using the account. A different workspace has its own account even on the same server.
-
-`configure --server`, `--server`, and `OPENCRAWL_URL` can change an unpinned command's destination, but never carry a saved key to a different origin. Sign in explicitly to authorize that destination. `--account` pins both server and account: it ignores ambient `OPENCRAWL_URL` and `OPENCRAWL_API_KEY`, and rejects a conflicting `--server`.
-
-`OPENCRAWL_URL`, `OPENCRAWL_API_KEY`, and `OPENCRAWL_CONFIG_DIR` support automation and isolated configurations. An explicitly supplied environment key authorizes the selected destination for an unpinned command; set its intended server alongside it. An empty environment key fails instead of falling back to a saved key. Environment keys are not saved unless `login` succeeds. `--config-dir DIRECTORY` selects a configuration directory explicitly.
-
-Older configurations stored an unbound key. They now require a fresh `login` with an explicitly supplied key; the CLI does not guess which server originally issued it. A failed login preserves the old file. After successful login, rerun each project's installer so existing MCP and hook entries gain account pins. An old unpinned MCP entry using a saved key stops with a reinstall message; an old prompt hook quietly skips retrieval until reinstalled.
-
-## Connect and index an app
-
-```sh
-opencrawl connect --query GitHub
-opencrawl connect start github --account PROFILE_ID
-# Open the returned URL yourself and finish provider authorization.
-opencrawl connect check SOURCE_ID --account PROFILE_ID
-opencrawl connect configure SOURCE_ID --repository OWNER/REPOSITORY \
-  --max-pages 1 --include-comments false --sync-every 0 --account PROFILE_ID
-opencrawl connect sync SOURCE_ID --account PROFILE_ID
-opencrawl jobs wait JOB_ID --account PROFILE_ID
-opencrawl connect setup SOURCE_ID --account PROFILE_ID
-```
-
-Use IDs returned by your own commands. `connect TOOLKIT` remains a start alias. Authorization, required source scope, background processing, listing coverage and searchable document count are separate states. `check` can queue indexing immediately when the scope is already configured. `configure` changes only explicit settings and can cancel an active sync; it never changes provider account identity. `sync` is explicit, can resume a paused source, and returns an already-active sync instead of submitting another. `--sync-every 0` is manual; a positive interval enables scheduled syncs. Unsupported indexing adapters remain identified in setup even when provider authorization works.
-
-Private nonce receipts are saved atomically with owner-only permissions under `connections/` in the selected config directory, bound to verified server/user/workspace identity. A logged-out browser callback can be completed with `connect check`; nonce values are never printed or passed in arguments. Do not share authorization URLs. `status` and `setup` only read server state. Unknown writes are never automatically repeated. If the initial link response was lost, inspect `connect status`, select the intended source, and explicitly `connect reconnect SOURCE_ID`; its missing nonce cannot be recovered or guessed. If no source ID was returned and you cannot select an intended source, status followed by `start TOOLKIT --acknowledge-unknown` permits one new explicit authorization that may replace or create a pending attempt. After an unknown sync, inspect jobs before using `sync --acknowledge-unknown` to request another potentially repeated run. See [connection recovery and bounds](../docs/cli.md#upload-files-and-connect-sources).
-
-## Import conversations
-
-```sh
-opencrawl import conversations.json --provider chatgpt
-opencrawl import chatgpt-export.zip --provider chatgpt
-opencrawl import claude-export.json --provider claude
-opencrawl import session.jsonl --provider claude-code
-opencrawl import rollout.jsonl --provider codex
-opencrawl import notes.md
-opencrawl watch ./exports --provider auto
-```
-
-The watcher uploads only JSON/JSONL files below the directory you explicitly choose, skips hidden paths and symbolic links, and records file signatures after upload acceptance to avoid repeats. Use `--once` for a single pass. Queued imports return a durable upload receipt; use `--wait` or inspect the job for parsing completion and errors. Small plain-text imports normally index immediately. If parsing fails, retry that job or explicitly import the file again: an unchanged file is not automatically reuploaded. Watcher state is bound to the verified account, folder, provider, source and visibility; key rotation preserves that identity. Changing source or visibility is a distinct explicit watched scope and can import the same files there. Older unscoped state is preserved and requires review before starting a new scoped import. Pending request keys are saved before submission; uncertain acceptance is looked up after restart and never triggers automatic retransmission. Explicitly retry a prepared/retryable receipt with `import --request-key` and its original file/scope. Expired or unresolved watcher entries remain blocked until reviewed. The watcher holds a per-folder lock and stays in the foreground until Ctrl+C; it does not install a background service. Exported ChatGPT branches and timestamps are preserved. Attachments are labeled in conversation text; their binary content needs a separate upload.
-
-JSON, JSONL, and ZIP chat exports upload their original bytes as multipart files. ZIP exports are unpacked by the server's bounded archive parser and must contain `conversations.json` or `chat.json`. JSON, JSONL, and plain-text imports are limited to 25,000,000 bytes; ZIP uploads are limited to 200 MiB and have additional expanded-size safety limits on the server. Plain text that would exceed the 8 MiB JSON request limit also uses the background file-upload pipeline. Use `--source SOURCE_ID` or `--visibility workspace` when needed; imports default to private.
-
-## Wait for indexing and recover acceptance
+Once published:
 
 ```sh
-opencrawl ingest product-spec.pdf --wait --timeout 300 --json
-opencrawl import conversations.json --provider chatgpt --wait
-opencrawl jobs list --status active
-opencrawl jobs show JOB_ID --json
-opencrawl jobs wait JOB_ID --timeout 300 --json
-opencrawl uploads lookup REQUEST_KEY --json
-opencrawl uploads show RECEIPT_ID --json
+npm install -g @getopencrawl/cli@next
+opencrawl login
+opencrawl status --json
 ```
 
-Copy IDs and request keys from your own receipts. Queued uploads hash and prepare a checked descriptor, then send file bytes at most once per invocation. The request key is printed before sending; after a dropped response, only a lookup is attempted. Keep an unconfirmed request key instead of starting a duplicate upload. Explicitly retry a prepared/retryable receipt with the same file, source, visibility and provider plus `--request-key REQUEST_KEY`. Accepted keys return their existing job. Receiving requests are only checked, and expired requests require a new explicit upload intent. Small text normally stays synchronous; `--request-key` selects the queued receipt pipeline.
-
-`--wait` and `jobs wait` stop after 1–3,600 seconds (default 300), measured after the upload attempt. The separate transfer deadline is five minutes. Ctrl+C stops local waiting, not server work. Only completed jobs exit successfully from wait; failure, cancellation, discard, timeout or access loss keep the known IDs in the unsuccessful receipt. No automatic cancellation, parsing retry, or reupload occurs. `--json` leaves stdout as one JSON result, with progress on stderr. Source listing coverage, file parsing and provider cleanup remain separate from an individual job's completion. Human search/job/conversation displays escape terminal controls; JSON and context export bytes are unchanged.
-
-## Read imported conversations
+For SSH, remote machines and coding agents:
 
 ```sh
-opencrawl sessions --help
-opencrawl sessions list --limit 20
-opencrawl sessions read DOCUMENT_ID --branch selected --limit 30
-opencrawl sessions read DOCUMENT_ID --branch selected --cursor OPAQUE_CURSOR
-opencrawl sessions export DOCUMENT_ID --revision REVISION --branch selected \
-  --selection t3:120:400 --budget 8000 > conversation-context.txt
+opencrawl login --no-browser --json
 ```
 
-Human-readable pages show terminal controls as visible escapes; JSON and exported context retain source text. Use the document ID from the list, then copy the revision and exact fragment selectors returned by `read`. A selector identifies a visible character range; do not guess it from a turn number or select unseen text. Every export requires an explicit revision, branch, one or more ranges (up to 20), and byte budget. Alternate branches require `--branch all`; missing branch information remains labeled unknown. Old records with no validated structured turns can be read as plain text but cannot be exported as guessed turn ranges.
-
-Export stdout contains only the server's cited reference block, without an added newline, bounded to the requested 256–32,000 UTF-8 bytes including headers and wrapper. Omitted/truncated selections are reported on stderr. `--json` preserves the complete selection receipt; that JSON envelope is outside the context byte budget. Permissions and revisions are checked again at export. A conflict requires a fresh read, not an automatic retry using stale ranges. `--revision` is the conversation version; global `--version` remains the package version. All commands honor `--account`, `--server` and `--config-dir` account binding.
-
-This exports imported evidence for a task; it does not continue a hosted conversation or restore hidden model state. See [the CLI guide](../docs/cli.md#read-and-export-selected-conversation-evidence) for pagination and exact-range semantics.
-
-## Ingest documents and recordings
-
-```sh
-opencrawl ingest product-spec.pdf
-opencrawl ingest customer-call.m4a
-opencrawl ingest demo.mp4 --source SOURCE_ID
-opencrawl ingest forecast.xlsx --visibility workspace
-```
-
-Uploads stream binary files up to 200 MiB into the background parsing queue. Visibility defaults to private. The server selects document parsing, OCR, or transcription and records progress in **Activity**; available formats and advanced parsing depend on server configuration. `import` also recognizes common document and media extensions and routes them through this upload pipeline. Use `ingest` for documents and media, and `import --provider` for conversation exports.
-
-## Optional concise answers
-
-When a matching server supplies a valid optional concise answer, the human
-research-run display prints its question-ordered sections before the exact
-findings. Every original qualification, citation, quote and gap remains in the
-output. Missing or invalid concise metadata falls back to the existing display;
-it never triggers another request. `--json` retains the original run exactly.
-Concise prose has a separate fallible review and is not a completeness claim.
-Source-scoped unknown and missing delivered answers remain distinct.
+Open the printed URL on any machine, confirm the displayed code and sign in.
+The CLI waits for approval. It needs no client secret, inbound port, localhost
+callback or browser on the machine running the command. No authentication runs
+during package installation. Installation does not configure agents or telemetry.
 
-## Feedback on research findings
-
-With a feedback-capable server and CLI, read the feedback on an exact saved finding before changing it:
-
-```sh
-opencrawl research agent feedback AGENT_ID RUN_ID FINDING_ID --json
-opencrawl research agent feedback AGENT_ID RUN_ID FINDING_ID set --revision FEEDBACK_REVISION --rating useful --note 'Prioritize pricing changes for this customer.' --json
-opencrawl research agent feedback AGENT_ID RUN_ID FINDING_ID delete --revision LATEST_FEEDBACK_REVISION --json
-```
-
-Ratings are `useful`, `irrelevant`, `incorrect` or `outdated`. Writes send once; after a conflict or uncertain response, read the same finding's feedback and review its value/revision before another write. Feedback guides later research selection without editing the saved finding, verifying facts or starting another run. See [current release status](../STATUS.md) and the [feedback contract](../docs/research-agents.md#finding-feedback-and-bounded-memory).
-
-## Bring context into agents
-
-Run from the project where you want the integration:
-
-```sh
-opencrawl install claude-code
-opencrawl install cursor
-opencrawl install codex
-```
-
-For new private conversation capture in this project, use `opencrawl install claude-code --capture` or `opencrawl install codex --capture`. This records future user prompts and final assistant messages from client hooks; it never scans old history, transcripts, tools or other projects. Codex requires project trust and explicit hook review with `/hooks`. Restart the client, then inspect delivery:
-
-```sh
-opencrawl capture status --provider codex --json
-opencrawl capture flush --provider codex --timeout 10 --json
-opencrawl capture pause --provider codex
-opencrawl capture resume --provider codex
-```
+`whoami` verifies your identity with the service. `status` reports the version,
+sign-in state and unavailable index. `logout` removes this deployment's local
+credentials; it does not revoke other devices or the server session. Use your
+identity provider's session controls to revoke access remotely.
 
-Use `--provider claude-code` for Claude Code. The first acknowledged message is immediately indexed for keyword search; semantic readiness is reported separately. Queued immutable events survive a lost response in an owner-only local spool and replay idempotently. Current-read failures are distinct from historical acknowledgements. Deleted sessions or conflicting event identities are blocked rather than silently recreated. Reinstalling preserves existing capture hooks; pause explicitly to stop new collection. See [capture limits, recovery and sandbox setup](../docs/cli.md#capture-new-project-conversations).
+`--api https://your-deployment.example` or `OPENCRAWL_API_URL` selects a deployment.
+Credentials stay separate for each origin. The deployment publishes the public
+OAuth configuration at `/.well-known/opencrawl-cli`; the CLI has no provider SDK,
+cloud credential, server secret or dependency on the hosted shell. Only HTTPS
+is accepted, except HTTP loopback for local development.
 
-Installers normally use a saved, verified account; the explicit environment-key option below supports sandbox runtimes. They preserve existing settings, back up files before changes, and are idempotent. Each generated entry pins the account identifier, server, and absolute configuration directory, so later logins, default changes, and shell environment changes do not retarget that project. Saved credentials remain in the OpenCrawl configuration; they are not copied into project files. Rerun the installer with `--account` to choose a different account. If its saved key is removed, the integration fails without falling back to another account. Restart the client. Codex project settings require a trusted project.
+Credentials live in `$XDG_CONFIG_HOME/opencrawl` (default `~/.config/opencrawl`),
+with directory mode 0700 and file mode 0600 on POSIX. `OPENCRAWL_CONFIG_HOME`
+overrides the configuration base directory. Refresh-token rotation uses a
+per-deployment process lock and atomic writes. Tokens never appear in output.
+Windows uses the user's profile directory and its inherited filesystem ACLs.
 
-Without `--capture`, Claude Code receives a `UserPromptSubmit` hook that retrieves relevant knowledge for each prompt and a local MCP connection. That hook sends up to the first 1,000 prompt characters for retrieval, waits up to 3.5 seconds, and fails open if retrieval fails. It does not upload the conversation. With `--capture`, Claude Code and Codex retrieve previous context before capturing the current prompt; Stop captures the final response and does not inject context. Explicit historical imports and `watch` remain available separately.
+`--json` returns `{schemaVersion:1,data:...}` or
+`{schemaVersion:1,error:{code,message}}`. Progress and approval instructions stay
+on stderr. Exit codes: 0 success, 1 service/network failure, 2 sign-in required
+or forbidden, 64 invalid arguments, 130 interrupted.
 
-`context` reports temporary keyword fallback or partial semantic coverage on stderr, keeping stdout for cited evidence within `--budget` UTF-8 bytes. `--json` preserves the complete response and coverage metadata. The Claude Code hook includes coverage warnings in its reference block and reserves space for them: its complete injected text, including the wrapper, is limited to 1,500 UTF-8 bytes.
+## Updates and distribution
 
-Cursor and Codex receive MCP tools through the API-key bridge. Their agents can call `opencrawl_search`, `opencrawl_context`, `opencrawl_list_conversations`, `opencrawl_read_conversation`, and `opencrawl_conversation_context`. API-key connections also expose `opencrawl_remember`; read-only OAuth connections cannot save notes, capture sessions or administer the workspace. Prompt retrieval is installed for Claude Code and also for Codex with `--capture`. Hosted ChatGPT and Claude history require user exports; installing a CLI does not grant access to those histories. Use the [hosted chat onboarding guide](../docs/onboarding.md) for OAuth setup; ChatGPT does not use the manual API-key header flow. Claude should select automatic client registration because OpenCrawl does not implement client metadata documents.
+TypeScript compiled with Bun 1.3.14 produces a self-contained binary. Users need
+Node 22+ and npm for the bootstrap, never Bun. npm platform packages cover macOS
+arm64/x64, Linux glibc arm64/x64 and Windows x64. Alpine/musl and Windows arm64
+are not supported in v0. Platform packages are optional dependencies at the exact
+launcher version; there are no install scripts.
 
-In disposable sandboxes, install inside the runtime and persist a private OpenCrawl configuration volume for queued events. An explicit `--credential-env OPENCRAWL_API_KEY --server ORIGIN` installer option verifies and pins the environment key's identity without copying the key to project files. Keep that environment variable available to the client and review hooks normally; no trust bypass is installed. Capture never needs native transcript persistence or access to the user's home directory.
+After a command, the launcher starts a detached updater at most once per day.
+It checks the npm `next` release channel, installs into a private user-owned cache with
+npm integrity verification and install scripts disabled, then atomically selects
+the complete release. The next invocation uses it. No sudo or global-directory
+writes are needed. Offline/update failures preserve the working copy and retry
+after 15 minutes. Concurrent updater processes share a lock; updates never
+downgrade. An unused installation updates when it is next invoked, not on a timer.
 
-Conversation tools let an agent discover imported chats, inspect an explicit `selected` or `all` branch, then retrieve exact character ranges using the returned version. Reads return at most two fragments per call; opaque cursors continue the same version and branch. Context requires 1–20 ranges and an explicit 256–32,000 UTF-8 byte budget including citation headers and wrapper. Missing permissions or a changed version require a fresh read, never a silent fallback to another branch. No tool selects unseen turns or resumes a hosted chat. The bridge forwards the server tools unchanged; refresh the client's tool list after a server update. Only a nonempty selected context response counts as successful conversation evidence retrieval for activation.
+`OPENCRAWL_NO_UPDATE=1` disables automatic updates for reproducible agent runs.
+`opencrawl update [--json]` explicitly updates from that same channel. Cached releases live under
+`$XDG_CACHE_HOME/opencrawl` (default `~/.cache/opencrawl`); previous releases remain
+available for recovery. Install a newer npm version directly to override an older
+cached version. Set the opt-out and remove `current.json` to return to the npm
+installed version. Standalone binary installation/self-update is not offered in
+this release; distribute the npm command.
 
-## Other commands
+## Development and release
 
 ```sh
-opencrawl connect
-opencrawl connect gmail
-opencrawl connect --query Salesforce
-opencrawl status
-opencrawl doctor
-opencrawl logout
+bun install --cwd cli --frozen-lockfile --ignore-scripts
+bun run --cwd cli typecheck
+bun test cli
+cd cli
+bun run src/cli.ts --help
+bun scripts/build.ts --current   # local smoke
+bun scripts/build.ts             # all five platform packages
 ```
 
-`connect` without a toolkit discovers the live Composio catalogue. Choose a returned slug to get its hosted authorization link. An authorized source still needs the relevant indexing adapter; the catalogue reports this separately.
-
-`logout` removes the default account's saved local key, or the account selected with `--account`. Other saved accounts remain available. It does not revoke the key on the server, unset environment credentials, stop watchers, or remove browser profiles/client settings. Revoke a key in **Connect an agent → API keys and developer tools**. `doctor` checks server health and API-key access, not provider ingestion or client installation; it exits unsuccessfully when the selected destination has no authorized key.
-
-`opencrawl mcp --account PROFILE_ID` is a newline-framed JSON-RPC stdio bridge to the authenticated `/mcp` endpoint. Standard output contains protocol messages only. Manual automation with an explicit environment key can also run `opencrawl mcp`; saved credentials require an explicit account pin.
-
-The CLI package uses an immutable `/downloads/sha256/<digest>/opencrawl-cli.tgz` URL supplied by the deployment. The old rolling address remains a compatibility alias. See [package installation and upgrades](../docs/client-packages.md); no public npm registry release is assumed.
-
-## Capture authorized browser pages
-
-```sh
-opencrawl browser setup
-opencrawl browser login https://your-company.my.salesforce.com --profile salesforce
-opencrawl browser capture https://your-company.my.salesforce.com/record-page --profile salesforce --interactive
-opencrawl browser crawl https://crm.example/accounts/ --profile crm --path-prefix /accounts/ --max-pages 20 --headless
-```
-
-Browser support uses the optional Playwright package. `setup` installs its Chromium runtime. Login happens in a separate OpenCrawl browser profile; it does not attach to your everyday browser. Each profile is bound to one origin. You enter credentials yourself, and local session state stays inside your OpenCrawl configuration directory.
-
-Capture imports visible page text with the original URL and private visibility. `--urls pages.txt` accepts an explicit list of URLs on the same origin. Crawl follows links only inside the chosen origin and path prefix, with a bounded page count; it does not click controls. During capture, network writes are blocked except explicitly read-only GraphQL queries. For applications whose rendering uses other POST reads, use `--interactive`: render the page manually, then press Enter to capture its current content. Some applications require a dedicated adapter for complete automated extraction.
-
-`--screenshots ./captures` saves private screenshots with form fields masked. Add `--ocr` to queue those screenshots through the server's upload/OCR pipeline. OCR completion appears in background jobs and requires the server's OCR dependencies. Screenshots and the browser profile may contain sensitive business data; keep them outside version control. `--headless` is available for previously authenticated unattended reads.
-
-For exact project files, credential/workspace selection, import recovery, and manual removal instructions, see [the CLI guide](../docs/cli.md).
-
-### Saved questions, current evidence
-
-Create a private task in OpenCrawl, then reuse its saved question and source scope:
-
-```sh
-opencrawl tasks list
-opencrawl tasks show task_0123456789abcdef0123456789abcdef
-opencrawl tasks run task_0123456789abcdef0123456789abcdef --revision 2 > context.txt
-```
-
-`--revision` pins a reviewed positive integer revision; omit it to run the current
-descriptor. Selected-source access loss blocks the run instead of widening its
-scope. Runs retrieve current indexed evidence without syncing sources or caching
-answers. The saved byte budget is at most 16,384 UTF-8 bytes. Plain stdout is exact
-context; notices go to stderr. `--json` returns the complete receipt or an error
-object. An empty successful context means no current evidence was returned.
-Read-only MCP clients can use `opencrawl_list_saved_tasks` and
-`opencrawl_run_saved_task` with the same account and permissions.
-
-## Monitor selected public pages
-
-Create a monitor in OpenCrawl, then inspect its exact page observations:
-
-```sh
-opencrawl monitors list
-opencrawl monitors show SOURCE_ID
-opencrawl monitors check SOURCE_ID --generation N
-opencrawl monitors changes --source SOURCE_ID
-opencrawl monitors context CHANGE_ID --revision VERSION_HASH --budget 4096 > change-context.txt
-```
-
-Use IDs, the generation and immutable version from your own receipts. A check only queues work; follow its job and inspect per-page outcomes. Static public pages are checked without sign-in, JavaScript or link discovery. Failed checks and missing pages never prove removals or business events. Latest attempt and last good observation are shown separately.
-
-Context exports one exact retained before/after comparison within 256–16,384 UTF-8 bytes. Stdout is unchanged evidence; omissions, truncation and warnings go to stderr. `--json` preserves the complete receipt. A version/access conflict stops without a newer substitution or retry. Changes support bounded `--limit` and opaque `--cursor` pagination. `opencrawl monitors --help` lists exact options; global account pins remain enforced.
-
-The server also exposes read-only MCP tools `opencrawl_list_monitored_changes` and `opencrawl_monitor_change_context`. Context takes 1–3 explicit change ID/version selections and a byte budget. These retrieve observed historical evidence; saved-task searches use current indexed observations instead. Neither operation infers when a remote author changed a page or whether a job opened or closed.
-
-## Private product-screen monitoring (source feature)
-
-With a matching server/client release, `opencrawl browser monitor --help` provides local login, masked preview, one-shot check, foreground watch, exact-receipt recovery and retained comparison/image reads. Cookies stay in an account-bound local profile. A watch runs only while the local process and computer remain available; it is not a hosted browser schedule. Unknown capture uploads stop and recover by GET without automatic resend. See [the complete CLI workflow and limits](../docs/product-monitoring-cli.md). Local synthetic tests do not establish compatibility with a real third-party account or current deployment availability.
-
-### Local Letter denial diagnostics
+The `next` dist-tag keeps this new client separate from the existing v1 `latest`
+release. Publishing must never move that v1 tag. The dedicated CLI workflow builds and signs on macOS, tests those same
+binaries on macOS/Linux/Windows, then publishes the five platform packages first, then publishes `@getopencrawl/cli` last. A failed platform
+publication cannot advertise an incomplete stable launcher. Retry skips already
+published immutable versions. Increment `cli/package.json` for every release.
+The main core package retains its separate private GitHub Packages release.
 
-The private local diagnostic report may include `blocked_requests` version1: up to eight fatal-denial samples with finite method, rejection reason, origin class and recognized operation labels, plus a SHA-256 fingerprint of the parsed origin/path. Queries, fragments, URL credentials, bodies, headers, cookies, arbitrary origins/paths and account markers are not retained. The first eight distinct samples are kept; repeated and omitted observations saturate at64. No operation is newly permitted or treated as harmless because it appears in diagnostics. Existing allow/deny decisions, required reads and deadlines remain authoritative.
+Create the free npm `getopencrawl` organization and grant the publisher access to
+its packages before the first release. Bootstrap publication uses an npm token
+or interactive npm authentication; configure npm trusted publishing for each
+package afterwards. A public npm package does not require a public GitHub repo,
+but its tarballs and binaries are public. Review the package allowlist with
+`npm pack --dry-run` before publishing. The current construction license remains
+UNLICENSED pending the repository's separate public-license decision.
 
-Fingerprints are query-free and pseudonymous, not anonymous; guessed low-entropy paths can be compared offline. URL parser normalization is a diagnostic grouping, never an authorization rule. A hash can match a separately inspected route, but does not identify an unknown operation by itself. Diagnostic reports remain local0600 files under the existing operation-history limits, at most16KiB each; nothing is added to capture uploads or command error text. No generic POST body operation is inferred or logged.
+Prior art: OnePatch's npm/native launcher, device login, automatic refresh and
+background upgrades; Glen's small npm entry point for agents. No query command,
+agent hooks, MCP integration or background daemon is included in this release.

package/bin/opencrawl.cjs

@@ -0,0 +1,168 @@
+#!/usr/bin/env node
+'use strict';
+// npm owns the bootstrap. Updates live in a private per-user cache so global
+// install permissions and running processes never prevent the next upgrade.
+const fs = require('node:fs');
+const path = require('node:path');
+const os = require('node:os');
+const {createRequire} = require('node:module');
+const {spawn, spawnSync} = require('node:child_process');
+const {randomUUID} = require('node:crypto');
+const {parseArgs} = require('node:util');
+const ROOT = path.resolve(__dirname, '..');
+const PACKAGE = '@getopencrawl/cli';
+const REGISTRY = 'https://registry.npmjs.org';
+const CHANNEL = 'next';
+const DAY = 24 * 60 * 60 * 1000;
+const cacheRoot = () => path.join(process.env.XDG_CACHE_HOME || path.join(os.homedir(), '.cache'), 'opencrawl');
+const stable = value => typeof value === 'string' && /^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$/.test(value);
+function compare(a, b) {
+  if (!stable(a) || !stable(b)) throw new Error('Invalid release version.');
+  const left = a.split('.').map(BigInt), right = b.split('.').map(BigInt);
+  for (let i = 0; i < 3; i++) if (left[i] !== right[i]) return left[i] > right[i] ? 1 : -1;
+  return 0;
+}
+function read(file) { try { return JSON.parse(fs.readFileSync(file, 'utf8')); } catch { return null; } }
+function write(file, value) {
+  fs.mkdirSync(path.dirname(file), {recursive: true, mode: 0o700});
+  const tmp = `${file}.${randomUUID()}.tmp`;
+  try { fs.writeFileSync(tmp, JSON.stringify(value) + '\n', {mode: 0o600, flag: 'wx'}); fs.renameSync(tmp, file); }
+  finally { try { fs.unlinkSync(tmp); } catch {} }
+}
+function target(platform = process.platform, arch = process.arch) {
+  const key = `${platform}-${arch}`;
+  if (!['darwin-arm64', 'darwin-x64', 'linux-arm64', 'linux-x64', 'win32-x64'].includes(key)) throw new Error(`No OpenCrawl binary for ${key}.`);
+  return key;
+}
+function binary(root) {
+  const key = target();
+  const localRequire = createRequire(path.join(root, 'package.json'));
+  let manifest;
+  try { manifest = localRequire.resolve(`${PACKAGE}-${key}/package.json`); }
+  catch { throw new Error('Missing platform package. Reinstall with npm install -g @getopencrawl/cli@next --include=optional.'); }
+  const pkg = read(manifest);
+  if (pkg?.name !== `${PACKAGE}-${key}` || pkg.version !== read(path.join(root, 'package.json'))?.version) throw new Error('The platform package version does not match the CLI.');
+  const file = path.join(path.dirname(manifest), 'bin', process.platform === 'win32' ? 'opencrawl.exe' : 'opencrawl');
+  if (!fs.existsSync(file)) throw new Error('The OpenCrawl platform binary is missing. Reinstall @getopencrawl/cli.');
+  return file;
+}
+function activeRoot(root = ROOT) {
+  const pkg = read(path.join(root, 'package.json'));
+  const current = read(path.join(cacheRoot(), 'current.json'));
+  if (stable(current?.version) && stable(pkg?.version) && compare(current.version, pkg.version) > 0) {
+    const cached = path.join(cacheRoot(), 'releases', current.version, 'node_modules', '@getopencrawl', 'cli');
+    if (read(path.join(cached, 'package.json'))?.version === current.version) {
+      try { binary(cached); return cached; } catch {} // a damaged cache must not break the installed copy
+    }
+  }
+  return root;
+}
+function npmCli(nodeExecutable = process.execPath) {
+  const candidates = [process.env.npm_execpath, path.join(path.dirname(nodeExecutable), 'node_modules/npm/bin/npm-cli.js'), path.resolve(path.dirname(nodeExecutable), '../lib/node_modules/npm/bin/npm-cli.js')];
+  for (const directory of (process.env.PATH || '').split(path.delimiter)) candidates.push(path.join(directory, 'npm'));
+  for (const file of candidates) {
+    if (!file) continue;
+    try { const resolved = fs.realpathSync(file); if (path.basename(resolved) === 'npm-cli.js') return resolved; } catch {}
+  }
+  throw new Error('npm is required for updates. Install Node.js with npm and retry.');
+}
+function acquireUpdateLock() {
+  const lock = path.join(cacheRoot(), 'update.lock');
+  fs.mkdirSync(cacheRoot(), {recursive: true, mode: 0o700});
+  try { const fd = fs.openSync(lock, 'wx', 0o600); fs.writeFileSync(fd, String(process.pid)); fs.closeSync(fd); }
+  catch (error) {
+    if (error.code !== 'EEXIST') throw error;
+    const pid = Number(fs.readFileSync(lock, 'utf8'));
+    if (pid > 0) {
+      try { process.kill(pid, 0); return null; }
+      catch (check) { if (check.code !== 'ESRCH') return null; }
+    } else if (Date.now() - fs.statSync(lock).mtimeMs < 5 * 60_000) return null;
+    try { fs.unlinkSync(lock); } catch {}
+    return acquireUpdateLock();
+  }
+  return () => fs.unlinkSync(lock);
+}
+async function update({force = false, root = ROOT, fetcher = fetch, install = installRelease} = {}) {
+  const unlock = acquireUpdateLock();
+  if (!unlock) return {updated: false, busy: true};
+  try {
+    const stateFile = path.join(cacheRoot(), 'update.json');
+    const state = read(stateFile);
+    const current = read(path.join(activeRoot(root), 'package.json')).version;
+    if (!force && state?.checkedAt > Date.now() - DAY) return {updated: false, version: current};
+    const response = await fetcher(`${REGISTRY}/@getopencrawl%2fcli/${CHANNEL}`, {redirect: 'error', signal: AbortSignal.timeout(10_000)});
+    if (!response.ok) throw new Error('The npm registry could not check for updates.');
+    const latest = await response.json();
+    if (latest.name !== PACKAGE || !stable(latest.version)) throw new Error('The registry returned invalid release metadata.');
+    if (compare(latest.version, current) <= 0) { write(stateFile, {checkedAt: Date.now()}); return {updated: false, version: current}; }
+    await install(latest.version);
+    write(path.join(cacheRoot(), 'current.json'), {version: latest.version});
+    write(stateFile, {checkedAt: Date.now()});
+    return {updated: true, version: latest.version};
+  } finally { unlock(); }
+}
+async function installRelease(version, {registry = REGISTRY} = {}) {
+  const origin = new URL(registry);
+  if (registry !== REGISTRY && !(origin.protocol === 'http:' && ['127.0.0.1', 'localhost'].includes(origin.hostname))) throw new Error('Invalid release registry.');
+  const releases = path.join(cacheRoot(), 'releases');
+  fs.mkdirSync(releases, {recursive: true, mode: 0o700});
+  const destination = path.join(releases, version);
+  // Reuse a fully staged version if pointer publication was interrupted.
+  if (fs.existsSync(destination)) { binary(path.join(destination, 'node_modules/@getopencrawl/cli')); return; }
+  const stage = fs.mkdtempSync(path.join(releases, '.stage-'));
+  try {
+    fs.writeFileSync(path.join(stage, 'package.json'), '{"private":true}\n');
+    fs.writeFileSync(path.join(stage, '.npmrc'), `@getopencrawl:registry=${registry}\n`);
+    const args = [npmCli(), 'install', '--prefix', stage, '--ignore-scripts', '--include=optional', '--no-audit', '--no-fund', '--no-package-lock', `--registry=${registry}`, `${PACKAGE}@${version}`];
+    const result = spawnSync(process.execPath, args, {cwd: stage, stdio: 'pipe', timeout: 180_000, windowsHide: true});
+    if (result.status !== 0) throw new Error('Could not install the update. The existing CLI is unchanged.');
+    const installed = path.join(stage, 'node_modules/@getopencrawl/cli');
+    const pkg = read(path.join(installed, 'package.json'));
+    if (pkg?.name !== PACKAGE || pkg.version !== version) throw new Error('The installed release does not match the requested version.');
+    binary(installed);
+    fs.renameSync(stage, destination);
+  } finally { fs.rmSync(stage, {recursive: true, force: true}); }
+}
+function scheduleUpdate() {
+  if (process.env.OPENCRAWL_NO_UPDATE === '1') return;
+  try {
+    if (read(path.join(cacheRoot(), 'update.json'))?.checkedAt > Date.now() - DAY) return;
+    const child = spawn(process.execPath, [__filename, '--internal-update'], {detached: true, stdio: 'ignore', windowsHide: true});
+    child.on('error', () => {});
+    child.unref();
+  } catch {} // updating can never make an ordinary command fail
+}
+async function main(args = process.argv.slice(2)) {
+  if (args[0] === '--internal-update') {
+    try { await update(); } catch { write(path.join(cacheRoot(), 'update.json'), {checkedAt: Date.now() - DAY + 15 * 60_000}); }
+    return 0;
+  }
+  const selected = activeRoot();
+  // Delegate the launcher too, so later releases can fix bootstrap behavior.
+  if (selected !== ROOT) {
+    const delegated = spawnSync(process.execPath, [path.join(selected, 'bin/opencrawl.cjs'), ...args], {stdio: 'inherit', windowsHide: true});
+    if (delegated.error) throw new Error('Could not start the cached launcher.');
+    return delegated.signal === 'SIGINT' ? 130 : delegated.status ?? 1;
+  }
+  let parsed;
+  try { parsed = parseArgs({args, allowPositionals: true, options: {json: {type: 'boolean'}, 'no-browser': {type: 'boolean'}, api: {type: 'string'}, version: {type: 'boolean', short: 'v'}, help: {type: 'boolean', short: 'h'}}}); }
+  catch {} // The native parser produces the public, versioned usage error.
+  if (parsed?.positionals.length === 1 && parsed.positionals[0] === 'update' && !parsed.values.help && !parsed.values.version) {
+    const data = await update({force: true});
+    process.stdout.write(args.includes('--json') ? JSON.stringify({schemaVersion: 1, data}) + '\n' : data.busy ? 'An update is already running.\n' : data.updated ? `Installed opencrawl ${data.version}. Your next invocation will use it.\n` : `opencrawl ${data.version} is current.\n`);
+    return 0;
+  }
+  const root = activeRoot();
+  const result = spawnSync(binary(root), args, {stdio: 'inherit', windowsHide: true});
+  scheduleUpdate();
+  if (result.error) throw new Error('Could not start the OpenCrawl binary. Reinstall @getopencrawl/cli.');
+  if (result.signal === 'SIGINT') return 130;
+  return result.status ?? 1;
+}
+module.exports = {channel: CHANNEL, target, stable, compare, activeRoot, update, installRelease, binary, npmCli, main};
+if (require.main === module) main().then(code => { process.exitCode = code; }).catch(() => {
+  const message = 'Could not run or update OpenCrawl. Retry, or reinstall with npm install -g @getopencrawl/cli@next --include=optional.';
+  if (process.argv.includes('--json')) process.stdout.write(JSON.stringify({schemaVersion: 1, error: {code: 'LAUNCHER_ERROR', message}}) + '\n');
+  else process.stderr.write(`opencrawl: ${message}\n`);
+  process.exitCode = 1;
+});