octocrawl 0.3.0 → 0.3.1

package/README.md

@@ -14,7 +14,7 @@ Every option of the REST API is a flag under its kebab-case name (`maxAge` is `-
 
 - Node.js 22.13 or later. The browser lane uses Playwright's Chromium: run `npx playwright install chromium` once; without it, pages are read over HTTP only.
 - `better-sqlite3` builds or downloads its native binding when installed. If your npm holds install scripts back, allow it (`npm install-scripts approve better-sqlite3`, or `allowScripts` in your package.json).
-- robots.txt is obeyed. By default pages are requested with the user agent and client hints of the Chrome the browser lane runs; `--mode research` names itself as a research crawler instead, with your contact from `W2L_CONTACT`. There is no stealth mode.
+- robots.txt is read for every page and recorded. A URL you name to `scrape` or `batch` is fetched whatever it says, and the result says so (`overrideBasis: "user_named_url"`, a `robots_overridden` warning); the links `crawl` and `map` discover obey it unless you pass `--ignore-robots-txt`. A rule robots.txt writes for `User-agent: Octocrawl` holds in every case. By default pages are requested with the user agent and client hints of the Chrome the browser lane runs; `--mode research` names itself as a research crawler instead, with your contact from `W2L_CONTACT`. There is no stealth option on a request: a provider's stealth or challenge solving runs only on a server started with an access grant that names it (`octocrawl serve --access-grant <file>`; see the main README).
 - Paid browser services (Browserbase, Steel) are used only when you name them in `W2L_VENDORS` (for example `W2L_VENDORS=browserbase`) and their key is set; a key alone does nothing.
 - `octocrawl serve` listens on 127.0.0.1. On any other address it needs a token (`--token` or `W2L_API_TOKEN`), since other machines could otherwise use it to reach your localhost and network.