getdocs 0.1.0__tar.gz

getdocs-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 jonbakerfish
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

getdocs-0.1.0/PKG-INFO

@@ -0,0 +1,169 @@
+Metadata-Version: 2.4
+Name: getdocs
+Version: 0.1.0
+Summary: Documentation crawler: recursively crawl a docs site and emit clean markdown
+Author-email: jonbakerfish <jonbakerfish@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/jonbakerfish/getdocs
+Project-URL: Repository, https://github.com/jonbakerfish/getdocs
+Project-URL: Issues, https://github.com/jonbakerfish/getdocs/issues
+Project-URL: Documentation, https://github.com/jonbakerfish/getdocs/blob/main/docs/USAGE.md
+Keywords: documentation,crawler,scraper,markdown,docs,llm,agents,rag
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: scrapy>=2.11
+Requires-Dist: markdownify>=0.13
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: trafilatura>=1.12
+Requires-Dist: scrapy-playwright>=0.0.40
+Provides-Extra: server
+Requires-Dist: fastapi>=0.110; extra == "server"
+Requires-Dist: uvicorn>=0.27; extra == "server"
+Requires-Dist: httpx>=0.27; extra == "server"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: httpx>=0.27; extra == "dev"
+Requires-Dist: fastapi>=0.110; extra == "dev"
+Requires-Dist: uvicorn>=0.27; extra == "dev"
+Dynamic: license-file
+
+# getdocs
+
+**Turn any documentation site into a clean, local markdown copy your coding agent can actually read.**
+
+```bash
+getdocs crawl https://example.com/docs -o ./out
+```
+
+Coding agents are only as good as the docs they can see. Pointing an agent at a
+live docs URL means it burns tokens on nav bars, cookie banners, and HTML
+chrome — or can't reach the page at all. `getdocs` gives the agent a local,
+offline, markdown mirror instead: the actual content, structured to match the
+original site, ready to drop into a repo or feed to a model.
+
+## Why getdocs
+
+- **Richer context for coding agents.** A local copy is greppable, indexable,
+  and always available — the agent reads the whole library at once instead of
+  fetching one rendered page at a time. No rate limits, no network flakiness,
+  no JS that won't hydrate.
+- **Clean markdown → fewer tokens.** Each page is reduced to its content (the
+  nav, headers, footers, and ad chrome stripped) and written as plain markdown.
+  Agents consume it directly, and you spend tokens on docs, not `<div>` soup.
+- **Structure preserved.** Files mirror the URL hierarchy
+  (`example.com/docs/auth` → `out/docs/auth.md`), each with YAML frontmatter
+  (`url`, `title`, `crawled_at`, `status`), plus a `crawl.json` Manifest that
+  captures the site's nav order and reading order — so an agent can follow the
+  docs in the order the authors intended.
+- **Source-first: clone over crawl.** If the docs site is open-source, getdocs
+  detects the "Edit this page" link, clones the repo, and serves you the
+  original markdown source instead of scraping HTML — the highest-fidelity copy
+  there is. Falls back to crawling automatically when there's no repo.
+
+## When to reach for it
+
+- **Coding against an unfamiliar library or API.** Mirror its docs into your
+  repo (or a scratch dir) so your agent can ground its answers in the real
+  reference instead of hallucinating from memory.
+- **RAG / knowledge bases.** Get a clean markdown corpus to chunk and embed,
+  without writing a bespoke scraper-and-cleaner for every site.
+- **Offline or air-gapped work.** Take a docs set with you; read and search it
+  with no network.
+- **Pinning a version.** Snapshot today's docs so your agent isn't tripped up
+  when the upstream site changes underneath you.
+- **Migrating or archiving docs.** Pull an entire site down as markdown to move,
+  diff, or keep.
+
+## Output
+
+```
+out/
+├── crawl.json            ← the Manifest: nav order, reading order, what ran
+└── docs/
+    ├── index.md
+    ├── auth.md
+    └── guide/
+        └── intro.md
+```
+
+```markdown
+---
+url: https://example.com/docs/auth
+title: Authentication
+status: 200
+crawled_at: 2026-06-12T10:00:00Z
+---
+
+# Authentication
+...
+```
+
+Sitemap discovery, JavaScript rendering, source-repo cloning, polite
+throttling, JSONL output, and resumable crawls are all built in — see
+[docs/USAGE.md](docs/USAGE.md).
+
+## Install
+
+Requires **Python 3.12+**.
+
+```bash
+pip install getdocs
+```
+
+Or from source, for the latest unreleased changes:
+
+```bash
+git clone https://github.com/jonbakerfish/getdocs
+cd getdocs
+pip install -e .
+```
+
+That's enough to crawl. Two optional pieces unlock more:
+
+```bash
+# JavaScript rendering — the headless browser used to hydrate SPA docs
+playwright install chromium
+
+# Serve a crawled/cloned copy locally as a browsable site
+pip install mkdocs mkdocs-material
+```
+
+**`git`** must be on your `PATH` for source-first cloning (it almost always
+already is); without it, getdocs simply falls back to crawling. To run the
+optional API service, install the server extra: `pip install "getdocs[server]"`.
+
+## Development
+
+```bash
+git clone https://github.com/jonbakerfish/getdocs
+cd getdocs
+pip install -e ".[dev]"
+pytest
+```
+
+## Responsible use
+
+getdocs is a tool; how you point it is on you. By default it **honors
+`robots.txt`**, throttles itself politely, and **identifies itself honestly**
+in the `User-Agent` (`getdocs/<version> (+project-url)`) — please keep it that
+way. For high-volume crawls, add `--contact you@example.com` so site operators
+can reach you (it's appended to the User-Agent; optional but courteous).
+
+getdocs is intended for personal, reference, and agent/RAG use on documentation
+you have the right to access. **You are solely responsible for complying with
+each site's Terms of Service, its `robots.txt`, applicable law, and the
+copyright of the content you fetch** — getdocs is provided as-is, with no
+warranty (see [LICENSE](LICENSE)). Crawled documentation belongs to its authors:
+use it for your own reference, but don't redistribute someone else's docs as
+your own. Crawl only what you have the right to.

getdocs-0.1.0/README.md

@@ -0,0 +1,129 @@
+# getdocs
+
+**Turn any documentation site into a clean, local markdown copy your coding agent can actually read.**
+
+```bash
+getdocs crawl https://example.com/docs -o ./out
+```
+
+Coding agents are only as good as the docs they can see. Pointing an agent at a
+live docs URL means it burns tokens on nav bars, cookie banners, and HTML
+chrome — or can't reach the page at all. `getdocs` gives the agent a local,
+offline, markdown mirror instead: the actual content, structured to match the
+original site, ready to drop into a repo or feed to a model.
+
+## Why getdocs
+
+- **Richer context for coding agents.** A local copy is greppable, indexable,
+  and always available — the agent reads the whole library at once instead of
+  fetching one rendered page at a time. No rate limits, no network flakiness,
+  no JS that won't hydrate.
+- **Clean markdown → fewer tokens.** Each page is reduced to its content (the
+  nav, headers, footers, and ad chrome stripped) and written as plain markdown.
+  Agents consume it directly, and you spend tokens on docs, not `<div>` soup.
+- **Structure preserved.** Files mirror the URL hierarchy
+  (`example.com/docs/auth` → `out/docs/auth.md`), each with YAML frontmatter
+  (`url`, `title`, `crawled_at`, `status`), plus a `crawl.json` Manifest that
+  captures the site's nav order and reading order — so an agent can follow the
+  docs in the order the authors intended.
+- **Source-first: clone over crawl.** If the docs site is open-source, getdocs
+  detects the "Edit this page" link, clones the repo, and serves you the
+  original markdown source instead of scraping HTML — the highest-fidelity copy
+  there is. Falls back to crawling automatically when there's no repo.
+
+## When to reach for it
+
+- **Coding against an unfamiliar library or API.** Mirror its docs into your
+  repo (or a scratch dir) so your agent can ground its answers in the real
+  reference instead of hallucinating from memory.
+- **RAG / knowledge bases.** Get a clean markdown corpus to chunk and embed,
+  without writing a bespoke scraper-and-cleaner for every site.
+- **Offline or air-gapped work.** Take a docs set with you; read and search it
+  with no network.
+- **Pinning a version.** Snapshot today's docs so your agent isn't tripped up
+  when the upstream site changes underneath you.
+- **Migrating or archiving docs.** Pull an entire site down as markdown to move,
+  diff, or keep.
+
+## Output
+
+```
+out/
+├── crawl.json            ← the Manifest: nav order, reading order, what ran
+└── docs/
+    ├── index.md
+    ├── auth.md
+    └── guide/
+        └── intro.md
+```
+
+```markdown
+---
+url: https://example.com/docs/auth
+title: Authentication
+status: 200
+crawled_at: 2026-06-12T10:00:00Z
+---
+
+# Authentication
+...
+```
+
+Sitemap discovery, JavaScript rendering, source-repo cloning, polite
+throttling, JSONL output, and resumable crawls are all built in — see
+[docs/USAGE.md](docs/USAGE.md).
+
+## Install
+
+Requires **Python 3.12+**.
+
+```bash
+pip install getdocs
+```
+
+Or from source, for the latest unreleased changes:
+
+```bash
+git clone https://github.com/jonbakerfish/getdocs
+cd getdocs
+pip install -e .
+```
+
+That's enough to crawl. Two optional pieces unlock more:
+
+```bash
+# JavaScript rendering — the headless browser used to hydrate SPA docs
+playwright install chromium
+
+# Serve a crawled/cloned copy locally as a browsable site
+pip install mkdocs mkdocs-material
+```
+
+**`git`** must be on your `PATH` for source-first cloning (it almost always
+already is); without it, getdocs simply falls back to crawling. To run the
+optional API service, install the server extra: `pip install "getdocs[server]"`.
+
+## Development
+
+```bash
+git clone https://github.com/jonbakerfish/getdocs
+cd getdocs
+pip install -e ".[dev]"
+pytest
+```
+
+## Responsible use
+
+getdocs is a tool; how you point it is on you. By default it **honors
+`robots.txt`**, throttles itself politely, and **identifies itself honestly**
+in the `User-Agent` (`getdocs/<version> (+project-url)`) — please keep it that
+way. For high-volume crawls, add `--contact you@example.com` so site operators
+can reach you (it's appended to the User-Agent; optional but courteous).
+
+getdocs is intended for personal, reference, and agent/RAG use on documentation
+you have the right to access. **You are solely responsible for complying with
+each site's Terms of Service, its `robots.txt`, applicable law, and the
+copyright of the content you fetch** — getdocs is provided as-is, with no
+warranty (see [LICENSE](LICENSE)). Crawled documentation belongs to its authors:
+use it for your own reference, but don't redistribute someone else's docs as
+your own. Crawl only what you have the right to.

getdocs-0.1.0/pyproject.toml

@@ -0,0 +1,52 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "getdocs"
+version = "0.1.0"
+description = "Documentation crawler: recursively crawl a docs site and emit clean markdown"
+readme = "README.md"
+requires-python = ">=3.12"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "jonbakerfish", email = "jonbakerfish@gmail.com" }]
+keywords = ["documentation", "crawler", "scraper", "markdown", "docs", "llm", "agents", "rag"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
+    "Topic :: Software Development :: Documentation",
+    "Topic :: Text Processing :: Markup :: Markdown",
+]
+dependencies = [
+    "scrapy>=2.11",
+    "markdownify>=0.13",
+    "beautifulsoup4>=4.12",
+    "pyyaml>=6.0",
+    "trafilatura>=1.12",
+    "scrapy-playwright>=0.0.40",
+]
+
+[project.urls]
+Homepage = "https://github.com/jonbakerfish/getdocs"
+Repository = "https://github.com/jonbakerfish/getdocs"
+Issues = "https://github.com/jonbakerfish/getdocs/issues"
+Documentation = "https://github.com/jonbakerfish/getdocs/blob/main/docs/USAGE.md"
+
+[project.optional-dependencies]
+server = ["fastapi>=0.110", "uvicorn>=0.27", "httpx>=0.27"]
+dev = ["pytest>=8.0", "httpx>=0.27", "fastapi>=0.110", "uvicorn>=0.27"]
+
+[project.scripts]
+getdocs = "getdocs.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

getdocs-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

getdocs-0.1.0/src/getdocs/__main__.py

@@ -0,0 +1,3 @@
+from getdocs.cli import main
+
+raise SystemExit(main())

getdocs-0.1.0/src/getdocs/api.py

@@ -0,0 +1,95 @@
+"""API service: Firecrawl-style async Crawl jobs over the engine (ADR-0002)."""
+
+from fastapi import FastAPI, HTTPException, WebSocket
+from pydantic import BaseModel, model_validator
+
+from getdocs.jobs import CrawlJob, JobManager
+
+
+class CrawlRequest(BaseModel):
+    url: str | None = None
+    urls: list[str] | None = None
+    limit: int | None = None
+    depth: int | None = None
+    allow_backward: bool = False
+    allow_subdomains: bool = False
+    include_paths: list[str] | None = None
+    exclude_paths: list[str] | None = None
+    sitemap: str | None = None  # "both" | "off" | "only"
+    render: str | None = None  # "auto" | "always" | "never"
+    selector: str | None = None
+    ignore_robots: bool = False
+    keep_html: bool = False
+    delay: float | None = None
+    concurrency: int | None = None
+    webhook: str | None = None  # URL POSTed started/page/completed events
+
+    @model_validator(mode="after")
+    def _require_some_url(self):
+        if not self.url and not self.urls:
+            raise ValueError("either url or urls is required")
+        return self
+
+
+def _serialize(job: CrawlJob) -> dict:
+    return {
+        "id": job.id,
+        "status": job.status,
+        "seeds": job.seeds,
+        "page_count": len(job.pages),
+        "pages": job.pages,
+        "manifest": job.manifest,
+        "error": job.error,
+        "webhook_failures": job.webhook_failures,
+    }
+
+
+def create_app(manager: JobManager | None = None) -> FastAPI:
+    manager = manager or JobManager()
+    app = FastAPI(title="getdocs", version="0.1.0")
+    app.state.manager = manager
+
+    @app.post("/v1/crawl", status_code=202)
+    async def start_crawl(request: CrawlRequest):
+        job = manager.start(request.model_dump(exclude_none=True))
+        return {"id": job.id, "status": job.status}
+
+    @app.get("/v1/crawl")
+    async def list_crawls():
+        return {
+            "jobs": [
+                {
+                    "id": job.id,
+                    "status": job.status,
+                    "seeds": job.seeds,
+                    "page_count": len(job.pages),
+                }
+                for job in manager.jobs.values()
+            ]
+        }
+
+    @app.get("/v1/crawl/{job_id}")
+    async def get_crawl(job_id: str):
+        job = manager.get(job_id)
+        if job is None:
+            raise HTTPException(status_code=404, detail="no such Crawl job")
+        return _serialize(job)
+
+    @app.websocket("/v1/crawl/{job_id}/ws")
+    async def stream_crawl(websocket: WebSocket, job_id: str):
+        await websocket.accept()
+        if manager.get(job_id) is None:
+            await websocket.close(code=4404, reason="no such Crawl job")
+            return
+        async for event in manager.stream(job_id):
+            await websocket.send_json(event)
+        await websocket.close()
+
+    @app.delete("/v1/crawl/{job_id}")
+    async def cancel_crawl(job_id: str):
+        job = manager.cancel(job_id)
+        if job is None:
+            raise HTTPException(status_code=404, detail="no such Crawl job")
+        return {"id": job.id, "status": job.status}
+
+    return app