@hasna/skills 0.1.60 → 0.1.62

package/skills/apidocs/CLAUDE.md

@@ -1,278 +0,0 @@
-# service-apidocs
-
-Agentic web crawler for API documentation indexing and semantic search.
-
-## Overview
-
-This service uses an intelligent web crawler to index API documentation from any website, stores vector embeddings in AWS S3 Vectors or locally, and provides semantic search capabilities for AI assistants. It also extracts structured API endpoint information for easy reference.
-
-## Architecture
-
-```
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                         PHASE 1: DOCS DISCOVERY                              │
-│  ┌─────────────┐    ┌──────────────────┐    ┌───────────────────────┐       │
-│  │  User:      │───▶│  Discovery Agent │───▶│  Check common paths   │       │
-│  │  stripe.com │    │  (find docs)     │    │  /docs, /api, etc.    │       │
-│  └─────────────┘    └──────────────────┘    └───────────────────────┘       │
-│                              │                                               │
-│                              ▼                                               │
-│                     ┌──────────────────┐                                     │
-│                     │  Found:          │                                     │
-│                     │  stripe.com/docs │                                     │
-│                     └──────────────────┘                                     │
-└─────────────────────────────────────────────────────────────────────────────┘
-                              │
-                              ▼
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                         PHASE 2: PAGE CRAWLING                               │
-│  ┌──────────────────┐    ┌─────────────────────┐    ┌───────────────────┐   │
-│  │  BFS Crawler     │───▶│  All Doc Pages      │───▶│  Save to Cache    │   │
-│  │  (from docs URL) │    │  (HTML + Markdown)  │    │  pages-*.json     │   │
-│  └──────────────────┘    └─────────────────────┘    └───────────────────┘   │
-└─────────────────────────────────────────────────────────────────────────────┘
-                              │
-                              ▼
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                         PHASE 3: ENDPOINT EXTRACTION                         │
-│  ┌──────────────────┐    ┌─────────────────────┐    ┌───────────────────┐   │
-│  │  Claude Agent    │───▶│  Structured         │───▶│  Save Endpoints   │   │
-│  │  (extract APIs)  │    │  Endpoint Data      │    │  endpoints.json   │   │
-│  └──────────────────┘    └─────────────────────┘    └───────────────────┘   │
-└─────────────────────────────────────────────────────────────────────────────┘
-                              │
-                              ▼
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                         PHASE 4: CHUNKING & EMBEDDING                        │
-│  ┌──────────────────┐    ┌─────────────────────┐    ┌───────────────────┐   │
-│  │  Parser/Chunker  │───▶│  OpenAI Embeddings  │───▶│  Vector Store     │   │
-│  │  (remark)        │    │  (text-embed-3-sm)  │    │  (Local/S3)       │   │
-│  └──────────────────┘    └─────────────────────┘    └───────────────────┘   │
-└─────────────────────────────────────────────────────────────────────────────┘
-```
-
-## Tech Stack
-
-- **Runtime**: Bun
-- **Language**: TypeScript
-- **CLI**: Commander.js
-- **AI Agent**: Claude Sonnet 4 (@anthropic-ai/sdk) - optional, for content cleanup
-- **HTML→Markdown**: turndown
-- **Markdown Parser**: remark + mdast
-- **Embeddings**: OpenAI text-embedding-3-small (1536 dimensions)
-- **Vector Store**: Local JSON files (default) or AWS S3 Vectors
-
-## Project Structure
-
-```
-service-apidocs/
-├── bin/
-│   └── cli.ts                 # CLI entry point
-├── src/
-│   ├── commands/              # CLI commands
-│   │   ├── add.ts             # Add documentation from website
-│   │   ├── query.ts           # Query documentation
-│   │   ├── list.ts            # List libraries
-│   │   ├── sync.ts            # Re-crawl documentation
-│   │   ├── remove.ts          # Remove library
-│   │   └── endpoints.ts       # List API endpoints
-│   ├── lib/                   # Core modules
-│   │   ├── crawler.ts         # Web crawler with discovery & extraction
-│   │   ├── discovery.ts       # Agentic docs URL discovery
-│   │   ├── endpoint-extractor.ts # Claude-powered endpoint extraction
-│   │   ├── html-to-md.ts      # HTML to Markdown conversion
-│   │   ├── parser.ts          # Markdown parser/chunker
-│   │   ├── embeddings.ts      # OpenAI embeddings
-│   │   ├── vectors.ts         # Vector store abstraction
-│   │   ├── vectors-local.ts   # Local file-based vectors
-│   │   ├── vectors-s3.ts      # AWS S3 Vectors client
-│   │   ├── search.ts          # Semantic search
-│   │   └── storage.ts         # Local storage
-│   ├── types/
-│   │   └── index.ts           # TypeScript types
-│   └── utils/
-│       ├── paths.ts           # Path/URL utilities
-│       └── output.ts          # CLI output helpers
-```
-
-## Environment Variables
-
-Required:
-- `ANTHROPIC_API_KEY` - For Claude agent (docs discovery, endpoint extraction)
-- `OPENAI_API_KEY` - For embeddings
-
-Optional:
-- `USE_S3_VECTORS` - Set to `true` to use AWS S3 Vectors (default: local JSON files)
-- `AWS_REGION` - AWS region (default: us-east-1)
-- `S3_VECTORS_BUCKET` - Vector bucket name (default: service-apidocs-vectors)
-
-## CLI Commands
-
-```bash
-# Add documentation from a website
-# If URL is not a docs URL, discovery will find the docs automatically
-service-apidocs add https://stripe.com --name stripe          # Auto-discovers stripe.com/docs
-service-apidocs add https://platform.openai.com/docs          # Uses URL directly (already docs)
-service-apidocs add https://docs.anthropic.com --name anthropic
-service-apidocs add https://hono.dev/docs --max-pages 100
-
-# Query documentation (semantic search)
-service-apidocs query openai "how to use function calling"
-service-apidocs query anthropic "tool use examples" --tokens 5000
-service-apidocs query hono "create a route" --json
-
-# List indexed libraries (shows endpoint counts)
-service-apidocs list
-
-# List API endpoints from indexed documentation
-service-apidocs endpoints openai                              # Group by resource (default)
-service-apidocs endpoints openai --group-by method            # Group by HTTP method
-service-apidocs endpoints openai --filter POST                # Show only POST endpoints
-service-apidocs endpoints openai --json                       # Output as JSON
-
-# Sync documentation (re-crawl)
-service-apidocs sync openai
-service-apidocs sync --all
-service-apidocs sync openai --max-pages 200
-
-# Remove library
-service-apidocs remove openai
-```
-
-## Local Storage
-
-Data is stored in `~/.service/service-apidocs/`:
-
-```
-~/.service/service-apidocs/
-├── config.json              # Global config
-├── vectors/                 # Local vector store (when USE_S3_VECTORS is not set)
-│   └── {index-name}.json    # Vectors with embeddings and metadata
-└── libraries/
-    └── {library-id}/
-        ├── metadata.json    # Library metadata (name, URL, docsUrl, endpoint count)
-        ├── endpoints.json   # Extracted API endpoints (structured data)
-        └── cache/
-            ├── pages-0.json # Crawled pages
-            └── chunks-0.json # Parsed chunks
-```
-
-## Vector Metadata
-
-Each vector stores:
-- `libraryId` - e.g., "openai"
-- `version` - Date indexed (e.g., "2024-01-15")
-- `filePath` - URL path (e.g., "/docs/api/chat")
-- `chunkIndex` - Position in document
-- `title` - Section heading
-- `type` - "code" or "text"
-- `content` - Full chunk content
-
-## Processing Pipeline
-
-1. Parse website URL
-2. **Discover docs URL** (if base URL provided, finds /docs, /api, etc.)
-3. Initialize crawler
-4. Crawl documentation pages (BFS within domain)
-5. Convert HTML to Markdown (turndown)
-6. **Extract API endpoints** (Claude agent extracts method, path, params, etc.)
-7. Parse and chunk by headings (~500 tokens)
-8. Generate OpenAI embeddings
-9. Create vector index
-10. Upsert vectors with metadata
-11. Save endpoints and metadata locally
-
-## Crawler Design
-
-The crawler uses a breadth-first search approach:
-
-1. **Discovery**: Aggressively follows ALL links within the same domain
-2. **Filtering**: Skips obvious non-doc URLs (blog, pricing, careers, etc.)
-3. **Validation**: Checks if page content looks like documentation (code blocks, API terms, etc.)
-4. **Extraction**: Extracts main content area, removes nav/header/footer
-5. **Conversion**: Converts HTML to clean Markdown via turndown
-6. **Deduplication**: Tracks content hashes to avoid duplicates
-
-### URL Filtering
-- Follows all links within the same domain
-- Skips: images, PDFs, auth pages, cart/checkout, search pages
-- Skips non-doc patterns: /blog/, /careers/, /pricing/, /terms/, etc.
-
-### Content Detection
-Pages are included if they have 2+ documentation indicators:
-- Code blocks (`<pre>`, `<code>`, triple backticks)
-- API terms (endpoint, parameter, request, response)
-- Doc terms (example, usage, guide, tutorial, reference)
-
-### Optional: Claude Content Cleanup
-Use `enhanceContentWithAgent()` after crawling to have Claude clean up extracted content:
-- Removes navigation remnants
-- Removes ads and cookie notices
-- Preserves code examples and API references
-
-## API Endpoint Extraction
-
-The endpoint extractor uses Claude to analyze documentation pages and extract structured API information:
-
-### Extracted Data
-- **method**: HTTP method (GET, POST, PUT, DELETE, PATCH, etc.)
-- **path**: Endpoint path (e.g., `/users/{id}`, `/v1/chat/completions`)
-- **title**: Short description of what the endpoint does
-- **description**: Full description from documentation
-- **parameters**: Path, query, header, and cookie parameters
-- **requestBody**: Content type, schema, and examples
-- **responses**: Status codes and response descriptions
-- **codeExamples**: Code snippets in various languages (curl, Python, JavaScript, etc.)
-
-### Grouping
-Endpoints are automatically grouped by:
-- **Resource**: Extracted from the path (e.g., `/users/{id}` → `users`)
-- **Method**: HTTP method (GET, POST, etc.)
-
-### Output Formats
-```bash
-# Human-readable grouped by resource (default)
-service-apidocs endpoints stripe
-
-# Human-readable grouped by method
-service-apidocs endpoints stripe --group-by method
-
-# JSON for programmatic use
-service-apidocs endpoints stripe --json
-```
-
-## Docs URL Discovery
-
-When you provide a base URL (like `stripe.com`), the discovery agent:
-
-1. **Checks common patterns**: `/docs`, `/api`, `/api-reference`, `/documentation`, `/developers`
-2. **Checks subdomains**: `docs.`, `api.`, `developers.`
-3. **Analyzes page content**: Uses Claude to find documentation links in the page
-
-This allows you to simply provide a company URL and have the tool find the API docs automatically.
-
-## Development
-
-```bash
-# Install dependencies
-bun install
-
-# Run CLI in development
-bun run bin/cli.ts add https://hono.dev/docs --max-pages 20
-
-# Type check
-bun run typecheck
-
-# Link globally
-bun link
-```
-
-## Notes
-
-- Chunks are sized to ~500 tokens for optimal retrieval
-- Code blocks are extracted as separate chunks with language info
-- Heading hierarchy is preserved for context
-- Results are deduplicated by content similarity
-- Response is formatted as markdown with sources
-- Crawler respects same-domain policy
-- Non-documentation pages (blog, pricing, etc.) are automatically skipped

package/skills/apidocs/LICENSE

@@ -1,192 +0,0 @@
-
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to the Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by the Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding any notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   Copyright 2025 Hasna
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
-

package/skills/apidocs/README.md

@@ -1,196 +0,0 @@
-# service-apidocs
-
-An agentic web crawler CLI tool for indexing and querying API documentation from any website using semantic search. Automatically discovers documentation URLs and extracts structured API endpoint information.
-
-## Features
-
-- **Automatic docs discovery** - Provide a base URL (`stripe.com`) and it finds the API docs
-- **API endpoint extraction** - Extracts structured endpoint data (method, path, parameters, etc.)
-- **Semantic search** - Query documentation using natural language
-- **Multiple storage backends** - Local JSON files or AWS S3 Vectors
-
-## Installation
-
-```bash
-# Install globally from npm
-bun install -g @hasnaxyz/service-apidocs
-
-# Or clone and install locally
-git clone https://github.com/hasnaxyz/service-apidocs.git
-cd service-apidocs
-bun install
-bun link
-```
-
-## Setup
-
-Set required environment variables:
-
-```bash
-# Required - add to ~/.secrets
-export OPENAI_API_KEY=sk-...
-export ANTHROPIC_API_KEY=sk-...  # For docs discovery and endpoint extraction
-
-# Optional - use AWS S3 Vectors instead of local storage
-export USE_S3_VECTORS=true
-```
-
-By default, vectors are stored locally in `~/.service/service-apidocs/vectors/`. Set `USE_S3_VECTORS=true` to use AWS S3 Vectors (requires AWS credentials with S3 Vectors permissions).
-
-## Usage
-
-### Add Documentation
-
-Index documentation from any website:
-
-```bash
-# Add documentation - auto-discovers docs URL from base URL
-service-apidocs add https://stripe.com --name stripe
-# -> Discovers and crawls stripe.com/docs
-
-# Add when URL is already the docs page
-service-apidocs add https://platform.openai.com/docs
-
-# Add with custom name
-service-apidocs add https://docs.anthropic.com --name anthropic
-
-# Limit number of pages to crawl
-service-apidocs add https://hono.dev/docs --max-pages 100
-```
-
-The add command now:
-1. Discovers the documentation URL if you provide a base URL
-2. Crawls all documentation pages
-3. Extracts structured API endpoints using Claude
-4. Generates embeddings for semantic search
-
-### Query Documentation
-
-Search documentation semantically:
-
-```bash
-# Basic query
-service-apidocs query openai "how to use function calling"
-
-# Limit response size
-service-apidocs query anthropic "tool use examples" --tokens 5000
-
-# JSON output
-service-apidocs query hono "create a route" --json
-```
-
-### List Indexed Libraries
-
-```bash
-service-apidocs list
-```
-
-Output:
-```
-Indexed Libraries
-────────────────────────────────────────────────────────────────────────────────
-openai     platform.openai.com  47 pages   234 chunks  47 endpoints  1/29/2026
-anthropic  docs.anthropic.com   32 pages   156 chunks  23 endpoints  1/29/2026
-
-Documentation URLs:
-  anthropic: https://docs.anthropic.com/api
-
-Total: 2 libraries
-```
-
-### List API Endpoints
-
-View extracted API endpoints from indexed documentation:
-
-```bash
-# Group by resource (default)
-service-apidocs endpoints openai
-
-# Output:
-# openai API Endpoints (47 total)
-# Docs: https://platform.openai.com/docs
-#
-# Chat
-#   POST    /v1/chat/completions  Create a chat completion
-#
-# Embeddings
-#   POST    /v1/embeddings        Create embeddings
-#
-# Files
-#   GET     /v1/files             List files
-#   POST    /v1/files             Upload a file
-#   DELETE  /v1/files/{file_id}   Delete a file
-
-# Group by HTTP method
-service-apidocs endpoints openai --group-by method
-
-# Filter by method
-service-apidocs endpoints openai --filter POST
-
-# JSON output for programmatic use
-service-apidocs endpoints openai --json
-```
-
-### Sync Documentation
-
-Re-crawl and update an existing library's index:
-
-```bash
-# Sync single library
-service-apidocs sync openai
-
-# Sync all libraries
-service-apidocs sync --all
-
-# Sync with custom page limit
-service-apidocs sync openai --max-pages 200
-```
-
-### Remove a Library
-
-```bash
-service-apidocs remove openai
-```
-
-## How It Works
-
-1. **Discovers** documentation URL from base website (checks /docs, /api, subdomains, etc.)
-2. **Crawls** documentation website following links within the same domain
-3. **Extracts** main content and converts HTML to Markdown
-4. **Extracts endpoints** using Claude to identify API endpoints with structured data
-5. **Parses** content and splits into semantic chunks (~500 tokens each)
-6. **Generates** embeddings using OpenAI text-embedding-3-small
-7. **Stores** vectors and endpoints locally or in AWS S3 Vectors
-8. **Queries** using semantic similarity and returns relevant documentation
-
-## Crawler Features
-
-- **Intelligent page detection**: Automatically identifies documentation pages vs marketing/blog content
-- **Same-domain crawling**: Only follows links within the documentation site
-- **Content extraction**: Uses common documentation selectors to extract main content
-- **HTML to Markdown**: Converts crawled HTML to clean Markdown for better chunking
-- **Duplicate prevention**: Tracks visited URLs to avoid re-crawling
-
-## API Endpoint Extraction
-
-The tool extracts structured endpoint information including:
-
-- **Method**: GET, POST, PUT, DELETE, PATCH, etc.
-- **Path**: `/users/{id}`, `/v1/chat/completions`
-- **Parameters**: Path, query, header parameters with types and descriptions
-- **Request body**: Content type and example payloads
-- **Responses**: Status codes and descriptions
-- **Code examples**: curl, Python, JavaScript, etc.
-
-Endpoints are automatically grouped by resource (extracted from path) for easy browsing.
-
-## Requirements
-
-- Bun runtime
-- OpenAI API key (for embeddings)
-- Anthropic API key (for docs discovery and endpoint extraction)
-- AWS credentials (optional, only if using S3 Vectors)
-
-## License
-
-MIT