@nshipster/sosumi 1.0.0

package/LICENSE.md

@@ -0,0 +1,19 @@
+Copyright 2025 NSHipster (https://nshipster.com)
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of this software and associated documentation files (the "Software"),
+to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense,
+and/or sell copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.

package/README.md

@@ -0,0 +1,304 @@
+# sosumi.ai
+
+Making Apple docs AI-readable.
+
+[sosumi.ai](https://sosumi.ai) 
+provides Apple Developer documentation in an AI-readable format 
+by converting JavaScript-rendered pages into Markdown.
+
+## Usage
+
+### HTTP API
+
+Replace `developer.apple.com` with `sosumi.ai` 
+in any Apple Developer documentation URL:
+
+**Original:**
+```
+https://developer.apple.com/documentation/swift/array
+```
+
+**AI-readable:**
+```
+https://sosumi.ai/documentation/swift/array
+```
+
+This works for all API reference docs, 
+as well as Apple's [Human Interface Guidelines](https://developer.apple.com/design/human-interface-guidelines/) (HIG).
+
+WWDC session transcripts are also supported by replacing the same host for video URLs:
+
+**Original:**
+```
+https://developer.apple.com/videos/play/wwdc2021/10133/
+```
+
+**AI-readable:**
+```
+https://sosumi.ai/videos/play/wwdc2021/10133
+```
+
+Sosumi can also proxy public non-Apple Swift-DocC pages using:
+
+**Original:**
+```
+https://apple.github.io/swift-argument-parser/documentation/argumentparser
+```
+
+**AI-readable:**
+```
+https://sosumi.ai/external/https://apple.github.io/swift-argument-parser/documentation/argumentparser
+```
+
+> [!NOTE]
+> Sosumi resolves the URL to the site's underlying DocC JSON endpoint
+> and renders Markdown, preserving any base path from the original URL.
+> External hosts can opt out via `robots.txt`
+> by disallowing user-agent `sosumi-ai`
+> (full UA: `sosumi-ai/1.0 (+https://sosumi.ai/#bot)`).
+> See `/bot` for the crawler policy and contact details.
+
+### MCP Integration
+
+Sosumi's MCP server supports Streamable HTTP and Server-Sent Events (SSE) transport. 
+If your client supports either of these, 
+configure it to connect directly to `https://sosumi.ai/mcp`.
+
+Otherwise,
+you can run this command to proxy over stdio:
+
+```json
+{
+  "mcpServers": {
+    "sosumi": {
+      "command": "npx",
+      "args": ["-y", "mcp-remote", "https://sosumi.ai/mcp"]
+    }
+  }
+}
+```
+
+See [the website](https://sosumi.ai/#clients) for client-specific instructions.
+
+#### Available Tools
+
+- `searchAppleDocumentation` - Searches Apple Developer documentation
+  - Parameters: `query` (string)
+  - Returns structured results with titles, URLs, descriptions, breadcrumbs, and tags
+
+- `fetchAppleDocumentation` - Fetches Apple Developer documentation and Human Interface Guidelines by path
+  - Parameters: `path` (string) - Documentation path (e.g., '/documentation/swift', 'swiftui/view', 'design/human-interface-guidelines/foundations/color')
+  - Returns content as Markdown
+
+- `fetchAppleVideoTranscript` - Fetches video transcripts, including WWDC sessions
+  - Parameters: `path` (string) - video path (e.g., `/videos/play/wwdc2021/10133`)
+  - Returns transcript content as Markdown
+
+- `fetchExternalDocumentation` - Fetches external Swift-DocC documentation by absolute HTTPS URL
+  - Parameters: `url` (string) - External URL (e.g., `https://apple.github.io/swift-argument-parser/documentation/argumentparser`)
+  - Returns content as Markdown
+
+### CLI
+
+Sosumi also provides a CLI that complements MCP:
+
+```bash
+npx @nshipster/sosumi fetch https://developer.apple.com/documentation/swift/array
+```
+
+If you use it regularly, install once:
+
+```bash
+npm i -g @nshipster/sosumi
+```
+
+Then use `sosumi` directly:
+
+```bash
+sosumi fetch https://developer.apple.com/documentation/swift/array
+```
+
+You can fetch all content types covered by MCP tools:
+
+```bash
+# Apple documentation / HIG / videos
+sosumi fetch /documentation/swift/array
+sosumi fetch /design/human-interface-guidelines/color
+sosumi fetch /videos/play/wwdc2021/10133
+
+# External Swift-DocC pages
+sosumi fetch https://apple.github.io/swift-argument-parser/documentation/argumentparser
+
+# Apple documentation search
+sosumi search "SwiftData"
+```
+
+Run a local server from the published package:
+
+```bash
+sosumi serve
+sosumi serve --port 8787
+```
+
+By default, output is plain text / Markdown.
+Use JSON output for scripts:
+
+```bash
+sosumi fetch https://developer.apple.com/documentation/swift/array --json
+sosumi search "SwiftData" --json
+```
+
+### Chrome Extension
+
+You can also use Sosumi from a community-contributed 
+[Chrome extension](https://chromewebstore.google.com/detail/donffakeimppgoehccpfhlchmbfdmfpj?utm_source=item-share-cb),
+which adds a "Copy sosumi Link" button 
+to Apple Developer documentation pages.
+[Source code](https://github.com/FromAtom/Link-Generator-for-sosumi.ai) is available on GitHub.
+
+## Self-Hosting
+
+This project is designed to be easily run on your own machine
+or deployed to a hosting provider.
+
+Sosumi.ai is currently hosted by 
+[Cloudflare Workers](https://workers.cloudflare.com).
+
+> [!NOTE]  
+> The application is built with Hono, 
+> making it compatible with various runtimes.
+>
+> See the [Hono docs](https://hono.dev/docs/getting-started/basic)
+> for more information about deploying to different platforms.
+
+### Prerequisites
+
+- Node.js 18+
+- npm
+
+### Quick Start
+
+1. **Clone the repository:**
+   ```bash
+   git clone https://github.com/nshipster/sosumi.ai.git
+   cd sosumi.ai
+   ```
+
+2. **Install dependencies:**
+   ```bash
+   npm install
+   ```
+
+3. **Start development server:**
+   ```bash
+   npm run dev
+   ```
+
+Once the application is up and running, press the <kbd>b</kbd>
+to open the URL in your browser.
+
+To configure MCP clients to use your development server, 
+replace `sosumi.ai` with the local server address
+(`http://localhost:8787` by default).
+
+### External Host Restrictions
+
+You can restrict which external Swift-DocC hosts are reachable
+with two environment variables (both newline-delimited):
+
+- `EXTERNAL_DOC_HOST_ALLOWLIST` — only listed hosts are permitted
+- `EXTERNAL_DOC_HOST_BLOCKLIST` — listed hosts are always denied
+
+> [!IMPORTANT]
+> Hostname-based private-network checks cannot fully prevent DNS rebinding.
+> Set an explicit `EXTERNAL_DOC_HOST_ALLOWLIST` in production.
+
+## Development
+
+### Testing
+
+This project uses [vitest](https://vitest.dev)
+for unit and integration testing.
+
+```bash
+npm run test          # Run tests
+npm run test:ui       # Run tests with UI
+npm run test:run      # Run tests once
+```
+
+> [!TIP]
+> When running the CLI through npm scripts during local development, 
+> use `-s` (`--silent`)
+> to suppress npm's script preamble so output pipes cleanly:
+>
+> ```bash
+> npm run -s cli -- fetch https://developer.apple.com/documentation/swift/array | bat -l md
+> ```
+
+### Code Quality
+
+This project uses [Biome](https://biomejs.dev/) 
+for code formatting, linting, and import organization.
+
+- `npm run format` - Format all code files
+- `npm run lint` - Lint and fix code issues
+- `npm run check` - Format, lint, and organize imports (recommended)
+- `npm run check:ci` - Check code without making changes (for CI)
+
+### Editor Integration
+
+For the best development experience, install the Biome extension for your editor:
+
+- [VSCode](https://marketplace.visualstudio.com/items?itemName=biomejs.biome)
+- [Vim/Neovim](https://github.com/biomejs/biome/tree/main/editors/vim)
+- [Emacs](https://github.com/biomejs/biome/tree/main/editors/emacs)
+
+### Cloudflare Workers
+
+Whenever you update your `wrangler.toml` or change your Worker bindings, 
+be sure to re-run:
+
+```bash
+npm run cf-typegen
+```
+
+### Publishing
+
+Publishing is handled by `.github/workflows/release.yml`.
+
+- Trigger: pushed tags matching `v*` or manual dispatch with `tag`
+- Release step: `gh release create "$TAG_NAME" --generate-notes`
+- Publish auth: npm trusted publishing via OIDC (`id-token: write`)
+- Publish command: `npm publish --provenance --access public`
+
+## License
+
+This project is available under the MIT license.
+See the LICENSE file for more info.
+
+## Legal
+
+This is an unofficial,
+independent project and is not affiliated with or endorsed by Apple Inc.
+"Apple", "Xcode", and related marks are trademarks of Apple Inc.
+
+This service is an accessibility-first,
+on‑demand renderer.
+It converts a single Apple Developer page to Markdown only when requested by a user.
+It does not crawl, spider, or bulk download;
+it does not attempt to bypass authentication or security;
+and it implements rate limiting to avoid imposing unreasonable load.
+
+For external Swift-DocC hosts, access can be denied by `robots.txt`
+and opt-out response directives such as `X-Robots-Tag: noai`.
+
+Content is fetched transiently and may be cached briefly to improve performance.
+No permanent archives are maintained.
+All copyrights and other rights in the underlying content remain with Apple Inc.
+Each page links back to the original source.
+
+Your use of this service must comply with Apple's Terms of Use and applicable law.
+You are solely responsible for how you access and use Apple's content through this tool.
+Do not use this service to circumvent technical measures or for redistribution.
+
+**Contact:** <info@sosumi.ai>

package/bin/sosumi.mjs

@@ -0,0 +1,76 @@
+#!/usr/bin/env node
+
+import { spawn } from "node:child_process"
+import { dirname, resolve } from "node:path"
+import { fileURLToPath } from "node:url"
+
+async function runServe(args) {
+  const scriptDir = dirname(fileURLToPath(import.meta.url))
+  const packageRoot = resolve(scriptDir, "..")
+  const configPath = resolve(packageRoot, "wrangler.jsonc")
+  const wranglerArgs = ["-y", "wrangler@^4", "dev", "--config", configPath, ...args]
+
+  await new Promise((resolvePromise, rejectPromise) => {
+    const child = spawn("npx", wranglerArgs, {
+      cwd: packageRoot,
+      stdio: "inherit",
+      shell: process.platform === "win32",
+    })
+
+    child.on("error", rejectPromise)
+    child.on("exit", (code, signal) => {
+      if (signal) {
+        rejectPromise(new Error(`wrangler dev terminated by signal: ${signal}`))
+        return
+      }
+      if (code && code !== 0) {
+        rejectPromise(new Error(`wrangler dev exited with code ${code}`))
+        return
+      }
+      resolvePromise(undefined)
+    })
+  })
+}
+
+async function runCliFromSource(args) {
+  const scriptDir = dirname(fileURLToPath(import.meta.url))
+  const packageRoot = resolve(scriptDir, "..")
+  const cliPath = resolve(packageRoot, "src/cli.ts")
+
+  return await new Promise((resolvePromise, rejectPromise) => {
+    const child = spawn(process.execPath, ["--import", "tsx/esm", cliPath, ...args], {
+      cwd: packageRoot,
+      stdio: "inherit",
+      shell: process.platform === "win32",
+    })
+
+    child.on("error", rejectPromise)
+    child.on("exit", (code, signal) => {
+      if (signal) {
+        rejectPromise(new Error(`CLI terminated by signal: ${signal}`))
+        return
+      }
+      resolvePromise(code ?? 0)
+    })
+  })
+}
+
+async function run() {
+  try {
+    const args = process.argv.slice(2)
+    if (args[0] === "serve") {
+      await runServe(args.slice(1))
+      process.exit(0)
+    }
+    const exitCode = await runCliFromSource(args)
+    if (typeof exitCode === "number" && exitCode !== 0) {
+      process.exit(exitCode)
+    }
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error)
+    console.error(`sosumi: ${message}`)
+    process.exit(1)
+  }
+}
+
+await run()

package/package.json

@@ -0,0 +1,53 @@
+{
+  "name": "@nshipster/sosumi",
+  "version": "1.0.0",
+  "license": "MIT",
+  "type": "module",
+  "engines": {
+    "node": ">=20.18.1"
+  },
+  "files": [
+    "bin",
+    "src",
+    "public",
+    "wrangler.jsonc",
+    "README.md",
+    "LICENSE.md"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "bin": {
+    "sosumi": "./bin/sosumi.mjs"
+  },
+  "scripts": {
+    "dev": "wrangler dev",
+    "deploy": "wrangler deploy --minify",
+    "cli": "node ./bin/sosumi.mjs",
+    "cf-typegen": "wrangler types --env-interface CloudflareBindings",
+    "test": "vitest",
+    "test:ui": "vitest --ui",
+    "test:run": "vitest run",
+    "format": "biome check --write --linter-enabled=false .",
+    "lint": "biome check --formatter-enabled=false .",
+    "check": "biome check --write .",
+    "check:ci": "biome check ."
+  },
+  "dependencies": {
+    "@hono/mcp": "^0.1.2",
+    "@modelcontextprotocol/sdk": "^1.17.4",
+    "cheerio": "^1.2.0",
+    "hono": "^4.9.4",
+    "robots-parser": "^3.0.1",
+    "tsx": "^4.21.0",
+    "zod": "^3"
+  },
+  "devDependencies": {
+    "@biomejs/biome": "2.2.2",
+    "@cloudflare/puppeteer": "^1.0.4",
+    "@cloudflare/vitest-pool-workers": "^0.8.68",
+    "@vitest/ui": "^3.2.4",
+    "vitest": "^3.2.4",
+    "wrangler": "^4.32.0"
+  }
+}

package/public/_headers

@@ -0,0 +1,2 @@
+/sosumi.m4a
+  Cache-Control: public, max-age=31556952, immutable

package/public/favicon.svg

@@ -0,0 +1,7 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24">
+  <!-- https://www.streamlinehq.com/icons/pixel?icon=ico_girK3NdW2U5HOWO0 -->
+  <path fill="currentColor"
+    d="M21.72 4.575V0H10.29v1.14H5.715v1.148H2.287v8.002h-1.14v2.28H0V24h24V4.575zM5.715 22.86H1.148v-6.863h4.567zm0-8.002h-2.28V3.428h2.28zm1.148-12.57h3.427v1.14H8.003v11.43h-1.14zm3.427 2.287v10.282H9.15V4.575zm1.14 18.285H6.863v-6.863h4.567zm11.43 0H12.578v-8.003H11.43V1.14h9.143v3.435h-3.428v1.14h5.715z" />
+  <path fill="currentColor"
+    d="M19.433 15.997h1.14v1.148h-1.14zM19.433 13.717h1.14v1.14h-1.14zM17.145 17.145h2.287v1.14h-2.287zM16.005 15.997h1.14v1.148h-1.14zM16.005 13.717h1.14v1.14h-1.14zM16.005 5.715h1.14v2.287h-1.14zM14.857 8.002h1.148v2.288h-1.147zM13.717 10.29h1.14v2.28h-1.14zM13.717 2.287h1.14v1.14h-1.14zM12.578 12.57h1.14v2.287h-1.14zM8.002 18.285h2.288v2.287H8.003zM2.287 18.285h2.288v2.287H2.287z" />
+</svg>

package/public/icons/square.and.pencil.svg

@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--Generator: Apple Native CoreSVG 326-->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"
+    viewBox="0 0 23.6475 23.3041">
+    <g>
+        <rect height="23.3041" opacity="0" width="23.6475" x="0" y="0" />
+        <path
+            d="M15.5591 4.88935L6.08643 4.88935C5.10986 4.88935 4.56299 5.41669 4.56299 6.43232L4.56299 17.5163C4.56299 18.5319 5.10986 19.0495 6.08643 19.0495L17.2095 19.0495C18.186 19.0495 18.7231 18.5319 18.7231 17.5163L18.7231 8.12957L20.2954 6.55445L20.2954 17.5944C20.2954 19.6159 19.27 20.6218 17.229 20.6218L6.05713 20.6218C4.02588 20.6218 2.99072 19.6159 2.99072 17.5944L2.99072 6.34443C2.99072 4.33271 4.02588 3.31708 6.05713 3.31708L17.1313 3.31708Z"
+            fill="white" fill-opacity="0.85" />
+        <path
+            d="M9.61182 14.2936L11.5161 13.4636L20.6372 4.35224L19.2993 3.03388L10.188 12.1452L9.30908 13.9811C9.23096 14.1472 9.42627 14.3718 9.61182 14.2936ZM21.3599 3.63935L22.063 2.91669C22.395 2.56513 22.395 2.09638 22.063 1.77412L21.8384 1.53974C21.5356 1.23701 21.0571 1.27607 20.7349 1.58857L20.022 2.29169Z"
+            fill="white" fill-opacity="0.85" />
+    </g>
+</svg>