@wenathlan/saddle 1.8.2 → 1.8.5

package/docs/plans/38.flow.md

@@ -0,0 +1,155 @@
+# Scraping Pipeline Flow
+
+## High-Level Pipeline
+
+```
+User Input (URL or HTML)
+        │
+        ▼
+┌──────────────────────┐
+│  scrapeUrl()         │
+│  ┌────────────────┐  │
+│  │ AgentBrowser    │  │
+│  │  ┌──────────┐   │  │
+│  │  │ Launch   │   │  │
+│  │  │ Browser  │   │  │
+│  │  └────┬─────┘   │  │
+│  │       ▼         │  │
+│  │  ┌──────────┐   │  │
+│  │  │ Navigate │   │  │
+│  │  │ to URL   │   │  │
+│  │  └────┬─────┘   │  │
+│  │       ▼         │  │
+│  │  ┌──────────┐   │  │
+│  │  │ Scroll / │   │  │
+│  │  │ Interact │   │  │
+│  │  └────┬─────┘   │  │
+│  │       ▼         │  │
+│  │  ┌──────────┐   │  │
+│  │  │ Extract  │   │  │
+│  │  │ HTML     │   │  │
+│  │  └──────────┘   │  │
+│  └────────────────┘  │
+│         │            │
+│         ▼            │
+│  ┌────────────────┐  │
+│  │ extractContent  │  │
+│  │ (Cheerio parse) │  │
+│  └────────────────┘  │
+└──────────┬───────────┘
+           │ ScrapeResult
+           ▼
+┌──────────────────────┐
+│  serializeResult()   │
+│  Format: markdown /  │
+│  json / xml / redis  │
+│  / text              │
+└──────────┬───────────┘
+           │ SerializedOutput
+           ▼
+┌──────────────────────┐
+│  formatForAgent()    │
+│  (LLM-friendly)      │
+│  or render with      │
+│  PygameRenderer      │
+└──────────────────────┘
+```
+
+## Execution Paths
+
+### URL Scraping
+
+```
+scrapeUrl(url, options)
+  ├── AgentBrowser.launch('chromium')
+  ├── AgentBrowser.navigate(url)
+  ├── [optional] AgentBrowser.scrollToBottom()
+  ├── AgentBrowser.html()        → raw HTML
+  ├── AgentBrowser.screenshot()  → Buffer
+  ├── extractContent(html)       → ExtractedContent
+  ├── AgentBrowser.close()
+  └── returns ScrapeResult { url, title, html, text, links, images, tables, metadata, screenshot }
+```
+
+### HTML Scraping (offline)
+
+```
+scrapeHtml(html, options)
+  ├── extractContent(html)       → ExtractedContent
+  └── returns ScrapeResult { url: 'inline', html, ...extracted }
+```
+
+### Browser Reuse
+
+```
+scrapeWithBrowser(browser, url, options)
+  ├── browser.navigate(url)
+  ├── browser.html()
+  ├── extractContent(html)
+  └── returns ScrapeResult
+```
+
+## Data Flow
+
+```
+Raw HTML
+   │
+   ▼
+Cheerio.parse(html)
+   │
+   ├── Title:    $('title').text() / $('h1').first().text()
+   ├── Author:   $('[rel=author]') / meta tags
+   ├── Text:     $('p, li, blockquote').text() (joined)
+   ├── Links:    $('a[href]') → { href, text }
+   ├── Images:   $('img[src]') → { src, alt }
+   ├── Tables:   $('table') → { headers, rows }
+   └── Metadata: meta tags → { description, keywords, og:* }
+           │
+           ▼
+   ExtractedContent
+           │
+           ▼
+   serializeResult(result, { format })
+           │
+           ├── markdown:  TurndownService.turndown(html)
+           ├── json:      JSON.stringify(content)
+           ├── xml:       xml2js.Builder.buildObject(content)
+           ├── redis:     JSON.SET key path value
+           └── text:      strip tags + join
+```
+
+## Serialization Flow
+
+```
+serializeResult(result, options)
+  │
+  ├── resolveFormat(options.format)          → format name
+  ├── resolveExtension(formatName)           → file extension
+  │
+  ├── format === 'markdown'
+  │     └── TurndownService.turndown(html)
+  │
+  ├── format === 'json'
+  │     └── JSON.stringify(content, null, pretty ? 2 : 0)
+  │
+  ├── format === 'xml'
+  │     └── xml2js.Builder.buildObject({ root: content })
+  │
+  ├── format === 'redis'
+  │     └── "JSON.SET key $ '...escaped json...'\n"
+  │
+  └── format === 'text'
+        └── strip all HTML tags, join text nodes
+```
+
+## Agent Output Flow
+
+```
+formatForAgent(result)
+  │
+  ├── Build summary (first 200 chars of text)
+  ├── Extract key points (sentences with key phrases)
+  ├── Collect relevant URLs
+  ├── Chunk content for token limits
+  └── Returns { summary, keyPoints, content, relevantUrls, chunks, tokenCount }
+```

package/docs/plans/39.multi.platform.bot.md

@@ -0,0 +1,252 @@
+# Saddle Multi-Platform Bot
+
+> Bot that operates across GitHub, GitLab, Forgejo, Gitea, and Discord.
+> Unified interface for all platform integrations.
+
+## Overview
+
+The saddle robot operates as a bot across multiple code
+hosting platforms and messaging services through a unified
+adapter interface. Each platform has a dedicated adapter
+implementing the same interface contract.
+
+## Platform Adapters
+
+### GitHub Adapter
+
+```typescript
+interface GitHubAdapter {
+  type: 'github'
+  authenticate(token: string): Promise<boolean>
+  listRepos(query?: string): Promise<Repo[]>
+  getRepo(owner: string, repo: string): Promise<Repo>
+  createIssue(repo: string, title: string, body: string): Promise<Issue>
+  createPR(repo: string, title: string, body: string, branch: string): Promise<PR>
+  addComment(repo: string, issueNumber: number, body: string): Promise<Comment>
+  createWebhook(repo: string, url: string, events: string[]): Promise<Webhook>
+  executeWorkflow(repo: string, workflow: string): Promise<WorkflowRun>
+  uploadReleaseAsset(repo: string, tag: string, name: string, path: string): Promise<Asset>
+}
+```
+
+Authentication uses a personal access token or GitHub App token.
+
+Token scopes required:
+- `repo` (full control)
+- `workflow` (manage workflows)
+- `read:org` (read organization data)
+
+### GitLab Adapter
+
+```typescript
+interface GitLabAdapter {
+  type: 'gitlab'
+  authenticate(token: string): Promise<boolean>
+  listProjects(query?: string): Promise<Project[]>
+  getProject(id: number | string): Promise<Project>
+  createIssue(projectId: number, title: string, description: string): Promise<Issue>
+  createMergeRequest(projectId: number, title: string, description: string, sourceBranch: string, targetBranch: string): Promise<MR>
+  addComment(projectId: number, noteableType: string, noteableId: number, body: string): Promise<Note>
+  createWebhook(projectId: number, url: string, events: string[]): Promise<Webhook>
+  triggerPipeline(projectId: number, ref: string, variables: Record<string, string>): Promise<Pipeline>
+  uploadFile(projectId: number, filePath: string, content: Buffer): Promise<Attachment>
+}
+```
+
+Authentication uses a personal access token with `api` scope.
+
+### Forgejo Adapter
+
+Forgejo is fully compatible with the GitHub API, making the
+GitHub adapter usable with minimal configuration changes.
+
+```typescript
+interface ForgejoAdapter {
+  extends: GitHubAdapter
+  customBaseUrl: string
+}
+```
+
+Configuration:
+- Use GitHub adapter with custom `baseUrl` pointing to Forgejo instance
+- Token type: access token with `write:repository`, `write:issue`, `write:webhook` scopes
+
+### Gitea Adapter
+
+Gitea also provides GitHub-compatible API endpoints.
+
+```typescript
+interface GiteaAdapter {
+  extends: GitHubAdapter
+  customBaseUrl: string
+}
+```
+
+Configuration:
+- Use GitHub adapter with custom `baseUrl` pointing to Gitea instance
+- Token type: API token with full repository scope
+
+### Discord Adapter
+
+```typescript
+interface DiscordAdapter {
+  type: 'discord'
+  authenticate(token: string): Promise<boolean>
+  sendMessage(channelId: string, content: string): Promise<Message>
+  createWebhook(channelId: string, name: string): Promise<Webhook>
+  executeWebhook(webhookId: string, webhookToken: string, payload: WebhookPayload): Promise<Message>
+  addReaction(channelId: string, messageId: string, emoji: string): Promise<void>
+  createThread(channelId: string, name: string, messageId: string): Promise<Thread>
+}
+```
+
+## Unified Bot Interface
+
+```typescript
+interface SaddleBot {
+  platform: PlatformAdapter
+  config: BotConfig
+
+  start(): Promise<void>
+  stop(): Promise<void>
+  executeCommand(command: BotCommand): Promise<BotResult>
+  handleWebhook(event: WebhookEvent): Promise<void>
+  scheduleTask(task: ScheduledTask): Promise<ScheduledTaskHandle>
+  getStatus(): Promise<BotStatus>
+}
+```
+
+### Bot Config
+
+```typescript
+interface BotConfig {
+  platforms: PlatformConfig[]
+  memoryEngine: MemoryEngineConfig
+  deploy: DeployConfig
+  logging: LogConfig
+}
+
+interface PlatformConfig {
+  type: 'github' | 'gitlab' | 'forgejo' | 'gitea' | 'discord'
+  name: string
+  token: string
+  baseUrl?: string
+  webhookSecret?: string
+  repositories: string[]
+  channels: string[]
+}
+```
+
+## Bot Commands
+
+| Command | Description | Platforms |
+|---------|-------------|-----------|
+| `capture` | Start browser capture session | All |
+| `scrape` | Scrape URLs and extract content | All |
+| `review` | Review PRs/issues automatically | GitHub, GitLab, Forgejo, Gitea |
+| `deploy` | Deploy to target platform | GitHub, GitLab, Forgejo |
+| `memory` | Store/retrieve memory objects | All |
+| `status` | Report bot status | All |
+| `test` | Run test suite | GitHub, GitLab, Forgejo |
+| `release` | Create release with assets | GitHub, GitLab, Forgejo |
+| `webhook` | Manage webhooks | GitHub, GitLab, Forgejo, Gitea |
+| `schedule` | Schedule recurring tasks | All |
+| `npm publish` | Publish NPM package | GitHub |
+| `artifact` | Upload artifacts to storage | All |
+
+## Multi-Platform Workflow
+
+A typical workflow uses the saddle robot across platforms:
+
+```
+1. Discord webhook receives user command
+2. Saddle robot parses command
+3. Bot selects target platform adapter
+4. Executes command on target platform
+5. Stores result in memory backend
+6. Reports status back to Discord
+7. Optionally triggers CI/CD pipeline
+```
+
+## Webhook Integration
+
+### GitHub Webhooks
+
+```json
+{
+  "webhooks": [
+    {
+      "url": "https://saddle.example.com/webhook/github",
+      "content_type": "json",
+      "secret": "${WEBHOOK_SECRET}",
+      "events": [
+        "push",
+        "pull_request",
+        "issues",
+        "issue_comment",
+        "release",
+        "deployment"
+      ]
+    }
+  ]
+}
+```
+
+### GitLab Webhooks
+
+```json
+{
+  "webhooks": [
+    {
+      "url": "https://saddle.example.com/webhook/gitlab",
+      "secret_token": "${WEBHOOK_SECRET}",
+      "trigger_events": true,
+      "push_events": true,
+      "merge_request_events": true,
+      "issue_events": true,
+      "tag_push_events": true
+    }
+  ]
+}
+```
+
+## Deployment per Platform
+
+| Platform | Deploy Method | Best For |
+|----------|--------------|----------|
+| GitHub | GitHub Actions + NPM package | Team collaboration |
+| GitLab | GitLab CI + Docker | Self-hosted or cloud |
+| Forgejo | Forgejo Actions + Docker | Lightweight self-hosted |
+| Gitea | Gitea Actions + Docker | Minimal self-hosted |
+| Discord | Discord bot token + HTTPS | Messaging integration |
+
+## Error Handling
+
+Each adapter implements error handling with contextual error
+messages for later tracing. Adapter failures are caught and
+reported without crashing the bot.
+
+```typescript
+class SafeAdapter {
+  private adapter: PlatformAdapter
+
+  async execute(
+    method: string,
+    ...args: unknown[]
+  ): Promise<{ success: boolean; result?: unknown; error?: string }> {
+    try {
+      const result = await (this.adapter as Record<string, unknown>)[method](...args)
+      return { success: true, result }
+    } catch (error) {
+      return {
+        success: false,
+        error: error instanceof Error ? error.message : String(error),
+      }
+    }
+  }
+}
+```
+
+## License
+
+MIT

package/docs/plans/40.npm.publish.md

@@ -0,0 +1,250 @@
+# NPM Package Publishing for Saddle Robot
+
+> Package `@devthink/saddle` published to NPM registry.
+> Acts as both library and CLI binary.
+
+## Overview
+
+Saddle robot is published as an NPM package for easy
+distribution as a library and a CLI tool. The package
+includes browser automation, bot functionality, captcha
+bypass, and memory engine capabilities.
+
+## Package Configuration
+
+### package.json
+
+```json
+{
+  "name": "@devthink/saddle",
+  "version": "1.0.0",
+  "description": "Super bot: agent browser, captcha bypass, multi-platform GitHub/GitLab/Forgejo/Gitea automation with computational memory",
+  "main": "lib/index.js",
+  "bin": {
+    "saddle": "bin/saddle.js"
+  },
+  "type": "module",
+  "exports": {
+    ".": {
+      "import": "./lib/index.js",
+      "require": "./lib/index.cjs"
+    },
+    "./browser": {
+      "import": "./lib/browser.js"
+    },
+    "./bot": {
+      "import": "./lib/bot.js"
+    },
+    "./ captcha": {
+      "import": "./lib/captcha.js"
+    },
+    "./memory-engine": {
+      "import": "./lib/memory-engine.js"
+    },
+    "./deploy": {
+      "import": "./lib/deploy.js"
+    }
+  },
+  "scripts": {
+    "build": "tsc",
+    "test": "vitest run",
+    "lint": "biome check .",
+    "typecheck": "tsc --noEmit",
+    "prepublishOnly": "npm run build && npm test",
+    "prepare": "husky"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/username/saddle.git"
+  },
+  "keywords": [
+    "agent",
+    "browser",
+    "automation",
+    "bot",
+    "captcha-bypass",
+    "github-bot",
+    "gitlab-bot",
+    "scraper",
+    "computer-use",
+    "ai-agent"
+  ],
+  "author": "iakadion",
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/username/saddle/issues"
+  },
+  "homepage": "https://github.com/username/saddle#readme"
+}
+```
+
+## Publishing Steps
+
+### 1. Build and Test
+
+```bash
+npm run build
+npm test
+npm run lint
+npm run typecheck
+```
+
+All checks must pass before publishing.
+
+### 2. Login to NPM
+
+```bash
+npm login
+# Enter NPM credentials (token recommended)
+```
+
+### 3. Publish
+
+```bash
+# Publish as public package
+npm publish --access public
+
+# Publish as scoped package
+npm publish --access public
+```
+
+### 4. Version Bumping
+
+```bash
+# Patch version (1.0.0 → 1.0.1)
+npm version patch
+
+# Minor version (1.0.0 → 1.1.0)
+npm version minor
+
+# Major version (1.0.0 → 2.0.0)
+npm version major
+
+# Push tags after version bump
+git push --follow-tags
+```
+
+## Installation
+
+### As Library
+
+```bash
+npm install @devthink/saddle
+```
+
+```javascript
+import { SaddleRobot } from '@devthink/saddle'
+
+const robot = new SaddleRobot({
+  platform: 'github',
+  token: process.env.SBOT_TOKEN,
+})
+
+await robot.start()
+```
+
+### As CLI
+
+```bash
+npm install -g @devthink/saddle
+
+saddle --help
+saddle capture --url https://example.com
+saddle bot --platform github --token $SBOT_TOKEN
+saddle memory --load repo://owner/repo/path/file.json
+```
+
+## Binary Entry Point
+
+```javascript
+#!/usr/bin/env node
+
+import { SaddleRobot } from '../lib/index.js'
+import { parseArgs } from 'node:util'
+
+const options = parseArgs({
+  options: {
+    command: { type: 'string' },
+    platform: { type: 'string' },
+    token: { type: 'string' },
+    verbose: { type: 'boolean' },
+  },
+})
+
+const { values } = options
+const robot = new SaddleRobot({
+  platform: values.platform ?? 'auto',
+  token: values.token ?? '',
+  verbose: values.verbose ?? false,
+})
+
+async function main() {
+  try {
+    const command = values.command ?? 'help'
+    await robot.start()
+    const result = await robot.executeCommand(command)
+    if (values.verbose) {
+      console.log(JSON.stringify(result, null, 2))
+    }
+  } catch (error) {
+    console.error(
+      `saddle: ${error instanceof Error ? error.message : String(error)}`
+    )
+    process.exit(1)
+  }
+}
+
+main()
+```
+
+## Scoped Package Structure
+
+```
+@devthink/saddle/
+├── lib/
+│   ├── index.js          main entry point
+│   ├── index.cjs         CommonJS entry point
+│   ├── browser.js        browser automation engine
+│   ├── bot.js            multi-platform bot engine
+│   ├── captcha.js        captcha bypass module
+│   ├── memory-engine.js  storage-to-compute memory
+│   ├── deploy.js         deploy strategy implementations
+│   └── adapters/
+│       ├── github.js     GitHub platform adapter
+│       ├── gitlab.js     GitLab platform adapter
+│       ├── forgejo.js    Forgejo platform adapter
+│       ├── gitea.js      Gitea platform adapter
+│       └── discord.js    Discord messaging adapter
+├── bin/
+│   └── saddle.js         CLI entry point
+├── types/
+│   └── index.d.ts        TypeScript declarations
+├── docs/
+│   └── index.md          package documentation
+├── package.json
+└── README.md
+```
+
+## Versioning Strategy
+
+Follow semantic versioning (semver):
+
+| Change Type | Version Bump |
+|-------------|-------------|
+| Bug fix | Patch (1.x.X) |
+| New backward-compatible feature | Minor (1.X.0) |
+| Breaking change | Major (X.0.0) |
+
+## Pre-publish Checklist
+
+1. All tests pass (`npm test`)
+2. Linting passes (`npm run lint`)
+3. TypeScript checks pass (`npm run typecheck`)
+4. Build succeeds (`npm run build`)
+5. README is updated
+6. Changelog is updated
+7. No breaking changes without major version bump
+
+## License
+
+MIT