deepspider 0.1.0

package/.trellis/spec/backend/index.md

@@ -0,0 +1,36 @@
+# DeepSpider Development Guidelines
+
+> DeepSpider 项目开发规范
+
+---
+
+## Overview
+
+DeepSpider 是基于 DeepAgents + Patchright 的 JS 逆向分析引擎。
+本目录包含项目的开发规范和代码模式。
+
+---
+
+## Guidelines Index
+
+| Guide | Description | Status |
+|-------|-------------|--------|
+| [Directory Structure](./directory-structure.md) | 项目目录结构和模块组织 | Done |
+| [DeepAgents Guide](./deepagents-guide.md) | DeepAgents 框架使用指南 | Done |
+| [Tool Guidelines](./tool-guidelines.md) | LangChain 工具定义规范 | Done |
+| [Hook Guidelines](./hook-guidelines.md) | 浏览器 Hook 注入规范 | Done |
+| [State Management](./state-management.md) | Agent 状态与数据存储 | Done |
+| [Quality Guidelines](./quality-guidelines.md) | 代码质量规范 | Done |
+| [Type Safety](./type-safety.md) | Zod 类型验证规范 | Done |
+
+---
+
+## Quick Reference
+
+核心规范要点：
+
+1. **Agent 创建**: 使用 `createDeepAgent()` + 配置对象
+2. **工具定义**: 使用 `@langchain/core/tools` + Zod schema
+3. **浏览器交互**: 优先使用 CDP，避免 `page.evaluate()`
+4. **AST 遍历**: 使用 `@babel/traverse`
+5. **数据存储**: 使用 `getDataStore()` 单例

package/.trellis/spec/backend/quality-guidelines.md

@@ -0,0 +1,201 @@
+# Quality Guidelines
+
+> DeepSpider 代码质量规范
+
+---
+
+## Overview
+
+DeepSpider 遵循 CLAUDE.md 中定义的代码规范，重点关注：
+- CDP 优先的浏览器交互
+- Babel AST 遍历模式
+- LangChain 工具定义规范
+
+---
+
+## Forbidden Patterns
+
+### 1. 使用 page.evaluate 代替 CDP
+
+```javascript
+// ❌ 禁止
+const result = await page.evaluate(() => { ... });
+
+// ✅ 使用 CDP
+const cdp = await browser.getCDPSession();
+const result = await cdp.send('Runtime.evaluate', { ... });
+```
+
+### 2. 直接访问封装类的内部属性
+
+```javascript
+// ❌ 禁止：暴露内部实现
+cdpSession.client.on('Debugger.paused', handler);
+
+// ✅ 使用封装类提供的方法
+cdpSession.on('Debugger.paused', handler);
+```
+
+**原因**: 直接访问 `.client` 会导致封装泄漏，当内部实现变化时调用方会报错。
+
+### 3. 子代理不配置中间件
+
+```javascript
+// ❌ 禁止：只在主 Agent 配置中间件，子代理不配置
+// index.js
+const agent = createDeepAgent({
+  middleware: [createFilterToolsMiddleware()],
+  subagents: [subagent1, subagent2],
+});
+
+// subagent1.js - 没有中间件
+export const subagent1 = {
+  name: 'subagent1',
+  tools: [...],
+  middleware: [],  // 空！
+};
+
+// ✅ 子代理也需要配置相同的中间件
+export const subagent1 = {
+  name: 'subagent1',
+  tools: [...],
+  middleware: [
+    createFilterToolsMiddleware(),  // 必须添加
+    createSkillsMiddleware({ ... }),
+  ],
+};
+```
+
+**原因**: DeepAgents 子代理不会继承主 Agent 的中间件配置。如果主 Agent 过滤了内置工具，子代理也必须单独配置过滤中间件，否则子代理仍会使用被过滤的工具。
+
+### 4. setInterval 中使用 async 回调
+
+```javascript
+// ❌ 禁止：async 回调不会被等待，可能导致并发问题
+setInterval(async () => {
+  const result = await detectCaptcha();
+  await handleResult(result);
+}, 30000);
+
+// ✅ 保持同步，只做状态检查和标记
+let needsCheck = false;
+setInterval(() => {
+  const elapsed = Date.now() - lastEventTime;
+  if (elapsed > timeout) {
+    console.log('[提示] 超时，请检查页面');
+  }
+}, 30000);
+```
+
+**原因**: setInterval 不会等待 async 回调完成，多次触发会导致并发执行。
+
+### 5. spawn 使用不存在的 timeout 选项
+
+```javascript
+// ❌ 禁止：spawn 不支持 timeout 选项，超时不会生效
+const proc = spawn('node', ['-e', code], {
+  timeout: 10000,  // 无效！
+});
+
+// ✅ 手动实现超时
+const proc = spawn('node', ['-e', code]);
+let killed = false;
+
+const timer = setTimeout(() => {
+  killed = true;
+  proc.kill('SIGTERM');
+}, 10000);
+
+proc.on('close', () => {
+  clearTimeout(timer);
+});
+```
+
+**原因**: `spawn` 的 options 不包含 `timeout`，这是 `execSync` 的选项。使用 spawn 时必须手动实现超时逻辑。
+
+---
+
+## Required Patterns
+
+### 1. Babel AST 遍历
+
+```javascript
+import traverse from '@babel/traverse';
+
+traverse.default(ast, {
+  FunctionDeclaration(path) {
+    // 处理
+  }
+});
+```
+
+### 2. CDP Session 复用
+
+```javascript
+const cdp = await browser.getCDPSession();
+```
+
+---
+
+## Testing Requirements
+
+运行测试：
+
+```bash
+pnpm test
+```
+
+---
+
+## Code Review Checklist
+
+- [ ] 工具名称使用 snake_case
+- [ ] 参数有 describe 描述
+- [ ] 浏览器交互使用 CDP
+- [ ] AST 遍历使用 Babel
+- [ ] 数组访问前检查边界
+- [ ] 对象访问前检查空值
+
+---
+
+## Defensive Programming
+
+### 1. 数组索引边界检查
+
+```javascript
+// ❌ 禁止：直接访问可能越界
+const stage = stages[parseInt(index)];
+stage.fields.push(field);
+
+// ✅ 先检查边界
+const idx = parseInt(index);
+if (idx < 0 || idx >= stages.length) return;
+const stage = stages[idx];
+```
+
+### 2. 工厂函数避免重复结构
+
+```javascript
+// ❌ 禁止：多处重复对象字面量
+stages.push({ name: 'list', fields: [], entry: null });
+// ... 另一处
+stages = [{ name: 'list', fields: [], entry: null }];
+
+// ✅ 使用工厂函数
+function createStage(name) {
+  return { name, fields: [], entry: null, pagination: null };
+}
+stages.push(createStage('list'));
+```
+
+### 3. 空值检查
+
+```javascript
+// ❌ 禁止：假设对象存在
+currentStage.fields.splice(index, 1);
+
+// ✅ 先检查
+if (!currentStage) return;
+if (index < 0 || index >= currentStage.fields.length) return;
+currentStage.fields.splice(index, 1);
+```

package/.trellis/spec/backend/state-management.md

@@ -0,0 +1,76 @@
+# State Management
+
+> Agent 状态与数据存储规范
+
+---
+
+## Overview
+
+DeepSpider 使用 DeepAgents 的状态后端和文件系统存储管理数据。
+Agent 状态通过 FilesystemBackend 持久化，采集数据通过 DataStore 存储。
+
+---
+
+## State Categories
+
+| 类型 | 存储方式 | 示例 |
+|------|----------|------|
+| Agent 状态 | FilesystemBackend | `.deepspider-agent/` |
+| 采集数据 | DataStore | `.deepspider-data/` |
+| 会话状态 | MemorySaver | 内存中 |
+
+---
+
+## DataStore Pattern
+
+数据存储使用单例模式：
+
+```javascript
+import { getDataStore } from '../store/DataStore.js';
+
+const store = getDataStore();
+await store.saveResponse(data);
+```
+
+**示例**: `src/store/DataStore.js:699-706`
+
+---
+
+## Agent Backend
+
+Agent 状态后端配置：
+
+```javascript
+import { FilesystemBackend } from 'deepagents';
+
+const backend = new FilesystemBackend({
+  rootDir: './.deepspider-agent'
+});
+```
+
+**示例**: `src/agent/index.js:59-62`
+
+---
+
+## Common Mistakes
+
+### 1. 未使用单例
+
+```javascript
+// ❌ 错误：每次创建新实例
+const store = new DataStore();
+
+// ✅ 正确：使用单例
+const store = getDataStore();
+```
+
+### 2. 忘记启动会话
+
+```javascript
+// ❌ 错误：直接保存
+await store.saveResponse(data);
+
+// ✅ 正确：先启动会话
+store.startSession();
+await store.saveResponse(data);
+```

package/.trellis/spec/backend/tool-guidelines.md

@@ -0,0 +1,144 @@
+# Tool Guidelines
+
+> LangChain 工具定义规范
+
+---
+
+## Overview
+
+DeepSpider 使用 `@langchain/core/tools` 定义 Agent 工具。
+每个工具是一个独立的功能单元，通过 Zod schema 定义参数类型。
+
+---
+
+## Tool Structure
+
+标准工具定义结构：
+
+```javascript
+import { z } from 'zod';
+import { tool } from '@langchain/core/tools';
+
+export const myTool = tool(
+  async ({ param1, param2 }) => {
+    // 工具逻辑
+    return JSON.stringify(result, null, 2);
+  },
+  {
+    name: 'tool_name',           // snake_case 命名
+    description: '工具描述',      // 简洁明确
+    schema: z.object({
+      param1: z.string().describe('参数描述'),
+      param2: z.number().optional().default(100),
+    }),
+  }
+);
+
+// 导出工具数组
+export const myTools = [myTool];
+```
+
+**示例**: `src/agent/tools/analyzer.js:14-38`
+
+---
+
+## Schema Conventions
+
+使用 Zod 定义参数 schema：
+
+```javascript
+schema: z.object({
+  // 必填参数
+  code: z.string().describe('JS代码'),
+
+  // 可选参数带默认值
+  extractFunctions: z.boolean().optional().default(true),
+
+  // 枚举类型
+  mode: z.enum(['fast', 'deep']).optional().default('fast'),
+
+  // 数组类型
+  patterns: z.array(z.string()).optional(),
+})
+```
+
+**示例**: `src/agent/tools/analyzer.js:32-36`
+
+---
+
+## Return Value Patterns
+
+工具返回值规范：
+
+```javascript
+// 返回 JSON 字符串（推荐）
+return JSON.stringify(result, null, 2);
+
+// 返回简单字符串
+return `分析完成: ${count} 个函数`;
+
+// 错误处理
+try {
+  // ...
+} catch (e) {
+  return JSON.stringify({ error: e.message });
+}
+```
+
+---
+
+## Tool Organization
+
+工具文件组织：
+
+```javascript
+// src/agent/tools/analyzer.js
+
+// 1. 导入依赖
+import { z } from 'zod';
+import { tool } from '@langchain/core/tools';
+import { ASTAnalyzer } from '../../analyzer/ASTAnalyzer.js';
+
+// 2. 定义各个工具
+export const analyzeAst = tool(...);
+export const analyzeCallstack = tool(...);
+
+// 3. 导出工具数组
+export const analyzerTools = [analyzeAst, analyzeCallstack];
+```
+
+**示例**: `src/agent/tools/analyzer.js`
+
+---
+
+## Common Mistakes
+
+### 1. 工具名称不规范
+
+```javascript
+// ❌ 错误：使用 camelCase
+name: 'analyzeAst'
+
+// ✅ 正确：使用 snake_case
+name: 'analyze_ast'
+```
+
+### 2. 缺少参数描述
+
+```javascript
+// ❌ 错误：无描述
+param1: z.string()
+
+// ✅ 正确：有描述
+param1: z.string().describe('JS代码')
+```
+
+### 3. 返回非字符串
+
+```javascript
+// ❌ 错误：返回对象
+return result;
+
+// ✅ 正确：返回 JSON 字符串
+return JSON.stringify(result, null, 2);
+```

package/.trellis/spec/backend/type-safety.md

@@ -0,0 +1,71 @@
+# Type Safety
+
+> Zod 类型验证规范
+
+---
+
+## Overview
+
+DeepSpider 是纯 JavaScript 项目，使用 Zod 进行运行时类型验证。
+主要用于 LangChain 工具的参数 schema 定义。
+
+---
+
+## Zod Schema
+
+工具参数使用 Zod 定义：
+
+```javascript
+import { z } from 'zod';
+
+const schema = z.object({
+  code: z.string().describe('JS代码'),
+  mode: z.enum(['fast', 'deep']).default('fast'),
+});
+```
+
+---
+
+## Validation
+
+常用 Zod 类型：
+
+| 类型 | 用法 |
+|------|------|
+| 字符串 | `z.string()` |
+| 数字 | `z.number()` |
+| 布尔 | `z.boolean()` |
+| 枚举 | `z.enum(['a', 'b'])` |
+| 数组 | `z.array(z.string())` |
+| 可选 | `.optional()` |
+| 默认值 | `.default(value)` |
+
+---
+
+## Common Patterns
+
+参数描述模式：
+
+```javascript
+schema: z.object({
+  // 必填 + 描述
+  code: z.string().describe('JS代码'),
+
+  // 可选 + 默认值
+  deep: z.boolean().optional().default(false),
+})
+```
+
+---
+
+## Forbidden Patterns
+
+### 1. 缺少 describe
+
+```javascript
+// ❌ 错误
+code: z.string()
+
+// ✅ 正确
+code: z.string().describe('JS代码')
+```

package/.trellis/spec/guides/code-reuse-thinking-guide.md

@@ -0,0 +1,92 @@
+# Code Reuse Thinking Guide
+
+> **Purpose**: Stop and think before creating new code - does it already exist?
+
+---
+
+## The Problem
+
+**Duplicated code is the #1 source of inconsistency bugs.**
+
+When you copy-paste or rewrite existing logic:
+- Bug fixes don't propagate
+- Behavior diverges over time
+- Codebase becomes harder to understand
+
+---
+
+## Before Writing New Code
+
+### Step 1: Search First
+
+```bash
+# Search for similar function names
+grep -r "functionName" .
+
+# Search for similar logic
+grep -r "keyword" .
+```
+
+### Step 2: Ask These Questions
+
+| Question | If Yes... |
+|----------|-----------|
+| Does a similar function exist? | Use or extend it |
+| Is this pattern used elsewhere? | Follow the existing pattern |
+| Could this be a shared utility? | Create it in the right place |
+| Am I copying code from another file? | **STOP** - extract to shared |
+
+---
+
+## Common Duplication Patterns
+
+### Pattern 1: Copy-Paste Functions
+
+**Bad**: Copying a validation function to another file
+
+**Good**: Extract to shared utilities, import where needed
+
+### Pattern 2: Similar Components
+
+**Bad**: Creating a new component that's 80% similar to existing
+
+**Good**: Extend existing component with props/variants
+
+### Pattern 3: Repeated Constants
+
+**Bad**: Defining the same constant in multiple files
+
+**Good**: Single source of truth, import everywhere
+
+---
+
+## When to Abstract
+
+**Abstract when**:
+- Same code appears 3+ times
+- Logic is complex enough to have bugs
+- Multiple people might need this
+
+**Don't abstract when**:
+- Only used once
+- Trivial one-liner
+- Abstraction would be more complex than duplication
+
+---
+
+## After Batch Modifications
+
+When you've made similar changes to multiple files:
+
+1. **Review**: Did you catch all instances?
+2. **Search**: Run grep to find any missed
+3. **Consider**: Should this be abstracted?
+
+---
+
+## Checklist Before Commit
+
+- [ ] Searched for existing similar code
+- [ ] No copy-pasted logic that should be shared
+- [ ] Constants defined in one place
+- [ ] Similar patterns follow same structure

package/.trellis/spec/guides/cross-layer-thinking-guide.md

@@ -0,0 +1,94 @@
+# Cross-Layer Thinking Guide
+
+> **Purpose**: Think through data flow across layers before implementing.
+
+---
+
+## The Problem
+
+**Most bugs happen at layer boundaries**, not within layers.
+
+Common cross-layer bugs:
+- API returns format A, frontend expects format B
+- Database stores X, service transforms to Y, but loses data
+- Multiple layers implement the same logic differently
+
+---
+
+## Before Implementing Cross-Layer Features
+
+### Step 1: Map the Data Flow
+
+Draw out how data moves:
+
+```
+Source → Transform → Store → Retrieve → Transform → Display
+```
+
+For each arrow, ask:
+- What format is the data in?
+- What could go wrong?
+- Who is responsible for validation?
+
+### Step 2: Identify Boundaries
+
+| Boundary | Common Issues |
+|----------|---------------|
+| API ↔ Service | Type mismatches, missing fields |
+| Service ↔ Database | Format conversions, null handling |
+| Backend ↔ Frontend | Serialization, date formats |
+| Component ↔ Component | Props shape changes |
+
+### Step 3: Define Contracts
+
+For each boundary:
+- What is the exact input format?
+- What is the exact output format?
+- What errors can occur?
+
+---
+
+## Common Cross-Layer Mistakes
+
+### Mistake 1: Implicit Format Assumptions
+
+**Bad**: Assuming date format without checking
+
+**Good**: Explicit format conversion at boundaries
+
+### Mistake 2: Scattered Validation
+
+**Bad**: Validating the same thing in multiple layers
+
+**Good**: Validate once at the entry point
+
+### Mistake 3: Leaky Abstractions
+
+**Bad**: Component knows about database schema
+
+**Good**: Each layer only knows its neighbors
+
+---
+
+## Checklist for Cross-Layer Features
+
+Before implementation:
+- [ ] Mapped the complete data flow
+- [ ] Identified all layer boundaries
+- [ ] Defined format at each boundary
+- [ ] Decided where validation happens
+
+After implementation:
+- [ ] Tested with edge cases (null, empty, invalid)
+- [ ] Verified error handling at each boundary
+- [ ] Checked data survives round-trip
+
+---
+
+## When to Create Flow Documentation
+
+Create detailed flow docs when:
+- Feature spans 3+ layers
+- Multiple teams are involved
+- Data format is complex
+- Feature has caused bugs before