@pzy560117/codex-harness 0.1.4 → 0.1.5

package/package-source/docs/codex-harness-engineering/templates/package-assets/skills/error-handling/SKILL.md

@@ -0,0 +1,245 @@
+---
+name: error-handling
+description: Error Handling Skill
+---
+
+# Error Handling Skill
+
+异常处理、日志记录和监控最佳实践。
+
+## 错误处理原则
+
+### 错误分类
+| 类型 | 示例 | 处理方式 |
+|------|------|----------|
+| 业务错误 | 用户不存在、余额不足 | 返回明确错误码和消息 |
+| 验证错误 | 参数格式错误 | 返回字段级错误详情 |
+| 系统错误 | 数据库连接失败 | 记录日志，返回通用错误 |
+| 未知错误 | 未捕获异常 | 记录完整堆栈，告警 |
+
+### 错误响应格式
+```json
+{
+  "code": 40001,
+  "message": "用户名或密码错误",
+  "details": null,
+  "requestId": "req-abc123",
+  "timestamp": "2024-01-01T12:00:00Z"
+}
+```
+
+### 验证错误响应
+```json
+{
+  "code": 42201,
+  "message": "参数验证失败",
+  "details": [
+    {
+      "field": "email",
+      "message": "邮箱格式不正确"
+    },
+    {
+      "field": "password",
+      "message": "密码长度至少8位"
+    }
+  ]
+}
+```
+
+## 异常处理模式
+
+### 全局异常处理
+```python
+# Python FastAPI
+from fastapi import Request
+from fastapi.responses import JSONResponse
+
+@app.exception_handler(Exception)
+async def global_exception_handler(request: Request, exc: Exception):
+    logger.error(f"Unhandled exception: {exc}", exc_info=True)
+    return JSONResponse(
+        status_code=500,
+        content={
+            "code": 50000,
+            "message": "服务器内部错误",
+            "requestId": request.state.request_id
+        }
+    )
+```
+
+```javascript
+// Node.js Express
+app.use((err, req, res, next) => {
+  logger.error('Unhandled error', {
+    error: err.message,
+    stack: err.stack,
+    requestId: req.id
+  });
+  
+  res.status(500).json({
+    code: 50000,
+    message: '服务器内部错误',
+    requestId: req.id
+  });
+});
+```
+
+### 自定义业务异常
+```python
+class BusinessError(Exception):
+    def __init__(self, code: int, message: str):
+        self.code = code
+        self.message = message
+
+class UserNotFoundError(BusinessError):
+    def __init__(self, user_id: int):
+        super().__init__(40401, f"用户 {user_id} 不存在")
+
+# 使用
+raise UserNotFoundError(user_id=123)
+```
+
+## 日志规范
+
+### 日志级别
+| 级别 | 用途 |
+|------|------|
+| DEBUG | 调试信息，生产环境关闭 |
+| INFO | 关键业务流程记录 |
+| WARN | 潜在问题，不影响功能 |
+| ERROR | 错误，需要关注 |
+| FATAL | 严重错误，服务不可用 |
+
+### 结构化日志
+```json
+{
+  "timestamp": "2024-01-01T12:00:00.000Z",
+  "level": "ERROR",
+  "message": "Failed to process order",
+  "service": "order-service",
+  "requestId": "req-abc123",
+  "userId": 12345,
+  "orderId": "ORD-001",
+  "error": {
+    "type": "PaymentError",
+    "message": "Insufficient balance",
+    "stack": "..."
+  },
+  "duration": 150
+}
+```
+
+### 日志配置
+```javascript
+// Winston 配置
+const logger = winston.createLogger({
+  level: process.env.LOG_LEVEL || 'info',
+  format: winston.format.combine(
+    winston.format.timestamp(),
+    winston.format.json()
+  ),
+  defaultMeta: { service: 'my-service' },
+  transports: [
+    new winston.transports.Console(),
+    new winston.transports.File({ filename: 'error.log', level: 'error' }),
+    new winston.transports.File({ filename: 'combined.log' })
+  ]
+});
+```
+
+## 监控告警
+
+### 关键指标
+```
+错误率 = 错误请求数 / 总请求数
+P99 延迟 = 99% 请求的响应时间
+可用性 = 正常运行时间 / 总时间
+```
+
+### 告警规则
+```yaml
+# Prometheus AlertManager 示例
+groups:
+  - name: api-alerts
+    rules:
+      - alert: HighErrorRate
+        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "API 错误率超过 5%"
+          
+      - alert: HighLatency
+        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "P99 延迟超过 1 秒"
+```
+
+## 前端错误处理
+
+### 全局错误捕获
+```javascript
+// React Error Boundary
+class ErrorBoundary extends React.Component {
+  state = { hasError: false };
+  
+  static getDerivedStateFromError(error) {
+    return { hasError: true };
+  }
+  
+  componentDidCatch(error, errorInfo) {
+    logger.error('React error', { error, errorInfo });
+  }
+  
+  render() {
+    if (this.state.hasError) {
+      return <ErrorFallback />;
+    }
+    return this.props.children;
+  }
+}
+```
+
+### API 错误处理
+```javascript
+// Axios 拦截器
+axios.interceptors.response.use(
+  response => response,
+  error => {
+    const { response } = error;
+    
+    if (response?.status === 401) {
+      // 跳转登录
+      router.push('/login');
+    } else if (response?.status >= 500) {
+      // 显示通用错误
+      toast.error('服务器错误，请稍后重试');
+    }
+    
+    return Promise.reject(error);
+  }
+);
+```
+
+## 重试策略
+
+```javascript
+// 指数退避重试
+async function retryWithBackoff(fn, maxRetries = 3) {
+  for (let i = 0; i < maxRetries; i++) {
+    try {
+      return await fn();
+    } catch (error) {
+      if (i === maxRetries - 1) throw error;
+      
+      const delay = Math.min(1000 * Math.pow(2, i), 10000);
+      await new Promise(resolve => setTimeout(resolve, delay));
+    }
+  }
+}
+```
+

package/package-source/docs/codex-harness-engineering/templates/package-assets/skills/eval/SKILL.md

@@ -0,0 +1,125 @@
+---
+name: eval
+description: Workflow skill from eval.md
+---
+
+# Eval Command
+
+Manage eval-driven development workflow.
+
+## Usage
+
+`/eval [define|check|report|list] [feature-name]`
+
+## Define Evals
+
+`/eval define feature-name`
+
+Create a new eval definition:
+
+1. Create `.claude/evals/feature-name.md` with template:
+
+```markdown
+## EVAL: feature-name
+Created: $(date)
+
+### Capability Evals
+- [ ] [Description of capability 1]
+- [ ] [Description of capability 2]
+
+### Regression Evals
+- [ ] [Existing behavior 1 still works]
+- [ ] [Existing behavior 2 still works]
+
+### Success Criteria
+- pass@3 > 90% for capability evals
+- pass^3 = 100% for regression evals
+```
+
+2. Prompt user to fill in specific criteria
+
+## Check Evals
+
+`/eval check feature-name`
+
+Run evals for a feature:
+
+1. Read eval definition from `.claude/evals/feature-name.md`
+2. For each capability eval:
+   - Attempt to verify criterion
+   - Record PASS/FAIL
+   - Log attempt in `.claude/evals/feature-name.log`
+3. For each regression eval:
+   - Run relevant tests
+   - Compare against baseline
+   - Record PASS/FAIL
+4. Report current status:
+
+```
+EVAL CHECK: feature-name
+========================
+Capability: X/Y passing
+Regression: X/Y passing
+Status: IN PROGRESS / READY
+```
+
+## Report Evals
+
+`/eval report feature-name`
+
+Generate comprehensive eval report:
+
+```
+EVAL REPORT: feature-name
+=========================
+Generated: $(date)
+
+CAPABILITY EVALS
+----------------
+[eval-1]: PASS (pass@1)
+[eval-2]: PASS (pass@2) - required retry
+[eval-3]: FAIL - see notes
+
+REGRESSION EVALS
+----------------
+[test-1]: PASS
+[test-2]: PASS
+[test-3]: PASS
+
+METRICS
+-------
+Capability pass@1: 67%
+Capability pass@3: 100%
+Regression pass^3: 100%
+
+NOTES
+-----
+[Any issues, edge cases, or observations]
+
+RECOMMENDATION
+--------------
+[SHIP / NEEDS WORK / BLOCKED]
+```
+
+## List Evals
+
+`/eval list`
+
+Show all eval definitions:
+
+```
+EVAL DEFINITIONS
+================
+feature-auth      [3/5 passing] IN PROGRESS
+feature-search    [5/5 passing] READY
+feature-export    [0/4 passing] NOT STARTED
+```
+
+## Arguments
+
+$ARGUMENTS:
+- `define <name>` - Create new eval definition
+- `check <name>` - Run and check evals
+- `report <name>` - Generate full report
+- `list` - Show all evals
+- `clean` - Remove old eval logs (keeps last 10 runs)

package/package-source/docs/codex-harness-engineering/templates/package-assets/skills/eval-harness/SKILL.md

@@ -0,0 +1,226 @@
+---
+name: eval-harness
+description: A formal evaluation framework implementing eval-driven development principles.
+---
+
+# Eval Harness Skill
+
+A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.
+
+## Philosophy
+
+Eval-Driven Development treats evals as the "unit tests of AI development":
+- Define expected behavior BEFORE implementation
+- Run evals continuously during development
+- Track regressions with each change
+- Use pass@k metrics for reliability measurement
+
+## Eval Types
+
+### Capability Evals
+Test if Claude can do something it couldn't before:
+```markdown
+[CAPABILITY EVAL: feature-name]
+Task: Description of what Claude should accomplish
+Success Criteria:
+  - [ ] Criterion 1
+  - [ ] Criterion 2
+  - [ ] Criterion 3
+Expected Output: Description of expected result
+```
+
+### Regression Evals
+Ensure changes don't break existing functionality:
+```markdown
+[REGRESSION EVAL: feature-name]
+Baseline: SHA or checkpoint name
+Tests:
+  - existing-test-1: PASS/FAIL
+  - existing-test-2: PASS/FAIL
+  - existing-test-3: PASS/FAIL
+Result: X/Y passed (previously Y/Y)
+```
+
+## Grader Types
+
+### 1. Code-Based Grader
+Deterministic checks using code:
+```bash
+# Check if file contains expected pattern
+grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
+
+# Check if tests pass
+npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
+
+# Check if build succeeds
+npm run build && echo "PASS" || echo "FAIL"
+```
+
+### 2. Model-Based Grader
+Use Claude to evaluate open-ended outputs:
+```markdown
+[MODEL GRADER PROMPT]
+Evaluate the following code change:
+1. Does it solve the stated problem?
+2. Is it well-structured?
+3. Are edge cases handled?
+4. Is error handling appropriate?
+
+Score: 1-5 (1=poor, 5=excellent)
+Reasoning: [explanation]
+```
+
+### 3. Human Grader
+Flag for manual review:
+```markdown
+[HUMAN REVIEW REQUIRED]
+Change: Description of what changed
+Reason: Why human review is needed
+Risk Level: LOW/MEDIUM/HIGH
+```
+
+## Metrics
+
+### pass@k
+"At least one success in k attempts"
+- pass@1: First attempt success rate
+- pass@3: Success within 3 attempts
+- Typical target: pass@3 > 90%
+
+### pass^k
+"All k trials succeed"
+- Higher bar for reliability
+- pass^3: 3 consecutive successes
+- Use for critical paths
+
+## Eval Workflow
+
+### 1. Define (Before Coding)
+```markdown
+## EVAL DEFINITION: feature-xyz
+
+### Capability Evals
+1. Can create new user account
+2. Can validate email format
+3. Can hash password securely
+
+### Regression Evals
+1. Existing login still works
+2. Session management unchanged
+3. Logout flow intact
+
+### Success Metrics
+- pass@3 > 90% for capability evals
+- pass^3 = 100% for regression evals
+```
+
+### 2. Implement
+Write code to pass the defined evals.
+
+### 3. Evaluate
+```bash
+# Run capability evals
+[Run each capability eval, record PASS/FAIL]
+
+# Run regression evals
+npm test -- --testPathPattern="existing"
+
+# Generate report
+```
+
+### 4. Report
+```markdown
+EVAL REPORT: feature-xyz
+========================
+
+Capability Evals:
+  create-user:     PASS (pass@1)
+  validate-email:  PASS (pass@2)
+  hash-password:   PASS (pass@1)
+  Overall:         3/3 passed
+
+Regression Evals:
+  login-flow:      PASS
+  session-mgmt:    PASS
+  logout-flow:     PASS
+  Overall:         3/3 passed
+
+Metrics:
+  pass@1: 67% (2/3)
+  pass@3: 100% (3/3)
+
+Status: READY FOR REVIEW
+```
+
+## Integration Patterns
+
+### Pre-Implementation
+```
+/eval define feature-name
+```
+Creates eval definition file at `.claude/evals/feature-name.md`
+
+### During Implementation
+```
+/eval check feature-name
+```
+Runs current evals and reports status
+
+### Post-Implementation
+```
+/eval report feature-name
+```
+Generates full eval report
+
+## Eval Storage
+
+Store evals in project:
+```
+.claude/
+  evals/
+    feature-xyz.md      # Eval definition
+    feature-xyz.log     # Eval run history
+    baseline.json       # Regression baselines
+```
+
+## Best Practices
+
+1. **Define evals BEFORE coding** - Forces clear thinking about success criteria
+2. **Run evals frequently** - Catch regressions early
+3. **Track pass@k over time** - Monitor reliability trends
+4. **Use code graders when possible** - Deterministic > probabilistic
+5. **Human review for security** - Never fully automate security checks
+6. **Keep evals fast** - Slow evals don't get run
+7. **Version evals with code** - Evals are first-class artifacts
+
+## Example: Adding Authentication
+
+```markdown
+## EVAL: add-authentication
+
+### Phase 1: Define (10 min)
+Capability Evals:
+- [ ] User can register with email/password
+- [ ] User can login with valid credentials
+- [ ] Invalid credentials rejected with proper error
+- [ ] Sessions persist across page reloads
+- [ ] Logout clears session
+
+Regression Evals:
+- [ ] Public routes still accessible
+- [ ] API responses unchanged
+- [ ] Database schema compatible
+
+### Phase 2: Implement (varies)
+[Write code]
+
+### Phase 3: Evaluate
+Run: /eval check add-authentication
+
+### Phase 4: Report
+EVAL REPORT: add-authentication
+==============================
+Capability: 5/5 passed (pass@3: 100%)
+Regression: 3/3 passed (pass^3: 100%)
+Status: SHIP IT
+```

package/package-source/docs/codex-harness-engineering/templates/package-assets/skills/example-skill/README.md

@@ -0,0 +1,37 @@
+# Example Skill
+
+这是一个示例skill，展示了完整的skill文件结构。
+
+## 目录结构
+
+```
+example-skill/
+├── SKILL.md              # 主要技能文档（必需）
+├── README.md             # 本文件
+├── scripts/              # 辅助脚本目录
+│   └── helper.py         # 示例辅助脚本
+├── examples/             # 使用示例
+│   └── usage.md          # 使用示例文档
+└── resources/            # 资源文件
+    └── config.json       # 配置模板
+```
+
+## 使用方法
+
+1. 查看 `SKILL.md` 了解技能详细说明
+2. 参考 `examples/usage.md` 查看使用示例
+3. 使用 `scripts/` 目录中的辅助工具
+4. 根据 `resources/` 中的模板进行配置
+
+## 快速开始
+
+```bash
+# 查看主要文档
+cat SKILL.md
+
+# 运行示例脚本
+python scripts/helper.py
+
+# 查看使用示例
+cat examples/usage.md
+```