@aion0/forge 0.6.1 → 0.8.0

package/docs/Jobs-Design.md

@@ -0,0 +1,240 @@
+# Jobs — scheduled connector polls that fan out to pipelines or chat
+
+## What a Job is
+
+A Job is a new Forge primitive (alongside Task and Pipeline). It periodically
+calls a **connector tool** and, for every new item in the result, **dispatches**
+to either a Pipeline run or a Chat session.
+
+Typical use cases:
+- Every 30 min: `teams.list_messages_in_current_chat` → for each unread → POST
+  to a chat session that asks the assistant to summarize and reply.
+- Every hour: `mantis.search_bugs query="version:25.4 status:open"` → for each
+  new bug → trigger the `bug-triage` pipeline bound to project `my-app` with
+  `bug_id`, `summary` as params.
+
+Job vs the other two:
+
+| Concept | Triggered by | Does what |
+|---|---|---|
+| Task (existing) | manual / Telegram / API | one CLI-agent invocation |
+| Pipeline (existing) | manual / schedule / GitHub issue / **Job** | DAG of CLI-agent tasks |
+| **Job** (new) | scheduler tick (every `schedule_interval_minutes`) | calls one connector tool, dedups, dispatches per item |
+
+The CLI keeps `forge task` / `forge pipeline` unchanged. A new `forge jobs`
+subcommand handles Job CRUD + manual trigger.
+
+## Storage (all sqlite)
+
+The Job definition lives in sqlite — no yaml files — so the extension can
+create / edit / delete jobs without writing to disk.
+
+```sql
+CREATE TABLE IF NOT EXISTS jobs (
+  id                          TEXT PRIMARY KEY,         -- short uuid
+  name                        TEXT NOT NULL,
+  enabled                     INTEGER NOT NULL DEFAULT 1,
+  schedule_interval_minutes   INTEGER NOT NULL DEFAULT 30,
+
+  -- Source: which connector tool to call each tick
+  source_connector            TEXT NOT NULL,            -- plugin_id, e.g. 'mantis'
+  source_tool                 TEXT NOT NULL,            -- tool name, e.g. 'search_bugs'
+  source_input                TEXT NOT NULL DEFAULT '{}', -- JSON: passed verbatim as tool input
+
+  -- How to find items in the tool result + dedup key
+  items_path                  TEXT,                     -- dotted path; empty = whole result if array
+  dedup_field                 TEXT NOT NULL,            -- dotted path on each item, e.g. 'id' or 'meta.uuid'
+
+  -- Dispatch (one of two modes)
+  dispatch_type               TEXT NOT NULL,            -- 'pipeline' | 'chat'
+  dispatch_params             TEXT NOT NULL DEFAULT '{}', -- JSON; shape depends on type — see below
+
+  -- Bookkeeping
+  last_run_at                 TEXT,
+  next_run_at                 TEXT,
+  created_at                  TEXT NOT NULL DEFAULT (datetime('now')),
+  updated_at                  TEXT NOT NULL DEFAULT (datetime('now'))
+);
+
+CREATE TABLE IF NOT EXISTS job_runs (
+  id              TEXT PRIMARY KEY,
+  job_id          TEXT NOT NULL REFERENCES jobs(id) ON DELETE CASCADE,
+  started_at      TEXT NOT NULL DEFAULT (datetime('now')),
+  finished_at     TEXT,
+  status          TEXT NOT NULL DEFAULT 'running',  -- running | ok | error
+  items_seen      INTEGER NOT NULL DEFAULT 0,
+  items_new       INTEGER NOT NULL DEFAULT 0,
+  items_dispatched INTEGER NOT NULL DEFAULT 0,
+  error           TEXT,
+  trigger         TEXT NOT NULL DEFAULT 'schedule'  -- schedule | manual
+);
+
+CREATE TABLE IF NOT EXISTS job_seen (
+  job_id          TEXT NOT NULL REFERENCES jobs(id) ON DELETE CASCADE,
+  dedup_key       TEXT NOT NULL,
+  first_seen_at   TEXT NOT NULL DEFAULT (datetime('now')),
+  PRIMARY KEY (job_id, dedup_key)
+);
+
+CREATE TABLE IF NOT EXISTS job_dispatches (
+  id                  TEXT PRIMARY KEY,
+  job_run_id          TEXT NOT NULL REFERENCES job_runs(id) ON DELETE CASCADE,
+  item_key            TEXT NOT NULL,
+  item_preview        TEXT,                       -- short rendering for the UI
+  dispatch_type       TEXT NOT NULL,              -- mirrored from job
+  dispatch_target_id  TEXT,                       -- pipeline_id OR chat session_id
+  status              TEXT NOT NULL DEFAULT 'dispatched',  -- dispatched | error
+  error               TEXT,
+  created_at          TEXT NOT NULL DEFAULT (datetime('now'))
+);
+
+CREATE INDEX IF NOT EXISTS idx_job_runs_job ON job_runs(job_id, started_at DESC);
+CREATE INDEX IF NOT EXISTS idx_job_dispatches_run ON job_dispatches(job_run_id, created_at DESC);
+```
+
+## dispatch_params shape
+
+### `dispatch_type = 'pipeline'`
+
+```json
+{
+  "workflow_name": "bug-triage",
+  "project_path": "/Users/me/code/my-app",
+  "project_name": "my-app",
+  "input_template": {
+    "bug_id": "{{item.id}}",
+    "summary": "{{item.summary}}",
+    "priority": "{{item.priority}}"
+  }
+}
+```
+
+Per item: render `input_template` (string substitution `{{item.<path>}}`), then
+call existing `triggerPipeline(project_path, project_name, workflow_name,
+rendered_input, dedup_key=item_key)`.
+
+### `dispatch_type = 'chat'`
+
+```json
+{
+  "agent_profile": "my-litellm",            // optional; falls back to settings.chatAgent
+  "session_title_template": "Bug {{item.id}}: {{item.summary}}",
+  "message_template": "Process Mantis bug {{item.id}} (priority {{item.priority}}).\nSummary: {{item.summary}}\n\nFind the relevant branch and prepare a triage note.",
+  "reuse_session": false                    // if true, all items append to one session; else fresh session per item
+}
+```
+
+Per item: POST `/api/sessions` (or reuse) on chat-standalone, then POST a
+message rendered from `message_template`. The chat agent gets the item and
+decides what tools to call (could be another connector call, code edit via
+shell, etc.).
+
+## Template syntax
+
+Values inside `{{ ... }}` use dotted paths into the item:
+- `{{item.id}}` → item.id
+- `{{item.assignee.name}}` → item.assignee.name
+- `{{item}}` → JSON.stringify(item) (for dumping into a chat message)
+
+Missing paths render to empty string. Non-string values render via
+`String(...)` (numbers/booleans) or `JSON.stringify(...)` (objects).
+
+Only `{{item.*}}` is supported in v1. `{{now}}` / `{{settings.*}}` are not
+needed yet.
+
+## Scheduler
+
+Runs inside the same Next.js worker as `pipeline-scheduler` (started in
+`lib/init.ts`). Every 60s:
+
+1. Load all `enabled = 1` jobs where `next_run_at IS NULL OR next_run_at <= now`.
+2. For each: insert a `job_runs` row, dispatch the tick asynchronously, advance
+   `next_run_at = now + interval_minutes`, set `last_run_at = now`.
+3. Per-tick work happens in a background promise (HTTP / SSE / spawn don't block
+   the scheduler loop).
+
+A tick is **idempotent** — if the previous tick is still running for the same
+job (run row marked `status = 'running'` with no `finished_at`), the next tick
+skips it and logs.
+
+## A tick in detail
+
+1. Resolve the connector + tool definition via `lib/plugins/registry` (same
+   path the chat dispatcher uses).
+2. Call `dispatchTool({id: '<job-run-id>', name: '<connector>.<tool>', input:
+   source_input})` from `lib/chat/tool-dispatcher`. This handles `http` /
+   `shell` / `browser` uniformly. (Browser-protocol jobs require the extension
+   bridge to be connected at tick time — if not, surface as run error.)
+3. Parse the tool's `content` string as JSON. Apply `items_path` to find the
+   array (e.g. `bugs` → `result.bugs`). If parsed value is already an array,
+   use it directly.
+4. For each item:
+   - Compute `key = pickPath(item, dedup_field)`. Stringify.
+   - `INSERT OR IGNORE INTO job_seen (job_id, dedup_key)` — if a new row was
+     written, it's a new item.
+   - For new items only: dispatch via the configured type, insert into
+     `job_dispatches` with the resulting target id.
+5. Update `job_runs` row with counts + finished_at + status.
+
+## API surface
+
+```
+GET    /api/jobs                       list (enabled+disabled)
+POST   /api/jobs                       create
+GET    /api/jobs/:id                   detail (with last 20 runs)
+PATCH  /api/jobs/:id                   partial update
+DELETE /api/jobs/:id                   cascade delete
+POST   /api/jobs/:id/run               manual fire-now (returns run id)
+GET    /api/jobs/:id/runs?limit=20     run history
+GET    /api/jobs/:id/runs/:run_id      run detail (with dispatches)
+POST   /api/jobs/:id/reset_dedup       wipe job_seen rows (force re-process all)
+```
+
+All gated by the existing Forge middleware (X-Forge-Token).
+
+## CLI
+
+```
+forge jobs                              list
+forge jobs new <interactive prompts>    create
+forge jobs run <id>                     fire now
+forge jobs runs <id>                    history
+forge jobs rm <id>
+forge jobs disable <id> / enable <id>
+forge jobs reset <id>                   wipe dedup state
+```
+
+## Extension UI (Tasks → Jobs)
+
+`TasksTab.tsx` renames to `JobsTab.tsx`. Tab label "Tasks" → "Jobs".
+
+Sections:
+1. **Active jobs** — for each: name, schedule, last run time + items new/seen,
+   next run countdown, enable toggle, "Run now" button.
+2. **+ New job** — modal:
+   - name, interval
+   - source: connector dropdown (from `/api/connectors`), tool dropdown
+     (populated from selected connector's tools), input JSON
+   - items_path + dedup_field (with examples per connector)
+   - dispatch type radio: pipeline | chat
+     - pipeline: workflow dropdown (`/api/pipelines/workflows`), project
+       dropdown (`/api/projects`), input_template builder
+     - chat: agent profile dropdown (`/api/agents`), session_title template,
+       message_template (textarea), reuse_session toggle
+3. **Recent runs** (collapsed by default) — last 20 across all jobs.
+
+Per architecture rule: extension is display + DOM connectors only. All
+scheduling, dedup, dispatch happens server-side. The extension talks to Forge
+HTTP only — no local state for jobs.
+
+## Out of scope (v1)
+
+- Cron expressions (interval-only)
+- Item filtering before dispatch (just write the connector tool to return
+  filtered)
+- Conditional dispatch (always-fire on new item; for skip-logic, use the
+  pipeline / chat to decide internally)
+- Webhook triggers (still poll-based)
+- Backfill controls (initial tick auto-marks everything as seen without
+  dispatching, so a brand-new job doesn't fire on history — controlled by
+  `mark_existing_as_seen` flag at create time, default true)

package/docs/LOCAL-DEPLOY.md

@@ -90,14 +90,14 @@ caffeinate -d -i -s &
 ### 注册为系统服务（开机自启 + 崩溃自重启）
 
 ```xml
-<!-- ~/Library/LaunchAgents/com.zliu.my-workflow.plist -->
+<!-- ~/Library/LaunchAgents/com.forge.my-workflow.plist -->
 <?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
   "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 <plist version="1.0">
 <dict>
     <key>Label</key>
-    <string>com.zliu.my-workflow</string>
+    <string>com.forge.my-workflow</string>
     <key>ProgramArguments</key>
     <array>
         <string>/usr/local/bin/mw</string>
@@ -119,7 +119,7 @@ caffeinate -d -i -s &
 
 ```bash
 # 注册服务
-launchctl load ~/Library/LaunchAgents/com.zliu.my-workflow.plist
+launchctl load ~/Library/LaunchAgents/com.forge.my-workflow.plist
 
 # 查看状态
 launchctl list | grep my-workflow