djobs 0.1.0__tar.gz

djobs-0.1.0/.github/agents/durable-coder.agent.md

@@ -0,0 +1,108 @@
+---
+name: Durable Coder
+description: "Durable Coder — a crash-proof coding agent. Use when you have multi-file tasks like 'add docstrings to all files', 'refactor all tests', 'add type hints everywhere'. Work survives IDE crashes and session interruptions — pick up exactly where you left off."
+tools:
+  - djobs/*
+  - read
+  - edit
+  - search
+  - execute
+  - agent
+---
+
+# Durable Coder Agent
+
+You are a crash-proof coding agent. When the user gives you a multi-file task,
+you use a durable job queue to **checkpoint your progress**, so if the session
+crashes or the user closes the chat, the work can be resumed from where it stopped.
+
+The user does NOT interact with the queue directly. You manage it transparently.
+**Never ask the user whether to resume — just do it.**
+
+## Core Behavior
+
+### 1. Session Start — Silent Resume
+
+At the start of EVERY conversation, call `resume_session` with
+`correlation_id` = the workspace folder path. **Do this silently.**
+
+- If incomplete tasks exist AND the user's message is vague (e.g. "hi", "繼續", "continue"):
+  → **Automatically resume.** Say "接續上次未完成的工作（N 個檔案），繼續處理。" then start working.
+- If incomplete tasks exist BUT the user gives a NEW specific request:
+  → Ignore old tasks, start the new request. Old tasks stay in queue but don't block.
+- If no incomplete tasks → proceed normally with whatever the user asked.
+
+**NEVER ask "要繼續嗎？" — that wastes a turn. Just do it.**
+
+### 2. Multi-File Tasks — Checkpoint Automatically
+
+When the user asks you to do something across multiple files (>3 files):
+
+1. **Scan** — find all target files.
+2. **Plan** — tell the user: "找到 N 個檔案，開始處理。" then immediately start.
+3. **Enqueue** — create one task per file via `enqueue_task` (this is the checkpoint).
+4. **Execute** — process each task yourself:
+   - Read the file.
+   - Make the edit.
+   - Call `complete_task(task_id)` on success.
+   - Call `fail_task(task_id, error)` if the edit fails.
+5. **Report** — after each batch: "[3/12] ✓ src/djobs/core/models.py"
+
+### 3. Crash Recovery — Seamless Auto-Resume
+
+If the session is interrupted and the user opens a new chat:
+
+1. `resume_session` finds unfinished tasks.
+2. If user's message is vague → auto-resume immediately, no questions asked.
+3. If user gives a new request → do the new request instead.
+
+### 4. Small Tasks — Just Do It
+
+If the task involves ≤3 files, skip the queue and do it directly.
+Don't over-engineer simple requests.
+
+### 5. "What did you do?" — Use audit_log
+
+When the user asks about past work — "what changed yesterday?", "did any tasks
+fail?", "summarise today's AI work" — call `audit_log` instead of guessing:
+
+- `audit_log()` — 24h summary (task counts, failure list, event breakdown).
+- `audit_log(summary=False, limit=50)` — recent event timeline.
+- `audit_log(event_type="job_failed")` — just the failures.
+- `audit_log(correlation_id=<workspace>)` — scoped to this workspace.
+
+This gives accurate answers from the event log, not from memory.
+
+## Rules
+
+- **correlation_id**: Always use the workspace folder path.
+- **idempotency_key**: Use `"{task_type}:{file_path}"` to prevent duplicates.
+- **Lifecycle**: After editing each file, call `complete_task(task_id)`. On error, call `fail_task(task_id, error)`. This keeps `resume_session` and `audit_log` accurate.
+- **Progress**: Print `[n/total] ✓ file` after each file completes.
+- **Transparency**: Never ask the user to call queue tools. You are the worker.
+
+## Example
+
+```
+User: "幫所有 Python 檔案加 docstring"
+
+Agent:
+1. resume_session (silently) → no incomplete tasks
+2. Scan src/ → finds 12 .py files
+3. "找到 12 個 Python 檔案，開始加 docstring。"
+4. enqueue all 12 + immediately start processing
+5. "[1/12] ✓ src/djobs/core/models.py"
+   "[2/12] ✓ src/djobs/core/states.py"
+   ... (no pauses, no questions) ...
+6. "全部完成！12/12 檔案已加上 docstring。"
+```
+
+If session crashes after file 7, user opens new chat:
+
+```
+User: "hi"
+Agent:
+1. resume_session → found 5 incomplete tasks
+2. "接續上次未完成的 docstring 工作（5/12），繼續處理。"
+3. Immediately continues from file 8 — no questions asked.
+```

djobs-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,29 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - run: pip install ruff
+      - run: ruff check src/ tests/
+      - run: ruff format --check src/ tests/
+
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - run: pip install -e ".[dev,mcp]"
+      - run: pytest -q --tb=short

djobs-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,28 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+      contents: read
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - run: python -m pip install --upgrade build twine
+
+      - run: python -m build
+      - run: python -m twine check dist/*
+
+      - uses: pypa/gh-action-pypi-publish@release/v1

djobs-0.1.0/.gitignore

@@ -0,0 +1,41 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+*.egg
+
+# Virtual environment
+.venv/
+
+# SQLite databases
+*.db
+*.sqlite
+*.sqlite3
+
+# IDE
+.vscode/*
+!.vscode/mcp.json
+!.vscode/settings.json
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Environment
+.env
+.env.*
+
+# Test / coverage
+.pytest_cache/
+htmlcov/
+.coverage
+coverage.xml
+
+# Ruff
+.ruff_cache/
+
+# Distribution
+*.whl
+*.tar.gz

djobs-0.1.0/.vscode/mcp.json

@@ -0,0 +1,22 @@
+{
+  "servers": {
+    "djobs": {
+      "type": "stdio",
+      "command": "${workspaceFolder}/.venv/Scripts/python.exe",
+      "args": ["-m", "djobs.mcp_server"],
+      "env": {
+        "PYTHONPATH": "${workspaceFolder}/src"
+      },
+      "autoApprove": [
+        "health",
+        "resume_session",
+        "check_task",
+        "complete_task",
+        "fail_task",
+        "list_tasks",
+        "enqueue_task",
+        "audit_log"
+      ]
+    }
+  }
+}

djobs-0.1.0/.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+    "vscode-nmake-tools.workspaceBuildDirectories": []
+}

djobs-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,48 @@
+# Contributing to djobs
+
+Thanks for your interest! Here's how to get started.
+
+## Development Setup
+
+```bash
+git clone https://github.com/jhuang-tw/djobs.git
+cd djobs
+python -m venv .venv
+.venv/bin/activate        # Windows: .venv\Scripts\activate
+pip install -e ".[dev,mcp]"
+```
+
+## Running Tests
+
+```bash
+pytest -q                 # all tests (skips Postgres unless available)
+pytest tests/unit -q      # unit tests only
+ruff check src/ tests/    # lint
+ruff format src/ tests/   # auto-format
+```
+
+## Pull Request Guidelines
+
+1. Fork the repo and create a feature branch from `main`.
+2. Keep commits focused — one logical change per commit.
+3. Add or update tests for any new behavior.
+4. Ensure `pytest -q` and `ruff check` pass before submitting.
+5. Write a clear PR description explaining *what* and *why*.
+
+## Code Style
+
+- Python 3.13+, type hints everywhere.
+- `ruff` for linting and formatting (config in `pyproject.toml`).
+- `src` layout — all package code lives under `src/djobs/`.
+- Tests mirror the source tree: `tests/unit/`, `tests/integration/`.
+
+## Architecture
+
+See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for module boundaries and design decisions.
+
+## Reporting Issues
+
+Open a GitHub issue with:
+- What you expected vs. what happened.
+- Minimal reproduction steps.
+- Python version and OS.

djobs-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 djobs contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

djobs-0.1.0/PKG-INFO

@@ -0,0 +1,261 @@
+Metadata-Version: 2.4
+Name: djobs
+Version: 0.1.0
+Summary: SQLite-backed durable task queue with first-class MCP integration, purpose-built for AI coding agents (crash recovery, audit trail, embedded daemon).
+Project-URL: Homepage, https://github.com/jhuang-tw/djobs
+Project-URL: Repository, https://github.com/jhuang-tw/djobs
+Project-URL: Issues, https://github.com/jhuang-tw/djobs/issues
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ai-agent,audit-log,crash-recovery,durable,job-queue,mcp,sqlite,task-queue
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.13
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Provides-Extra: mcp
+Requires-Dist: mcp[cli]>=1.0; extra == 'mcp'
+Provides-Extra: pg
+Requires-Dist: psycopg[binary]>=3.1; extra == 'pg'
+Description-Content-Type: text/markdown
+
+# djobs
+
+**A SQLite-backed durable task queue with first-class MCP integration**, purpose-built for AI coding agents that need crash recovery, audit trails, and zero infrastructure.
+
+[![CI](https://github.com/jhuang-tw/djobs/actions/workflows/ci.yml/badge.svg)](https://github.com/jhuang-tw/djobs/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+[![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
+
+---
+
+## Why djobs?
+
+AI coding agents (GitHub Copilot, Cursor, Cline, etc.) often run multi-file tasks that can take several minutes. When the IDE crashes or the chat disconnects mid-way, in-flight progress is usually lost because the agent's state lives only in chat history.
+
+djobs gives agents a small, durable checkpoint queue so they can resume exactly where they stopped:
+
+```
+Agent: "Add docstrings to 12 files"
+  -> enqueue 12 tasks (crash-safe checkpoint)
+  -> edit file -> complete_task
+  -> edit file -> complete_task
+  -> ... IDE crashes after file 7 ...
+
+New chat: "hi"
+  -> resume_session -> 5 incomplete tasks found
+  -> auto-resume from file 8 — no questions asked
+```
+
+Under the hood it is a fairly conventional durable job queue (state machine, retry policy, lease, scheduler, event log). The interesting part is how it is wired to AI agents: an MCP server with `enqueue_task` / `complete_task` / `resume_session` / `audit_log`, plus an embedded background daemon, plus a `type_filter` so daemon-managed jobs and agent-managed jobs do not fight over the same queue.
+
+### What is in the box
+
+| Area | What you get |
+|------|--------------|
+| **MCP server** | 8 tools exposed via FastMCP / stdio — works in VS Code, Claude Desktop, etc. |
+| **Crash recovery** | `resume_session` returns incomplete tasks for a given workspace / correlation id |
+| **Audit trail** | `audit_log` aggregates `job_events` so you can answer "what did the AI do yesterday?" |
+| **Type isolation** | Built-in daemon only claims job types it has handlers for; AI-only types are left to the agent via `complete_task` / `fail_task` |
+| **SQLite first** | No Redis, RabbitMQ, Docker, or Postgres required for local use |
+| **Postgres path** | Same `JobRepository` protocol implemented on top of `SELECT ... FOR UPDATE SKIP LOCKED` for multi-worker setups |
+| **Test coverage** | 214 passing tests (16 skipped without Postgres), strict ruff lint |
+
+---
+
+## Quick Start
+
+### As a Python Library
+
+```bash
+pip install djobs
+```
+
+```python
+from djobs import SQLiteJobRepository, QueueService, HandlerRegistry, WorkerPool
+
+# 1. Set up
+repo = SQLiteJobRepository.from_path("jobs.db")
+queue = QueueService(repo)
+
+# 2. Submit a job
+job = queue.submit("send_email", {"to": "user@example.com"}, max_attempts=3)
+
+# 3. Process jobs
+registry = HandlerRegistry()
+registry.register("send_email", lambda payload: send_email(**payload))
+
+pool = WorkerPool(queue, registry, worker_id="worker-1", max_concurrent=4)
+pool.run_loop(stop_event)
+```
+
+### As an MCP Server (for AI Agents)
+
+Add to `.vscode/mcp.json`:
+
+```json
+{
+  "servers": {
+    "djobs": {
+      "type": "stdio",
+      "command": "python",
+      "args": ["-m", "djobs.mcp_server"],
+      "autoApprove": [
+        "health", "resume_session", "check_task", "complete_task",
+        "fail_task", "list_tasks", "enqueue_task", "audit_log"
+      ]
+    }
+  }
+}
+```
+
+Then any AI agent can call these MCP tools:
+
+| Tool | Purpose |
+|------|---------|
+| `enqueue_task` | Submit a durable task (survives crashes) |
+| `complete_task` | Mark task succeeded after agent finishes work |
+| `fail_task` | Mark task failed with error message |
+| `resume_session` | Find incomplete tasks from previous sessions |
+| `check_task` | Inspect task status, attempts, duration |
+| `list_tasks` | List tasks by correlation_id |
+| `audit_log` | Query event history — "what did the AI do?" |
+| `health` | Queue depth by status |
+
+---
+
+## How is this different from X?
+
+djobs is not the first project to expose a task queue to an AI agent over MCP. It targets a specific combination of properties: SQLite-first, MCP-driven, with crash recovery and audit-log style observability built in.
+
+| Project | Storage | Focus | Closest to djobs? |
+|---------|---------|-------|-------------------|
+| [TadMSTR/task-queue-mcp](https://github.com/TadMSTR/task-queue-mcp) | YAML files | Multi-agent task hand-off for Claude Code | Closest in spirit. Different storage model (YAML files + dispatcher), no `resume_session` / `audit_log` style observability. |
+| [midweste/mcp-cli-gateway](https://github.com/midweste/mcp-cli-gateway) | SQLite | Routing prompts to CLI agents (Gemini / Codex / Claude) with pacing | Overlaps on persistence + observability, but the unit of work is "dispatch a prompt to a CLI", not "durable user task with retry / lease". |
+| [j0j1j2/claude-tunnel](https://github.com/j0j1j2/claude-tunnel) | In-memory | Pub/sub + 1:1 request/reply + job queue between Claude Code sessions | Different problem: inter-session messaging, not durable work tracking. |
+| Celery / RQ / Dramatiq / Hatchet | Redis / Postgres | General-purpose distributed task queues | Strictly more capable as general queues, but not designed to be driven directly by an AI agent over MCP. |
+| Temporal / Inngest / DBOS | Server / SaaS | Durable workflow / execution engines | Much more powerful and much heavier; no MCP integration; not aimed at single-developer laptop use. |
+
+In one sentence: **djobs is what you reach for when you want a small, Celery-shaped Python job queue, driven mostly by an AI agent through MCP, on a single developer machine, with SQLite.**
+
+---
+
+## Architecture
+
+```
+┌─────────────┐     MCP tools      ┌──────────────┐
+│  AI Agent   │ ──────────────────> │  MCP Server  │
+│  (Copilot)  │ <────────────────── │  (FastMCP)   │
+└─────────────┘                     └──────┬───────┘
+                                           │
+                              ┌────────────┼────────────┐
+                              │            │            │
+                        ┌─────▼─────┐ ┌────▼────┐ ┌────▼─────┐
+                        │  Queue    │ │ Daemon  │ │ Audit    │
+                        │  Service  │ │ (Pool + │ │ Log      │
+                        │           │ │ Sched)  │ │          │
+                        └─────┬─────┘ └─────────┘ └──────────┘
+                              │
+                        ┌─────▼─────┐
+                        │  SQLite   │
+                        │  (or PG)  │
+                        └───────────┘
+```
+
+### Job State Machine
+
+```
+pending ──────► running ──────► succeeded
+   │               │
+   │               ├──────► failed
+   │               │
+   │               ├──────► retry_scheduled ──► pending (retry)
+   │               │
+   │               └──────► dead_lettered
+   │
+   ├──────► succeeded  (AI agent direct complete)
+   └──────► failed     (AI agent direct fail)
+```
+
+### Module Map
+
+| Module | Responsibility |
+|--------|---------------|
+| `djobs.core` | Job model, state machine, domain errors |
+| `djobs.queue` | Submit, claim, complete, fail, retry logic |
+| `djobs.storage` | SQLite & PostgreSQL repositories, event log |
+| `djobs.worker` | Handler registry, WorkerPool, WorkerRunner |
+| `djobs.scheduler` | Retry promotion, expired lease recovery |
+| `djobs.daemon` | Composes WorkerPool + Scheduler into one process |
+| `djobs.observability` | Metrics, structured logging, job inspection |
+| `djobs.mcp_server` | MCP tool definitions, embedded daemon |
+| `djobs.cli` | `djobs serve` CLI entry point |
+
+---
+
+## Examples
+
+```bash
+# Basic job lifecycle
+python examples/run_echo_job.py
+
+# Retry with exponential backoff
+python examples/run_retry_job.py
+
+# Concurrent worker pool
+python examples/run_pool_demo.py
+
+# Scheduler loop (retry promotion + lease recovery)
+python examples/run_scheduler_demo.py
+
+# AI task platform (batch submit + cost tracking)
+python examples/run_ai_demo.py
+
+# Durable crash recovery demo
+python examples/run_durable_demo.py
+```
+
+---
+
+## Development
+
+```bash
+git clone https://github.com/jhuang-tw/djobs.git
+cd djobs
+python -m venv .venv && .venv/bin/activate
+pip install -e ".[dev,mcp]"
+
+pytest -q              # 214 tests (16 skipped without Postgres)
+ruff check src/ tests/ # lint
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+---
+
+## Roadmap
+
+- [x] Durable job queue with retry, lease, heartbeat
+- [x] SQLite + PostgreSQL backends
+- [x] Worker pool with concurrency control
+- [x] Scheduler (retry promotion + lease recovery)
+- [x] Event sourcing & audit trail
+- [x] MCP server with 8 tools
+- [x] Embedded daemon (auto-start with MCP)
+- [x] Type isolation (daemon vs. AI agent tasks)
+- [ ] `pip install djobs` on PyPI
+- [ ] Async worker support
+- [ ] Priority queues
+- [ ] Web dashboard for audit trail
+- [ ] Rate limiting per job type
+
+---
+
+## License
+
+[MIT](LICENSE)