agentexec 0.2.0rc1__tar.gz → 0.3.0__tar.gz

{agentexec-0.2.0rc1 → agentexec-0.3.0}/CHANGELOG.md

@@ -1,5 +1,112 @@
 # Changelog
 
+## v0.3.0
+
+### Breaking Changes
+
+**Permanent task failure raises `TaskFailedError`**
+- Callers that previously relied on `TimeoutError` to detect a task that exhausted its retries now receive `ax.TaskFailedError` as soon as the failure is recorded
+
+### New Features
+
+**Terminal failure contract for `ax.gather` / `ax.get_result`**
+- When a task exhausts its retries, the pool stores a `TaskFailure` record at the task's result key (same convention and TTL as success results)
+- `ax.get_result` raises `ax.TaskFailedError` — carrying `task_name`, `agent_id`, `error`, and `attempts` — on its next poll (~0.5s) instead of polling until the timeout
+- `ax.gather` fails fast on the first permanent failure; remaining tasks keep running and their results stay retrievable, so re-gathering survivors resolves instantly
+- `TaskFailedError` is exported on the `ax` namespace
+
+## v0.2.0
+
+Major refactor of the backend, queue, activity, worker, and database layers.
+If you're upgrading from 0.1.x, read the **Breaking Changes** section closely.
+
+### Breaking Changes
+
+**Fully async database layer**
+- `configure_engine()` and `get_session()` now require an async SQLAlchemy engine (`AsyncEngine`) and return `AsyncSession`
+- Database URLs must use async drivers (e.g. `sqlite+aiosqlite://`, `postgresql+asyncpg://`)
+- `sqlalchemy[asyncio]` is now a core dependency
+
+**Async activity API**
+- All activity functions are async: `await ax.activity.create(...)`, `await ax.activity.update(...)`, `await ax.activity.complete(...)`, `await ax.activity.error(...)`
+- `activity.list()`, `activity.detail()`, and `activity.count_active()` are async and accept `AsyncSession`
+- Activity handlers are async (`async def __call__`)
+- The `session` parameter was removed from activity mutations — the handler owns its own session lifecycle
+
+**Pool entry point**
+- `pool.run()` was removed. Use `await pool.start()` in an asyncio loop, or the new `agentexec run mymodule:pool` CLI
+- `AGENTEXEC_QUEUE_NAME` renamed to `AGENTEXEC_QUEUE_PREFIX` (old name still accepted as alias)
+- `agentexec.state.redis_backend` renamed to `agentexec.state.redis` — update `AGENTEXEC_STATE_BACKEND` if set explicitly
+
+**Task context serialization**
+- `Task.context` is now `Mapping[str, Any]` (raw dict), not a typed BaseModel — hydration happens at execution time
+- `Task.create()` is now async
+
+**Queue backend protocol**
+- `BaseQueueBackend.push()` signature changed from `high_priority: bool` to `priority: Priority | None` — affects Redis, Kafka, and any custom queue backend
+
+**Removed APIs**
+- `set_global_session`/`get_global_session`/`remove_global_session` — use `configure_engine`/`get_session`
+- `state.backend.publish`/`subscribe` (pubsub), `index_add`/`index_range`/`index_remove`, `clear`, `configure`
+- `worker/logging.py` and `core/logging.py` — all modules use stdlib `logging.getLogger(__name__)` directly
+
+### New Features
+
+**CLI entrypoint**
+- New `agentexec` CLI command: `agentexec run mymodule:pool --create-tables --workers 4`
+
+**Partitioned Redis queues**
+- Tasks with `lock_key` route to dedicated partition queues with per-partition locking and SCAN-based fair dequeue
+
+**Activity handler pattern**
+- Pluggable persistence via `PostgresHandler` (default) and `IPCHandler` (worker processes)
+
+**Task retry**
+- Failed tasks requeue as high priority with `AGENTEXEC_MAX_TASK_RETRIES` (default 3)
+
+**Kafka backend (experimental)**
+- `pip install agentexec[kafka]` for queue and schedule via Kafka
+
+**Typed worker IPC**
+- `TaskFailed` and `ActivityEvent` messages flow over `multiprocessing.Queue` with pydantic validation
+
+**Schedule composite keys**
+- `{task_name}:{cron}:{context_hash}` for unique schedule identity
+
+**Activity model `create()` classmethod**
+- `Activity.create()` encapsulates record + initial log entry creation in one async call
+
+**Async engine disposal**
+- `dispose_engine()` ensures the async engine's background threads exit cleanly on shutdown
+
+### Architecture Changes
+
+**Worker pool refactor**
+- Workers use the `spawn` multiprocessing start method with explicit context — no inherited state
+- Event handling and scheduling extracted into `_EventHandler` and `_Scheduler` classes
+- `StateEvent` replaced with stdlib `multiprocessing.Event` — removes dependency on the state backend for shutdown coordination
+- Class-based backend architecture with ABCs (`BaseStateBackend`, `BaseQueueBackend`, `BaseScheduleBackend`)
+- `Task` is pure data, `TaskDefinition` owns behavior
+- Status enum extracted to `activity/status.py` (no SQLAlchemy dependency)
+
+**Logging**
+- All modules use stdlib `logging.getLogger(__name__)`
+- Spawned workers bootstrap a `StreamHandler` on the root logger so logs reach stderr
+- Pool messages use `logger.info`/`logger.error` instead of `print()`
+
+### Bug Fixes
+
+- **Orphaned worker processes on shutdown.** SIGTERM (systemd/docker stop), SIGKILL, and SIGHUP were leaving worker processes running. Fixed via an asyncio SIGTERM handler in the CLI and `prctl(PR_SET_PDEATHSIG)` in each worker so the kernel terminates workers when the pool dies
+- **Worker and scheduler error loops throttled.** Infra failures (e.g. Redis unreachable) were producing 100k+ log lines per second. Added a 1s sleep after outer-loop exceptions
+- **Unregistered task name crash.** Worker now logs an error and skips instead of crashing when it receives a task for an unknown name
+- Failed tasks now log full tracebacks via `logger.exception` instead of `logger.error`
+- Kafka consumer handles `None` message values without crashing
+- `ActivityUpdated.status` is a `Status` enum instead of raw string
+
+### Documentation
+
+- Full documentation sweep for the async API — connection strings, CLI usage, `await` on activity calls across all guides and API references
+
 ## v0.1.7
 
 ### New Features

{agentexec-0.2.0rc1 → agentexec-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentexec
-Version: 0.2.0rc1
+Version: 0.3.0
 Summary: Production-ready orchestration for OpenAI Agents with Redis-backed coordination, activity tracking, and workflow management
 Project-URL: Homepage, https://github.com/Agent-CI/agentexec
 Project-URL: Documentation, https://github.com/Agent-CI/agentexec#readme
@@ -21,7 +21,7 @@ Requires-Dist: openai-agents>=0.1.0
 Requires-Dist: pydantic-settings>=2.5.0
 Requires-Dist: pydantic>=2.12.0
 Requires-Dist: redis>=7.0.1
-Requires-Dist: sqlalchemy>=2.0.44
+Requires-Dist: sqlalchemy[asyncio]>=2.0.44
 Provides-Extra: kafka
 Requires-Dist: aiokafka>=0.11.0; extra == 'kafka'
 Description-Content-Type: text/markdown

agentexec-0.3.0/RELEASE.md

@@ -0,0 +1,91 @@
+# Release Process
+
+This document describes how to publish a new release of agentexec.
+
+## Overview
+
+Creating a **GitHub Release** triggers three publish workflows automatically:
+
+| Workflow | Target | Secret Required |
+|---|---|---|
+| `publish.yml` | PyPI | `PYPI_API_TOKEN` |
+| `docker-publish.yml` | ghcr.io (`agentexec-worker`) | `GITHUB_TOKEN` (built-in) |
+| `npm-publish.yml` | npm (`agentexec-ui`) | `NPM_TOKEN` |
+
+Docker and npm publishing can also be triggered manually via workflow dispatch.
+
+## Steps
+
+### 1. Ensure CI is green
+
+Push all changes to `main` and verify the CI workflow passes (unit tests on
+Python 3.12/3.13 and Kafka integration tests).
+
+### 2. Bump the version
+
+Update the version in `pyproject.toml`:
+
+```toml
+version = "X.Y.Z"      # stable release
+version = "X.Y.ZrcN"   # release candidate
+```
+
+For the UI package, also update `ui/package.json`.
+
+Commit and push:
+
+```bash
+git add pyproject.toml
+git commit -m "Bump version to X.Y.Z"
+git push
+```
+
+### 3. Update CHANGELOG.md
+
+Add a new section at the top of `CHANGELOG.md` with release notes covering
+breaking changes, new features, improvements, bug fixes, and testing updates.
+
+### 4. Tag the release
+
+```bash
+git tag vX.Y.Z
+git push origin vX.Y.Z
+```
+
+### 5. Create the GitHub Release
+
+This is what triggers the publish workflows.
+
+```bash
+gh release create vX.Y.Z --title "vX.Y.Z" --notes-file CHANGELOG.md
+```
+
+For release candidates, mark as a pre-release:
+
+```bash
+gh release create vX.Y.ZrcN --title "vX.Y.ZrcN" --generate-notes --prerelease
+```
+
+### 6. Verify
+
+Check that all three publish workflows succeeded:
+
+```bash
+gh run list --limit 5
+```
+
+- **PyPI**: https://pypi.org/project/agentexec/
+- **Docker**: https://ghcr.io/agent-ci/agentexec-worker
+- **npm**: https://www.npmjs.com/package/agentexec-ui
+
+## Manual Publishing
+
+Docker and npm workflows support manual dispatch for one-off builds:
+
+```bash
+# Docker with a custom tag
+gh workflow run docker-publish.yml -f tag=dev
+
+# npm with a version override
+gh workflow run npm-publish.yml -f version=0.2.0-beta.1
+```

agentexec-0.3.0/docs/api-reference/activity.md

@@ -0,0 +1,401 @@
+# Activity API Reference
+
+This document covers the activity tracking API for monitoring task execution.
+
+All activity functions are `async` and operate on `AsyncSession` from
+`sqlalchemy.ext.asyncio`. The session parameter is optional — when omitted,
+functions fall back to `get_session()` from `agentexec.core.db`, which
+requires `configure_engine()` to have been called.
+
+## Module: agentexec.activity
+
+```python
+import agentexec as ax
+
+# Lifecycle (async)
+await ax.activity.create(task_name, message)
+await ax.activity.update(agent_id, message)
+await ax.activity.complete(agent_id)
+await ax.activity.error(agent_id)
+await ax.activity.cancel_pending()
+
+# Query (async)
+await ax.activity.list()
+await ax.activity.detail(agent_id=...)
+await ax.activity.count_active()
+```
+
+---
+
+## create()
+
+Create a new activity record with an initial `QUEUED` log entry.
+
+```python
+async def create(
+    task_name: str,
+    message: str = "Agent queued",
+    agent_id: str | uuid.UUID | None = None,
+    metadata: dict[str, Any] | None = None,
+) -> uuid.UUID
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `task_name` | `str` | required | Name of the task |
+| `message` | `str` | `"Agent queued"` | Initial log message |
+| `agent_id` | `str \| UUID \| None` | `None` | Pre-generated ID (auto-generated if omitted) |
+| `metadata` | `dict \| None` | `None` | Arbitrary metadata attached to the activity |
+
+### Returns
+
+`uuid.UUID` — the activity's `agent_id`.
+
+### Example
+
+```python
+import agentexec as ax
+
+agent_id = await ax.activity.create(
+    task_name="my_task",
+    message="Task created",
+    metadata={"organization_id": "org-123"},
+)
+```
+
+Usually called automatically by `ax.enqueue()`. Call it directly only for
+advanced use cases where you need to register an activity outside the normal
+enqueue flow.
+
+---
+
+## update()
+
+Append a log entry to an existing activity.
+
+```python
+async def update(
+    agent_id: str | uuid.UUID,
+    message: str,
+    percentage: int | None = None,
+    status: Status | None = None,
+) -> bool
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `agent_id` | `str \| UUID` | required | Activity identifier |
+| `message` | `str` | required | Log message |
+| `percentage` | `int \| None` | `None` | Progress (0-100) |
+| `status` | `Status \| None` | `None` | Status override (defaults to `RUNNING`) |
+
+### Example
+
+```python
+# Progress update
+await ax.activity.update(agent_id, "Processing data...", percentage=30)
+
+# Explicit status
+from agentexec.activity import Status
+await ax.activity.update(agent_id, "Starting", status=Status.RUNNING)
+```
+
+---
+
+## complete()
+
+Mark an activity as successfully completed.
+
+```python
+async def complete(
+    agent_id: str | uuid.UUID,
+    message: str = "Agent completed",
+    percentage: int = 100,
+) -> bool
+```
+
+### Example
+
+```python
+await ax.activity.complete(agent_id)
+await ax.activity.complete(agent_id, "Partial completion", percentage=75)
+```
+
+---
+
+## error()
+
+Mark an activity as failed.
+
+```python
+async def error(
+    agent_id: str | uuid.UUID,
+    message: str = "Agent failed",
+    percentage: int = 100,
+) -> bool
+```
+
+### Example
+
+```python
+try:
+    await risky_operation()
+except Exception as e:
+    await ax.activity.error(agent_id, f"Failed: {e}")
+    raise
+```
+
+---
+
+## cancel_pending()
+
+Cancel all queued and running activities. Typically called at pool startup
+to reconcile state from a previous run.
+
+```python
+async def cancel_pending(
+    session: AsyncSession | None = None,
+) -> int
+```
+
+### Returns
+
+`int` — number of activities canceled.
+
+### Example
+
+```python
+from agentexec.core.db import get_session
+
+async with get_session() as db:
+    canceled = await ax.activity.cancel_pending(session=db)
+    print(f"Canceled {canceled} activities")
+```
+
+---
+
+## list()
+
+Get a paginated list of activities.
+
+```python
+async def list(
+    session: AsyncSession | None = None,
+    page: int = 1,
+    page_size: int = 50,
+    metadata_filter: dict[str, Any] | None = None,
+) -> ActivityListSchema
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `session` | `AsyncSession \| None` | `None` | Session (falls back to `get_session()`) |
+| `page` | `int` | `1` | Page number (1-indexed) |
+| `page_size` | `int` | `50` | Items per page |
+| `metadata_filter` | `dict \| None` | `None` | Filter by metadata fields |
+
+### Example
+
+```python
+result = await ax.activity.list(page=1, page_size=20)
+
+print(f"Total: {result.total}")
+print(f"Page {result.page}, total_pages: {result.total_pages}")
+
+for item in result.items:
+    print(f"{item.agent_id}: {item.status}")
+```
+
+### With Metadata Filter
+
+```python
+# Only activities for org-123
+result = await ax.activity.list(
+    metadata_filter={"organization_id": "org-123"},
+)
+```
+
+---
+
+## detail()
+
+Get detailed activity with full log history.
+
+```python
+async def detail(
+    session: AsyncSession | None = None,
+    agent_id: str | uuid.UUID | None = None,
+    metadata_filter: dict[str, Any] | None = None,
+) -> ActivityDetailSchema | None
+```
+
+### Returns
+
+`ActivityDetailSchema | None` — activity details, or `None` if not found (or
+the metadata filter didn't match).
+
+### Example
+
+```python
+activity = await ax.activity.detail(agent_id=agent_id)
+
+if activity:
+    print(f"Task: {activity.agent_type}")
+    for log in activity.logs:
+        print(f"  [{log.created_at}] {log.status} {log.message}")
+```
+
+---
+
+## count_active()
+
+Return the number of queued or running activities.
+
+```python
+async def count_active(
+    session: AsyncSession | None = None,
+) -> int
+```
+
+---
+
+## Status Enum
+
+```python
+class Status(str, Enum):
+    QUEUED = "QUEUED"
+    RUNNING = "RUNNING"
+    COMPLETE = "COMPLETE"
+    ERROR = "ERROR"
+    CANCELED = "CANCELED"
+```
+
+| Status | Description |
+|--------|-------------|
+| `QUEUED` | Task is waiting in queue |
+| `RUNNING` | Task is being executed |
+| `COMPLETE` | Task finished successfully |
+| `ERROR` | Task failed with error |
+| `CANCELED` | Task was canceled |
+
+---
+
+## Response Schemas
+
+### ActivityLogSchema
+
+```python
+class ActivityLogSchema(BaseModel):
+    id: uuid.UUID
+    message: str
+    status: Status
+    percentage: int | None = 0
+    created_at: datetime
+```
+
+### ActivityDetailSchema
+
+```python
+class ActivityDetailSchema(BaseModel):
+    id: uuid.UUID | None
+    agent_id: uuid.UUID
+    agent_type: str
+    created_at: datetime
+    updated_at: datetime
+    logs: list[ActivityLogSchema]
+```
+
+### ActivityListItemSchema
+
+```python
+class ActivityListItemSchema(BaseModel):
+    agent_id: uuid.UUID
+    agent_type: str
+    status: Status
+    latest_log_message: str | None
+    latest_log_timestamp: datetime | None
+    percentage: int | None
+    started_at: datetime | None
+    elapsed_time_seconds: int  # computed
+```
+
+### ActivityListSchema
+
+```python
+class ActivityListSchema(BaseModel):
+    items: list[ActivityListItemSchema]
+    total: int
+    page: int
+    page_size: int
+    total_pages: int  # computed
+```
+
+---
+
+## ORM Models
+
+For advanced queries, use the ORM models directly with an `AsyncSession`.
+
+### Activity
+
+```python
+class Activity(Base):
+    __tablename__ = "{prefix}activity"
+
+    id: Mapped[UUID]
+    agent_id: Mapped[UUID]       # Unique, indexed
+    agent_type: Mapped[str | None]
+    created_at: Mapped[datetime]
+    updated_at: Mapped[datetime]
+    metadata_: Mapped[dict | None]  # Column is "metadata"
+    logs: Mapped[list[ActivityLog]]
+```
+
+#### Classmethods (all async)
+
+- `Activity.create(session, agent_id, task_name, message, metadata=None)`
+- `Activity.append_log(session, agent_id, message, status, percentage=None)`
+- `Activity.get_by_agent_id(session, agent_id, metadata_filter=None)`
+- `Activity.get_list(session, page=1, page_size=50, metadata_filter=None)`
+- `Activity.get_pending_ids(session)`
+- `Activity.get_active_count(session)`
+
+### ActivityLog
+
+```python
+class ActivityLog(Base):
+    __tablename__ = "{prefix}activity_log"
+
+    id: Mapped[UUID]
+    activity_id: Mapped[UUID]
+    message: Mapped[str]
+    status: Mapped[Status]
+    percentage: Mapped[int | None]
+    created_at: Mapped[datetime]
+```
+
+---
+
+## Direct ORM Usage
+
+```python
+from sqlalchemy import select
+from agentexec.activity.models import Activity, ActivityLog, Status
+from agentexec.core.db import get_session
+
+async with get_session() as db:
+    # Latest log for a given agent
+    result = await db.execute(
+        select(ActivityLog)
+        .where(ActivityLog.activity_id == activity_id)
+        .order_by(ActivityLog.created_at.desc())
+        .limit(1)
+    )
+    latest = result.scalar_one_or_none()
+```