agentexec 0.2.0rc2__tar.gz → 0.3.0__tar.gz

{agentexec-0.2.0rc2 → agentexec-0.3.0}/CHANGELOG.md

@@ -1,93 +1,60 @@
 # Changelog
 
-## v0.2.0rc2
-
-Second release candidate for 0.2.0. Completes the async migration and
-stabilizes the worker pool architecture.
+## v0.3.0
 
 ### Breaking Changes
 
-**Fully async database layer**
-- `configure_engine()` and `get_session()` now require an async SQLAlchemy engine (`AsyncEngine`) and return `AsyncSession`
-- Database URLs must use async drivers (e.g. `sqlite+aiosqlite://`, `postgresql+asyncpg://`)
-- `sqlalchemy[asyncio]` is now a core dependency; `aiosqlite` added as dev dependency
-
-**Async activity query API**
-- `activity.list()`, `activity.detail()`, and `activity.count_active()` are now async and accept `AsyncSession`
-- Activity handlers are now async (`async def __call__`)
-
-**Removed `session` parameter from activity mutations**
-- `activity.create()`, `activity.update()`, `activity.complete()`, and `activity.error()` no longer accept a `session` parameter — the handler owns its own session lifecycle
-
-**Queue priority parameter**
-- `BaseQueueBackend.push()` signature changed from `high_priority: bool` to `priority: Priority | None`
-- Affects Redis, Kafka, and any custom queue backend implementations
+**Permanent task failure raises `TaskFailedError`**
+- Callers that previously relied on `TimeoutError` to detect a task that exhausted its retries now receive `ax.TaskFailedError` as soon as the failure is recorded
 
 ### New Features
 
-**CLI entrypoint**
-- New `agentexec` CLI command via `[project.scripts]`
-
-**Activity model `create()` classmethod**
-- `Activity.create()` encapsulates record + initial log entry creation in one async call
-
-**Async engine disposal**
-- `dispose_engine()` ensures the async engine's background threads exit cleanly on shutdown
-
-### Architecture Changes
-
-**Worker pool refactor**
-- Workers now use the `spawn` multiprocessing start method with explicit context — no inherited state
-- `StateEvent` replaced with stdlib `multiprocessing.Event` — removes dependency on the state backend for shutdown coordination
-- Event handling and scheduling extracted into `_EventHandler` and `_Scheduler` classes
-- Worker processes are now daemonic with `SIGINT` ignored — clean shutdown driven by the event
-- `pool.start()` handles `CancelledError` directly for Ctrl+C shutdown
-
-**Logging overhaul**
-- Removed `worker/logging.py` and `core/logging.py` — all modules use stdlib `logging.getLogger(__name__)`
-- Spawned workers bootstrap a `StreamHandler` on the root logger so logs reach stderr
-- Pool messages use `logger.info`/`logger.error` instead of `print()`
-
-### Bug Fixes
-
-- Fixed crash when worker receives a task for an unregistered task name (now logs error and skips)
-- Failed tasks now log full tracebacks via `logger.exception` instead of `logger.error`
-- Kafka consumer handles `None` message values without crashing
-- `ActivityUpdated.status` is now `Status` enum instead of raw string
-- `raise e` instead of bare `raise` in task execution for clearer tracebacks
+**Terminal failure contract for `ax.gather` / `ax.get_result`**
+- When a task exhausts its retries, the pool stores a `TaskFailure` record at the task's result key (same convention and TTL as success results)
+- `ax.get_result` raises `ax.TaskFailedError` — carrying `task_name`, `agent_id`, `error`, and `attempts` — on its next poll (~0.5s) instead of polling until the timeout
+- `ax.gather` fails fast on the first permanent failure; remaining tasks keep running and their results stay retrievable, so re-gathering survivors resolves instantly
+- `TaskFailedError` is exported on the `ax` namespace
 
-### Improvements
-
-- Dependency groups use PEP 735 `[dependency-groups]` instead of `[tool.uv]`
-- Ruff line-length increased to 110
-- Removed verbose docstring examples and redundant comments throughout activity models
-- `selectinload` for eager loading of activity logs in `get_by_agent_id`
-- Redis backend adds `type: ignore` annotations for redis-py async stubs
+## v0.2.0
 
-## v0.2.0rc1
-
-First release candidate for 0.2.0. Major refactor of the backend, queue,
-activity, and worker systems.
+Major refactor of the backend, queue, activity, worker, and database layers.
+If you're upgrading from 0.1.x, read the **Breaking Changes** section closely.
 
 ### Breaking Changes
 
-**Backend module restructure**
-- `agentexec.state.redis_backend` renamed to `agentexec.state.redis` — update `AGENTEXEC_STATE_BACKEND` if set explicitly
-- `AGENTEXEC_QUEUE_NAME` renamed to `AGENTEXEC_QUEUE_PREFIX` (old name still accepted as alias)
+**Fully async database layer**
+- `configure_engine()` and `get_session()` now require an async SQLAlchemy engine (`AsyncEngine`) and return `AsyncSession`
+- Database URLs must use async drivers (e.g. `sqlite+aiosqlite://`, `postgresql+asyncpg://`)
+- `sqlalchemy[asyncio]` is now a core dependency
 
 **Async activity API**
-- Activity functions are now async: `await ax.activity.create(...)`, `await ax.activity.update(...)`, etc.
+- All activity functions are async: `await ax.activity.create(...)`, `await ax.activity.update(...)`, `await ax.activity.complete(...)`, `await ax.activity.error(...)`
+- `activity.list()`, `activity.detail()`, and `activity.count_active()` are async and accept `AsyncSession`
+- Activity handlers are async (`async def __call__`)
+- The `session` parameter was removed from activity mutations — the handler owns its own session lifecycle
+
+**Pool entry point**
+- `pool.run()` was removed. Use `await pool.start()` in an asyncio loop, or the new `agentexec run mymodule:pool` CLI
+- `AGENTEXEC_QUEUE_NAME` renamed to `AGENTEXEC_QUEUE_PREFIX` (old name still accepted as alias)
+- `agentexec.state.redis_backend` renamed to `agentexec.state.redis` — update `AGENTEXEC_STATE_BACKEND` if set explicitly
 
 **Task context serialization**
 - `Task.context` is now `Mapping[str, Any]` (raw dict), not a typed BaseModel — hydration happens at execution time
 - `Task.create()` is now async
 
+**Queue backend protocol**
+- `BaseQueueBackend.push()` signature changed from `high_priority: bool` to `priority: Priority | None` — affects Redis, Kafka, and any custom queue backend
+
 **Removed APIs**
 - `set_global_session`/`get_global_session`/`remove_global_session` — use `configure_engine`/`get_session`
 - `state.backend.publish`/`subscribe` (pubsub), `index_add`/`index_range`/`index_remove`, `clear`, `configure`
+- `worker/logging.py` and `core/logging.py` — all modules use stdlib `logging.getLogger(__name__)` directly
 
 ### New Features
 
+**CLI entrypoint**
+- New `agentexec` CLI command: `agentexec run mymodule:pool --create-tables --workers 4`
+
 **Partitioned Redis queues**
 - Tasks with `lock_key` route to dedicated partition queues with per-partition locking and SCAN-based fair dequeue
 
@@ -101,18 +68,45 @@ activity, and worker systems.
 - `pip install agentexec[kafka]` for queue and schedule via Kafka
 
 **Typed worker IPC**
-- `TaskFailed`, `LogEntry`, `ActivityUpdated` messages over `multiprocessing.Queue`
+- `TaskFailed` and `ActivityEvent` messages flow over `multiprocessing.Queue` with pydantic validation
 
 **Schedule composite keys**
 - `{task_name}:{cron}:{context_hash}` for unique schedule identity
 
-### Improvements
+**Activity model `create()` classmethod**
+- `Activity.create()` encapsulates record + initial log entry creation in one async call
+
+**Async engine disposal**
+- `dispose_engine()` ensures the async engine's background threads exit cleanly on shutdown
 
+### Architecture Changes
+
+**Worker pool refactor**
+- Workers use the `spawn` multiprocessing start method with explicit context — no inherited state
+- Event handling and scheduling extracted into `_EventHandler` and `_Scheduler` classes
+- `StateEvent` replaced with stdlib `multiprocessing.Event` — removes dependency on the state backend for shutdown coordination
 - Class-based backend architecture with ABCs (`BaseStateBackend`, `BaseQueueBackend`, `BaseScheduleBackend`)
 - `Task` is pure data, `TaskDefinition` owns behavior
-- Session management via `configure_engine`/`get_session` (Pool owns the engine)
 - Status enum extracted to `activity/status.py` (no SQLAlchemy dependency)
 
+**Logging**
+- All modules use stdlib `logging.getLogger(__name__)`
+- Spawned workers bootstrap a `StreamHandler` on the root logger so logs reach stderr
+- Pool messages use `logger.info`/`logger.error` instead of `print()`
+
+### Bug Fixes
+
+- **Orphaned worker processes on shutdown.** SIGTERM (systemd/docker stop), SIGKILL, and SIGHUP were leaving worker processes running. Fixed via an asyncio SIGTERM handler in the CLI and `prctl(PR_SET_PDEATHSIG)` in each worker so the kernel terminates workers when the pool dies
+- **Worker and scheduler error loops throttled.** Infra failures (e.g. Redis unreachable) were producing 100k+ log lines per second. Added a 1s sleep after outer-loop exceptions
+- **Unregistered task name crash.** Worker now logs an error and skips instead of crashing when it receives a task for an unknown name
+- Failed tasks now log full tracebacks via `logger.exception` instead of `logger.error`
+- Kafka consumer handles `None` message values without crashing
+- `ActivityUpdated.status` is a `Status` enum instead of raw string
+
+### Documentation
+
+- Full documentation sweep for the async API — connection strings, CLI usage, `await` on activity calls across all guides and API references
+
 ## v0.1.7
 
 ### New Features

{agentexec-0.2.0rc2 → agentexec-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentexec
-Version: 0.2.0rc2
+Version: 0.3.0
 Summary: Production-ready orchestration for OpenAI Agents with Redis-backed coordination, activity tracking, and workflow management
 Project-URL: Homepage, https://github.com/Agent-CI/agentexec
 Project-URL: Documentation, https://github.com/Agent-CI/agentexec#readme

agentexec-0.3.0/docs/api-reference/activity.md

@@ -0,0 +1,401 @@
+# Activity API Reference
+
+This document covers the activity tracking API for monitoring task execution.
+
+All activity functions are `async` and operate on `AsyncSession` from
+`sqlalchemy.ext.asyncio`. The session parameter is optional — when omitted,
+functions fall back to `get_session()` from `agentexec.core.db`, which
+requires `configure_engine()` to have been called.
+
+## Module: agentexec.activity
+
+```python
+import agentexec as ax
+
+# Lifecycle (async)
+await ax.activity.create(task_name, message)
+await ax.activity.update(agent_id, message)
+await ax.activity.complete(agent_id)
+await ax.activity.error(agent_id)
+await ax.activity.cancel_pending()
+
+# Query (async)
+await ax.activity.list()
+await ax.activity.detail(agent_id=...)
+await ax.activity.count_active()
+```
+
+---
+
+## create()
+
+Create a new activity record with an initial `QUEUED` log entry.
+
+```python
+async def create(
+    task_name: str,
+    message: str = "Agent queued",
+    agent_id: str | uuid.UUID | None = None,
+    metadata: dict[str, Any] | None = None,
+) -> uuid.UUID
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `task_name` | `str` | required | Name of the task |
+| `message` | `str` | `"Agent queued"` | Initial log message |
+| `agent_id` | `str \| UUID \| None` | `None` | Pre-generated ID (auto-generated if omitted) |
+| `metadata` | `dict \| None` | `None` | Arbitrary metadata attached to the activity |
+
+### Returns
+
+`uuid.UUID` — the activity's `agent_id`.
+
+### Example
+
+```python
+import agentexec as ax
+
+agent_id = await ax.activity.create(
+    task_name="my_task",
+    message="Task created",
+    metadata={"organization_id": "org-123"},
+)
+```
+
+Usually called automatically by `ax.enqueue()`. Call it directly only for
+advanced use cases where you need to register an activity outside the normal
+enqueue flow.
+
+---
+
+## update()
+
+Append a log entry to an existing activity.
+
+```python
+async def update(
+    agent_id: str | uuid.UUID,
+    message: str,
+    percentage: int | None = None,
+    status: Status | None = None,
+) -> bool
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `agent_id` | `str \| UUID` | required | Activity identifier |
+| `message` | `str` | required | Log message |
+| `percentage` | `int \| None` | `None` | Progress (0-100) |
+| `status` | `Status \| None` | `None` | Status override (defaults to `RUNNING`) |
+
+### Example
+
+```python
+# Progress update
+await ax.activity.update(agent_id, "Processing data...", percentage=30)
+
+# Explicit status
+from agentexec.activity import Status
+await ax.activity.update(agent_id, "Starting", status=Status.RUNNING)
+```
+
+---
+
+## complete()
+
+Mark an activity as successfully completed.
+
+```python
+async def complete(
+    agent_id: str | uuid.UUID,
+    message: str = "Agent completed",
+    percentage: int = 100,
+) -> bool
+```
+
+### Example
+
+```python
+await ax.activity.complete(agent_id)
+await ax.activity.complete(agent_id, "Partial completion", percentage=75)
+```
+
+---
+
+## error()
+
+Mark an activity as failed.
+
+```python
+async def error(
+    agent_id: str | uuid.UUID,
+    message: str = "Agent failed",
+    percentage: int = 100,
+) -> bool
+```
+
+### Example
+
+```python
+try:
+    await risky_operation()
+except Exception as e:
+    await ax.activity.error(agent_id, f"Failed: {e}")
+    raise
+```
+
+---
+
+## cancel_pending()
+
+Cancel all queued and running activities. Typically called at pool startup
+to reconcile state from a previous run.
+
+```python
+async def cancel_pending(
+    session: AsyncSession | None = None,
+) -> int
+```
+
+### Returns
+
+`int` — number of activities canceled.
+
+### Example
+
+```python
+from agentexec.core.db import get_session
+
+async with get_session() as db:
+    canceled = await ax.activity.cancel_pending(session=db)
+    print(f"Canceled {canceled} activities")
+```
+
+---
+
+## list()
+
+Get a paginated list of activities.
+
+```python
+async def list(
+    session: AsyncSession | None = None,
+    page: int = 1,
+    page_size: int = 50,
+    metadata_filter: dict[str, Any] | None = None,
+) -> ActivityListSchema
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `session` | `AsyncSession \| None` | `None` | Session (falls back to `get_session()`) |
+| `page` | `int` | `1` | Page number (1-indexed) |
+| `page_size` | `int` | `50` | Items per page |
+| `metadata_filter` | `dict \| None` | `None` | Filter by metadata fields |
+
+### Example
+
+```python
+result = await ax.activity.list(page=1, page_size=20)
+
+print(f"Total: {result.total}")
+print(f"Page {result.page}, total_pages: {result.total_pages}")
+
+for item in result.items:
+    print(f"{item.agent_id}: {item.status}")
+```
+
+### With Metadata Filter
+
+```python
+# Only activities for org-123
+result = await ax.activity.list(
+    metadata_filter={"organization_id": "org-123"},
+)
+```
+
+---
+
+## detail()
+
+Get detailed activity with full log history.
+
+```python
+async def detail(
+    session: AsyncSession | None = None,
+    agent_id: str | uuid.UUID | None = None,
+    metadata_filter: dict[str, Any] | None = None,
+) -> ActivityDetailSchema | None
+```
+
+### Returns
+
+`ActivityDetailSchema | None` — activity details, or `None` if not found (or
+the metadata filter didn't match).
+
+### Example
+
+```python
+activity = await ax.activity.detail(agent_id=agent_id)
+
+if activity:
+    print(f"Task: {activity.agent_type}")
+    for log in activity.logs:
+        print(f"  [{log.created_at}] {log.status} {log.message}")
+```
+
+---
+
+## count_active()
+
+Return the number of queued or running activities.
+
+```python
+async def count_active(
+    session: AsyncSession | None = None,
+) -> int
+```
+
+---
+
+## Status Enum
+
+```python
+class Status(str, Enum):
+    QUEUED = "QUEUED"
+    RUNNING = "RUNNING"
+    COMPLETE = "COMPLETE"
+    ERROR = "ERROR"
+    CANCELED = "CANCELED"
+```
+
+| Status | Description |
+|--------|-------------|
+| `QUEUED` | Task is waiting in queue |
+| `RUNNING` | Task is being executed |
+| `COMPLETE` | Task finished successfully |
+| `ERROR` | Task failed with error |
+| `CANCELED` | Task was canceled |
+
+---
+
+## Response Schemas
+
+### ActivityLogSchema
+
+```python
+class ActivityLogSchema(BaseModel):
+    id: uuid.UUID
+    message: str
+    status: Status
+    percentage: int | None = 0
+    created_at: datetime
+```
+
+### ActivityDetailSchema
+
+```python
+class ActivityDetailSchema(BaseModel):
+    id: uuid.UUID | None
+    agent_id: uuid.UUID
+    agent_type: str
+    created_at: datetime
+    updated_at: datetime
+    logs: list[ActivityLogSchema]
+```
+
+### ActivityListItemSchema
+
+```python
+class ActivityListItemSchema(BaseModel):
+    agent_id: uuid.UUID
+    agent_type: str
+    status: Status
+    latest_log_message: str | None
+    latest_log_timestamp: datetime | None
+    percentage: int | None
+    started_at: datetime | None
+    elapsed_time_seconds: int  # computed
+```
+
+### ActivityListSchema
+
+```python
+class ActivityListSchema(BaseModel):
+    items: list[ActivityListItemSchema]
+    total: int
+    page: int
+    page_size: int
+    total_pages: int  # computed
+```
+
+---
+
+## ORM Models
+
+For advanced queries, use the ORM models directly with an `AsyncSession`.
+
+### Activity
+
+```python
+class Activity(Base):
+    __tablename__ = "{prefix}activity"
+
+    id: Mapped[UUID]
+    agent_id: Mapped[UUID]       # Unique, indexed
+    agent_type: Mapped[str | None]
+    created_at: Mapped[datetime]
+    updated_at: Mapped[datetime]
+    metadata_: Mapped[dict | None]  # Column is "metadata"
+    logs: Mapped[list[ActivityLog]]
+```
+
+#### Classmethods (all async)
+
+- `Activity.create(session, agent_id, task_name, message, metadata=None)`
+- `Activity.append_log(session, agent_id, message, status, percentage=None)`
+- `Activity.get_by_agent_id(session, agent_id, metadata_filter=None)`
+- `Activity.get_list(session, page=1, page_size=50, metadata_filter=None)`
+- `Activity.get_pending_ids(session)`
+- `Activity.get_active_count(session)`
+
+### ActivityLog
+
+```python
+class ActivityLog(Base):
+    __tablename__ = "{prefix}activity_log"
+
+    id: Mapped[UUID]
+    activity_id: Mapped[UUID]
+    message: Mapped[str]
+    status: Mapped[Status]
+    percentage: Mapped[int | None]
+    created_at: Mapped[datetime]
+```
+
+---
+
+## Direct ORM Usage
+
+```python
+from sqlalchemy import select
+from agentexec.activity.models import Activity, ActivityLog, Status
+from agentexec.core.db import get_session
+
+async with get_session() as db:
+    # Latest log for a given agent
+    result = await db.execute(
+        select(ActivityLog)
+        .where(ActivityLog.activity_id == activity_id)
+        .order_by(ActivityLog.created_at.desc())
+        .limit(1)
+    )
+    latest = result.scalar_one_or_none()
+```