chaoscypher-neuron 0.1.0__tar.gz

chaoscypher_neuron-0.1.0/PKG-INFO

@@ -0,0 +1,185 @@
+Metadata-Version: 2.4
+Name: chaoscypher-neuron
+Version: 0.1.0
+Summary: Chaos Cypher Neuron - Background task processing workers (worker cells)
+Author-email: Denis MacPherson <denis@chaoscypher.com>
+License-Expression: AGPL-3.0-only
+Project-URL: Homepage, https://github.com/chaoscypherinc/chaoscypher
+Project-URL: Documentation, https://github.com/chaoscypherinc/chaoscypher#readme
+Project-URL: Repository, https://github.com/chaoscypherinc/chaoscypher
+Project-URL: Issues, https://github.com/chaoscypherinc/chaoscypher/issues
+Keywords: knowledge-graph,workers,background-tasks,queue,ai
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.14
+Description-Content-Type: text/markdown
+Requires-Dist: chaoscypher-core>=0.1.0
+Requires-Dist: structlog<26,>=25.5.0
+Requires-Dist: pyyaml<7,>=6.0.3
+Provides-Extra: dev
+Requires-Dist: pytest<10,>=9.0.0; extra == "dev"
+Requires-Dist: pytest-cov<8,>=7.1.0; extra == "dev"
+Requires-Dist: pytest-asyncio<2,>=1.3.0; extra == "dev"
+
+# ChaosCypher Neuron
+
+**Background task processing — unified worker cell**
+
+Neuron is ChaosCypher's background worker process. A single `cc-neuron`
+entry point runs two independently-paced queues in one process:
+
+- **LLM queue** — serialized (default 1 concurrent) for chat, embeddings,
+  tool calls, and chunk extraction. Sequential execution avoids overwhelming
+  the LLM provider and keeps priority queueing fair.
+- **Operations queue** — parallel (default 8 concurrent) for source
+  processing, exports, workflows, vision-page work, and other CPU/IO-bound
+  tasks that benefit from concurrency.
+
+Tasks are pulled from [Valkey](https://valkey.io/) via ChaosCypher's own
+queue client (`chaoscypher_core.queue`); there is no ARQ, Celery, or RQ
+dependency.
+
+## Installation
+
+```bash
+# Workspace sync — installs core + cortex + neuron + dev tools
+uv sync --all-packages --extra dev
+
+# Single-package mode (neuron + its core dependency only)
+uv sync --package chaoscypher-neuron
+```
+
+The repo uses uv workspaces (see root `pyproject.toml` `[tool.uv.workspace]`).
+Install uv via [the official installer](https://docs.astral.sh/uv/getting-started/installation/)
+before running these commands; `pip install -e .` is not supported.
+
+## Usage
+
+```bash
+# Start the unified worker (both queues)
+cc-neuron
+
+# With a custom Valkey instance
+QUEUE_HOST=localhost QUEUE_PORT=6379 cc-neuron
+```
+
+The worker auto-detects which queue handlers are registered and polls
+each queue at its configured concurrency. There is no separate
+`cc-neuron-llm` / `cc-neuron-ops` binary — both queues run in the same
+process.
+
+### Docker
+
+In the all-in-one image, `cc-neuron` is launched by supervisord alongside
+cortex, valkey, and nginx. In the multi-container deployments
+(`packages/docker/multi-container/docker-compose.{dev,prod}.yml`) the
+worker runs in its own `worker` service:
+
+```bash
+docker run -e QUEUE_HOST=valkey -e QUEUE_PORT=6379 chaoscypher-neuron
+```
+
+## Configuration
+
+Worker behaviour is configured through ChaosCypher's settings layer
+(env var → `packages/docker/data/settings.yaml` → package default).
+Concurrency and timeouts live under `QueueSettings`; only the Valkey
+connection itself is read directly from the environment so the worker
+can bootstrap before settings are loaded:
+
+```bash
+QUEUE_HOST=localhost           # Valkey hostname
+QUEUE_PORT=6379                # Valkey port
+QUEUE_PASSWORD=chaoscypher     # Valkey password
+QUEUE_DB=0                     # Valkey logical DB
+LOG_LEVEL=INFO
+USE_JSON_LOGGING=false
+```
+
+Per-queue concurrency, retry limits, and timeouts are not env-driven —
+edit `settings.yaml` and restart the worker, or change them through the
+Settings UI (cortex pushes updates via the `settings_changes` channel and
+`cc-neuron` picks them up at the next poll boundary).
+
+## Architecture
+
+Neuron is part of the ChaosCypher neural architecture:
+
+- **Core** — domain logic + operational substrate (queue, settings, handlers)
+- **Cortex** — FastAPI backend that enqueues work
+- **Neuron** — unified worker that consumes both queues 👈 You are here
+- **Interface** — React UI
+
+### Why one process, two queues?
+
+- **Separate concurrency control.** The LLM queue intentionally throttles
+  to 1; the Operations queue parallelises to 8. Running them in the same
+  process avoids container overhead and the need for two scaler knobs
+  while keeping each queue independently paced.
+- **Shared resources.** Database adapter, LLM provider, settings cache,
+  recovery loops, and the Valkey connection pool are initialised once
+  per process instead of duplicated across two worker images.
+- **Startup recovery.** A single recovery sweep (`SourceRecovery` +
+  orphan-task rehydration) runs at boot, not twice.
+
+## Task types
+
+### LLM queue
+- `llm_chat` — chat completion
+- `llm_embedding` — generate embeddings
+- `llm_tool_call` — execute tool with LLM
+- `llm_extract_chunk` — entity / relationship extraction over a chunk
+
+### Operations queue
+- `operations_indexing` — chunking + entity prep
+- `operations_embedding` — bulk embedding writeback
+- `operations_commit` — graph commit + search index update
+- `operations_vision_page` / `operations_vision_finalize` — per-page vision
+- `operations_export` / `operations_import` — package export / import
+- `operations_workflow` — workflow step execution
+
+Canonical mapping lives in `chaoscypher_core.constants.OPERATION_QUEUE_ROUTING`
+(enforced by lint rule CC044).
+
+## Monitoring
+
+```bash
+# Container logs
+docker compose logs -f chaoscypher           # all-in-one (supervisord muxes)
+docker compose logs -f worker                # multi-container worker service
+
+# Queue depth + per-task status via cortex
+curl http://localhost/api/v1/queue/status
+```
+
+## Development
+
+```bash
+# Workspace install (from repo root)
+uv sync --all-packages --extra dev
+
+# Run tests
+uv run pytest packages/neuron
+
+# Auto-restart on source edits
+uv run watchmedo auto-restart -d packages/neuron/src -p '*.py' -- cc-neuron
+```
+
+## Scaling
+
+The all-in-one image runs exactly one `cc-neuron` instance under
+supervisord. For horizontal scale-out, use the multi-container
+deployment and replicate the `worker` service:
+
+```bash
+cd packages/docker/multi-container
+docker compose -f docker-compose.prod.yml up -d --scale worker=4
+```
+
+Each replica polls both queues; Valkey distributes tasks across them.
+
+## License
+
+AGPL-3.0 — see the repository `LICENSE` file.

chaoscypher_neuron-0.1.0/README.md

@@ -0,0 +1,160 @@
+# ChaosCypher Neuron
+
+**Background task processing — unified worker cell**
+
+Neuron is ChaosCypher's background worker process. A single `cc-neuron`
+entry point runs two independently-paced queues in one process:
+
+- **LLM queue** — serialized (default 1 concurrent) for chat, embeddings,
+  tool calls, and chunk extraction. Sequential execution avoids overwhelming
+  the LLM provider and keeps priority queueing fair.
+- **Operations queue** — parallel (default 8 concurrent) for source
+  processing, exports, workflows, vision-page work, and other CPU/IO-bound
+  tasks that benefit from concurrency.
+
+Tasks are pulled from [Valkey](https://valkey.io/) via ChaosCypher's own
+queue client (`chaoscypher_core.queue`); there is no ARQ, Celery, or RQ
+dependency.
+
+## Installation
+
+```bash
+# Workspace sync — installs core + cortex + neuron + dev tools
+uv sync --all-packages --extra dev
+
+# Single-package mode (neuron + its core dependency only)
+uv sync --package chaoscypher-neuron
+```
+
+The repo uses uv workspaces (see root `pyproject.toml` `[tool.uv.workspace]`).
+Install uv via [the official installer](https://docs.astral.sh/uv/getting-started/installation/)
+before running these commands; `pip install -e .` is not supported.
+
+## Usage
+
+```bash
+# Start the unified worker (both queues)
+cc-neuron
+
+# With a custom Valkey instance
+QUEUE_HOST=localhost QUEUE_PORT=6379 cc-neuron
+```
+
+The worker auto-detects which queue handlers are registered and polls
+each queue at its configured concurrency. There is no separate
+`cc-neuron-llm` / `cc-neuron-ops` binary — both queues run in the same
+process.
+
+### Docker
+
+In the all-in-one image, `cc-neuron` is launched by supervisord alongside
+cortex, valkey, and nginx. In the multi-container deployments
+(`packages/docker/multi-container/docker-compose.{dev,prod}.yml`) the
+worker runs in its own `worker` service:
+
+```bash
+docker run -e QUEUE_HOST=valkey -e QUEUE_PORT=6379 chaoscypher-neuron
+```
+
+## Configuration
+
+Worker behaviour is configured through ChaosCypher's settings layer
+(env var → `packages/docker/data/settings.yaml` → package default).
+Concurrency and timeouts live under `QueueSettings`; only the Valkey
+connection itself is read directly from the environment so the worker
+can bootstrap before settings are loaded:
+
+```bash
+QUEUE_HOST=localhost           # Valkey hostname
+QUEUE_PORT=6379                # Valkey port
+QUEUE_PASSWORD=chaoscypher     # Valkey password
+QUEUE_DB=0                     # Valkey logical DB
+LOG_LEVEL=INFO
+USE_JSON_LOGGING=false
+```
+
+Per-queue concurrency, retry limits, and timeouts are not env-driven —
+edit `settings.yaml` and restart the worker, or change them through the
+Settings UI (cortex pushes updates via the `settings_changes` channel and
+`cc-neuron` picks them up at the next poll boundary).
+
+## Architecture
+
+Neuron is part of the ChaosCypher neural architecture:
+
+- **Core** — domain logic + operational substrate (queue, settings, handlers)
+- **Cortex** — FastAPI backend that enqueues work
+- **Neuron** — unified worker that consumes both queues 👈 You are here
+- **Interface** — React UI
+
+### Why one process, two queues?
+
+- **Separate concurrency control.** The LLM queue intentionally throttles
+  to 1; the Operations queue parallelises to 8. Running them in the same
+  process avoids container overhead and the need for two scaler knobs
+  while keeping each queue independently paced.
+- **Shared resources.** Database adapter, LLM provider, settings cache,
+  recovery loops, and the Valkey connection pool are initialised once
+  per process instead of duplicated across two worker images.
+- **Startup recovery.** A single recovery sweep (`SourceRecovery` +
+  orphan-task rehydration) runs at boot, not twice.
+
+## Task types
+
+### LLM queue
+- `llm_chat` — chat completion
+- `llm_embedding` — generate embeddings
+- `llm_tool_call` — execute tool with LLM
+- `llm_extract_chunk` — entity / relationship extraction over a chunk
+
+### Operations queue
+- `operations_indexing` — chunking + entity prep
+- `operations_embedding` — bulk embedding writeback
+- `operations_commit` — graph commit + search index update
+- `operations_vision_page` / `operations_vision_finalize` — per-page vision
+- `operations_export` / `operations_import` — package export / import
+- `operations_workflow` — workflow step execution
+
+Canonical mapping lives in `chaoscypher_core.constants.OPERATION_QUEUE_ROUTING`
+(enforced by lint rule CC044).
+
+## Monitoring
+
+```bash
+# Container logs
+docker compose logs -f chaoscypher           # all-in-one (supervisord muxes)
+docker compose logs -f worker                # multi-container worker service
+
+# Queue depth + per-task status via cortex
+curl http://localhost/api/v1/queue/status
+```
+
+## Development
+
+```bash
+# Workspace install (from repo root)
+uv sync --all-packages --extra dev
+
+# Run tests
+uv run pytest packages/neuron
+
+# Auto-restart on source edits
+uv run watchmedo auto-restart -d packages/neuron/src -p '*.py' -- cc-neuron
+```
+
+## Scaling
+
+The all-in-one image runs exactly one `cc-neuron` instance under
+supervisord. For horizontal scale-out, use the multi-container
+deployment and replicate the `worker` service:
+
+```bash
+cd packages/docker/multi-container
+docker compose -f docker-compose.prod.yml up -d --scale worker=4
+```
+
+Each replica polls both queues; Valkey distributes tasks across them.
+
+## License
+
+AGPL-3.0 — see the repository `LICENSE` file.

chaoscypher_neuron-0.1.0/pyproject.toml

@@ -0,0 +1,48 @@
+[build-system]
+requires = ["setuptools>=82.0.0,<90", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "chaoscypher-neuron"
+version = "0.1.0"
+description = "Chaos Cypher Neuron - Background task processing workers (worker cells)"
+readme = "README.md"
+license = "AGPL-3.0-only"
+authors = [
+    {name = "Denis MacPherson", email = "denis@chaoscypher.com"}
+]
+keywords = ["knowledge-graph", "workers", "background-tasks", "queue", "ai"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.14",
+]
+requires-python = ">=3.14"
+dependencies = [
+    "chaoscypher-core>=0.1.0",
+    "structlog>=25.5.0,<26",
+    "pyyaml>=6.0.3,<7",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=9.0.0,<10",
+    "pytest-cov>=7.1.0,<8",
+    "pytest-asyncio>=1.3.0,<2",
+]
+
+[project.scripts]
+cc-neuron = "chaoscypher_neuron.worker:main"
+
+[project.urls]
+Homepage = "https://github.com/chaoscypherinc/chaoscypher"
+Documentation = "https://github.com/chaoscypherinc/chaoscypher#readme"
+Repository = "https://github.com/chaoscypherinc/chaoscypher"
+Issues = "https://github.com/chaoscypherinc/chaoscypher/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+chaoscypher_neuron = ["py.typed"]

chaoscypher_neuron-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

chaoscypher_neuron-0.1.0/src/chaoscypher_neuron/__init__.py

@@ -0,0 +1,49 @@
+# Copyright (C) 2024-2026 Chaos Cypher, Inc.
+# SPDX-License-Identifier: AGPL-3.0-only
+
+"""Chaos Cypher Neuron - Background Task Processing Workers.
+
+Background task processing capabilities with a unified worker that handles
+both LLM and Operations queues concurrently. Part of the Neural Architecture.
+
+Queue Configuration:
+    - LLM queue: Serialized AI operations (default 1 concurrent task)
+      (chat, embeddings, tool calls, chunk extraction)
+    - Operations queue: Parallel processing (default 8 concurrent tasks)
+      (source processing, exports, workflows, bulk operations)
+
+Components:
+    - worker: Unified entry point (cc-neuron) running both queues
+    - config: Worker configuration management
+
+Example:
+    Start the unified worker via CLI entry point::
+
+        # Start unified worker (both LLM and Operations queues)
+        cc-neuron
+
+    For programmatic access to configuration::
+
+        from chaoscypher_neuron.config import load_worker_config
+
+        config = load_worker_config("llm_worker")
+        llm_timeout = config["timeout"]
+
+Note:
+    Workers require Valkey connection. Configure via environment:
+    - QUEUE_HOST: Valkey hostname (default: valkey)
+    - QUEUE_PORT: Valkey port (default: 6379)
+
+See Also:
+    - packages/docker/multi-container/docker-compose.dev.yml for service orchestration
+    - packages/cortex/src/chaoscypher_cortex/shared/config for timeout/retry settings
+"""
+
+__version__ = "0.1.0"
+
+# Export config functions and types for external use
+from chaoscypher_neuron.config import load_worker_config
+from chaoscypher_neuron.types import WorkerContext
+
+
+__all__ = ["WorkerContext", "load_worker_config"]