sibyld 1.0.0__tar.gz

sibyld-1.0.0/.gitignore

@@ -0,0 +1,103 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+/lib/
+/lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# Node
+node_modules/
+
+# VitePress
+docs/.vitepress/cache/
+docs/.vitepress/dist/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# Testing
+.coverage
+coverage.xml
+.pytest_cache/
+htmlcov/
+.tox/
+.nox/
+
+# MyPy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Ruff
+.ruff_cache/
+
+# Environment
+.env
+.env.local
+
+# Docker
+falkordb_data/
+
+# Kubernetes / Tilt
+infra/local/secrets.yaml
+.tiltbuild/
+tilt_modules/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Logs
+*.log
+logs/
+
+.serena
+.coverage.*
+
+# moon
+.moon/cache
+.moon/docker
+
+# Backups and database dumps
+backups/
+!apps/web/src/app/(main)/settings/admin/backups/
+!apps/web/src/app/(main)/settings/admin/backups/page.tsx
+*.sql
+!docs/schemas/*.sql
+
+# generated docs
+docs/research/*
+!docs/research/rust-port/
+docs/research/rust-port/*
+!docs/research/rust-port/INVENTORY.md
+
+# Local strategy and planning notes (kept on disk, referenced by Sibyl memory)
+docs/architecture/COMPETITIVE_ANALYSIS_*.md
+docs/architecture/DATA_MODEL_*_2026-*.md
+docs/architecture/LAUNCH_AND_MONETIZATION_*
+docs/architecture/SAAS_HOSTING_STRATEGY_*.md

sibyld-1.0.0/PKG-INFO

@@ -0,0 +1,228 @@
+Metadata-Version: 2.4
+Name: sibyld
+Version: 1.0.0
+Summary: Sibyl daemon - knowledge graph and task workflow server
+Project-URL: Homepage, https://github.com/hyperb1iss/sibyl
+Project-URL: Repository, https://github.com/hyperb1iss/sibyl
+Project-URL: Issues, https://github.com/hyperb1iss/sibyl/issues
+Author-email: Stefanie Jane <stef@hyperbliss.tech>
+License-Expression: Apache-2.0
+Keywords: developer-tools,graph-rag,knowledge-graph,persistent-memory,semantic-search,task-workflow
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Typing :: Typed
+Requires-Python: >=3.13
+Requires-Dist: argon2-cffi>=25.1.0
+Requires-Dist: arq>=0.26.3
+Requires-Dist: authlib<1.8,>=1.7.2
+Requires-Dist: email-validator>=2.3.0
+Requires-Dist: fastapi>=0.115.0
+Requires-Dist: google-genai>=2.8.0
+Requires-Dist: greenlet>=3.3.0
+Requires-Dist: itsdangerous<3,>=2.2
+Requires-Dist: mcp>=1.27.2
+Requires-Dist: pyjwt[crypto]<3,>=2.13.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: sibyl-core[crawler,embeddings,graph,graphrag,llm]==1.0.0
+Requires-Dist: slowapi>=0.1.9
+Requires-Dist: starlette>=1.3.1
+Requires-Dist: surrealdb<3.0,>=2.0.0
+Requires-Dist: tomli-w>=1.2.0
+Requires-Dist: typer>=0.26.7
+Requires-Dist: uvicorn[standard]>=0.38
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
+Requires-Dist: mkdocs>=1.6; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=0.27; extra == 'docs'
+Provides-Extra: generator
+Requires-Dist: anthropic>=0.75; extra == 'generator'
+Provides-Extra: graphrag
+Requires-Dist: networkx>=3.0; extra == 'graphrag'
+Requires-Dist: python-louvain>=0.16; extra == 'graphrag'
+Provides-Extra: graphrag-leiden
+Requires-Dist: igraph>=0.11; extra == 'graphrag-leiden'
+Requires-Dist: leidenalg>=0.10; extra == 'graphrag-leiden'
+Requires-Dist: networkx>=3.0; extra == 'graphrag-leiden'
+Description-Content-Type: text/markdown
+
+# Sibyl API Server
+
+`sibyld` is the FastAPI + FastMCP server behind Sibyl's knowledge graph, agent memory loop, task
+workflow, search, synthesis, and real-time updates.
+
+## Quick Reference
+
+```bash
+# Install the embedded daemon without the web UI
+curl -fsSL https://raw.githubusercontent.com/hyperb1iss/sibyl/main/install.sh | sh -s -- --daemon
+
+# Start server from the monorepo
+moon run api:serve        # or: uv run sibyld serve
+
+# Start worker (Redis coordination only)
+moon run api:worker       # or: uv run sibyld worker
+
+# Quality checks
+moon run api:test         # Run tests
+moon run api:lint         # Lint
+moon run api:typecheck    # Type check
+```
+
+## What's Here
+
+- **MCP Server:** eleven tools for search, context packs, exploration, capture, memory, synthesis,
+  and management
+- **REST API:** 29 routers covering entities, tasks, projects, memory, synthesis, sources, auth,
+  settings, and admin
+- **Auth System:** JWT sessions, GitHub OAuth, API keys with scopes, MCP OAuth clients, RBAC,
+  SMTP-backed password reset
+- **Background Jobs:** in-process local runtime or Redis-backed `arq` workers, including the nightly
+  reflection dream-cycle
+- **WebSocket:** Real-time updates for entities and tasks
+
+## Architecture
+
+```
+Sibyl API (port 3334)
+├── /api/*              → FastAPI REST endpoints
+├── /api/openapi.json   → OpenAPI schema
+├── /mcp                → MCP server (streamable-http, 11 tools)
+├── /api/ws             → WebSocket for real-time updates
+└── Lifespan            → Background jobs + coordination broker
+```
+
+## Key Directories
+
+| Directory       | Purpose                                                                                          |
+| --------------- | ------------------------------------------------------------------------------------------------ |
+| `api/routes/`   | REST endpoints (29 routers: tasks, entities, memory, synthesis, crawler, ingestion, auth, admin) |
+| `ai/`           | DB-backed LLM settings, model validation routes, runtime invalidation                            |
+| `auth/`         | JWT, sessions, API keys, RBAC, MCP OAuth clients                                                 |
+| `persistence/`  | SurrealDB-native runtimes for auth, content, graph, and backups                                  |
+| `crawler/`      | Documentation crawl and ingestion pipeline                                                       |
+| `ingestion/`    | Source import pipeline (mailbox and other adapters)                                              |
+| `jobs/`         | Background jobs (reflection dream-cycle, crawl, backups)                                         |
+| `coordination/` | Local and Redis brokers for jobs, locks, and pub/sub                                             |
+| `email/`        | Transactional email delivery                                                                     |
+| `generator/`    | Synthetic test-data generation                                                                   |
+
+## Configuration
+
+**Required:**
+
+```bash
+SIBYL_JWT_SECRET=...              # Auth (required in production; dev auto-generates)
+SIBYL_ANTHROPIC_API_KEY=...       # Required when LLM provider=anthropic
+# SIBYL_OPENAI_API_KEY=sk-...     # Required when LLM provider=openai
+# SIBYL_GEMINI_API_KEY=...        # Required when LLM provider=gemini
+
+# Embeddings: choose OpenAI or Gemini
+SIBYL_EMBEDDING_PROVIDER=openai   # openai | gemini
+SIBYL_OPENAI_API_KEY=sk-...       # Required when embedding provider=openai
+# SIBYL_GEMINI_API_KEY=...        # Required when embedding provider=gemini
+```
+
+**Optional:**
+
+```bash
+SIBYL_STORE=surreal                   # default; legacy is migration/source-side only
+SIBYL_COORDINATION_BACKEND=auto       # auto | local | redis
+SIBYL_SURREAL_URL=ws://127.0.0.1:8000/rpc
+SIBYL_SURREAL_USERNAME=root
+SIBYL_SURREAL_PASSWORD=root
+SIBYL_REDIS_HOST=127.0.0.1            # only needed for Redis coordination
+SIBYL_REDIS_PORT=6381
+SIBYL_LLM_PROVIDER=anthropic          # anthropic | openai
+SIBYL_LLM_MODEL=claude-haiku-4-5
+SIBYL_LLM_CRAWLER_MODEL=claude-haiku-4-5
+SIBYL_LLM_SYNTHESIS_MODEL=claude-sonnet-4-6
+SIBYL_LLM_TEMPERATURE=0
+SIBYL_LLM_TIMEOUT_SECONDS=60
+SIBYL_EMBEDDING_MODEL=text-embedding-3-small
+SIBYL_EMBEDDING_DIMENSIONS=1536
+SIBYL_GRAPH_EMBEDDING_PROVIDER=openai
+SIBYL_GRAPH_EMBEDDING_MODEL=text-embedding-3-small
+SIBYL_GRAPH_EMBEDDING_DIMENSIONS=1024
+```
+
+PostgreSQL settings are only for historical archive rehearsal commands that explicitly restore a
+retained `postgres.sql` payload against an operator-managed database. They are not part of default
+Surreal runtime startup.
+
+Gemini keys can also be supplied through `GEMINI_API_KEY` or `GOOGLE_API_KEY`. Changing embedding
+provider, model, or dimensions changes vector spaces; re-crawl sources and rebuild graph indexes
+before mixing old and new search results.
+
+LLM settings are instance-wide. Environment variables win over database settings field by field;
+env-backed fields return `409 LOCKED_BY_ENV` on write. Database settings are managed under:
+
+```text
+GET  /api/settings/ai/llm
+PUT  /api/settings/ai/llm/{surface}
+POST /api/settings/ai/llm/{surface}/test
+POST /api/settings/ai/keys/{provider}/test
+POST /api/settings/ai/models/{model_alias}/test
+GET  /api/settings/ai/registry?kind=llm
+```
+
+Crawler extraction and synthesis generation call `sibyl_core.ai` rather than provider SDKs directly.
+Custom model IDs are accepted as database settings with an `unverified_model` warning; validate them
+with the model test endpoint before using them in production flows.
+
+## CLI Commands
+
+```bash
+sibyld serve              # Start the HTTP server
+sibyld serve -t stdio     # Start a stdio server (MCP subprocess mode)
+sibyld worker             # Start the job worker (local mode exits cleanly)
+sibyld up                 # Start data services + API
+sibyld down               # Stop all services
+sibyld db backup          # Back up the graph database
+sibyld migrate import ... # Import a migration archive
+sibyld generate realistic # Generate sample data
+```
+
+## Runtime Modes
+
+For single-machine Surreal development, run `sibyld serve` or `sibyld up` with
+`SIBYL_STORE=surreal`. The default `coordination_backend=auto` resolves to `local`, so background
+jobs, pending state, locks, pub/sub, and schedules all stay in-process with no Redis requirement.
+
+Redis remains available for distributed or multi-process dev. Set `SIBYL_COORDINATION_BACKEND=redis`
+when you want the `arq` worker model, then run `sibyld worker` or `moon run api:worker` separately.
+
+## Key Patterns
+
+**Multi-tenancy:** Every operation requires org context.
+
+```python
+manager = EntityManager(client, group_id=str(org.id))
+```
+
+Write concurrency: the SurrealDB driver serializes WebSocket operations per client. Clone graph
+drivers per organization rather than sharing one driver across org scopes.
+
+**SurrealDB access model:** The API server, worker, CLI, and schema bootstrap flows use configured
+SurrealDB system credentials (`SIBYL_SURREAL_USERNAME` / `SIBYL_SURREAL_PASSWORD`) so they can run
+migrations, background jobs, and admin workflows. Route code must keep explicit org, project, and
+principal predicates because system users sit above table-level permissions.
+
+Auth and content schema migrations also define table permissions for future scoped Surreal record
+users. Tenant-owned tables accept rows where `organization_id` (or `organizations.uuid`) matches
+either `$token.org` from an external JWT access method or `$auth.organization_id` from a Surreal
+record session. Secret-heavy and global tables, such as API keys, sessions, OAuth tokens, system
+settings, and telemetry rollups, remain `PERMISSIONS NONE` for direct scoped DB access.
+
+**Request context:** Auth middleware injects user and org.
+
+```python
+from sibyl.auth.dependencies import get_current_user, get_current_organization
+```
+
+## Dependencies
+
+Depends on `sibyl-core` for models, graph client, AI substrate, and tool implementations.

sibyld-1.0.0/README.md

@@ -0,0 +1,177 @@
+# Sibyl API Server
+
+`sibyld` is the FastAPI + FastMCP server behind Sibyl's knowledge graph, agent memory loop, task
+workflow, search, synthesis, and real-time updates.
+
+## Quick Reference
+
+```bash
+# Install the embedded daemon without the web UI
+curl -fsSL https://raw.githubusercontent.com/hyperb1iss/sibyl/main/install.sh | sh -s -- --daemon
+
+# Start server from the monorepo
+moon run api:serve        # or: uv run sibyld serve
+
+# Start worker (Redis coordination only)
+moon run api:worker       # or: uv run sibyld worker
+
+# Quality checks
+moon run api:test         # Run tests
+moon run api:lint         # Lint
+moon run api:typecheck    # Type check
+```
+
+## What's Here
+
+- **MCP Server:** eleven tools for search, context packs, exploration, capture, memory, synthesis,
+  and management
+- **REST API:** 29 routers covering entities, tasks, projects, memory, synthesis, sources, auth,
+  settings, and admin
+- **Auth System:** JWT sessions, GitHub OAuth, API keys with scopes, MCP OAuth clients, RBAC,
+  SMTP-backed password reset
+- **Background Jobs:** in-process local runtime or Redis-backed `arq` workers, including the nightly
+  reflection dream-cycle
+- **WebSocket:** Real-time updates for entities and tasks
+
+## Architecture
+
+```
+Sibyl API (port 3334)
+├── /api/*              → FastAPI REST endpoints
+├── /api/openapi.json   → OpenAPI schema
+├── /mcp                → MCP server (streamable-http, 11 tools)
+├── /api/ws             → WebSocket for real-time updates
+└── Lifespan            → Background jobs + coordination broker
+```
+
+## Key Directories
+
+| Directory       | Purpose                                                                                          |
+| --------------- | ------------------------------------------------------------------------------------------------ |
+| `api/routes/`   | REST endpoints (29 routers: tasks, entities, memory, synthesis, crawler, ingestion, auth, admin) |
+| `ai/`           | DB-backed LLM settings, model validation routes, runtime invalidation                            |
+| `auth/`         | JWT, sessions, API keys, RBAC, MCP OAuth clients                                                 |
+| `persistence/`  | SurrealDB-native runtimes for auth, content, graph, and backups                                  |
+| `crawler/`      | Documentation crawl and ingestion pipeline                                                       |
+| `ingestion/`    | Source import pipeline (mailbox and other adapters)                                              |
+| `jobs/`         | Background jobs (reflection dream-cycle, crawl, backups)                                         |
+| `coordination/` | Local and Redis brokers for jobs, locks, and pub/sub                                             |
+| `email/`        | Transactional email delivery                                                                     |
+| `generator/`    | Synthetic test-data generation                                                                   |
+
+## Configuration
+
+**Required:**
+
+```bash
+SIBYL_JWT_SECRET=...              # Auth (required in production; dev auto-generates)
+SIBYL_ANTHROPIC_API_KEY=...       # Required when LLM provider=anthropic
+# SIBYL_OPENAI_API_KEY=sk-...     # Required when LLM provider=openai
+# SIBYL_GEMINI_API_KEY=...        # Required when LLM provider=gemini
+
+# Embeddings: choose OpenAI or Gemini
+SIBYL_EMBEDDING_PROVIDER=openai   # openai | gemini
+SIBYL_OPENAI_API_KEY=sk-...       # Required when embedding provider=openai
+# SIBYL_GEMINI_API_KEY=...        # Required when embedding provider=gemini
+```
+
+**Optional:**
+
+```bash
+SIBYL_STORE=surreal                   # default; legacy is migration/source-side only
+SIBYL_COORDINATION_BACKEND=auto       # auto | local | redis
+SIBYL_SURREAL_URL=ws://127.0.0.1:8000/rpc
+SIBYL_SURREAL_USERNAME=root
+SIBYL_SURREAL_PASSWORD=root
+SIBYL_REDIS_HOST=127.0.0.1            # only needed for Redis coordination
+SIBYL_REDIS_PORT=6381
+SIBYL_LLM_PROVIDER=anthropic          # anthropic | openai
+SIBYL_LLM_MODEL=claude-haiku-4-5
+SIBYL_LLM_CRAWLER_MODEL=claude-haiku-4-5
+SIBYL_LLM_SYNTHESIS_MODEL=claude-sonnet-4-6
+SIBYL_LLM_TEMPERATURE=0
+SIBYL_LLM_TIMEOUT_SECONDS=60
+SIBYL_EMBEDDING_MODEL=text-embedding-3-small
+SIBYL_EMBEDDING_DIMENSIONS=1536
+SIBYL_GRAPH_EMBEDDING_PROVIDER=openai
+SIBYL_GRAPH_EMBEDDING_MODEL=text-embedding-3-small
+SIBYL_GRAPH_EMBEDDING_DIMENSIONS=1024
+```
+
+PostgreSQL settings are only for historical archive rehearsal commands that explicitly restore a
+retained `postgres.sql` payload against an operator-managed database. They are not part of default
+Surreal runtime startup.
+
+Gemini keys can also be supplied through `GEMINI_API_KEY` or `GOOGLE_API_KEY`. Changing embedding
+provider, model, or dimensions changes vector spaces; re-crawl sources and rebuild graph indexes
+before mixing old and new search results.
+
+LLM settings are instance-wide. Environment variables win over database settings field by field;
+env-backed fields return `409 LOCKED_BY_ENV` on write. Database settings are managed under:
+
+```text
+GET  /api/settings/ai/llm
+PUT  /api/settings/ai/llm/{surface}
+POST /api/settings/ai/llm/{surface}/test
+POST /api/settings/ai/keys/{provider}/test
+POST /api/settings/ai/models/{model_alias}/test
+GET  /api/settings/ai/registry?kind=llm
+```
+
+Crawler extraction and synthesis generation call `sibyl_core.ai` rather than provider SDKs directly.
+Custom model IDs are accepted as database settings with an `unverified_model` warning; validate them
+with the model test endpoint before using them in production flows.
+
+## CLI Commands
+
+```bash
+sibyld serve              # Start the HTTP server
+sibyld serve -t stdio     # Start a stdio server (MCP subprocess mode)
+sibyld worker             # Start the job worker (local mode exits cleanly)
+sibyld up                 # Start data services + API
+sibyld down               # Stop all services
+sibyld db backup          # Back up the graph database
+sibyld migrate import ... # Import a migration archive
+sibyld generate realistic # Generate sample data
+```
+
+## Runtime Modes
+
+For single-machine Surreal development, run `sibyld serve` or `sibyld up` with
+`SIBYL_STORE=surreal`. The default `coordination_backend=auto` resolves to `local`, so background
+jobs, pending state, locks, pub/sub, and schedules all stay in-process with no Redis requirement.
+
+Redis remains available for distributed or multi-process dev. Set `SIBYL_COORDINATION_BACKEND=redis`
+when you want the `arq` worker model, then run `sibyld worker` or `moon run api:worker` separately.
+
+## Key Patterns
+
+**Multi-tenancy:** Every operation requires org context.
+
+```python
+manager = EntityManager(client, group_id=str(org.id))
+```
+
+Write concurrency: the SurrealDB driver serializes WebSocket operations per client. Clone graph
+drivers per organization rather than sharing one driver across org scopes.
+
+**SurrealDB access model:** The API server, worker, CLI, and schema bootstrap flows use configured
+SurrealDB system credentials (`SIBYL_SURREAL_USERNAME` / `SIBYL_SURREAL_PASSWORD`) so they can run
+migrations, background jobs, and admin workflows. Route code must keep explicit org, project, and
+principal predicates because system users sit above table-level permissions.
+
+Auth and content schema migrations also define table permissions for future scoped Surreal record
+users. Tenant-owned tables accept rows where `organization_id` (or `organizations.uuid`) matches
+either `$token.org` from an external JWT access method or `$auth.organization_id` from a Surreal
+record session. Secret-heavy and global tables, such as API keys, sessions, OAuth tokens, system
+settings, and telemetry rollups, remain `PERMISSIONS NONE` for direct scoped DB access.
+
+**Request context:** Auth middleware injects user and org.
+
+```python
+from sibyl.auth.dependencies import get_current_user, get_current_organization
+```
+
+## Dependencies
+
+Depends on `sibyl-core` for models, graph client, AI substrate, and tool implementations.