@bodhi-ventures/aiocs 0.1.2 → 0.3.0

package/docs/2026-03-26-agent-json-and-daemon-design.md

@@ -1,157 +0,0 @@
-# AI Agent JSON CLI And Daemon Design
-
-## Summary
-
-`aiocs` will add a machine-oriented JSON contract across the CLI and a first-class long-running daemon mode for scheduled refreshes. The CLI remains human-friendly by default, but every command will support a global `--json` flag so agents can consume one stable structured payload instead of parsing text. The daemon will live inside the same binary as `docs daemon`, and a Docker image will run that command in a loop with environment-configured cadence.
-
-## Goals
-
-- make every one-shot CLI command safe for direct agent use
-- keep a single canonical implementation path inside the existing CLI
-- avoid introducing a separate HTTP service or second control plane
-- support a long-running local container that keeps the shared catalog warm
-
-## Non-Goals
-
-- MCP in this change
-- remote registries or distributed scheduling
-- a second machine API beyond the CLI
-
-## JSON Output Contract
-
-### Scope
-
-`--json` is a root-level global flag that applies to every CLI command.
-
-### One-shot commands
-
-These commands emit exactly one JSON document to stdout:
-
-- `source upsert`
-- `source list`
-- `fetch`
-- `refresh due`
-- `snapshot list`
-- `project link`
-- `project unlink`
-- `search`
-- `show`
-
-### Envelope
-
-Every command returns:
-
-```json
-{
-  "ok": true,
-  "command": "source.list",
-  "data": {}
-}
-```
-
-Failures also emit a single JSON document to stdout, with exit code `1`:
-
-```json
-{
-  "ok": false,
-  "command": "search",
-  "error": {
-    "message": "No linked project scope found. Use --source or --all."
-  }
-}
-```
-
-### Command payloads
-
-- `source.upsert`: upserted source metadata
-- `source.list`: array of sources with due/snapshot fields
-- `fetch`: array of per-source fetch results, even for a single source
-- `refresh.due`: array of per-source fetch results; empty array when nothing is due
-- `snapshot.list`: array of snapshots
-- `project.link`: canonical project path and linked source ids
-- `project.unlink`: canonical project path and removed scope
-- `search`: array of chunk results
-- `show`: one chunk result
-
-### Daemon exception
-
-`docs daemon` is long-running, so a single final JSON document is the wrong shape. In JSON mode it will emit newline-delimited JSON event objects to stdout, one event per lifecycle action. This is the one intended exception to the single-document rule.
-
-## Daemon Design
-
-### Command
-
-Add `docs daemon`.
-
-### Responsibilities
-
-- ensure config and data directories exist
-- optionally bootstrap source specs from configured directories
-- optionally run an immediate refresh cycle on startup
-- loop forever:
-  - upsert any source spec files from configured directories
-  - run refresh for due sources
-  - sleep until the next cycle
-
-### Environment variables
-
-- `AIOCS_DAEMON_INTERVAL_MINUTES`
-  - required positive integer semantics, default `60`
-- `AIOCS_DAEMON_FETCH_ON_START`
-  - `true` by default
-- `AIOCS_SOURCE_SPEC_DIRS`
-  - comma-separated list of directories to scan for `.yaml`, `.yml`, and `.json` source specs
-  - default points at the bundled `sources/` directory in the image and local repo
-
-### Logging
-
-- human mode: concise single-line operational logs
-- JSON mode: one JSON event per line with `event`, `timestamp`, and event-specific fields
-
-### Failure model
-
-- invalid env config fails fast at startup
-- invalid source spec files fail the cycle and are logged explicitly
-- fetch failures for one source do not kill the daemon process unless startup config is invalid
-
-## Docker Design
-
-### Image
-
-Ship a Dockerfile that builds `aiocs`, includes the bundled `sources/` directory, and runs:
-
-```bash
-./dist/cli.js daemon
-```
-
-### Runtime contract
-
-- mount persistent data to `/root/.aiocs/data` or provide `AIOCS_DATA_DIR`
-- optional config mount for `/root/.aiocs/config`
-- source specs available from bundled `/app/sources` by default
-- allow overriding `AIOCS_SOURCE_SPEC_DIRS` with mounted custom directories
-
-### Compose
-
-Ship a compose example that:
-
-- builds the image locally
-- mounts a persistent volume for the data directory
-- sets `AIOCS_DAEMON_INTERVAL_MINUTES`
-- optionally mounts a host directory of custom source specs
-
-## Testing Strategy
-
-- CLI tests for `--json` across representative commands and failure paths
-- unit tests for daemon env parsing and cycle behavior
-- integration tests for daemon bootstrap + due refresh behavior with a short injected interval
-- existing CLI/fetch regression suite stays green in human mode
-
-## Risks And Mitigations
-
-- daemon JSON logs differ from one-shot JSON
-  - mitigate by documenting daemon as the explicit streaming exception
-- source spec drift inside long-running containers
-  - mitigate by re-upserting source specs each cycle
-- duplicated output logic across commands
-  - mitigate by centralizing response/error emission in one CLI output path

package/docs/2026-03-28-hybrid-search-design.md

@@ -1,423 +0,0 @@
-# aiocs Hybrid Search Design
-
-Date: 2026-03-28
-
-## Goal
-
-Add hybrid retrieval to `aiocs` so agents get better docs results for fuzzy and conceptual queries without weakening the current source/snapshot semantics.
-
-This design keeps `aiocs` as the canonical docs system:
-
-- `aiocs` owns fetching, normalization, chunking, snapshots, canaries, diffs, and ranking policy
-- SQLite FTS5 remains the primary lexical index and source of truth
-- a dedicated `aiocs-qdrant` container stores only derived embedding vectors for `aiocs`
-- local Ollama generates embeddings
-- SocratiCode remains separate and may later consume selected `aiocs` snapshots, but it is not the runtime for `aiocs` hybrid search
-
-## Non-Goals
-
-- No replacement of SQLite FTS5 with vector-only search
-- No reuse of SocratiCode's Qdrant collection or deployment
-- No new hosted service or remote dependency
-- No cross-repo code search in this phase
-- No backup/restore of vector state; vectors are rebuildable
-
-## Why This Shape
-
-The current `aiocs` search path in [catalog.ts](../src/catalog/catalog.ts) is pure FTS5 BM25 over the latest successful snapshots. That is excellent for exact docs lookups, versioned terms, and API names. It is weaker for:
-
-- synonym-heavy prompts
-- conceptual questions
-- vague agent prompts
-- recall across wording shifts in docs
-
-Hybrid retrieval is the right improvement, but the ranking policy must remain docs-aware. That is why `aiocs` itself should own the hybrid query plan instead of delegating search semantics to a generic vector layer.
-
-## Recommended Architecture
-
-### 1. Canonical storage remains SQLite
-
-SQLite remains the system of record for:
-
-- sources
-- snapshots
-- pages
-- chunks
-- project links
-- fetch/canary/daemon metadata
-
-Add embedding-specific metadata tables in the same catalog:
-
-- `embedding_models`
-- `embedding_jobs`
-- `embedding_state`
-
-These track derived vector work, not source content.
-
-### 2. Dedicated `aiocs-qdrant`
-
-Ship a separate Qdrant container in `aiocs/docker-compose.yml`:
-
-- service name: `aiocs-qdrant`
-- dedicated persistent volume
-- default local URL from `aiocs` runtime
-
-This container is strictly for `aiocs`. It must not share collections or lifecycle with SocratiCode.
-
-### 3. Ollama as embedding provider
-
-Use Ollama locally for embeddings, with explicit config:
-
-- provider: `ollama`
-- model: configurable
-- default model chosen from the local setup you already use for embeddings
-
-`aiocs` should own its own embedding config even if the model matches SocratiCode.
-
-### 4. Hybrid retrieval strategy
-
-Search modes:
-
-- `lexical`
-- `hybrid`
-- `semantic`
-- `auto`
-
-Default: `auto`
-
-Behavior:
-
-- if vector infra is healthy and the target scope has embeddings, use hybrid
-- otherwise fall back to lexical
-
-Hybrid query plan:
-
-1. Run FTS5 BM25 against SQLite
-2. Run vector similarity search in Qdrant
-3. Fuse result sets with Reciprocal Rank Fusion
-4. Return the existing `aiocs` chunk shape plus hybrid metadata
-
-RRF is preferred over weighted score mixing because:
-
-- BM25 and cosine/dot-product scores are not directly comparable
-- RRF is robust across model swaps
-- RRF is simple and stable for agents
-
-## Data Model
-
-### SQLite additions
-
-#### `embedding_models`
-
-Tracks the embedding configuration currently in use.
-
-Columns:
-
-- `id`
-- `provider`
-- `model`
-- `dimension`
-- `distance_metric`
-- `created_at`
-- `active`
-
-#### `embedding_state`
-
-Tracks per-chunk embedding lifecycle.
-
-Columns:
-
-- `chunk_id`
-- `source_id`
-- `snapshot_id`
-- `embedding_model_id`
-- `content_hash`
-- `vector_id`
-- `status` (`pending`, `embedded`, `stale`, `failed`)
-- `last_embedded_at`
-- `last_error`
-
-This avoids guessing whether a vector is current.
-
-#### `embedding_jobs`
-
-Persistent queue for background embedding work.
-
-Columns:
-
-- `id`
-- `source_id`
-- `snapshot_id`
-- `job_type` (`snapshot_latest`, `snapshot_remove`, `reindex_model`)
-- `status` (`pending`, `running`, `failed`, `completed`)
-- `attempt_count`
-- `last_error`
-- `created_at`
-- `started_at`
-- `finished_at`
-
-This queue is important because embedding is slower and more failure-prone than lexical indexing.
-
-### Qdrant payload
-
-Each vector point stores:
-
-- `chunk_id`
-- `source_id`
-- `snapshot_id`
-- `page_url`
-- `page_title`
-- `section_title`
-- `embedding_model_id`
-- `content_hash`
-- `is_latest_snapshot`
-
-The point id should be stable and deterministic per chunk/model, for example:
-
-- `${embedding_model_id}:${chunk_id}:${content_hash}`
-
-That makes reindexing idempotent.
-
-## Indexing Lifecycle
-
-### Snapshot write path
-
-When `recordSuccessfulSnapshot()` writes chunks:
-
-1. normal SQLite snapshot/page/chunk write happens first
-2. `aiocs` marks the new latest snapshot as requiring embeddings
-3. an embedding job is enqueued
-
-The fetch path must not block on embedding completion. Search must continue working lexically even with zero vectors.
-
-### Latest-only vector policy
-
-For this phase, vectors should be generated only for the latest successful snapshot per source.
-
-Rationale:
-
-- minimizes vector volume
-- aligns with how current `search()` already targets latest successful snapshots by default
-- avoids wasting GPU/CPU on historical snapshots rarely used in normal retrieval
-
-Historical snapshot diffs remain SQLite-only.
-
-If a source gets a new latest snapshot:
-
-- mark prior latest vectors as stale
-- enqueue cleanup/remove for stale vectors
-- enqueue embedding for the new latest snapshot
-
-### Embedding worker
-
-Add an embedding worker loop to the daemon process:
-
-- fetch/canary cycle remains unchanged in purpose
-- after refresh work, process a bounded number of embedding jobs
-- retry failed jobs with capped attempts and backoff
-
-The daemon becomes the single operational background process for both freshness and vector health.
-
-## Query Path
-
-### Lexical path
-
-Keep the existing query path as-is for:
-
-- `searchMode=lexical`
-- `searchMode=auto` when vectors are unavailable
-
-### Semantic path
-
-For `searchMode=semantic`:
-
-1. embed the query with Ollama
-2. query Qdrant with the same scope constraints
-3. fetch result chunk records from SQLite by `chunk_id`
-4. return ordered results
-
-### Hybrid path
-
-For `searchMode=hybrid`:
-
-1. run lexical query for top `N`
-2. run semantic query for top `K`
-3. fuse with RRF
-4. fetch canonical chunk data from SQLite
-5. return result rows with mode metadata
-
-Initial defaults:
-
-- BM25 candidate window: `40`
-- vector candidate window: `40`
-- final page size: existing `limit`
-- RRF `k`: `60`
-
-These should be configurable, but not user-tuned in the first release.
-
-## Filtering and Invariants
-
-All source/snapshot/project scoping remains authoritative in `aiocs`, not in Qdrant.
-
-The runtime must:
-
-- resolve project scope in SQLite first
-- resolve latest snapshot ids in SQLite first
-- constrain vector retrieval to those snapshot ids
-
-This preserves the current guarantee that source/project/snapshot filters are exact.
-
-Qdrant is a retrieval backend, not a source of truth.
-
-## CLI and MCP Changes
-
-### CLI
-
-Extend `docs search` with:
-
-- `--mode lexical|hybrid|semantic|auto`
-
-Add operational commands:
-
-- `docs embeddings status`
-- `docs embeddings backfill [source-id|all]`
-- `docs embeddings clear [source-id|all]`
-
-Optional later:
-
-- `docs embeddings doctor`
-
-### MCP
-
-Extend `search` input with `mode`.
-
-Add tools:
-
-- `embeddings_status`
-- `embeddings_backfill`
-- `embeddings_clear`
-
-The existing JSON envelope remains unchanged.
-
-## Docker and Runtime
-
-### `docker-compose.yml`
-
-Add:
-
-- `aiocs-qdrant`
-- volume for Qdrant storage
-- daemon env vars for Qdrant/Ollama config
-
-The daemon container should depend on Qdrant health, not just startup ordering.
-
-### Config
-
-Add environment variables:
-
-- `AIOCS_SEARCH_MODE_DEFAULT`
-- `AIOCS_QDRANT_URL`
-- `AIOCS_QDRANT_COLLECTION`
-- `AIOCS_EMBEDDING_PROVIDER`
-- `AIOCS_OLLAMA_BASE_URL`
-- `AIOCS_OLLAMA_EMBEDDING_MODEL`
-- `AIOCS_EMBEDDING_BATCH_SIZE`
-- `AIOCS_EMBEDDING_JOB_LIMIT_PER_CYCLE`
-
-Defaults should make local Docker + local Ollama work without extra ceremony.
-
-## Doctor and Health
-
-Extend `doctor` with new checks:
-
-- `qdrant`
-- `embedding-provider`
-- `embedding-coverage`
-- `embedding-backlog`
-
-Examples:
-
-- pass: vectors are healthy and mostly current
-- warn: lexical search works, but vectors are unavailable or backlog is growing
-- fail: `searchMode=hybrid` default is configured but vector infra is broken
-
-## Backups
-
-Backups remain SQLite/config only.
-
-Do not export Qdrant state in `backup export`.
-
-After `backup import`:
-
-- mark embeddings stale
-- enqueue re-embedding for current latest snapshots
-
-This keeps backup semantics simple and avoids trying to synchronize two storage engines.
-
-## Testing Strategy
-
-### Unit
-
-- query mode parsing
-- embedding config validation
-- RRF fusion
-- deterministic vector id generation
-- embedding-state transitions
-
-### Integration
-
-- snapshot creation enqueues embedding work
-- daemon processes embedding jobs
-- hybrid search falls back cleanly when vector infra is absent
-- hybrid search respects source/project/snapshot filters
-- `backup import` triggers rebuild behavior
-
-### Docker/runtime
-
-- compose config includes dedicated Qdrant service
-- doctor reports degraded state when Qdrant is unreachable
-
-## Migration Strategy
-
-1. add schema and config surfaces
-2. add Qdrant/Ollama client integration
-3. add embedding queue and daemon worker
-4. add hybrid query mode
-5. add doctor/docs/tests
-
-This keeps lexical search live throughout the rollout.
-
-## Risks
-
-### 1. Ranking regressions
-
-Mitigation:
-
-- lexical remains available
-- `auto` falls back safely
-- RRF instead of fragile score blending
-
-### 2. Embedding backlog growth
-
-Mitigation:
-
-- latest-only vector policy
-- bounded jobs per cycle
-- explicit backlog health checks
-
-### 3. Vector/schema drift
-
-Mitigation:
-
-- embedding model registry in SQLite
-- deterministic point ids
-- explicit stale/rebuild lifecycle
-
-## Recommended Next Step
-
-Turn this design into an execution plan and implement it in phases, starting with:
-
-1. embedding config + schema
-2. dedicated Qdrant runtime
-3. daemon embedding worker
-4. hybrid search mode

package/docs/superpowers/specs/2026-03-29-tag-driven-release-pipeline-design.md

@@ -1,135 +0,0 @@
-# Tag-Driven Release Pipeline Design
-
-## Summary
-
-`aiocs` should release as the public scoped npm package `@bodhi-ventures/aiocs` through a stable, tag-driven GitHub Actions workflow. The repository remains the source of truth for versioning. Releases are created only from pushed stable tags of the form `vX.Y.Z`.
-
-The workflow must never mutate git state. It should validate the tag against `package.json`, run the full release verification stack, publish publicly to npm with provenance, and create a GitHub release from the same tag.
-
-## Goals
-
-- publish `aiocs` publicly under `@bodhi-ventures/aiocs`
-- eliminate workflow-managed version bumps, commits, and tag creation
-- remove improvised git author configuration from the release workflow
-- make the release contract deterministic and easy to reason about
-- keep CI and release validation aligned with the actual shipped package surface
-
-## Non-Goals
-
-- prerelease publishing
-- automatic releases from `main`
-- workspace/multi-package publishing
-- alternative binary distribution outside npm
-
-## Current State
-
-- `package.json` still uses the unscoped package name `aiocs`
-- the current release workflow is `workflow_dispatch`-driven
-- the workflow edits `package.json`, creates commits and tags, and configures a bot git identity
-- release automation is more complex than necessary and mixes version mutation with publishing
-
-## Chosen Design
-
-### Package Identity
-
-- rename the npm package to `@bodhi-ventures/aiocs`
-- keep the package public
-- keep CLI command names unchanged:
-  - `docs`
-  - `aiocs-mcp`
-- add explicit `publishConfig`:
-  - `access: public`
-  - `provenance: true`
-
-### Release Trigger
-
-- release workflow triggers only on pushed stable semver tags matching `v*.*.*`
-- the tag version must match `package.json.version` exactly
-- only stable releases are supported; prerelease tags are rejected
-
-### Release Workflow Behavior
-
-- check out the tagged revision
-- install dependencies and release prerequisites
-- fail fast unless all of these are true:
-  - tag matches `vX.Y.Z`
-  - `package.json.version === X.Y.Z`
-  - `package.json.name === @bodhi-ventures/aiocs`
-- run full release validation:
-  - `pnpm lint`
-  - `pnpm test`
-  - `pnpm build`
-  - `npm pack --dry-run`
-  - built CLI smoke, at minimum `node dist/cli.js --version`
-- publish to npm with the existing `NPM_TOKEN` org secret
-- create a GitHub release for the pushed tag
-- if a GitHub release already exists for that tag, do not recreate it
-
-### Rerun and Partial-Failure Policy
-
-Tag releases must be safely rerunnable.
-
-- `workflow_dispatch` is removed entirely; tag pushes are the only release trigger
-- if a rerun sees that `@bodhi-ventures/aiocs@X.Y.Z` is already published on npm, it must skip `npm publish` instead of failing
-- if a rerun sees that the GitHub release for `vX.Y.Z` already exists, it must skip release creation instead of failing
-- validation steps always run on every attempt, even when publish/release steps are skipped
-- if npm already contains `@bodhi-ventures/aiocs@X.Y.Z` but the checked-out tag does not match `package.json.version === X.Y.Z` or `package.json.name === @bodhi-ventures/aiocs`, the workflow must fail fast
-- the workflow should treat npm and GitHub release publication as independently idempotent so a partial success can be completed by rerunning the same tag job
-
-### Git Behavior
-
-- the workflow never edits files
-- the workflow never commits
-- the workflow never creates tags
-- the workflow never configures a synthetic git author
-
-Version bumps happen in normal development flow:
-
-1. update `package.json.version`
-2. commit the version bump
-3. create tag `vX.Y.Z`
-4. push commit and tag
-
-## CI Alignment
-
-CI remains the pre-release gate and should stay close to the release workflow:
-
-- install dependencies
-- install Playwright Chromium
-- run lint, tests, build, and `npm pack --dry-run`
-- validate Docker image and compose config
-- smoke test the packaged CLI surface
-
-CI should also assert package metadata consistency where useful, especially the scoped package name.
-
-## Documentation Changes
-
-Update repository docs to reflect the scoped public package and release process:
-
-- install command becomes `npm install -g @bodhi-ventures/aiocs`
-- add a short release section to the README describing the tag-based flow
-- keep the CLI-facing command examples unchanged
-
-## Risks
-
-- npm org publishing can still fail if org/package permissions are not configured correctly on npmjs
-- the first release is the highest-risk run because the scoped package has not been proven yet
-- GitHub Actions can only validate the workflow structure locally; final proof requires one live tag release
-
-## Acceptance Criteria
-
-- `package.json` is updated to `@bodhi-ventures/aiocs`
-- release workflow triggers on stable tags only
-- release workflow does not mutate git state
-- release workflow validates tag/version/name consistency before publishing
-- README/install docs reference the scoped package name
-- local verification passes on the updated tree:
-  - lint
-  - tests
-  - build
-  - `npm pack --dry-run`
-- the first real release can be executed by bumping version, pushing a `vX.Y.Z` tag, and observing npm + GitHub release creation
-
-## Follow-Up
-
-After implementation, do one real tagged stable release to prove the pipeline end to end.