@bodhi-ventures/aiocs 0.1.2 → 0.3.0

package/README.md

@@ -5,6 +5,7 @@ Local-only documentation fetch, versioning, and search CLI for AI agents.
 ## What it does
 
 - fetches docs from websites with Playwright
+- snapshots curated external git repositories as shared local reference sources
 - supports authenticated sources via environment-backed headers and cookies
 - runs lightweight canaries to detect source drift before full refreshes
 - normalizes them into Markdown
@@ -12,6 +13,7 @@ Local-only documentation fetch, versioning, and search CLI for AI agents.
 - diffs snapshots to show what changed between fetches
 - indexes heading-aware chunks in SQLite FTS5
 - adds optional hybrid retrieval with local Ollama embeddings and a dedicated Qdrant vector index
+- compiles derived research workspaces into Markdown wiki artifacts with LM Studio
 - links docs sources to local projects for scoped search
 - exports and imports manifest-backed backups for `~/.aiocs`
 
@@ -110,10 +112,21 @@ GitHub Actions publishes `@bodhi-ventures/aiocs` publicly to npm and creates the
 
 For Codex-first setup, automatic-use guidance, MCP recommendations, and agent definitions, see [docs/codex-integration.md](./docs/codex-integration.md).
 
+Canonical Codex setup:
+
+- register `aiocs-mcp` in `~/.codex/config.toml`
+- link `skills/aiocs` into `~/.codex/skills/aiocs`
+- link `skills/aiocs-curation` into `~/.codex/skills/aiocs-curation`
+- keep the optional specialist subagent linked only as a fallback for heavier docs workflows
+
 ## Managed sources
 
-The open-source repo bundles `hyperliquid` in `sources/`. Additional machine-local source specs
-belong in `~/.aiocs/sources`.
+The open-source repo bundles both web and git sources in `sources/`:
+
+- `hyperliquid` for the public docs site
+- `nktkas-hyperliquid` for the `nktkas/hyperliquid` GitHub repository
+
+Additional machine-local source specs belong in `~/.aiocs/sources`.
 
 `docs init` bootstraps both managed locations, so source behavior is the same regardless of
 whether a spec lives in the repo or in `~/.aiocs/sources`.
@@ -170,6 +183,7 @@ Search and inspect results:
 
 ```bash
 docs search "maker flow" --source hyperliquid
+docs search "WebSocketTransport" --source nktkas-hyperliquid --path "src/**" --language typescript
 docs search "maker flow" --source hyperliquid --mode lexical
 docs search "maker flow" --source hyperliquid --mode hybrid
 docs search "maker flow" --source hyperliquid --mode semantic
@@ -187,6 +201,83 @@ docs verify coverage hyperliquid /absolute/path/to/reference.md
 
 When `docs search` runs inside a linked project, it automatically scopes to that project's linked sources unless `--source` or `--all` is provided.
 
+### Research workspaces
+
+`aiocs` can compile a derived workspace wiki on top of canonical source snapshots. Workspace artifacts are stored separately from fetched sources under `~/.aiocs/data/workspaces/<workspace-id>/` and keep provenance links back to source chunks.
+
+LM Studio is the v1 compiler runtime. Load `google/gemma-4-26b-a4b` in LM Studio, start the local API server, and make sure `doctor` can see it:
+
+```bash
+docs --json doctor
+```
+
+If LM Studio is running on a non-default host or model, override:
+
+```bash
+export AIOCS_LMSTUDIO_BASE_URL=ws://127.0.0.1:1234
+export AIOCS_LMSTUDIO_MODEL=google/gemma-4-26b-a4b
+```
+
+These environment overrides apply at runtime, including for already-created workspaces.
+
+Create, bind, compile, and search a workspace:
+
+```bash
+docs workspace create market-structure --label "Market Structure" --auto-compile
+docs workspace bind market-structure hyperliquid nktkas-hyperliquid
+docs workspace compile market-structure
+docs workspace status market-structure
+docs workspace queue-run
+docs workspace search market-structure "transport and websocket design" --scope mixed
+docs workspace artifact list market-structure
+docs workspace artifact show market-structure derived/index.md
+docs workspace lint market-structure
+docs workspace output market-structure report --name weekly-brief
+docs workspace output market-structure slides --name weekly-brief
+docs workspace answer market-structure note "What changed in websocket transport?" --name websocket-note
+```
+
+Workspace compile and maintenance are incremental:
+
+- canonical source snapshots remain the source of truth
+- unchanged sources are skipped on later compiles
+- changed sources regenerate only their derived summaries/concepts plus the shared index
+- changed raw inputs regenerate only their derived summaries/concepts plus the shared index
+- compiled artifacts get deterministic graph navigation with outgoing relations and backlinks
+- workspace lint persists durable suggestion artifacts for duplicate concepts, missing articles, and follow-up questions
+- generated reports/slides keep provenance to the bound source snapshots
+- generated notes/reports/slides are marked stale when their inputs change
+
+Opt-in automatic recompilation:
+
+- set `--auto-compile` when creating the workspace, or later with `docs workspace configure <workspace-id> --auto-compile true`
+- the daemon queues recompiles after bound source refreshes
+- run `docs workspace queue-run` to process queued jobs outside the daemon
+
+Workspace-scoped raw ingest:
+
+```bash
+docs workspace ingest add market-structure markdown-dir /absolute/path/to/notes --label "Research notes"
+docs workspace ingest add market-structure pdf /absolute/path/to/paper.pdf --label "Paper PDF"
+docs workspace ingest add market-structure image /absolute/path/to/diagram.png --label "Market diagram"
+docs workspace ingest add market-structure csv /absolute/path/to/fills.csv --label "Fills CSV"
+docs workspace ingest add market-structure json /absolute/path/to/manifest.json --label "Research manifest"
+docs workspace ingest add market-structure jsonl /absolute/path/to/events.jsonl --label "Events JSONL"
+docs workspace ingest list market-structure
+docs workspace ingest search market-structure "fee tier" --kind markdown-dir
+docs workspace ingest show market-structure markdown-dir-notes-<hash>
+```
+
+Obsidian sync:
+
+```bash
+docs workspace sync obsidian market-structure /absolute/path/to/obsidian-vault
+```
+
+The workspace is synced under `aiocs/<workspace-id>/` by default so the raw, derived, and output trees stay grouped inside the vault.
+
+If any bound source snapshot changes, rerun `docs workspace compile <workspace-id>` before `docs workspace output ...`. Output generation fails closed when required derived artifacts are stale.
+
 For agents, the intended decision order is:
 
 1. check `source list` or scoped `search` first
@@ -194,6 +285,34 @@ For agents, the intended decision order is:
 3. if the source is missing but worth reusing, add a spec under `~/.aiocs/sources`, then upsert and refresh only that source
 4. avoid `fetch all` unless the user explicitly asks or the daemon is doing maintenance
 
+### Git repo sources
+
+`aiocs` supports first-class `kind: git` sources for curated external repositories that should be
+reused across multiple local projects.
+
+Example:
+
+```yaml
+kind: git
+id: nktkas-hyperliquid
+label: nktkas/hyperliquid Repo
+repo:
+  url: https://github.com/nktkas/hyperliquid.git
+  ref: main
+  include:
+    - README.md
+    - docs/**
+    - src/**
+  exclude:
+    - .github/**
+    - dist/**
+schedule:
+  everyHours: 24
+```
+
+Git source snapshots are commit-based, stored under the shared local catalog, and searchable with
+the same project linking, diffing, canary, and hybrid search flows as website docs.
+
 ### Hybrid search
 
 `aiocs` keeps SQLite FTS5/BM25 as the canonical lexical index and adds an optional hybrid layer:
@@ -409,7 +528,7 @@ The repo ships two GitHub Actions workflows:
 - [ci.yml](./.github/workflows/ci.yml): validation for lint, tests, build, pack, and Docker smoke coverage
 - [release.yml](./.github/workflows/release.yml): tag-driven stable release flow that validates the tagged package state, publishes to npm, and creates a GitHub release
 
-The release workflow is triggered only by pushed stable tags matching `vX.Y.Z` and expects `NPM_TOKEN` in repository secrets. The release job is retryable: if `@bodhi-ventures/aiocs@X.Y.Z` already exists on npm or the GitHub release already exists for `vX.Y.Z`, the workflow skips the completed publication step and finishes the remaining one.
+The release workflow is triggered only by pushed stable tags matching `vX.Y.Z` and expects an npm publish token in GitHub repository secrets. The release job is retryable: if `@bodhi-ventures/aiocs@X.Y.Z` already exists on npm or the GitHub release already exists for `vX.Y.Z`, the workflow skips the completed publication step and finishes the remaining one.
 
 Successful MCP results use an envelope: