@zokizuan/satori-mcp 4.11.17 → 6.2.0

package/README.md

@@ -1,59 +1,29 @@
 # @zokizuan/satori-mcp
 
-Read-only MCP server for Satori. It gives coding agents six deterministic tools for repo search, symbol navigation, call graph context, bounded file reads, and index lifecycle management.
+The MCP server for [Satori](https://github.com/ham-zax/satori): freshness-aware hybrid code search, symbol navigation, advisory call graphs, bounded source reads, and index lifecycle management.
 
-## Install
+Most users should install Satori through `@zokizuan/satori-cli`. The installer writes a stable local launcher and configures supported MCP clients; this package does not manage client configuration by itself.
 
-Installer ownership is **`@zokizuan/satori-cli` only** (this package serves MCP tools; it does not install client configs). Use the CLI installer for normal setup:
+## Install
 
 ```bash
-npx -y @zokizuan/satori-cli@latest install --client all
+npx -y @zokizuan/satori-cli@latest install --client all --runtime offline
 npx -y @zokizuan/satori-cli@latest doctor
 ```
 
-The CLI installer supports `codex`, `claude`, `opencode`, and `all`. It creates the runtime cache, writes the stable launcher, and writes client config for you. Avoid using `npx` as the resident MCP server command; first-run package resolution can exceed normal MCP startup timeouts.
+The local Potion runtime currently supports Linux x64, including Windows through WSL2. Connected Voyage and explicit local Ollama configurations are also available. See the [main README](https://github.com/ham-zax/satori#quick-start) for runtime choices.
 
-Use `--profile default|minimal|all-text` to write repo-local `satori.toml` during install:
+Direct package execution is intended for inspection and custom harnesses:
 
 ```bash
-npx -y @zokizuan/satori-cli@latest install --client all --profile minimal
-```
-
-Profiles control indexing breadth, not search scope. `default` is safe-broad, `minimal` indexes source plus docs/text, and `all-text` indexes additional UTF-8 text files under the size limit. `search_codebase` still defaults to `scope=runtime`.
-
-The repo-local config shape is:
-
-```toml
-[index]
-profile = "minimal"
+npx -y @zokizuan/satori-mcp@latest --help
 ```
 
-`satori.toml` is repository index policy, not MCP client config and not provider config. Do not put API keys, model names, Milvus endpoints, or tokens in it. Provider settings belong in the MCP client's runtime environment.
-
-Profile behavior:
-
-- `default`: source, docs/text, config, scripts, infra/query files, and known extensionless files such as `Dockerfile`, `Makefile`, `Justfile`, `Taskfile`, `Procfile`, `Jenkinsfile`, and `.dockerignore`.
-- `minimal`: source plus docs/text only.
-- `all-text`: default plus unknown UTF-8 text files under the size limit. `SATORI_ALL_TEXT_MAX_BYTES` can override the text-file cap.
-
-All profiles still honor `.gitignore`, `.satoriignore`, and the hard denylist for secrets, lockfiles, generated output, dependency folders, binaries, bundles, logs, database dumps, source maps, and snapshots. `satori.toml` is treated as an index-policy control file; search freshness and `manage_index action="sync"` can reconcile ordinary profile/ignore changes, while incompatible fingerprints still return `requires_reindex`.
-
-Codex installs write two companion artifacts by default: the first-party `satori` skill under `~/.codex/skills` and a marked Satori guidance block in `~/.codex/AGENTS.md`. The AGENTS block tells Codex to use Satori for semantic ownership/context discovery first, then use exact navigation and reads for proof.
+Do not use `npx` as the resident MCP command when the CLI installer supports your client; package resolution can exceed normal MCP startup timeouts.
 
-For Codex, add `--install-guidance-hook` only when you want an installer-managed `SessionStart` reminder in `~/.codex/config.toml`. The hook prints guidance only, suppresses duplicate startup prints for the same working directory, and does not run indexing, search, or provider-backed work.
-
-Advanced direct execution is available through the package bin:
-
-```bash
-npx -y @zokizuan/satori-mcp@4.11.17 --help
-```
-
-Use direct package execution for inspection, smoke tests, or unsupported harnesses. For supported clients, prefer `satori-cli install` so startup does not depend on package-manager resolution.
-
-## Agent Workflow
+## Workflow
 
 ```text
-list_codebases
 manage_index action="create" path="/absolute/path/to/repo"
 search_codebase path="/absolute/path/to/repo" query="where is auth refresh handled"
 file_outline path="/absolute/path/to/repo" file="src/auth.ts"
@@ -61,173 +31,47 @@ call_graph path="/absolute/path/to/repo" symbolRef={...} direction="both"
 read_file path="/absolute/path/to/repo/src/auth.ts" start_line=1 end_line=160
 ```
 
-Important defaults:
-
-- Public paths for tools are **absolute filesystem paths**. Relative paths are rejected (not CWD-resolved).
-- `search_codebase` starts with `scope=runtime`, `resultMode=grouped`, `groupBy=symbol`, `rankingMode=auto_changed_first`.
-- `search_codebase` is the sync-on-read freshness tool; other tools may gate compatibility but do not replace it.
-- Grouped search is symbol-owned: chunks are supporting evidence for an owner symbol, not the final navigation unit.
-- Exact symbol navigation uses `symbolInstanceId`. `symbolKey` is stable-ish candidate lookup only, not exact identity.
-- Ready/`indexed` means searchable-readable, not automatically symbol-rich. Check `manage_index status` `symbolQuality` or Ready-line `symbolQuality=…` before treating outline/graph as rich navigation evidence (`symbol_rich` \| `mixed` \| `symbol_sparse` \| `search_only` \| `unknown` — observed registry evidence, not a parser diagnosis).
-- Index profiles still honor `.satoriignore`, `.gitignore`, `satori.toml`, and the hard denylist for secrets, lockfiles, generated output, dependencies, binaries, bundles, logs, and database dumps.
-- `read_file` only reads under tracked searchable roots (`indexed` / `sync_completed`); it is not a general host filesystem reader. Relative paths and root escapes are denied.
-- Prefer `recommendedNextAction` when present; if `callGraphHint.supported=false`, follow `navigationFallback` args rather than inventing spans.
-- `requires_reindex` means reindex first, then retry the original call (do not substitute `sync`).
-- `manage_index action="clear"` is destructive and should be explicit. `repair` reports structured proof across collection, snapshot, marker, fingerprint, payload, stale remote chunks, and navigation. Only no related collection routes to create; existing incompatible, incomplete, stale, malformed, or unprovable generations route to reindex. Backend failures preserve partial proof when available and should be diagnosed before retrying repair. Successful repair may write a fresh completion marker and rebuild navigation, but never re-embeds or rewrites source chunks.
-- After changing `EMBEDDING_PROVIDER`, `EMBEDDING_MODEL`, embedding dimension, `HYBRID_MODE`, vector backend settings, or the Satori runtime version, restart **all** Satori MCP clients before running `manage_index create`, `reindex`, `sync`, `clear`, or `repair`.
-- Satori records live runtime owners in `~/.satori/runtime/owners.json` and blocks those index mutations with `status="blocked"` / `reason="runtime_owner_conflict"` if another live Satori MCP runtime has a different fingerprint, package version, or config identity.
-- On `runtime_owner_conflict`, the manage envelope lists conflicting **pids**, **versions**, and conflict reasons, plus a concrete `hints.nextStep`. MCP tools never kill other processes. Stop the listed host clients (or only orphaned Satori MCP node PIDs), leave a single package version/config running, then retry. `manage_index status` and `list_codebases` also show a compact **Runtime owners** line (live pids/versions); multi-version means mutations may block. `satori-cli doctor` reports multi-version live owners.
-- A separate canonical-root mutation lease permits multiple readers but only one create/reindex/sync/repair/clear writer across processes. Contention returns `reason="mutation_in_progress"`; `manage_index status` exposes `hints.activeMutation`. Leases do not have a time-based expiry, and a live owner is never evicted by age alone.
-- `zillizDropCollection` is fail-closed: Satori must prove the selected collection's owning codebase root and lease both affected roots before deletion. Unknown ownership or an active writer leaves the collection and local snapshot unchanged.
-- Grouped `search_codebase` results with supported call graph attach **`inboundRecovery`**: a ready-to-run `must:` search to verify callers before blast-radius edits (`call_graph` inbound stays advisory/low).
-
-## Navigation Sidecars
-
-Completed full indexes write a derived symbol registry and relationship sidecar. Files remain the source of truth; the registry is the deterministic navigation view for the indexed snapshot.
-
-- The symbol registry stores candidate owner keys, exact symbol instances, file-owner fallback symbols, and outline records used by grouped search, `file_outline`, and exact reads.
-- The relationship sidecar stores conservative `CALLS v0` edges plus TypeScript/JavaScript `IMPORTS`/`EXPORTS v0` edges with manifest compatibility gates.
-- Runtime navigation still serves canonical JSON sidecars by default. When the default shared runtime store is created at process startup, `SATORI_NAVIGATION_BACKEND=sqlite` can opt that shared store into SQLite-backed reads only after SQLite proves parity with the canonical JSON symbol registry and relationship sidecars. If canonical JSON is missing or incompatible, SQLite is not served as truth; if SQLite is missing, stale, incompatible, or parity-mismatched while JSON is compatible, runtime falls back to JSON with a warning. When the default shared runtime store is created with `SATORI_NAVIGATION_DUAL_READ=1`, JSON remains the serving backend and the runtime emits once-per-root parity mismatch warnings without changing the served result.
-- `CALLS v0` is heuristic/name-based. Same-file unique targets can be high confidence; cross-file name-only targets start low and are upgraded when IMPORTS/EXPORTS evidence supports them, or when the imported file has a unique same-name target (class methods without top-level EXPORTS). Generic names like `push`/`get` stay suppressed without EXPORTS. Ambiguous same-name targets are skipped.
-- `IMPORTS`/`EXPORTS v0` records only resolvable relative module edges and unambiguous local export declarations. Package imports, unresolved paths, ambiguous exports, and multiline module syntax are skipped.
-- `call_graph` uses compatible relationship sidecars as the canonical source for symbol-owned traversal.
-- Successful incremental sync reuses changed-file symbol output, preserves unchanged registry state, and recomputes relationships against the merged registry without re-splitting unchanged files. If changed-file indexing stops early, navigation state is cleared instead of publishing a mixed generation.
-
-## Runtime Requirements
-
-Configure an embedding provider and Milvus-compatible backend before indexing. Supported embedding providers are OpenAI, VoyageAI, Gemini, and Ollama. Changing provider, model, dimension, vector store, or schema requires a reindex because those values are part of the index fingerprint.
-
-MCP startup, `tools/list`, and installer operations are lazy with respect to provider credentials. Missing provider values become `MISSING_PROVIDER_CONFIG` only when a provider-backed tool call needs them.
-
-Installer-managed client config starts the resident launcher. Runtime provider settings come from the MCP client's environment and are exposed in native client config:
-
-- Codex writes active `env_vars` forwarding plus an optional commented `[mcp_servers.satori.env]` template in `~/.codex/config.toml`.
-- Claude Code writes `mcpServers.satori.env` in `~/.claude.json` with `${VAR:-}` pass-through values.
-- OpenCode writes `mcp.satori.environment` in `~/.config/opencode/opencode.json` with `{env:VAR}` pass-through values.
-
-Users who want literal values in a client config can replace the generated pass-through value for that client. In Codex, uncomment or add this table outside the installer-managed launcher block so reinstalls keep edits:
-
-```toml
-[mcp_servers.satori.env]
-EMBEDDING_PROVIDER = "VoyageAI"
-EMBEDDING_MODEL = "voyage-code-3"
-EMBEDDING_OUTPUT_DIMENSION = "1024"
-VOYAGEAI_API_KEY = "pa-..."
-VOYAGEAI_RERANKER_MODEL = "rerank-2.5"
-MILVUS_ADDRESS = "https://your-zilliz-endpoint"
-MILVUS_TOKEN = "your-zilliz-token"
-```
-
-Cloud-quality setup:
-
-```bash
-EMBEDDING_PROVIDER=VoyageAI
-EMBEDDING_MODEL=voyage-code-3
-EMBEDDING_OUTPUT_DIMENSION=1024
-VOYAGEAI_API_KEY=your-api-key
-VOYAGEAI_RERANKER_MODEL=rerank-2.5
-MILVUS_ADDRESS=your-milvus-endpoint
-MILVUS_TOKEN=your-milvus-token
-```
-
-The full generated tool reference below is kept in the npm README for MCP clients and package consumers.
-
-<!-- TOOLS_START -->
-
-## Tool Reference
-
-### `manage_index`
-
-Manage index lifecycle operations (create/reindex/sync/status/clear/repair) for a codebase path. repair rebuilds local readiness only when existing vector payload and trusted runtime fingerprint proof match. Repair responses may include optional `repairProof` evidence for collection, snapshot, marker, fingerprint, payload, staleRemoteChunks, and navigation. No related collection routes to create; an existing incompatible, incomplete, stale, malformed, or unprovable generation routes to reindex; backend failures preserve partial proof when collection began and should be diagnosed before retrying repair. Successful repair may write a fresh completion marker and rebuild navigation, but it does not re-embed or rewrite source chunks. Ignore-rule edits in repo-root .satoriignore/.gitignore reconcile automatically in the normal sync path. Use action="sync" for immediate convergence and action="reindex" for full rebuild recovery (preflight may block unnecessary ignore-only reindex churn unless allowUnnecessaryReindex=true). Successful sync responses include `syncStats` with `added`, `removed`, and `modified` counts; consume those fields instead of parsing `humanText`. create/reindex return the kickoff response immediately and do not poll to terminal state; use action="status" to observe progress. Status may include `languageCapabilities`, which combines declared claims with compatible symbol-registry and relationship-sidecar evidence for indexed languages. Mutation responses may include a durable `operation` receipt with `id`, canonical root, generation, accepted time, current phase, last durable transition, runtime fingerprint, and writer identity. Status returns the latest persisted receipt after restart. In a status envelope, top-level `action` remains `status` while `operation.action` names the observed mutation. Terminal phases are `completed`, `failed`, and `blocked`; `operation` is absent when no durable operation exists or contention was rejected before lease acquisition.
-
-| Parameter | Type | Required | Default | Description |
-|---|---|---|---|---|
-| `action` | enum("create", "reindex", "sync", "status", "clear", "repair") | yes |  | Required operation to run. |
-| `path` | string | yes |  | ABSOLUTE filesystem path to the target codebase (relative paths are rejected). |
-| `force` | boolean | no |  | Only for action='create'. Force rebuild from scratch. |
-| `allowUnnecessaryReindex` | boolean | no |  | Only for action='reindex'. Override preflight block when reindex is detected as unnecessary ignore-only churn. |
-| `customExtensions` | array<string> | no |  | Only for action='create'. Additional file extensions to include. |
-| `ignorePatterns` | array<string> | no |  | Only for action='create'. Additional ignore patterns to apply. |
-| `zillizDropCollection` | string | no |  | Only for action='create'. Zilliz-only: drop this Satori-managed collection before creating the new index. |
-
-### `search_codebase`
+Public paths are absolute. Search is freshness-aware; exact reads are limited to indexed searchable roots. Follow `recommendedNextAction` when returned, and reindex before retrying a request that reports `requires_reindex`.
 
-Unified semantic search with runtime-first defaults (start with scope="runtime"), grouped/raw output modes, and deterministic ranking/freshness behavior. Operators are parsed from a query prefix block: lang:, path:, -path:, must:, exclude: (escape with \\ to keep literals). For high-precision queries such as exact identifiers, quoted literal phrases, and strict path filters, search_codebase can use an exact registry fast path or add a bounded tracked-file lexical recovery pass when semantic retrieval under-delivers. Grouped results expose legacy span plus explicit previewSpan/symbolSpan metadata, structured warnings, recommendedNextAction, per-result capabilities/fallbacks, executable nextActions/navigationFallbacks, and remediation hints such as .satoriignore noise handling. Use debug:true for explainability payloads, including debugSummary, exactRegistry, phaseTimingsMs, trackedLexical, and ranking provenance.
+## Measured Performance
 
-| Parameter | Type | Required | Default | Description |
-|---|---|---|---|---|
-| `path` | string | yes |  | ABSOLUTE filesystem path to an indexed codebase or subdirectory (relative paths are rejected). |
-| `query` | string | yes |  | Natural-language query. |
-| `scope` | enum("runtime", "mixed", "docs") | no | `"runtime"` | Search scope policy. runtime includes source/runtime code and tests while excluding docs/generated/artifacts/landing/fixtures; docs returns documentation paths only (not tests); mixed includes all. Docs scope skips reranker by policy in the current tool surface. |
-| `resultMode` | enum("grouped", "raw") | no | `"grouped"` | Output mode. grouped returns merged search groups, raw returns chunk hits. |
-| `groupBy` | enum("symbol", "file") | no | `"symbol"` | Grouping strategy in grouped mode. |
-| `rankingMode` | enum("default", "auto_changed_first") | no | `"auto_changed_first"` | Ranking policy. auto_changed_first boosts files changed in the current git working tree when available. |
-| `limit` | integer | no | `50` | Maximum groups (grouped mode) or chunks (raw mode). |
-| `debug` | boolean | no | `false` | Optional debug payload toggle for score and fusion breakdowns. |
+A checksum-sealed Potion/LanceDB run on Satori published 488 files and 10,830 chunks in 34.46 seconds on CPU. The later representative delta run measured 154.543 ms warm-search p95, 185.662 ms zero-change synchronization p95, and 789–865 ms p95 for one-file add/edit/delete operations.
 
-### `call_graph`
+On 30 frozen positive retrieval tasks, Potion placed the required owner in the top five on 23 tasks versus Voyage on 25. Potion's observed median search latency was 94.64 ms versus 1,009.46 ms for Voyage in that paired run, but the provider latency observations were descriptive and Potion showed weaker Java and configuration/runtime retrieval.
 
-Traverse registry-resolved caller/callee relationships for indexed TS/JS/Python code. Relationship-backed CALLS v0 is heuristic and name-based (not a compiler-grade call graph): unique same-file name matches are high confidence; cross-file edges stay low unless IMPORTS/EXPORTS evidence upgrades them, or an imported module has a unique same-name target (e.g. class methods without a top-level EXPORTS record; generic names like push/get stay suppressed). Traversal is bounded (depth/limit) and incomplete by design—empty or short edge lists are not proof of “no callers.” Output is advisory navigation context, not authoritative blast-radius proof; confirm impact with search_codebase, read_file, tests, and references before editing. In successful traversal responses, sidecar.nodeCount and sidecar.edgeCount report counts returned in that response, not whole-sidecar totals.
+In a fresh two-task OpenCode comparison where both arms answered correctly, Satori used 16 tool calls and returned 76,113 tool-output bytes versus 25 calls and 96,801 bytes for native `grep` / `glob` / `read`. Wall time was 51.65 versus 96.04 seconds; total model tokens were effectively unchanged. This is a small exploratory context-route result, not a universal token-reduction claim.
 
-| Parameter | Type | Required | Default | Description |
-|---|---|---|---|---|
-| `path` | string | yes |  | ABSOLUTE filesystem path to the indexed codebase root or subdirectory (relative paths are rejected). |
-| `symbolRef` | object | yes |  | Symbol reference from a grouped search result callGraphHint. |
-| `direction` | enum("callers", "callees", "both") | no | `"both"` | Traversal direction from the starting symbol. |
-| `depth` | integer | no | `1` | Traversal depth (max 3). |
-| `limit` | integer | no | `20` | Maximum number of returned edges. |
-
-### `file_outline`
-
-Return a sidecar-backed symbol outline for one file, including call_graph jump handles.
-
-| Parameter | Type | Required | Default | Description |
-|---|---|---|---|---|
-| `path` | string | yes |  | ABSOLUTE filesystem path to the indexed codebase root (relative paths are rejected). |
-| `file` | string | yes |  | Repo-relative file path inside the codebase root (not absolute; resolved only against that root). |
-| `start_line` | integer | no |  | Optional start line filter (1-based, inclusive). |
-| `end_line` | integer | no |  | Optional end line filter (1-based, inclusive). |
-| `limitSymbols` | integer | no | `500` | Maximum number of returned symbols after line filtering. |
-| `resolveMode` | enum("outline", "exact") | no | `"outline"` | Outline mode returns all symbols (windowed/limited). Exact mode resolves deterministic symbol matches in this file. |
-| `symbolIdExact` | string | no |  | Used with resolveMode="exact": exact symbol identifier match in the target file. On symbol-owned flows, pass the symbol's symbolInstanceId. |
-| `symbolLabelExact` | string | no |  | Used with resolveMode="exact": exact symbol label match in the target file. |
-
-### `read_file`
-
-Read file content under an indexed/searchable Satori codebase root only (not a general host filesystem reader). Requires an absolute path whose canonical real path is inside a tracked root with status indexed or sync_completed. Supports optional 1-based inclusive line ranges and safe truncation.
-
-| Parameter | Type | Required | Default | Description |
-|---|---|---|---|---|
-| `path` | string | yes |  | ABSOLUTE path to the file under an indexed/searchable codebase root (relative paths are rejected). |
-| `start_line` | integer | no |  | Optional start line (1-based, inclusive). |
-| `end_line` | integer | no |  | Optional end line (1-based, inclusive). |
-| `mode` | enum("plain", "annotated") | no | `"plain"` | Output mode. plain returns text only; annotated returns content plus sidecar-backed outline metadata. |
-| `open_symbol` | object | no |  | Optional deterministic symbol jump request for this file path. Uses exact symbol resolution within `path` when symbolId/symbolLabel is provided, and only uses direct span opens when no symbol identity fields are supplied. On symbol-owned flows, symbolId should carry the symbolInstanceId. |
-
-### `list_codebases`
-
-List tracked codebases and their indexing state.
+<!-- TOOLS_START -->
 
-No parameters.
+## Tools
 
+| Tool | Purpose |
+|---|---|
+| `manage_index` | Create, synchronize, inspect, repair, reindex, or clear a repository index. Use status and repair guidance instead of guessing whether an index is ready. |
+| `search_codebase` | Run freshness-aware hybrid search and return symbol-owned evidence. Start here for behavior, ownership, configuration, or path discovery. |
+| `continue_search` | Reveal more of one frozen result set without rerunning retrieval. Use it when the initial disclosure is relevant but incomplete. |
+| `call_graph` | Inspect advisory callers, callees, imports, and exports when supported. Verify inbound leads before blast-radius changes. |
+| `file_outline` | List the indexed symbols and spans in one file. Use it to choose an exact owner before reading implementation. |
+| `read_file` | Read a bounded source span or one exact indexed symbol. Large ranges are compacted so agent UIs receive structure instead of implementation floods. |
+| `list_codebases` | List known indexed repositories, readiness, and runtime-owner state. Use it to discover existing publications before creating another one. |
 
 <!-- TOOLS_END -->
 
-## Notes
+## Runtime Boundaries
 
-- `open_symbol` resolves exact symbols inside the same file passed to `read_file.path`. On symbol-owned flows, `symbolId`/`symbolIdExact` should carry `symbolInstanceId`.
-- `MILVUS_TOKEN` is optional auth; local unauthenticated Milvus only needs `MILVUS_ADDRESS`.
-- MCP startup does not require provider credentials or a live Milvus backend. Provider-backed calls report `MISSING_PROVIDER_CONFIG` when setup is incomplete.
-- `MISSING_PROVIDER_CONFIG` is an active setup failure only when it appears as a tool response `code` or `reason`.
+- The server does not edit repository source.
+- `read_file` is not a general host-filesystem reader.
+- Inbound call-graph evidence is advisory and should be verified before blast-radius edits.
+- Provider, model, dimensions, projection, and vector backend are persisted compatibility identities; changing them requires a reindex.
+- Multiple incompatible live Satori runtimes are blocked from mutating the same publication.
+- Offline neural reranking is not shipped today. The candidate boundary permits a future complete-set local scorer to fail back atomically to exact + BM25 + single-vector ordering.
 
-## Local Development
+## Development
 
 ```bash
-pnpm --filter @zokizuan/satori-mcp start
 pnpm --filter @zokizuan/satori-mcp build
-pnpm --filter @zokizuan/satori-mcp typecheck
 pnpm --filter @zokizuan/satori-mcp test
 pnpm --filter @zokizuan/satori-mcp docs:check
 ```
 
-`build` regenerates the tool reference from live tool schemas.
+Node.js 22.13 or newer is required. Satori is MIT licensed.

package/{LICENSE → assets/potion/linux-x64/MODEL2VEC_RS_LICENSE}

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2026 Hamza (@ham-zax)
+Copyright (c) 2025 The Minish Lab
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/assets/potion/linux-x64/MODEL_CARD.md

@@ -0,0 +1,110 @@
+---
+language:
+- code
+license: mit
+library_name: model2vec
+tags:
+- model2vec
+- embeddings
+- code
+- retrieval
+- static-embeddings
+datasets:
+- minishlab/tokenlearn-cornstack-queries-coderankembed
+- minishlab/tokenlearn-cornstack-docs-coderankembed
+- nomic-ai/cornstack-python-v1
+- nomic-ai/cornstack-java-v1
+- nomic-ai/cornstack-php-v1
+- nomic-ai/cornstack-go-v1
+- nomic-ai/cornstack-javascript-v1
+- nomic-ai/cornstack-ruby-v1
+---
+
+# potion-code-16M-v2 Model Card
+
+## Overview
+
+**potion-code-16M-v2** is a fast static code embedding model optimized for code retrieval tasks. It powers [Semble](https://github.com/MinishLab/semble), a code search library for agents. It is distilled from [nomic-ai/CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed) and trained on the [CornStack](https://huggingface.co/datasets/nomic-ai/cornstack-python-v1) code corpus using [Tokenlearn](https://github.com/MinishLab/tokenlearn) and contrastive fine-tuning.
+It is the successor to [potion-code-16M](https://huggingface.co/minishlab/potion-code-16M).
+It uses static embeddings, allowing text and code embeddings to be computed orders of magnitude faster than transformer-based models on both GPU and CPU.
+
+## Installation
+
+```bash
+pip install model2vec
+```
+
+## Usage
+
+```python
+from model2vec import StaticModel
+
+model = StaticModel.from_pretrained("minishlab/potion-code-16M-v2")
+
+# Embed natural language queries
+query_embeddings = model.encode(["How to read a file in Python?"])
+
+# Embed code documents
+code_embeddings = model.encode(["def read_file(path):\n    with open(path) as f:\n        return f.read()"])
+```
+
+## How it works
+
+potion-code-16M-v2 is created using the following pipeline:
+
+1. **Vocabulary mining**: code-specific tokens are mined from CornStack and added to the base CodeRankEmbed tokenizer (43k extra tokens → ~63.5k total)
+2. **Distillation**: the extended vocabulary is distilled from CodeRankEmbed using Model2Vec (256-dimensional embeddings, PCA)
+3. **Tokenlearn**: the distilled model is fine-tuned on 1.2 million (query, document) pairs from CornStack using cosine similarity loss
+4. **Contrastive fine-tuning**: the model is further fine-tuned using MultipleNegativesRankingLoss on 1.2 million CornStack query-document pairs
+
+## Results
+
+Results on the [CoIR benchmark](https://github.com/CoIR-team/coir) on [MTEB](https://github.com/embeddings-benchmark/mteb) (NDCG@10, `mteb>=2.10`):
+
+| Model | Params | AVG | AppsRetrieval | COIRCodeSearchNet | CodeFeedbackMT | CodeFeedbackST | CodeSearchNetCC | CodeTransContest | CodeTransDL | CosQA | StackOverflow | Text2SQL |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| CodeRankEmbed | 137M | 59.14 | 23.46 | 94.70 | 42.61 | 78.11 | 76.39 | 66.43 | 34.84 | 35.92 | 80.53 | 58.37 |
+| **potion-code-16M-v2 + BM25 (hybrid)** | **16M** | **43.36** | **6.08** | **47.71** | **45.38** | **61.10** | **51.68** | **53.80** | **33.42** | **21.39** | **66.73** | **46.29** |
+| BM25 | — | 42.31 | 4.76 | 40.86 | 59.19 | 68.15 | 53.97 | 47.78 | 34.42 | 18.75 | 70.26 | 24.94 |
+| **potion-code-16M-v2** | **16M** | **39.08** | **5.19** | **46.37** | **38.02** | **53.22** | **43.66** | **43.66** | **32.64** | **24.36** | **59.57** | **44.07** |
+| potion-code-16M | 16M | 37.05 | 3.97 | 42.99 | 36.26 | 50.27 | 43.40 | 39.76 | 31.72 | 21.37 | 57.47 | 43.34 |
+| potion-retrieval-32M | 32M | 32.10 | 4.22 | 31.80 | 36.71 | 45.11 | 38.64 | 29.97 | 32.62 | 8.70 | 56.26 | 36.93 |
+| potion-base-32M | 32M | 31.42 | 3.37 | 29.58 | 34.77 | 42.69 | 37.88 | 28.51 | 30.55 | 14.61 | 53.36 | 38.88 |
+
+CoIR covers a broad range of code retrieval scenarios. For the use case of finding code given a natural language query, **CosQA** and **CodeFeedback (ST/MT)** are the most relevant tasks. Others are less so: **COIRCodeSearchNetRetrieval** retrieves text given a code query (the reverse direction), and the **CodeTransOcean** tasks target cross-language code translation.
+The hybrid row combines dense retrieval with BM25 using Reciprocal Rank Fusion (k=60).
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| Parameters | ~16M |
+| Embedding dimensions | 256 |
+| Vocabulary size | ~63,500 |
+| Teacher model | nomic-ai/CodeRankEmbed |
+| Training corpus | CornStack (6 languages: Python, Java, JavaScript, Go, PHP, Ruby) |
+| Max sequence length | 1,000,000 tokens (static, no limit in practice) |
+
+## Additional Resources
+
+- [Semble repository](https://github.com/MinishLab/semble)
+- [Model2Vec repository](https://github.com/MinishLab/model2vec)
+- [Tokenlearn repository](https://github.com/MinishLab/tokenlearn)
+- [Tokenlearn document dataset](https://huggingface.co/minishlab/tokenlearn-cornstack-docs-coderankembed-v2)
+- [Tokenlearn query dataset](https://huggingface.co/minishlab/tokenlearn-cornstack-queries-coderankembed-v2)
+- [CornStack dataset](https://huggingface.co/datasets/nomic-ai/cornstack-python-v1)
+- [CoIR benchmark](https://github.com/CoIR-team/coir)
+
+## Citation
+
+```bibtex
+@software{minishlab2024model2vec,
+  author       = {Stephan Tulkens and {van Dongen}, Thomas},
+  title        = {Model2Vec: Fast State-of-the-Art Static Embeddings},
+  year         = {2024},
+  publisher    = {Zenodo},
+  doi          = {10.5281/zenodo.17270888},
+  url          = {https://github.com/MinishLab/model2vec},
+  license      = {MIT}
+}
+```

package/assets/potion/linux-x64/manifest.json

@@ -0,0 +1,61 @@
+{
+  "schemaVersion": 1,
+  "platform": "linux",
+  "architecture": "x64",
+  "target": "x86_64-unknown-linux-gnu",
+  "model": {
+    "repository": "minishlab/potion-code-16M-v2",
+    "revision": "e9d2a44ca6a05ac6685f3b23709ea57eb7352d5b",
+    "identity": "minishlab/potion-code-16M-v2@e9d2a44ca6a05ac6685f3b23709ea57eb7352d5b",
+    "license": "MIT"
+  },
+  "helper": {
+    "sourceRepository": "satori/experiments/potion-l0-l1",
+    "sourceRevision": "d04eac35af2addad7b68525f91b0c6dacd8b94da",
+    "model2vecRsRevision": "6f51c7afe2436bcb76fc467bad54eaa94f8db30d",
+    "rustToolchain": "1.97.1",
+    "features": [
+      "default-features=false",
+      "local-only",
+      "fancy-regex"
+    ],
+    "dynamicRuntime": [
+      "glibc",
+      "libgcc_s"
+    ]
+  },
+  "embeddingInferenceContractDigest": "bfda80d97aeb585e20650b1c54e9063a65068ce284317f0e0a812e20964dcee7",
+  "files": [
+    {
+      "path": "satori-potion",
+      "bytes": 4224392,
+      "sha256": "2e42f3165b96927bb365f74a11b0495661ac3c44e1a194c55a8f0613b5bb2e12",
+      "executable": true
+    },
+    {
+      "path": "model/config.json",
+      "bytes": 59,
+      "sha256": "148e5691a6fcc553437156859701fba017a1ba5d340b170f17e0f3668fb861a7"
+    },
+    {
+      "path": "model/model.safetensors",
+      "bytes": 32490072,
+      "sha256": "75cf7a6c2171b230ad19b1e7d8e0b1aee86da5a02af8e7cacedd9921d227623c"
+    },
+    {
+      "path": "model/tokenizer.json",
+      "bytes": 1024340,
+      "sha256": "107bbdcbad4bff1d299b7a4c3a2fb17c52890688b7dd0e4c9deab79d3c4f3d45"
+    },
+    {
+      "path": "MODEL_CARD.md",
+      "bytes": 5301,
+      "sha256": "835e4161a3942cd5284d423dbdb4ca69024b671436273142d7512153e770794c"
+    },
+    {
+      "path": "MODEL2VEC_RS_LICENSE",
+      "bytes": 1071,
+      "sha256": "6869ad291a9df9866448579b7157bf337373a21ced5b4b315e68b2924aff2ab9"
+    }
+  ]
+}

package/assets/potion/linux-x64/model/config.json

@@ -0,0 +1,4 @@
+{
+    "normalize": true,
+    "embedding_dtype": "float16"
+}