delphi-code 0.1.1__tar.gz → 0.2.0__tar.gz

{delphi_code-0.1.1 → delphi_code-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: delphi-code
-Version: 0.1.1
+Version: 0.2.0
 Summary: Offline local code search for agents
 Author: Mark Dorofeev
 License-Expression: Apache-2.0
@@ -24,6 +24,7 @@ Requires-Dist: sqlite-vec==0.1.9
 Requires-Dist: sentence-transformers==6.1.0
 Requires-Dist: numpy==2.5.3
 Requires-Dist: pathspec==0.12.1
+Requires-Dist: questionary==2.1.1
 Requires-Dist: certifi
 Dynamic: license-file
 
@@ -107,18 +108,82 @@ delphi-code status -p /path/to/project
 - **`status`** reports stored index state. It needs no model and does not rescan source files to detect staleness.
 - **`doctor`** checks the model, SQLite extensions, and native storage, then runs a real local embedding and vector distance calculation.
 - **`setup`** provisions the model (see above).
+- **`add`**, **`sync`**, **`list`**, and **`remove`** manage tracked projects (see below).
 
 `python -m delphi_code` works as an alternative to the `delphi-code` executable.
 
 ### Selecting a project
 
 - `--project` / `-p` accepts a path, resolved relative to the current working directory. The project root is explicit; it is not inferred from Git.
-- After a project has been indexed once, its folder name works too, for example `delphi-code search -p myapp 'parse config'`. A bare name selects the uniquely matching indexed project; duplicate names produce an error listing their paths. Use `./name` or an absolute path to select a directory explicitly.
+- After a project has been indexed once, its key or folder name works too, for example `delphi-code search -p acme/api 'parse config'` or `-p myapp`. A key matches in full or by a trailing part (`bitbucket.org/acme/api`, `acme/api`, or `api`). A name that matches several indexes produces an error listing their keys. Use `./name` or an absolute path to select a directory explicitly.
 - Without `--project`, `search` searches all stored indexes; other commands use the current directory.
 
+### Tracking projects
+
+Instead of indexing projects one by one, keep a list of them and update them all together:
+
+```sh
+delphi-code add ~/src/api ~/src/web --path 'src/*'
+delphi-code sync
+delphi-code list
+delphi-code remove api
+```
+
+- **`add`** records projects and indexes them; `--no-sync` only records them. Filter options are stored with each project, and adding a project again replaces its options.
+- **`sync`** indexes every tracked project. One failing project does not stop the rest: the command then exits with `sync_failed` (code 5) and still reports each project's result under `data`.
+- **`list`** shows tracked projects and every stored index, including ones created with plain `index`. It needs no model.
+- **`remove`** stops tracking a project and deletes its index; `--keep-index` keeps the index.
+
+The list lives in `repos.toml` in the storage directory (`DELPHI_CODE_REGISTRY` overrides the location). You can edit it by hand; `add` and `remove` rewrite it without comments.
+
+```toml
+[[repo]]
+source = "/Users/me/src/api"
+key = "bitbucket.org/acme/api"
+paths = ["src/*"]
+
+[[repo]]
+source = "bitbucket.org/acme/billing"
+ref = "release"
+```
+
+### Bitbucket Cloud and GitHub repositories
+
+Repositories you have not cloned can be tracked too:
+
+```sh
+delphi-code add bitbucket.org/acme/billing
+delphi-code add git@bitbucket.org:acme/billing.git --ref release
+delphi-code add github.com/octo/tools
+```
+
+- `sync` asks the hosting service for the latest commit of the branch or tag, which is the default branch unless `--ref` names another. It skips the repository when the index already holds that commit with the same filters and model. Otherwise it makes a shallow clone into a temporary directory, indexes it, and deletes the clone. Unchanged files are not embedded again.
+- Search results from these repositories carry a `url`: a Bitbucket or GitHub link to the indexed commit and line range. The response also includes `ref` and `commit`, and `project` is `null`.
+- Use `sync`, not `index`, to update them.
+- Only `add` and `sync` touch the network. `index`, `search`, `status`, and `list` stay offline.
+
+**Picking repositories.** Run `add` without sources in a terminal to choose from the repositories your accounts can see:
+
+```sh
+delphi-code add
+```
+
+Choose Bitbucket or GitHub (asked only when both have credentials), then a Bitbucket workspace or GitHub account (skipped when there is only one). Then check repositories in a list you can filter by typing. Repositories you already track start checked, so unchecking one stops tracking it and deletes its index. After you confirm, the new repositories are indexed. Filter options and `--ref` apply to the repositories you add. The prompts are drawn on stderr, and stdout still receives one JSON object with the added repositories under `repos` and the untracked ones under `removed`. Cancelling changes nothing and reports `"cancelled": true`. Without a terminal, `add` requires sources.
+
+**Credentials.** Git never prompts. Credentials from the environment reach git through `GIT_ASKPASS`, never in URLs or arguments, and take precedence over configured git credentials. Listing repositories for the picker uses the services' REST APIs. The API requests run in a separate process, like the model download in `setup`, so the main process keeps its network guard.
+
+| | Bitbucket Cloud | GitHub |
+|---|---|---|
+| Cloning | `BITBUCKET_USERNAME` and `BITBUCKET_APP_PASSWORD`, otherwise git's own credentials | `GITHUB_TOKEN` or `GH_TOKEN`, otherwise git's own credentials (`gh auth setup-git` configures them) |
+| Listing for the picker | The same variables (with `BITBUCKET_EMAIL` in place of the username when set), otherwise the credentials git has stored for bitbucket.org | `GITHUB_TOKEN` or `GH_TOKEN`, otherwise `gh auth token`, otherwise the token git has stored for github.com |
+
+Bitbucket has replaced app passwords with API tokens. An API token also works in `BITBUCKET_APP_PASSWORD`. Git over HTTPS takes the username that Bitbucket shows for it, and the REST API takes your Atlassian account email, which you set in `BITBUCKET_EMAIL`.
+
+`DELPHI_CODE_BITBUCKET_GIT_BASE` and `DELPHI_CODE_GITHUB_GIT_BASE` send clones to another base URL, such as a mirror, and `DELPHI_CODE_BITBUCKET_API_BASE` and `DELPHI_CODE_GITHUB_API_BASE` do the same for the APIs. Result links still point to bitbucket.org and github.com.
+
 ### Searching across projects
 
-Cross-project search returns a single globally ranked list, with the full `project` path on each result. The query is embedded once, `--limit` applies to the combined list, and path/language filters apply within every index. All indexes must be readable, complete, and built with the selected model: a busy, broken, or incompatible index produces an error rather than silently partial results. Use `-p` to narrow the search when needed.
+Cross-project search returns a single globally ranked list, with the `key` and full `project` path on each result (`null` for Bitbucket and GitHub repositories, whose results carry a `url` instead). The query is embedded once, `--limit` applies to the combined list, and path/language filters apply within every index. All indexes must be readable, complete, and built with the selected model: a busy, broken, or incompatible index produces an error rather than silently partial results. Use `-p` to narrow the search when needed.
 
 ## Output
 
@@ -205,11 +270,18 @@ Models and indexes live outside the installation, so reinstalling or upgrading l
 
 The default model is under `models/all-MiniLM-L6-v2/` and indexes are under `indexes/`. `DELPHI_CODE_INDEX_ROOT` overrides the index location.
 
-Each project has one index at `indexes/<sha256 of the resolved project path>/`, independent of the working directory. It holds CocoIndex's incremental state, the SQLite vector table, and a manifest recording the project path; command output includes this `index_directory`.
+Each project has one index in `indexes/<id>/`, where the id is random. The index holds CocoIndex's incremental state, the SQLite vector table, and a manifest recording the project's key and its last indexed path. Command output includes this `index_directory` and the `key`.
+
+A project's key decides which index it uses:
+
+- The root of a Git repository with an `origin` remote is keyed by that remote, normalized to `host/owner/repo`. `git@bitbucket.org:acme/api.git` and `https://bitbucket.org/acme/api` both become `bitbucket.org/acme/api`. Moving or re-cloning the repository keeps its index, and unchanged files are not embedded again.
+- Any other directory, including a subdirectory of a repository, is keyed by its resolved path: `local:/path/to/project`.
+- Indexes created by earlier versions are given a key when they are first seen, without being rebuilt. If another index already has that remote key, the older index keeps a path key.
 
 - A file lock prevents reads during updates and concurrent writers.
 - An interrupted or failed update leaves the index marked `ready: false`, and search refuses it until indexing succeeds. There is no fallback to a last good snapshot.
-- Moving a project changes its index identity.
+- Two checkouts of the same repository share one index, which holds whichever of them was indexed last. Select a subdirectory to keep them apart.
+- Moving a project that has no Git remote changes its key, and indexing it again builds a new index.
 - To switch model assets or rebuild incompatible state, move that `index_directory` aside and run `index` again.
 
 ## Offline guarantees
@@ -223,7 +295,7 @@ A Python audit hook also rejects Internet socket operations and DNS lookups. Thi
   delphi-code index -p /path/to/project
 ```
 
-Delphi Code needs read access to source and model files, write access to its user index directory, threads, SQLite extensions, and CocoIndex's LMDB memory maps. It needs no network access. Some stricter sandboxes deny CocoIndex's native storage initialization with `EPERM`; `doctor` probes for this and reports `sandbox_storage_denied`. The application never escalates its own permissions. `doctor` reports the Python guard only; it does not claim that an external OS sandbox is active.
+Delphi Code needs read access to source and model files, write access to its user index directory, threads, SQLite extensions, and CocoIndex's LMDB memory maps. It needs no network access, except for `setup` and for `add` and `sync` of Bitbucket or GitHub repositories; local projects never need it. Some stricter sandboxes deny CocoIndex's native storage initialization with `EPERM`; `doctor` probes for this and reports `sandbox_storage_denied`. The application never escalates its own permissions. `doctor` reports the Python guard only; it does not claim that an external OS sandbox is active.
 
 ## Development
 

{delphi_code-0.1.1 → delphi_code-0.2.0}/README.md

@@ -78,18 +78,82 @@ delphi-code status -p /path/to/project
 - **`status`** reports stored index state. It needs no model and does not rescan source files to detect staleness.
 - **`doctor`** checks the model, SQLite extensions, and native storage, then runs a real local embedding and vector distance calculation.
 - **`setup`** provisions the model (see above).
+- **`add`**, **`sync`**, **`list`**, and **`remove`** manage tracked projects (see below).
 
 `python -m delphi_code` works as an alternative to the `delphi-code` executable.
 
 ### Selecting a project
 
 - `--project` / `-p` accepts a path, resolved relative to the current working directory. The project root is explicit; it is not inferred from Git.
-- After a project has been indexed once, its folder name works too, for example `delphi-code search -p myapp 'parse config'`. A bare name selects the uniquely matching indexed project; duplicate names produce an error listing their paths. Use `./name` or an absolute path to select a directory explicitly.
+- After a project has been indexed once, its key or folder name works too, for example `delphi-code search -p acme/api 'parse config'` or `-p myapp`. A key matches in full or by a trailing part (`bitbucket.org/acme/api`, `acme/api`, or `api`). A name that matches several indexes produces an error listing their keys. Use `./name` or an absolute path to select a directory explicitly.
 - Without `--project`, `search` searches all stored indexes; other commands use the current directory.
 
+### Tracking projects
+
+Instead of indexing projects one by one, keep a list of them and update them all together:
+
+```sh
+delphi-code add ~/src/api ~/src/web --path 'src/*'
+delphi-code sync
+delphi-code list
+delphi-code remove api
+```
+
+- **`add`** records projects and indexes them; `--no-sync` only records them. Filter options are stored with each project, and adding a project again replaces its options.
+- **`sync`** indexes every tracked project. One failing project does not stop the rest: the command then exits with `sync_failed` (code 5) and still reports each project's result under `data`.
+- **`list`** shows tracked projects and every stored index, including ones created with plain `index`. It needs no model.
+- **`remove`** stops tracking a project and deletes its index; `--keep-index` keeps the index.
+
+The list lives in `repos.toml` in the storage directory (`DELPHI_CODE_REGISTRY` overrides the location). You can edit it by hand; `add` and `remove` rewrite it without comments.
+
+```toml
+[[repo]]
+source = "/Users/me/src/api"
+key = "bitbucket.org/acme/api"
+paths = ["src/*"]
+
+[[repo]]
+source = "bitbucket.org/acme/billing"
+ref = "release"
+```
+
+### Bitbucket Cloud and GitHub repositories
+
+Repositories you have not cloned can be tracked too:
+
+```sh
+delphi-code add bitbucket.org/acme/billing
+delphi-code add git@bitbucket.org:acme/billing.git --ref release
+delphi-code add github.com/octo/tools
+```
+
+- `sync` asks the hosting service for the latest commit of the branch or tag, which is the default branch unless `--ref` names another. It skips the repository when the index already holds that commit with the same filters and model. Otherwise it makes a shallow clone into a temporary directory, indexes it, and deletes the clone. Unchanged files are not embedded again.
+- Search results from these repositories carry a `url`: a Bitbucket or GitHub link to the indexed commit and line range. The response also includes `ref` and `commit`, and `project` is `null`.
+- Use `sync`, not `index`, to update them.
+- Only `add` and `sync` touch the network. `index`, `search`, `status`, and `list` stay offline.
+
+**Picking repositories.** Run `add` without sources in a terminal to choose from the repositories your accounts can see:
+
+```sh
+delphi-code add
+```
+
+Choose Bitbucket or GitHub (asked only when both have credentials), then a Bitbucket workspace or GitHub account (skipped when there is only one). Then check repositories in a list you can filter by typing. Repositories you already track start checked, so unchecking one stops tracking it and deletes its index. After you confirm, the new repositories are indexed. Filter options and `--ref` apply to the repositories you add. The prompts are drawn on stderr, and stdout still receives one JSON object with the added repositories under `repos` and the untracked ones under `removed`. Cancelling changes nothing and reports `"cancelled": true`. Without a terminal, `add` requires sources.
+
+**Credentials.** Git never prompts. Credentials from the environment reach git through `GIT_ASKPASS`, never in URLs or arguments, and take precedence over configured git credentials. Listing repositories for the picker uses the services' REST APIs. The API requests run in a separate process, like the model download in `setup`, so the main process keeps its network guard.
+
+| | Bitbucket Cloud | GitHub |
+|---|---|---|
+| Cloning | `BITBUCKET_USERNAME` and `BITBUCKET_APP_PASSWORD`, otherwise git's own credentials | `GITHUB_TOKEN` or `GH_TOKEN`, otherwise git's own credentials (`gh auth setup-git` configures them) |
+| Listing for the picker | The same variables (with `BITBUCKET_EMAIL` in place of the username when set), otherwise the credentials git has stored for bitbucket.org | `GITHUB_TOKEN` or `GH_TOKEN`, otherwise `gh auth token`, otherwise the token git has stored for github.com |
+
+Bitbucket has replaced app passwords with API tokens. An API token also works in `BITBUCKET_APP_PASSWORD`. Git over HTTPS takes the username that Bitbucket shows for it, and the REST API takes your Atlassian account email, which you set in `BITBUCKET_EMAIL`.
+
+`DELPHI_CODE_BITBUCKET_GIT_BASE` and `DELPHI_CODE_GITHUB_GIT_BASE` send clones to another base URL, such as a mirror, and `DELPHI_CODE_BITBUCKET_API_BASE` and `DELPHI_CODE_GITHUB_API_BASE` do the same for the APIs. Result links still point to bitbucket.org and github.com.
+
 ### Searching across projects
 
-Cross-project search returns a single globally ranked list, with the full `project` path on each result. The query is embedded once, `--limit` applies to the combined list, and path/language filters apply within every index. All indexes must be readable, complete, and built with the selected model: a busy, broken, or incompatible index produces an error rather than silently partial results. Use `-p` to narrow the search when needed.
+Cross-project search returns a single globally ranked list, with the `key` and full `project` path on each result (`null` for Bitbucket and GitHub repositories, whose results carry a `url` instead). The query is embedded once, `--limit` applies to the combined list, and path/language filters apply within every index. All indexes must be readable, complete, and built with the selected model: a busy, broken, or incompatible index produces an error rather than silently partial results. Use `-p` to narrow the search when needed.
 
 ## Output
 
@@ -176,11 +240,18 @@ Models and indexes live outside the installation, so reinstalling or upgrading l
 
 The default model is under `models/all-MiniLM-L6-v2/` and indexes are under `indexes/`. `DELPHI_CODE_INDEX_ROOT` overrides the index location.
 
-Each project has one index at `indexes/<sha256 of the resolved project path>/`, independent of the working directory. It holds CocoIndex's incremental state, the SQLite vector table, and a manifest recording the project path; command output includes this `index_directory`.
+Each project has one index in `indexes/<id>/`, where the id is random. The index holds CocoIndex's incremental state, the SQLite vector table, and a manifest recording the project's key and its last indexed path. Command output includes this `index_directory` and the `key`.
+
+A project's key decides which index it uses:
+
+- The root of a Git repository with an `origin` remote is keyed by that remote, normalized to `host/owner/repo`. `git@bitbucket.org:acme/api.git` and `https://bitbucket.org/acme/api` both become `bitbucket.org/acme/api`. Moving or re-cloning the repository keeps its index, and unchanged files are not embedded again.
+- Any other directory, including a subdirectory of a repository, is keyed by its resolved path: `local:/path/to/project`.
+- Indexes created by earlier versions are given a key when they are first seen, without being rebuilt. If another index already has that remote key, the older index keeps a path key.
 
 - A file lock prevents reads during updates and concurrent writers.
 - An interrupted or failed update leaves the index marked `ready: false`, and search refuses it until indexing succeeds. There is no fallback to a last good snapshot.
-- Moving a project changes its index identity.
+- Two checkouts of the same repository share one index, which holds whichever of them was indexed last. Select a subdirectory to keep them apart.
+- Moving a project that has no Git remote changes its key, and indexing it again builds a new index.
 - To switch model assets or rebuild incompatible state, move that `index_directory` aside and run `index` again.
 
 ## Offline guarantees
@@ -194,7 +265,7 @@ A Python audit hook also rejects Internet socket operations and DNS lookups. Thi
   delphi-code index -p /path/to/project
 ```
 
-Delphi Code needs read access to source and model files, write access to its user index directory, threads, SQLite extensions, and CocoIndex's LMDB memory maps. It needs no network access. Some stricter sandboxes deny CocoIndex's native storage initialization with `EPERM`; `doctor` probes for this and reports `sandbox_storage_denied`. The application never escalates its own permissions. `doctor` reports the Python guard only; it does not claim that an external OS sandbox is active.
+Delphi Code needs read access to source and model files, write access to its user index directory, threads, SQLite extensions, and CocoIndex's LMDB memory maps. It needs no network access, except for `setup` and for `add` and `sync` of Bitbucket or GitHub repositories; local projects never need it. Some stricter sandboxes deny CocoIndex's native storage initialization with `EPERM`; `doctor` probes for this and reports `sandbox_storage_denied`. The application never escalates its own permissions. `doctor` reports the Python guard only; it does not claim that an external OS sandbox is active.
 
 ## Development
 

delphi_code-0.2.0/delphi_code/__init__.py

@@ -0,0 +1,3 @@
+from .offline import enforce_offline_policy
+
+enforce_offline_policy()

delphi_code-0.2.0/delphi_code/cli.py

@@ -0,0 +1,188 @@
+import argparse
+from collections.abc import Callable
+from contextlib import redirect_stdout
+import json
+import os
+from pathlib import Path
+import sys
+
+from . import projects
+from .errors import ExitCode, Failure
+from .paths import model_directory
+from .registry import Registry
+from .selection import DEFAULT_MAX_BYTES, FileSelection
+from .store import Store
+
+OUTPUT_SCHEMA_VERSION = 1
+MAX_SEARCH_LIMIT = 1000
+
+
+class Parser(argparse.ArgumentParser):
+    def error(self, message):
+        raise Failure("usage", message, ExitCode.USAGE)
+
+
+def arguments() -> argparse.Namespace:
+    parser = Parser(prog="delphi-code", description="Offline local code search")
+    commands = parser.add_subparsers(dest="command", required=True)
+    for name in ("index", "search", "status", "doctor"):
+        command = commands.add_parser(name)
+        command.add_argument(
+            "--project",
+            "-p",
+            default=None if name == "search" else str(Path.cwd()),
+            help="Project path, key, or indexed name; search defaults to all indexes",
+        )
+        if name != "status":
+            _add_model_option(command)
+        if name in {"index", "search"}:
+            _add_selection_options(command, indexing=name == "index")
+        if name == "search":
+            command.add_argument("query")
+            command.add_argument("--limit", type=_search_limit, default=10)
+    add = commands.add_parser("add", help="Track projects in the registry and index them")
+    add.add_argument(
+        "sources",
+        nargs="*",
+        metavar="SOURCE",
+        help="Project directory, bitbucket.org/workspace/repository, or github.com/owner/repository; "
+        "omit to pick repositories interactively",
+    )
+    add.add_argument("--ref", help="Branch or tag of remote repositories; defaults to the default branch")
+    _add_selection_options(add, indexing=True)
+    _add_model_option(add)
+    add.add_argument("--no-sync", action="store_true", help="Only update the registry")
+    sync = commands.add_parser("sync", help="Index every tracked project")
+    _add_model_option(sync)
+    commands.add_parser("list", help="List tracked projects and stored indexes")
+    remove = commands.add_parser("remove", help="Stop tracking a project and delete its index")
+    remove.add_argument("name", help="Project path, key, or indexed name")
+    remove.add_argument("--keep-index", action="store_true", help="Only untrack the project")
+    setup = commands.add_parser("setup", help="Download or import the pinned model and check the installation")
+    setup.add_argument("--from", dest="source", help="Import a prepared MiniLM model without network access")
+    _add_model_option(setup, help="Model destination")
+    return parser.parse_args()
+
+
+def execute(args: argparse.Namespace) -> dict:
+    return COMMANDS[args.command](args)
+
+
+def main():
+    command = None
+    exit_code = 0
+    try:
+        args = arguments()
+        command = args.command
+        with redirect_stdout(sys.stderr):
+            payload = {"ok": True, "command": command, "data": execute(args)}
+    except Exception as exc:
+        if os.environ.get("DELPHI_CODE_DEBUG") == "1":
+            import traceback
+
+            traceback.print_exc(file=sys.stderr)
+        failure = Failure.from_exception(exc)
+        exit_code = failure.exit_code
+        print(f"delphi-code: {failure}", file=sys.stderr)
+        payload = {"ok": False, "command": command, "error": failure.to_json()}
+        if failure.data is not None:
+            payload["data"] = failure.data
+    print(
+        json.dumps(
+            {"schema_version": OUTPUT_SCHEMA_VERSION, **payload}, ensure_ascii=False, sort_keys=True, allow_nan=False
+        )
+    )
+    raise SystemExit(exit_code)
+
+
+def _index(args):
+    return projects.index_local_project(Store(), args.project, _selection(args), args.model)
+
+
+def _search(args):
+    request = projects.SearchRequest(args.query, args.path, args.language, args.limit)
+    if args.project is None:
+        return projects.search_everywhere(Store(), request, args.model)
+    return projects.search_project(Store(), args.project, request, args.model)
+
+
+def _status(args):
+    return projects.status(Store(), args.project)
+
+
+def _doctor(args):
+    from .doctor import diagnose
+
+    return diagnose(args.model, Store(), args.project)
+
+
+def _add(args):
+    if not args.sources:
+        return projects.add_picked_repositories(
+            Store(), Registry(), args.ref, _selection(args), args.model, sync_now=not args.no_sync
+        )
+    return projects.add_sources(
+        Store(), Registry(), args.sources, args.ref, _selection(args), args.model, sync_now=not args.no_sync
+    )
+
+
+def _sync(args):
+    registry = Registry()
+    return projects.sync(Store(), registry, registry.entries(), args.model)
+
+
+def _list(args):
+    return projects.list_projects(Store(), Registry())
+
+
+def _remove(args):
+    return projects.remove_project(Store(), Registry(), args.name, keep_index=args.keep_index)
+
+
+def _setup(args):
+    from .setup import provision
+
+    return provision(args.model, args.source)
+
+
+COMMANDS: dict[str, Callable[[argparse.Namespace], dict]] = {
+    "index": _index,
+    "search": _search,
+    "status": _status,
+    "doctor": _doctor,
+    "add": _add,
+    "sync": _sync,
+    "list": _list,
+    "remove": _remove,
+    "setup": _setup,
+}
+
+
+def _selection(args) -> FileSelection:
+    return FileSelection(args.path, args.language, args.ignore, args.max_bytes)
+
+
+def _add_selection_options(command, indexing):
+    command.add_argument("--path", action="append", default=[], help="Project-relative glob; repeat for alternatives")
+    command.add_argument("--language", action="append", default=[])
+    if indexing:
+        command.add_argument("--ignore", action="append", default=[])
+        command.add_argument("--max-bytes", type=_positive_integer, default=DEFAULT_MAX_BYTES)
+
+
+def _add_model_option(command, help=None):
+    command.add_argument("--model", default=os.environ.get("DELPHI_CODE_MODEL") or model_directory(), help=help)
+
+
+def _positive_integer(text):
+    value = int(text)
+    if value < 1:
+        raise argparse.ArgumentTypeError("must be positive")
+    return value
+
+
+def _search_limit(text):
+    value = int(text)
+    if not 1 <= value <= MAX_SEARCH_LIMIT:
+        raise argparse.ArgumentTypeError(f"must be between 1 and {MAX_SEARCH_LIMIT}")
+    return value

delphi_code-0.2.0/delphi_code/doctor.py

@@ -0,0 +1,38 @@
+import asyncio
+from importlib.metadata import version
+from pathlib import Path
+import sqlite3
+import tempfile
+
+from .projects import open_model, resolve_existing
+from .store import Store, vector_database
+
+REPORTED_DEPENDENCIES = ("cocoindex", "sqlite-vec", "sentence-transformers", "torch")
+PROBE_TEXT = "local code search"
+
+
+def diagnose(model_location: str, store: Store, project_name: str) -> dict:
+    from .indexing import check_storage
+
+    project, key, index = resolve_existing(store, project_name)
+    model = open_model(model_location)
+    with tempfile.TemporaryDirectory(prefix="delphi-code-doctor-") as scratch:
+        asyncio.run(check_storage(Path(scratch)))
+    vector = model.embed([PROBE_TEXT])[0]
+    with vector_database(":memory:") as db:
+        self_distance = db.execute("SELECT vec_distance_L2(?, ?)", (vector.tobytes(), vector.tobytes())).fetchone()[0]
+    return {
+        "project": str(project) if project else None,
+        "key": key,
+        "index_directory": str(index.directory) if index else None,
+        "model": str(model.directory),
+        "model_sha256": model.sha256,
+        "dimensions": len(vector),
+        "self_distance": self_distance,
+        "device": "cpu",
+        "cocoindex_storage": "ok",
+        "offline": True,
+        "network_guard": "python_audit",
+        "sqlite": sqlite3.sqlite_version,
+        "dependencies": {name: version(name) for name in REPORTED_DEPENDENCIES},
+    }

{delphi_code-0.1.1 → delphi_code-0.2.0}/delphi_code/download.py

@@ -15,7 +15,9 @@ def main():
     manifest = json.loads(Path(__file__).with_name("model_provenance.json").read_text())
     destination = Path(sys.argv[1])
     snapshot_download(
-        manifest["repository"], revision=manifest["revision"], local_dir=destination,
+        manifest["repository"],
+        revision=manifest["revision"],
+        local_dir=destination,
         allow_patterns=list(manifest["sha256"]),
     )
     if not (destination / "LICENSE").exists():

delphi_code-0.2.0/delphi_code/errors.py

@@ -0,0 +1,27 @@
+from enum import IntEnum
+
+
+class ExitCode(IntEnum):
+    USAGE = 2
+    RUNTIME_ASSETS = 3
+    INDEX_STATE = 4
+    OPERATION = 5
+
+
+class Failure(Exception):
+    def __init__(self, code: str, message: str, exit_code: ExitCode, data: dict | None = None):
+        super().__init__(message)
+        self.code = code
+        self.exit_code = exit_code
+        self.data = data
+
+    @classmethod
+    def from_exception(cls, exc: BaseException) -> "Failure":
+        if isinstance(exc, Failure):
+            return exc
+        if isinstance(exc, ModuleNotFoundError):
+            return cls("dependency_missing", str(exc), ExitCode.RUNTIME_ASSETS)
+        return cls("runtime_error", str(exc), ExitCode.OPERATION)
+
+    def to_json(self) -> dict:
+        return {"code": self.code, "message": str(self)}

delphi_code-0.2.0/delphi_code/files.py

@@ -0,0 +1,92 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import NamedTuple
+
+from pathspec import GitIgnoreSpec
+
+from .selection import FileSelection
+
+ALWAYS_EXCLUDED_NAMES = {".git", ".delphi-code", ".venv", "venv", "node_modules", "__pycache__", ".models"}
+IGNORE_FILE_NAMES = (".gitignore", ".delphi-codeignore")
+FALLBACK_LANGUAGE = "text"
+
+SourceFile = tuple[str, str, str]
+
+
+class CollectedFiles(NamedTuple):
+    files: dict[str, SourceFile]
+    skipped: int
+
+
+def collect(root: Path, selection: FileSelection, excluded: Path | None = None) -> CollectedFiles:
+    collector = _Collector(root, selection, excluded)
+    collector.walk(root, [])
+    return CollectedFiles(collector.files, collector.skipped)
+
+
+class _Collector:
+    def __init__(self, root: Path, selection: FileSelection, excluded: Path | None):
+        self.root = root
+        self.selection = selection
+        self.excluded = excluded
+        self.extra_ignores = GitIgnoreSpec.from_lines(selection.ignores)
+        self.files: dict[str, SourceFile] = {}
+        self.skipped = 0
+
+    def walk(self, directory: Path, inherited_rules: list[tuple[Path, GitIgnoreSpec]]):
+        rules = inherited_rules + _ignore_rules_in(directory)
+        for entry in sorted(directory.iterdir()):
+            if self._is_skipped(entry, rules):
+                self.skipped += 1
+            elif entry.is_dir():
+                self.walk(entry, rules)
+            elif entry.is_file():
+                self._read(entry)
+
+    def _is_skipped(self, entry: Path, rules: list[tuple[Path, GitIgnoreSpec]]) -> bool:
+        if entry.is_symlink() or entry.name in ALWAYS_EXCLUDED_NAMES or entry == self.excluded:
+            return True
+        suffix = "/" if entry.is_dir() else ""
+        ignored = False
+        for base, spec in rules:
+            match = spec.check_file(entry.relative_to(base).as_posix() + suffix)
+            if match.include is not None:
+                ignored = match.include
+        return ignored or self.extra_ignores.match_file(self._relative(entry) + suffix)
+
+    def _read(self, entry: Path):
+        from cocoindex.ops.text import detect_code_language
+
+        relative = self._relative(entry)
+        language = detect_code_language(filename=entry.name) or FALLBACK_LANGUAGE
+        if not self.selection.includes(relative, language):
+            return
+        content = _text_content(entry, self.selection.max_bytes)
+        if content is None:
+            self.skipped += 1
+        elif content.strip():
+            self.files[relative] = (relative, language, content)
+
+    def _relative(self, entry: Path) -> str:
+        return entry.relative_to(self.root).as_posix()
+
+
+def _ignore_rules_in(directory: Path) -> list[tuple[Path, GitIgnoreSpec]]:
+    rules = []
+    for name in IGNORE_FILE_NAMES:
+        rule_file = directory / name
+        if rule_file.is_file() and not rule_file.is_symlink():
+            rules.append((directory, GitIgnoreSpec.from_lines(rule_file.read_text().splitlines())))
+    return rules
+
+
+def _text_content(path: Path, max_bytes: int) -> str | None:
+    with path.open("rb") as stream:
+        data = stream.read(max_bytes + 1)
+    if len(data) > max_bytes or b"\0" in data:
+        return None
+    try:
+        return data.decode("utf-8")
+    except UnicodeDecodeError:
+        return None