cocoindex-code 0.2.8__tar.gz → 0.2.10__tar.gz

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex-code
-Version: 0.2.8
+Version: 0.2.10
 Summary: MCP server for indexing and querying codebases using CocoIndex
 Project-URL: Homepage, https://github.com/cocoindex-io/cocoindex-code
 Project-URL: Repository, https://github.com/cocoindex-io/cocoindex-code
@@ -17,7 +17,7 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.11
-Requires-Dist: cocoindex[litellm]==1.0.0a37
+Requires-Dist: cocoindex[litellm]==1.0.0a38
 Requires-Dist: einops>=0.8.2
 Requires-Dist: mcp>=1.0.0
 Requires-Dist: msgspec>=0.19.0
@@ -226,6 +226,105 @@ ccc search --refresh database schema                 # update index first, then
 
 By default, `ccc search` scopes results to your current working directory (relative to the project root). Use `--path` to override.
 
+## Docker
+
+A Docker image is available for teams who want a reproducible, dependency-free
+setup — no Python, `uv`, or system dependencies required on the host.
+
+The recommended approach is a **persistent container**: start it once, and use
+`docker exec` to run CLI commands or connect MCP sessions to it. The daemon
+inside stays warm across sessions, so the embedding model is loaded only once.
+
+### Step 1 — Start the container
+
+```bash
+docker run -d --name cocoindex-code \
+  --volume "$(pwd):/workspace" \
+  --volume cocoindex-db:/db \
+  --volume cocoindex-model-cache:/root/.cache \
+  ghcr.io/cocoindex-io/cocoindex-code:latest
+```
+
+- `/workspace` — mount your project root here
+- `cocoindex-db` — index databases live inside the container (fast native I/O, no cross-OS volume issues)
+- `cocoindex-model-cache` — persists the embedding model across image upgrades
+
+### Step 2 — Index your codebase
+
+```bash
+docker exec -it cocoindex-code ccc index
+```
+
+### Step 3 — Connect your coding agent
+
+<details>
+<summary>Claude Code</summary>
+
+```bash
+claude mcp add cocoindex-code -- docker exec -i cocoindex-code ccc mcp
+```
+
+Or via `.mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "cocoindex-code": {
+      "type": "stdio",
+      "command": "docker",
+      "args": ["exec", "-i", "cocoindex-code", "ccc", "mcp"]
+    }
+  }
+}
+```
+</details>
+
+<details>
+<summary>Codex</summary>
+
+```bash
+codex mcp add cocoindex-code -- docker exec -i cocoindex-code ccc mcp
+```
+</details>
+
+### CLI usage inside the container
+
+All `ccc` commands work via `docker exec`:
+
+```bash
+docker exec -it cocoindex-code ccc index
+docker exec -it cocoindex-code ccc search "authentication logic"
+docker exec -it cocoindex-code ccc status
+```
+
+Or set an alias on your host so it feels native:
+
+```bash
+alias ccc='docker exec -it cocoindex-code ccc'
+```
+
+### Configuration via environment variables
+
+Pass configuration to `docker run` with `-e`:
+
+```bash
+# Extra extensions (e.g. Typesafe Config, SBT build files)
+-e COCOINDEX_CODE_EXTRA_EXTENSIONS="conf,sbt"
+
+# Exclude build artefacts (Scala/SBT example)
+-e COCOINDEX_CODE_EXCLUDE_PATTERNS='["**/target/**","**/.bloop/**","**/.metals/**"]'
+
+# Swap in a code-optimised embedding model
+-e COCOINDEX_CODE_EMBEDDING_MODEL=voyage/voyage-code-3
+-e VOYAGE_API_KEY=your-key
+```
+
+### Build the image locally
+
+```bash
+docker build -t cocoindex-code:local -f docker/Dockerfile .
+```
+
 ## Features
 - **Semantic Code Search**: Find relevant code using natural language queries when grep doesn't work well, and save tokens immediately.
 - **Ultra Performant**: ⚡ Built on top of ultra performant [Rust indexing engine](https://github.com/cocoindex-io/cocoindex). Only re-indexes changed files for fast updates.
@@ -276,10 +375,43 @@ exclude_patterns:
 language_overrides:
   - ext: inc               # treat .inc files as PHP
     lang: php
+
+chunkers:
+  - ext: toml              # use a custom chunker for .toml files
+    module: example_toml_chunker:toml_chunker
 ```
 
 > `.cocoindex_code/` is automatically added to `.gitignore` during init.
 
+Use `chunkers` when you want to control how a file type is split into chunks before indexing.
+
+`module: example_toml_chunker:toml_chunker` means:
+- `example_toml_chunker` is a local Python module
+- `toml_chunker` is the function inside that module
+
+In practice, this usually means:
+- you create a Python file in your project, for example `example_toml_chunker.py`
+- you add a function in that file
+- you point `settings.yml` at it with `module.path:function_name`
+
+The function should use this signature:
+
+```python
+from pathlib import Path
+from cocoindex_code.chunking import Chunk
+
+def my_chunker(path: Path, content: str) -> tuple[str | None, list[Chunk]]:
+    ...
+```
+
+- `path` is the file being indexed
+- `content` is the full text of that file
+- return `language_override` as a string like `"toml"` if you want to override language detection
+- return `None` as `language_override` if you want to keep the detected language
+- return a `list[Chunk]` with the chunks you want stored in the index
+
+See [`src/cocoindex_code/chunking.py`](./src/cocoindex_code/chunking.py) for the public types and [`tests/example_toml_chunker.py`](./tests/example_toml_chunker.py) for a complete example.
+
 ## Embedding Models
 
 By default, a local SentenceTransformers model ([sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)) is used — no API key required. To use a different model, edit `~/.cocoindex_code/global_settings.yml`.

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/README.md

@@ -187,6 +187,105 @@ ccc search --refresh database schema                 # update index first, then
 
 By default, `ccc search` scopes results to your current working directory (relative to the project root). Use `--path` to override.
 
+## Docker
+
+A Docker image is available for teams who want a reproducible, dependency-free
+setup — no Python, `uv`, or system dependencies required on the host.
+
+The recommended approach is a **persistent container**: start it once, and use
+`docker exec` to run CLI commands or connect MCP sessions to it. The daemon
+inside stays warm across sessions, so the embedding model is loaded only once.
+
+### Step 1 — Start the container
+
+```bash
+docker run -d --name cocoindex-code \
+  --volume "$(pwd):/workspace" \
+  --volume cocoindex-db:/db \
+  --volume cocoindex-model-cache:/root/.cache \
+  ghcr.io/cocoindex-io/cocoindex-code:latest
+```
+
+- `/workspace` — mount your project root here
+- `cocoindex-db` — index databases live inside the container (fast native I/O, no cross-OS volume issues)
+- `cocoindex-model-cache` — persists the embedding model across image upgrades
+
+### Step 2 — Index your codebase
+
+```bash
+docker exec -it cocoindex-code ccc index
+```
+
+### Step 3 — Connect your coding agent
+
+<details>
+<summary>Claude Code</summary>
+
+```bash
+claude mcp add cocoindex-code -- docker exec -i cocoindex-code ccc mcp
+```
+
+Or via `.mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "cocoindex-code": {
+      "type": "stdio",
+      "command": "docker",
+      "args": ["exec", "-i", "cocoindex-code", "ccc", "mcp"]
+    }
+  }
+}
+```
+</details>
+
+<details>
+<summary>Codex</summary>
+
+```bash
+codex mcp add cocoindex-code -- docker exec -i cocoindex-code ccc mcp
+```
+</details>
+
+### CLI usage inside the container
+
+All `ccc` commands work via `docker exec`:
+
+```bash
+docker exec -it cocoindex-code ccc index
+docker exec -it cocoindex-code ccc search "authentication logic"
+docker exec -it cocoindex-code ccc status
+```
+
+Or set an alias on your host so it feels native:
+
+```bash
+alias ccc='docker exec -it cocoindex-code ccc'
+```
+
+### Configuration via environment variables
+
+Pass configuration to `docker run` with `-e`:
+
+```bash
+# Extra extensions (e.g. Typesafe Config, SBT build files)
+-e COCOINDEX_CODE_EXTRA_EXTENSIONS="conf,sbt"
+
+# Exclude build artefacts (Scala/SBT example)
+-e COCOINDEX_CODE_EXCLUDE_PATTERNS='["**/target/**","**/.bloop/**","**/.metals/**"]'
+
+# Swap in a code-optimised embedding model
+-e COCOINDEX_CODE_EMBEDDING_MODEL=voyage/voyage-code-3
+-e VOYAGE_API_KEY=your-key
+```
+
+### Build the image locally
+
+```bash
+docker build -t cocoindex-code:local -f docker/Dockerfile .
+```
+
 ## Features
 - **Semantic Code Search**: Find relevant code using natural language queries when grep doesn't work well, and save tokens immediately.
 - **Ultra Performant**: ⚡ Built on top of ultra performant [Rust indexing engine](https://github.com/cocoindex-io/cocoindex). Only re-indexes changed files for fast updates.
@@ -237,10 +336,43 @@ exclude_patterns:
 language_overrides:
   - ext: inc               # treat .inc files as PHP
     lang: php
+
+chunkers:
+  - ext: toml              # use a custom chunker for .toml files
+    module: example_toml_chunker:toml_chunker
 ```
 
 > `.cocoindex_code/` is automatically added to `.gitignore` during init.
 
+Use `chunkers` when you want to control how a file type is split into chunks before indexing.
+
+`module: example_toml_chunker:toml_chunker` means:
+- `example_toml_chunker` is a local Python module
+- `toml_chunker` is the function inside that module
+
+In practice, this usually means:
+- you create a Python file in your project, for example `example_toml_chunker.py`
+- you add a function in that file
+- you point `settings.yml` at it with `module.path:function_name`
+
+The function should use this signature:
+
+```python
+from pathlib import Path
+from cocoindex_code.chunking import Chunk
+
+def my_chunker(path: Path, content: str) -> tuple[str | None, list[Chunk]]:
+    ...
+```
+
+- `path` is the file being indexed
+- `content` is the full text of that file
+- return `language_override` as a string like `"toml"` if you want to override language detection
+- return `None` as `language_override` if you want to keep the detected language
+- return a `list[Chunk]` with the chunks you want stored in the index
+
+See [`src/cocoindex_code/chunking.py`](./src/cocoindex_code/chunking.py) for the public types and [`tests/example_toml_chunker.py`](./tests/example_toml_chunker.py) for a complete example.
+
 ## Embedding Models
 
 By default, a local SentenceTransformers model ([sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)) is used — no API key required. To use a different model, edit `~/.cocoindex_code/global_settings.yml`.

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/pyproject.toml

@@ -23,7 +23,7 @@ classifiers = [
 
 dependencies = [
     "mcp>=1.0.0",
-    "cocoindex[litellm]==1.0.0a37",
+    "cocoindex[litellm]==1.0.0a38",
     "sentence-transformers>=2.2.0",
     "sqlite-vec>=0.1.0",
     "pydantic>=2.0.0",
@@ -91,9 +91,11 @@ select = ["E", "F", "I", "N", "W", "UP"]
 python_version = "3.11"
 strict = true
 ignore_missing_imports = true
+explicit_package_bases = true
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 python_files = ["test_*.py"]
 python_functions = ["test_*"]
 addopts = "-v --tb=short"
+asyncio_mode = "auto"

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/src/cocoindex_code/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.2.8'
-__version_tuple__ = version_tuple = (0, 2, 8)
+__version__ = version = '0.2.10'
+__version_tuple__ = version_tuple = (0, 2, 10)
 
 __commit_id__ = commit_id = None

cocoindex_code-0.2.10/src/cocoindex_code/chunking.py

@@ -0,0 +1,29 @@
+"""Public API for writing custom chunkers.
+
+Example usage::
+
+    from pathlib import Path
+    from cocoindex_code.chunking import Chunk, ChunkerFn, TextPosition
+
+    def my_chunker(path: Path, content: str) -> tuple[str | None, list[Chunk]]:
+        pos = TextPosition(byte_offset=0, char_offset=0, line=1, column=0)
+        return "mylang", [Chunk(text=content, start=pos, end=pos)]
+"""
+
+from __future__ import annotations
+
+import pathlib as _pathlib
+from collections.abc import Callable as _Callable
+
+import cocoindex as _coco
+from cocoindex.resources.chunk import Chunk, TextPosition
+
+# Callable alias (not Protocol) — consistent with codebase style.
+# language_override=None keeps the language detected by detect_code_language.
+# path is not resolved (no syscall); call path.resolve() inside the chunker if needed.
+ChunkerFn = _Callable[[_pathlib.Path, str], tuple[str | None, list[Chunk]]]
+
+# tracked=False: callables are not fingerprint-able; daemon restart re-indexes anyway.
+CHUNKER_REGISTRY = _coco.ContextKey[dict[str, ChunkerFn]]("chunker_registry", tracked=False)
+
+__all__ = ["Chunk", "ChunkerFn", "CHUNKER_REGISTRY", "TextPosition"]

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/src/cocoindex_code/cli.py

@@ -12,13 +12,16 @@ import typer as _typer
 from .client import DaemonStartError
 from .protocol import DoctorCheckResult, IndexingProgress, ProjectStatusResponse, SearchResponse
 from .settings import (
+    cocoindex_db_path,
     default_project_settings,
     default_user_settings,
     find_parent_with_marker,
     find_project_root,
+    project_settings_path,
     resolve_db_dir,
     save_project_settings,
     save_user_settings,
+    target_sqlite_db_path,
     user_settings_path,
 )
 
@@ -284,10 +287,8 @@ def init(
     force: bool = _typer.Option(False, "-f", "--force", help="Skip parent directory warning"),
 ) -> None:
     """Initialize a project for cocoindex-code."""
-    from .settings import project_settings_path as _project_settings_path
-
     cwd = Path.cwd().resolve()
-    settings_file = _project_settings_path(cwd)
+    settings_file = project_settings_path(cwd)
 
     # Always ensure user settings exist
     user_path = user_settings_path()
@@ -377,8 +378,15 @@ def status() -> None:
     """Show project status."""
     from . import client as _client
 
-    project_root = str(require_project_root())
+    project_root_path = require_project_root()
+    project_root = str(project_root_path)
     print_project_header(project_root)
+
+    _typer.echo(f"Settings: {project_settings_path(project_root_path)}")
+    db_path = target_sqlite_db_path(project_root_path)
+    if db_path.exists():
+        _typer.echo(f"Index DB: {db_path}")
+
     print_index_stats(_client.project_status(project_root))
 
 
@@ -393,10 +401,10 @@ def reset(
     db_dir = resolve_db_dir(project_root)
 
     db_files = [
-        db_dir / "cocoindex.db",
-        db_dir / "target_sqlite.db",
+        cocoindex_db_path(project_root),
+        target_sqlite_db_path(project_root),
     ]
-    settings_file = cocoindex_dir / "settings.yml"
+    settings_file = project_settings_path(project_root)
 
     # Determine what will be deleted
     to_delete = [f for f in db_files if f.exists()]
@@ -503,16 +511,10 @@ def doctor() -> None:
     from .settings import (
         load_user_settings as _load_user_settings,
     )
-    from .settings import (
-        project_settings_path as _project_settings_path,
-    )
-    from .settings import (
-        user_settings_path as _user_settings_path,
-    )
 
     # --- 1. Global settings (local, no daemon needed) ---
     _print_section("Global Settings")
-    settings_path = _user_settings_path()
+    settings_path = user_settings_path()
     _typer.echo(f"  Settings: {settings_path}")
     try:
         user_settings = _load_user_settings()
@@ -570,7 +572,7 @@ def doctor() -> None:
     # --- 6. Project settings (local, no daemon needed) ---
     if project_root is not None:
         _print_section("Project Settings")
-        ps_path = _project_settings_path(project_root)
+        ps_path = project_settings_path(project_root)
         _typer.echo(f"  Settings: {ps_path}")
         try:
             ps = _load_project_settings(project_root)
@@ -597,10 +599,9 @@ def doctor() -> None:
 
     # --- 8. Log files ---
     _print_section("Log Files")
-    from .daemon import daemon_dir as _daemon_dir
+    from .daemon import daemon_log_path as _daemon_log_path
 
-    log_dir = _daemon_dir()
-    _typer.echo(f"  Daemon logs: {log_dir / 'daemon.log'}")
+    _typer.echo(f"  Daemon logs: {_daemon_log_path()}")
     _typer.echo("  Check logs above for further troubleshooting.")
 
 

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/src/cocoindex_code/client.py

@@ -343,10 +343,10 @@ def start_daemon() -> subprocess.Popen[bytes]:
     Returns the ``Popen`` object so callers can detect early process death
     (via ``proc.poll()``) instead of waiting for a full timeout.
     """
-    from .daemon import daemon_dir
+    from .daemon import daemon_dir, daemon_log_path
 
     daemon_dir().mkdir(parents=True, exist_ok=True)
-    log_path = daemon_dir() / "daemon.log"
+    log_path = daemon_log_path()
 
     ccc_path = _find_ccc_executable()
     if ccc_path:

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/src/cocoindex_code/daemon.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import asyncio
+import importlib
 import logging
 import os
 import signal
@@ -15,6 +16,7 @@ from pathlib import Path
 from typing import Any
 
 from ._version import __version__
+from .chunking import ChunkerFn as _ChunkerFn
 from .project import Project
 from .protocol import (
     DaemonEnvRequest,
@@ -46,15 +48,37 @@ from .protocol import (
     encode_response,
 )
 from .settings import (
+    ChunkerMapping,
     global_settings_mtime_us,
+    load_project_settings,
     load_user_settings,
-    resolve_db_dir,
+    target_sqlite_db_path,
     user_settings_dir,
 )
 from .shared import Embedder, create_embedder
 
 logger = logging.getLogger(__name__)
 
+
+def _resolve_chunker_registry(mappings: list[ChunkerMapping]) -> dict[str, _ChunkerFn]:
+    """Resolve ``ChunkerMapping`` settings entries to a ``{suffix: fn}`` dict.
+
+    Each ``mapping.module`` must be a ``"module.path:callable"`` string importable
+    from the current environment.
+    """
+    registry: dict[str, _ChunkerFn] = {}
+    for cm in mappings:
+        module_path, _, attr = cm.module.partition(":")
+        if not attr:
+            raise ValueError(f"chunker module {cm.module!r} must use 'module.path:callable' format")
+        mod = importlib.import_module(module_path)
+        fn = getattr(mod, attr)
+        if not callable(fn):
+            raise ValueError(f"chunker {cm.module!r}: {attr!r} is not callable")
+        registry[f".{cm.ext}"] = fn
+    return registry
+
+
 # ---------------------------------------------------------------------------
 # Daemon paths
 # ---------------------------------------------------------------------------
@@ -111,7 +135,9 @@ class ProjectRegistry:
         """Get or create a Project for the given root. Lazy initialization."""
         if project_root not in self._projects:
             root = Path(project_root)
-            project = await Project.create(root, self._embedder)
+            project_settings = load_project_settings(root)
+            chunker_registry = _resolve_chunker_registry(project_settings.chunkers)
+            project = await Project.create(root, self._embedder, chunker_registry=chunker_registry)
             self._projects[project_root] = project
         return self._projects[project_root]
 
@@ -346,7 +372,7 @@ async def _check_index_status(project_root_str: str) -> DoctorCheckResult:
     from cocoindex.connectors import sqlite as coco_sqlite
 
     project_root = Path(project_root_str)
-    db_path = resolve_db_dir(project_root) / "target_sqlite.db"
+    db_path = target_sqlite_db_path(project_root)
     details = [f"Index: {db_path}"]
 
     if not db_path.exists():

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/src/cocoindex_code/indexer.py

@@ -14,6 +14,7 @@ from cocoindex.resources.file import FilePathMatcher, PatternFilePathMatcher
 from cocoindex.resources.id import IdGenerator
 from pathspec import GitIgnoreSpec
 
+from .chunking import CHUNKER_REGISTRY
 from .settings import load_gitignore_spec, load_project_settings
 from .shared import (
     CODEBASE_DIR,
@@ -158,13 +159,20 @@ async def process_file(
         or "text"
     )
 
-    chunks = splitter.split(
-        content,
-        chunk_size=CHUNK_SIZE,
-        min_chunk_size=MIN_CHUNK_SIZE,
-        chunk_overlap=CHUNK_OVERLAP,
-        language=language,
-    )
+    chunker_registry = coco.use_context(CHUNKER_REGISTRY)
+    chunker = chunker_registry.get(suffix)
+    if chunker is not None:
+        language_override, chunks = chunker(Path(file.file_path.path), content)
+        if language_override is not None:
+            language = language_override
+    else:
+        chunks = splitter.split(
+            content,
+            chunk_size=CHUNK_SIZE,
+            min_chunk_size=MIN_CHUNK_SIZE,
+            chunk_overlap=CHUNK_OVERLAP,
+            language=language,
+        )
 
     id_gen = IdGenerator()
 

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/src/cocoindex_code/project.py

@@ -10,6 +10,7 @@ from pathlib import Path
 import cocoindex as coco
 from cocoindex.connectors import sqlite as coco_sqlite
 
+from .chunking import CHUNKER_REGISTRY, ChunkerFn
 from .indexer import indexer_main
 from .protocol import (
     IndexingProgress,
@@ -21,7 +22,15 @@ from .protocol import (
     SearchResult,
 )
 from .query import query_codebase
-from .settings import resolve_db_dir
+from .settings import (
+    cocoindex_db_path as _cocoindex_db_path,
+)
+from .settings import (
+    resolve_db_dir,
+)
+from .settings import (
+    target_sqlite_db_path as _target_sqlite_db_path,
+)
 from .shared import (
     CODEBASE_DIR,
     EMBEDDER,
@@ -171,7 +180,7 @@ class Project:
         offset: int = 0,
     ) -> list[SearchResult]:
         """Search within this project."""
-        target_db = resolve_db_dir(self._project_root) / "target_sqlite.db"
+        target_db = _target_sqlite_db_path(self._project_root)
         results = await query_codebase(
             query=query,
             target_sqlite_db_path=target_db,
@@ -248,12 +257,20 @@ class Project:
     async def create(
         project_root: Path,
         embedder: Embedder,
+        chunker_registry: dict[str, ChunkerFn] | None = None,
     ) -> Project:
         """Create a project with explicit embedder.
 
         Project-level settings and .gitignore are NOT cached here — the
         indexer loads them fresh from disk on every run so that user edits
         take effect without restarting the daemon.
+
+        Args:
+            project_root: Root directory of the codebase to index.
+            embedder: Embedding model instance.
+            chunker_registry: Optional mapping of file suffix (e.g. ``".toml"``)
+                to a ``ChunkerFn``. When a suffix matches, the registered
+                chunker is called instead of the built-in splitter.
         """
         settings_dir = project_root / ".cocoindex_code"
         settings_dir.mkdir(parents=True, exist_ok=True)
@@ -261,15 +278,16 @@ class Project:
         db_dir = resolve_db_dir(project_root)
         db_dir.mkdir(parents=True, exist_ok=True)
 
-        cocoindex_db_path = db_dir / "cocoindex.db"
-        target_sqlite_db_path = db_dir / "target_sqlite.db"
+        cocoindex_db = _cocoindex_db_path(project_root)
+        target_sqlite_db = _target_sqlite_db_path(project_root)
 
-        settings = coco.Settings.from_env(cocoindex_db_path)
+        settings = coco.Settings.from_env(cocoindex_db)
 
         context = coco.ContextProvider()
         context.provide(CODEBASE_DIR, project_root)
-        context.provide(SQLITE_DB, coco_sqlite.connect(str(target_sqlite_db_path), load_vec=True))
+        context.provide(SQLITE_DB, coco_sqlite.connect(str(target_sqlite_db), load_vec=True))
         context.provide(EMBEDDER, embedder)
+        context.provide(CHUNKER_REGISTRY, dict(chunker_registry) if chunker_registry else {})
 
         env = coco.Environment(settings, context_provider=context)
         app = coco.App(

{cocoindex_code-0.2.8 → cocoindex_code-0.2.10}/src/cocoindex_code/settings.py

@@ -45,6 +45,29 @@ DEFAULT_INCLUDED_PATTERNS: list[str] = [
     "**/*.rst",  # reStructuredText
     "**/*.php",  # PHP
     "**/*.lua",  # Lua
+    "**/*.rb",  # Ruby
+    "**/*.swift",  # Swift
+    "**/*.kt",  # Kotlin
+    "**/*.kts",  # Kotlin script
+    "**/*.scala",  # Scala
+    "**/*.r",  # R
+    "**/*.html",  # HTML
+    "**/*.htm",  # HTML
+    "**/*.css",  # CSS
+    "**/*.scss",  # SCSS
+    "**/*.json",  # JSON
+    "**/*.xml",  # XML
+    "**/*.yaml",  # YAML
+    "**/*.yml",  # YAML
+    "**/*.toml",  # TOML
+    "**/*.sol",  # Solidity
+    "**/*.pas",  # Pascal
+    "**/*.dpr",  # Pascal/Delphi
+    "**/*.dtd",  # DTD
+    "**/*.f",  # Fortran
+    "**/*.f90",  # Fortran
+    "**/*.f95",  # Fortran
+    "**/*.f03",  # Fortran
 ]
 
 DEFAULT_EXCLUDED_PATTERNS: list[str] = [
@@ -83,11 +106,18 @@ class LanguageOverride:
     lang: str  # e.g. "php"
 
 
+@dataclass
+class ChunkerMapping:
+    ext: str  # without dot, e.g. "toml"
+    module: str  # "module.path:callable", e.g. "cocoindex_code.toml_chunker:toml_chunker"
+
+
 @dataclass
 class ProjectSettings:
     include_patterns: list[str] = field(default_factory=lambda: list(DEFAULT_INCLUDED_PATTERNS))
     exclude_patterns: list[str] = field(default_factory=lambda: list(DEFAULT_EXCLUDED_PATTERNS))
     language_overrides: list[LanguageOverride] = field(default_factory=list)
+    chunkers: list[ChunkerMapping] = field(default_factory=list)
 
 
 # ---------------------------------------------------------------------------
@@ -194,6 +224,20 @@ def _reset_db_path_mapping_cache() -> None:
     _db_path_mapping = None
 
 
+_TARGET_SQLITE_DB_NAME = "target_sqlite.db"
+_COCOINDEX_DB_NAME = "cocoindex.db"
+
+
+def target_sqlite_db_path(project_root: Path) -> Path:
+    """Return the path to the vector index SQLite database for a project."""
+    return resolve_db_dir(project_root) / _TARGET_SQLITE_DB_NAME
+
+
+def cocoindex_db_path(project_root: Path) -> Path:
+    """Return the path to the CocoIndex state database for a project."""
+    return resolve_db_dir(project_root) / _COCOINDEX_DB_NAME
+
+
 def user_settings_dir() -> Path:
     """Return ``~/.cocoindex_code/``.
 
@@ -238,7 +282,7 @@ def find_legacy_project_root(start: Path) -> Path | None:
     """
     current = start.resolve()
     while True:
-        if (current / _SETTINGS_DIR_NAME / "cocoindex.db").exists():
+        if (current / _SETTINGS_DIR_NAME / _COCOINDEX_DB_NAME).exists():
             return current
         parent = current.parent
         if parent == current:
@@ -337,6 +381,8 @@ def _project_settings_to_dict(settings: ProjectSettings) -> dict[str, Any]:
         d["language_overrides"] = [
             {"ext": lo.ext, "lang": lo.lang} for lo in settings.language_overrides
         ]
+    if settings.chunkers:
+        d["chunkers"] = [{"ext": cm.ext, "module": cm.module} for cm in settings.chunkers]
     return d
 
 
@@ -344,10 +390,12 @@ def _project_settings_from_dict(d: dict[str, Any]) -> ProjectSettings:
     overrides = [
         LanguageOverride(ext=lo["ext"], lang=lo["lang"]) for lo in d.get("language_overrides", [])
     ]
+    chunkers = [ChunkerMapping(ext=cm["ext"], module=cm["module"]) for cm in d.get("chunkers", [])]
     return ProjectSettings(
         include_patterns=d.get("include_patterns", list(DEFAULT_INCLUDED_PATTERNS)),
         exclude_patterns=d.get("exclude_patterns", list(DEFAULT_EXCLUDED_PATTERNS)),
         language_overrides=overrides,
+        chunkers=chunkers,
     )
 
 

cocoindex_code-0.2.8/src/cocoindex_code/config.py

@@ -1,144 +0,0 @@
-"""Configuration management for cocoindex-code."""
-
-from __future__ import annotations
-
-import json
-import os
-from dataclasses import dataclass
-from pathlib import Path
-
-from .settings import resolve_db_dir
-
-_DEFAULT_MODEL = "sbert/sentence-transformers/all-MiniLM-L6-v2"
-
-
-def _find_root_with_marker(start: Path, markers: list[str]) -> Path | None:
-    """Walk up from start, return first directory containing any marker."""
-    current = start
-    while True:
-        if any((current / m).exists() for m in markers):
-            return current
-        parent = current.parent
-        if parent == current:
-            return None
-        current = parent
-
-
-def _discover_codebase_root() -> Path:
-    """Discover the codebase root directory.
-
-    Discovery order:
-    1. Find nearest parent with `.cocoindex_code` directory (re-anchor to previously-indexed tree)
-    2. Find nearest parent with any common project root marker
-    3. Fall back to current working directory
-    """
-    cwd = Path.cwd()
-
-    # First, look for existing .cocoindex_code directory
-    root = _find_root_with_marker(cwd, [".cocoindex_code"])
-    if root is not None:
-        return root
-
-    # Then, look for common project root markers
-    markers = [".git", "pyproject.toml", "package.json", "Cargo.toml", "go.mod"]
-    root = _find_root_with_marker(cwd, markers)
-    return root if root is not None else cwd
-
-
-def _parse_json_string_list_env(var_name: str) -> list[str]:
-    """Parse an environment variable as a JSON array of strings."""
-    raw_value = os.environ.get(var_name, "")
-    if not raw_value.strip():
-        return []
-
-    try:
-        parsed = json.loads(raw_value)
-    except json.JSONDecodeError as exc:
-        raise ValueError(f"{var_name} must be a JSON array of strings, got invalid JSON") from exc
-
-    if not isinstance(parsed, list):
-        raise ValueError(f"{var_name} must be a JSON array of strings")
-
-    result: list[str] = []
-    for item in parsed:
-        if not isinstance(item, str):
-            raise ValueError(f"{var_name} must be a JSON array of strings")
-        item = item.strip()
-        if item:
-            result.append(item)
-
-    return result
-
-
-@dataclass
-class Config:
-    """Configuration loaded from environment variables."""
-
-    codebase_root_path: Path
-    embedding_model: str
-    index_dir: Path
-    device: str | None
-    extra_extensions: dict[str, str | None]
-    excluded_patterns: list[str]
-
-    @classmethod
-    def from_env(cls) -> Config:
-        """Load configuration from environment variables."""
-        # Get root path from env or discover it
-        root_path_str = os.environ.get("COCOINDEX_CODE_ROOT_PATH")
-        if root_path_str:
-            root = Path(root_path_str).resolve()
-        else:
-            root = _discover_codebase_root()
-
-        # Get embedding model
-        # Prefix "sbert/" for SentenceTransformers models, otherwise LiteLLM.
-        embedding_model = os.environ.get(
-            "COCOINDEX_CODE_EMBEDDING_MODEL",
-            _DEFAULT_MODEL,
-        )
-
-        # Index directory: apply DB path mapping if configured
-        index_dir = resolve_db_dir(root)
-
-        # Device: auto-detect CUDA or use env override
-        device = os.environ.get("COCOINDEX_CODE_DEVICE")
-
-        # Extra file extensions (format: "inc:php,yaml,toml" — optional lang after colon)
-        raw_extra = os.environ.get("COCOINDEX_CODE_EXTRA_EXTENSIONS", "")
-        extra_extensions: dict[str, str | None] = {}
-        for token in raw_extra.split(","):
-            token = token.strip()
-            if not token:
-                continue
-            if ":" in token:
-                ext, lang = token.split(":", 1)
-                extra_extensions[f".{ext.strip()}"] = lang.strip() or None
-            else:
-                extra_extensions[f".{token}"] = None
-
-        # Excluded file glob patterns
-        excluded_patterns = _parse_json_string_list_env("COCOINDEX_CODE_EXCLUDED_PATTERNS")
-
-        return cls(
-            codebase_root_path=root,
-            embedding_model=embedding_model,
-            index_dir=index_dir,
-            device=device,
-            extra_extensions=extra_extensions,
-            excluded_patterns=excluded_patterns,
-        )
-
-    @property
-    def target_sqlite_db_path(self) -> Path:
-        """Path to the vector index SQLite database."""
-        return self.index_dir / "target_sqlite.db"
-
-    @property
-    def cocoindex_db_path(self) -> Path:
-        """Path to the CocoIndex state database."""
-        return self.index_dir / "cocoindex.db"
-
-
-# Module-level singleton — imported directly by all modules that need configuration
-config: Config = Config.from_env()