cocoindex-code 0.2.7__tar.gz → 0.2.9__tar.gz

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex-code
-Version: 0.2.7
+Version: 0.2.9
 Summary: MCP server for indexing and querying codebases using CocoIndex
 Project-URL: Homepage, https://github.com/cocoindex-io/cocoindex-code
 Project-URL: Repository, https://github.com/cocoindex-io/cocoindex-code
@@ -276,10 +276,43 @@ exclude_patterns:
 language_overrides:
   - ext: inc               # treat .inc files as PHP
     lang: php
+
+chunkers:
+  - ext: toml              # use a custom chunker for .toml files
+    module: example_toml_chunker:toml_chunker
 ```
 
 > `.cocoindex_code/` is automatically added to `.gitignore` during init.
 
+Use `chunkers` when you want to control how a file type is split into chunks before indexing.
+
+`module: example_toml_chunker:toml_chunker` means:
+- `example_toml_chunker` is a local Python module
+- `toml_chunker` is the function inside that module
+
+In practice, this usually means:
+- you create a Python file in your project, for example `example_toml_chunker.py`
+- you add a function in that file
+- you point `settings.yml` at it with `module.path:function_name`
+
+The function should use this signature:
+
+```python
+from pathlib import Path
+from cocoindex_code.chunking import Chunk
+
+def my_chunker(path: Path, content: str) -> tuple[str | None, list[Chunk]]:
+    ...
+```
+
+- `path` is the file being indexed
+- `content` is the full text of that file
+- return `language_override` as a string like `"toml"` if you want to override language detection
+- return `None` as `language_override` if you want to keep the detected language
+- return a `list[Chunk]` with the chunks you want stored in the index
+
+See [`src/cocoindex_code/chunking.py`](./src/cocoindex_code/chunking.py) for the public types and [`tests/example_toml_chunker.py`](./tests/example_toml_chunker.py) for a complete example.
+
 ## Embedding Models
 
 By default, a local SentenceTransformers model ([sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)) is used — no API key required. To use a different model, edit `~/.cocoindex_code/global_settings.yml`.
@@ -457,6 +490,26 @@ embedding:
 | xml | | `.xml` |
 | yaml | | `.yaml`, `.yml` |
 
+### Custom Database Location
+
+By default, index databases (`cocoindex.db` and `target_sqlite.db`) live alongside settings in `<project>/.cocoindex_code/`. When running in Docker, you may want the databases on the container's native filesystem for performance (LMDB doesn't work well on mounted volumes) while keeping the source code and settings on a mounted volume.
+
+Set `COCOINDEX_CODE_DB_PATH_MAPPING` to remap database locations by path prefix:
+
+```bash
+COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/db-files
+```
+
+With this mapping, a project at `/workspace/myrepo` stores its databases in `/db-files/myrepo/` instead of `/workspace/myrepo/.cocoindex_code/`. Settings files remain in the original location.
+
+Multiple mappings are comma-separated and resolved in order (first match wins):
+
+```bash
+COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/db-files,/workspace2=/db-files2
+```
+
+Both source and target must be absolute paths. If no mapping matches, the default location is used.
+
 ## Troubleshooting
 
 Run `ccc doctor` to diagnose common issues. It checks your settings, daemon health, embedding model, file matching, and index status — all in one command.
@@ -501,6 +554,19 @@ If you previously configured `cocoindex-code` via environment variables, the `co
 
 If you need help with remote setup, please email our maintainer linghua@cocoindex.io, happy to help!
 
+## Contributing
+
+We welcome contributions! Before you start, please install the [pre-commit](https://pre-commit.com/) hooks so that linting, formatting, type checking, and tests run automatically before each commit:
+
+```bash
+pip install pre-commit
+pre-commit install
+```
+
+This catches common issues — trailing whitespace, lint errors (Ruff), type errors (mypy), and test failures — before they reach CI.
+
+For more details, see our [contributing guide](https://cocoindex.io/docs/contributing/guide).
+
 ## License
 
 Apache-2.0

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/README.md

@@ -237,10 +237,43 @@ exclude_patterns:
 language_overrides:
   - ext: inc               # treat .inc files as PHP
     lang: php
+
+chunkers:
+  - ext: toml              # use a custom chunker for .toml files
+    module: example_toml_chunker:toml_chunker
 ```
 
 > `.cocoindex_code/` is automatically added to `.gitignore` during init.
 
+Use `chunkers` when you want to control how a file type is split into chunks before indexing.
+
+`module: example_toml_chunker:toml_chunker` means:
+- `example_toml_chunker` is a local Python module
+- `toml_chunker` is the function inside that module
+
+In practice, this usually means:
+- you create a Python file in your project, for example `example_toml_chunker.py`
+- you add a function in that file
+- you point `settings.yml` at it with `module.path:function_name`
+
+The function should use this signature:
+
+```python
+from pathlib import Path
+from cocoindex_code.chunking import Chunk
+
+def my_chunker(path: Path, content: str) -> tuple[str | None, list[Chunk]]:
+    ...
+```
+
+- `path` is the file being indexed
+- `content` is the full text of that file
+- return `language_override` as a string like `"toml"` if you want to override language detection
+- return `None` as `language_override` if you want to keep the detected language
+- return a `list[Chunk]` with the chunks you want stored in the index
+
+See [`src/cocoindex_code/chunking.py`](./src/cocoindex_code/chunking.py) for the public types and [`tests/example_toml_chunker.py`](./tests/example_toml_chunker.py) for a complete example.
+
 ## Embedding Models
 
 By default, a local SentenceTransformers model ([sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)) is used — no API key required. To use a different model, edit `~/.cocoindex_code/global_settings.yml`.
@@ -418,6 +451,26 @@ embedding:
 | xml | | `.xml` |
 | yaml | | `.yaml`, `.yml` |
 
+### Custom Database Location
+
+By default, index databases (`cocoindex.db` and `target_sqlite.db`) live alongside settings in `<project>/.cocoindex_code/`. When running in Docker, you may want the databases on the container's native filesystem for performance (LMDB doesn't work well on mounted volumes) while keeping the source code and settings on a mounted volume.
+
+Set `COCOINDEX_CODE_DB_PATH_MAPPING` to remap database locations by path prefix:
+
+```bash
+COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/db-files
+```
+
+With this mapping, a project at `/workspace/myrepo` stores its databases in `/db-files/myrepo/` instead of `/workspace/myrepo/.cocoindex_code/`. Settings files remain in the original location.
+
+Multiple mappings are comma-separated and resolved in order (first match wins):
+
+```bash
+COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/db-files,/workspace2=/db-files2
+```
+
+Both source and target must be absolute paths. If no mapping matches, the default location is used.
+
 ## Troubleshooting
 
 Run `ccc doctor` to diagnose common issues. It checks your settings, daemon health, embedding model, file matching, and index status — all in one command.
@@ -462,6 +515,19 @@ If you previously configured `cocoindex-code` via environment variables, the `co
 
 If you need help with remote setup, please email our maintainer linghua@cocoindex.io, happy to help!
 
+## Contributing
+
+We welcome contributions! Before you start, please install the [pre-commit](https://pre-commit.com/) hooks so that linting, formatting, type checking, and tests run automatically before each commit:
+
+```bash
+pip install pre-commit
+pre-commit install
+```
+
+This catches common issues — trailing whitespace, lint errors (Ruff), type errors (mypy), and test failures — before they reach CI.
+
+For more details, see our [contributing guide](https://cocoindex.io/docs/contributing/guide).
+
 ## License
 
 Apache-2.0

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/pyproject.toml

@@ -91,9 +91,11 @@ select = ["E", "F", "I", "N", "W", "UP"]
 python_version = "3.11"
 strict = true
 ignore_missing_imports = true
+explicit_package_bases = true
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 python_files = ["test_*.py"]
 python_functions = ["test_*"]
 addopts = "-v --tb=short"
+asyncio_mode = "auto"

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/src/cocoindex_code/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.2.7'
-__version_tuple__ = version_tuple = (0, 2, 7)
+__version__ = version = '0.2.9'
+__version_tuple__ = version_tuple = (0, 2, 9)
 
 __commit_id__ = commit_id = None

cocoindex_code-0.2.9/src/cocoindex_code/chunking.py

@@ -0,0 +1,29 @@
+"""Public API for writing custom chunkers.
+
+Example usage::
+
+    from pathlib import Path
+    from cocoindex_code.chunking import Chunk, ChunkerFn, TextPosition
+
+    def my_chunker(path: Path, content: str) -> tuple[str | None, list[Chunk]]:
+        pos = TextPosition(byte_offset=0, char_offset=0, line=1, column=0)
+        return "mylang", [Chunk(text=content, start=pos, end=pos)]
+"""
+
+from __future__ import annotations
+
+import pathlib as _pathlib
+from collections.abc import Callable as _Callable
+
+import cocoindex as _coco
+from cocoindex.resources.chunk import Chunk, TextPosition
+
+# Callable alias (not Protocol) — consistent with codebase style.
+# language_override=None keeps the language detected by detect_code_language.
+# path is not resolved (no syscall); call path.resolve() inside the chunker if needed.
+ChunkerFn = _Callable[[_pathlib.Path, str], tuple[str | None, list[Chunk]]]
+
+# tracked=False: callables are not fingerprint-able; daemon restart re-indexes anyway.
+CHUNKER_REGISTRY = _coco.ContextKey[dict[str, ChunkerFn]]("chunker_registry", tracked=False)
+
+__all__ = ["Chunk", "ChunkerFn", "CHUNKER_REGISTRY", "TextPosition"]

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/src/cocoindex_code/cli.py

@@ -12,12 +12,16 @@ import typer as _typer
 from .client import DaemonStartError
 from .protocol import DoctorCheckResult, IndexingProgress, ProjectStatusResponse, SearchResponse
 from .settings import (
+    cocoindex_db_path,
     default_project_settings,
     default_user_settings,
     find_parent_with_marker,
     find_project_root,
+    project_settings_path,
+    resolve_db_dir,
     save_project_settings,
     save_user_settings,
+    target_sqlite_db_path,
     user_settings_path,
 )
 
@@ -283,10 +287,8 @@ def init(
     force: bool = _typer.Option(False, "-f", "--force", help="Skip parent directory warning"),
 ) -> None:
     """Initialize a project for cocoindex-code."""
-    from .settings import project_settings_path as _project_settings_path
-
     cwd = Path.cwd().resolve()
-    settings_file = _project_settings_path(cwd)
+    settings_file = project_settings_path(cwd)
 
     # Always ensure user settings exist
     user_path = user_settings_path()
@@ -376,8 +378,15 @@ def status() -> None:
     """Show project status."""
     from . import client as _client
 
-    project_root = str(require_project_root())
+    project_root_path = require_project_root()
+    project_root = str(project_root_path)
     print_project_header(project_root)
+
+    _typer.echo(f"Settings: {project_settings_path(project_root_path)}")
+    db_path = target_sqlite_db_path(project_root_path)
+    if db_path.exists():
+        _typer.echo(f"Index DB: {db_path}")
+
     print_index_stats(_client.project_status(project_root))
 
 
@@ -389,12 +398,13 @@ def reset(
     """Reset project databases and optionally remove settings."""
     project_root = require_project_root()
     cocoindex_dir = project_root / ".cocoindex_code"
+    db_dir = resolve_db_dir(project_root)
 
     db_files = [
-        cocoindex_dir / "cocoindex.db",
-        cocoindex_dir / "target_sqlite.db",
+        cocoindex_db_path(project_root),
+        target_sqlite_db_path(project_root),
     ]
-    settings_file = cocoindex_dir / "settings.yml"
+    settings_file = project_settings_path(project_root)
 
     # Determine what will be deleted
     to_delete = [f for f in db_files if f.exists()]
@@ -436,6 +446,12 @@ def reset(
             f.unlink(missing_ok=True)
 
     if all_:
+        # Remove db_dir if empty and different from cocoindex_dir
+        if db_dir != cocoindex_dir:
+            try:
+                db_dir.rmdir()
+            except OSError:
+                pass  # Not empty or doesn't exist
         # Remove .cocoindex_code/ if empty
         try:
             cocoindex_dir.rmdir()
@@ -495,16 +511,10 @@ def doctor() -> None:
     from .settings import (
         load_user_settings as _load_user_settings,
     )
-    from .settings import (
-        project_settings_path as _project_settings_path,
-    )
-    from .settings import (
-        user_settings_path as _user_settings_path,
-    )
 
     # --- 1. Global settings (local, no daemon needed) ---
     _print_section("Global Settings")
-    settings_path = _user_settings_path()
+    settings_path = user_settings_path()
     _typer.echo(f"  Settings: {settings_path}")
     try:
         user_settings = _load_user_settings()
@@ -539,6 +549,10 @@ def doctor() -> None:
             other_keys = [k for k in env_resp.env_names if k not in settings_keys]
             if other_keys:
                 _typer.echo(f"  Other env vars in daemon: {', '.join(sorted(other_keys))}")
+            if env_resp.db_path_mappings:
+                _typer.echo("  DB path mappings:")
+                for m in env_resp.db_path_mappings:
+                    _typer.echo(f"    {m.source} \u2192 {m.target}")
         except Exception as e:
             _print_error(f"Failed to get daemon env: {e}")
 
@@ -558,7 +572,7 @@ def doctor() -> None:
     # --- 6. Project settings (local, no daemon needed) ---
     if project_root is not None:
         _print_section("Project Settings")
-        ps_path = _project_settings_path(project_root)
+        ps_path = project_settings_path(project_root)
         _typer.echo(f"  Settings: {ps_path}")
         try:
             ps = _load_project_settings(project_root)
@@ -585,10 +599,9 @@ def doctor() -> None:
 
     # --- 8. Log files ---
     _print_section("Log Files")
-    from .daemon import daemon_dir as _daemon_dir
+    from .daemon import daemon_log_path as _daemon_log_path
 
-    log_dir = _daemon_dir()
-    _typer.echo(f"  Daemon logs: {log_dir / 'daemon.log'}")
+    _typer.echo(f"  Daemon logs: {_daemon_log_path()}")
     _typer.echo("  Check logs above for further troubleshooting.")
 
 

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/src/cocoindex_code/client.py

@@ -343,10 +343,10 @@ def start_daemon() -> subprocess.Popen[bytes]:
     Returns the ``Popen`` object so callers can detect early process death
     (via ``proc.poll()``) instead of waiting for a full timeout.
     """
-    from .daemon import daemon_dir
+    from .daemon import daemon_dir, daemon_log_path
 
     daemon_dir().mkdir(parents=True, exist_ok=True)
-    log_path = daemon_dir() / "daemon.log"
+    log_path = daemon_log_path()
 
     ccc_path = _find_ccc_executable()
     if ccc_path:

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/src/cocoindex_code/daemon.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import asyncio
+import importlib
 import logging
 import os
 import signal
@@ -15,6 +16,7 @@ from pathlib import Path
 from typing import Any
 
 from ._version import __version__
+from .chunking import ChunkerFn as _ChunkerFn
 from .project import Project
 from .protocol import (
     DaemonEnvRequest,
@@ -46,14 +48,37 @@ from .protocol import (
     encode_response,
 )
 from .settings import (
+    ChunkerMapping,
     global_settings_mtime_us,
+    load_project_settings,
     load_user_settings,
+    target_sqlite_db_path,
     user_settings_dir,
 )
 from .shared import Embedder, create_embedder
 
 logger = logging.getLogger(__name__)
 
+
+def _resolve_chunker_registry(mappings: list[ChunkerMapping]) -> dict[str, _ChunkerFn]:
+    """Resolve ``ChunkerMapping`` settings entries to a ``{suffix: fn}`` dict.
+
+    Each ``mapping.module`` must be a ``"module.path:callable"`` string importable
+    from the current environment.
+    """
+    registry: dict[str, _ChunkerFn] = {}
+    for cm in mappings:
+        module_path, _, attr = cm.module.partition(":")
+        if not attr:
+            raise ValueError(f"chunker module {cm.module!r} must use 'module.path:callable' format")
+        mod = importlib.import_module(module_path)
+        fn = getattr(mod, attr)
+        if not callable(fn):
+            raise ValueError(f"chunker {cm.module!r}: {attr!r} is not callable")
+        registry[f".{cm.ext}"] = fn
+    return registry
+
+
 # ---------------------------------------------------------------------------
 # Daemon paths
 # ---------------------------------------------------------------------------
@@ -110,7 +135,9 @@ class ProjectRegistry:
         """Get or create a Project for the given root. Lazy initialization."""
         if project_root not in self._projects:
             root = Path(project_root)
-            project = await Project.create(root, self._embedder)
+            project_settings = load_project_settings(root)
+            chunker_registry = _resolve_chunker_registry(project_settings.chunkers)
+            project = await Project.create(root, self._embedder, chunker_registry=chunker_registry)
             self._projects[project_root] = project
         return self._projects[project_root]
 
@@ -345,7 +372,7 @@ async def _check_index_status(project_root_str: str) -> DoctorCheckResult:
     from cocoindex.connectors import sqlite as coco_sqlite
 
     project_root = Path(project_root_str)
-    db_path = project_root / ".cocoindex_code" / "target_sqlite.db"
+    db_path = target_sqlite_db_path(project_root)
     details = [f"Index: {db_path}"]
 
     if not db_path.exists():
@@ -441,9 +468,16 @@ async def _dispatch(
             return StopResponse(ok=True)
 
         if isinstance(req, DaemonEnvRequest):
+            from .protocol import DbPathMappingEntry
+            from .settings import get_db_path_mappings
+
             return DaemonEnvResponse(
                 env_names=sorted(os.environ.keys()),
                 settings_env_names=settings_env_names,
+                db_path_mappings=[
+                    DbPathMappingEntry(source=str(m.source), target=str(m.target))
+                    for m in get_db_path_mappings()
+                ],
             )
 
         if isinstance(req, DoctorRequest):

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/src/cocoindex_code/indexer.py

@@ -14,6 +14,7 @@ from cocoindex.resources.file import FilePathMatcher, PatternFilePathMatcher
 from cocoindex.resources.id import IdGenerator
 from pathspec import GitIgnoreSpec
 
+from .chunking import CHUNKER_REGISTRY
 from .settings import load_gitignore_spec, load_project_settings
 from .shared import (
     CODEBASE_DIR,
@@ -158,13 +159,20 @@ async def process_file(
         or "text"
     )
 
-    chunks = splitter.split(
-        content,
-        chunk_size=CHUNK_SIZE,
-        min_chunk_size=MIN_CHUNK_SIZE,
-        chunk_overlap=CHUNK_OVERLAP,
-        language=language,
-    )
+    chunker_registry = coco.use_context(CHUNKER_REGISTRY)
+    chunker = chunker_registry.get(suffix)
+    if chunker is not None:
+        language_override, chunks = chunker(Path(file.file_path.path), content)
+        if language_override is not None:
+            language = language_override
+    else:
+        chunks = splitter.split(
+            content,
+            chunk_size=CHUNK_SIZE,
+            min_chunk_size=MIN_CHUNK_SIZE,
+            chunk_overlap=CHUNK_OVERLAP,
+            language=language,
+        )
 
     id_gen = IdGenerator()
 

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/src/cocoindex_code/project.py

@@ -10,6 +10,7 @@ from pathlib import Path
 import cocoindex as coco
 from cocoindex.connectors import sqlite as coco_sqlite
 
+from .chunking import CHUNKER_REGISTRY, ChunkerFn
 from .indexer import indexer_main
 from .protocol import (
     IndexingProgress,
@@ -21,6 +22,15 @@ from .protocol import (
     SearchResult,
 )
 from .query import query_codebase
+from .settings import (
+    cocoindex_db_path as _cocoindex_db_path,
+)
+from .settings import (
+    resolve_db_dir,
+)
+from .settings import (
+    target_sqlite_db_path as _target_sqlite_db_path,
+)
 from .shared import (
     CODEBASE_DIR,
     EMBEDDER,
@@ -170,7 +180,7 @@ class Project:
         offset: int = 0,
     ) -> list[SearchResult]:
         """Search within this project."""
-        target_db = self._project_root / ".cocoindex_code" / "target_sqlite.db"
+        target_db = _target_sqlite_db_path(self._project_root)
         results = await query_codebase(
             query=query,
             target_sqlite_db_path=target_db,
@@ -247,25 +257,37 @@ class Project:
     async def create(
         project_root: Path,
         embedder: Embedder,
+        chunker_registry: dict[str, ChunkerFn] | None = None,
     ) -> Project:
         """Create a project with explicit embedder.
 
         Project-level settings and .gitignore are NOT cached here — the
         indexer loads them fresh from disk on every run so that user edits
         take effect without restarting the daemon.
+
+        Args:
+            project_root: Root directory of the codebase to index.
+            embedder: Embedding model instance.
+            chunker_registry: Optional mapping of file suffix (e.g. ``".toml"``)
+                to a ``ChunkerFn``. When a suffix matches, the registered
+                chunker is called instead of the built-in splitter.
         """
-        index_dir = project_root / ".cocoindex_code"
-        index_dir.mkdir(parents=True, exist_ok=True)
+        settings_dir = project_root / ".cocoindex_code"
+        settings_dir.mkdir(parents=True, exist_ok=True)
+
+        db_dir = resolve_db_dir(project_root)
+        db_dir.mkdir(parents=True, exist_ok=True)
 
-        cocoindex_db_path = index_dir / "cocoindex.db"
-        target_sqlite_db_path = index_dir / "target_sqlite.db"
+        cocoindex_db = _cocoindex_db_path(project_root)
+        target_sqlite_db = _target_sqlite_db_path(project_root)
 
-        settings = coco.Settings.from_env(cocoindex_db_path)
+        settings = coco.Settings.from_env(cocoindex_db)
 
         context = coco.ContextProvider()
         context.provide(CODEBASE_DIR, project_root)
-        context.provide(SQLITE_DB, coco_sqlite.connect(str(target_sqlite_db_path), load_vec=True))
+        context.provide(SQLITE_DB, coco_sqlite.connect(str(target_sqlite_db), load_vec=True))
         context.provide(EMBEDDER, embedder)
+        context.provide(CHUNKER_REGISTRY, dict(chunker_registry) if chunker_registry else {})
 
         env = coco.Environment(settings, context_provider=context)
         app = coco.App(

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/src/cocoindex_code/protocol.py

@@ -158,9 +158,15 @@ class DoctorResponse(_msgspec.Struct, tag="doctor"):
     final: bool = False
 
 
+class DbPathMappingEntry(_msgspec.Struct):
+    source: str
+    target: str
+
+
 class DaemonEnvResponse(_msgspec.Struct, tag="daemon_env"):
     env_names: list[str]
     settings_env_names: list[str]
+    db_path_mappings: list[DbPathMappingEntry] = []
 
 
 class ErrorResponse(_msgspec.Struct, tag="error"):

{cocoindex_code-0.2.7 → cocoindex_code-0.2.9}/src/cocoindex_code/settings.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import os
 from dataclasses import dataclass, field
 from pathlib import Path
 from typing import Any
@@ -44,6 +45,29 @@ DEFAULT_INCLUDED_PATTERNS: list[str] = [
     "**/*.rst",  # reStructuredText
     "**/*.php",  # PHP
     "**/*.lua",  # Lua
+    "**/*.rb",  # Ruby
+    "**/*.swift",  # Swift
+    "**/*.kt",  # Kotlin
+    "**/*.kts",  # Kotlin script
+    "**/*.scala",  # Scala
+    "**/*.r",  # R
+    "**/*.html",  # HTML
+    "**/*.htm",  # HTML
+    "**/*.css",  # CSS
+    "**/*.scss",  # SCSS
+    "**/*.json",  # JSON
+    "**/*.xml",  # XML
+    "**/*.yaml",  # YAML
+    "**/*.yml",  # YAML
+    "**/*.toml",  # TOML
+    "**/*.sol",  # Solidity
+    "**/*.pas",  # Pascal
+    "**/*.dpr",  # Pascal/Delphi
+    "**/*.dtd",  # DTD
+    "**/*.f",  # Fortran
+    "**/*.f90",  # Fortran
+    "**/*.f95",  # Fortran
+    "**/*.f03",  # Fortran
 ]
 
 DEFAULT_EXCLUDED_PATTERNS: list[str] = [
@@ -82,11 +106,18 @@ class LanguageOverride:
     lang: str  # e.g. "php"
 
 
+@dataclass
+class ChunkerMapping:
+    ext: str  # without dot, e.g. "toml"
+    module: str  # "module.path:callable", e.g. "cocoindex_code.toml_chunker:toml_chunker"
+
+
 @dataclass
 class ProjectSettings:
     include_patterns: list[str] = field(default_factory=lambda: list(DEFAULT_INCLUDED_PATTERNS))
     exclude_patterns: list[str] = field(default_factory=lambda: list(DEFAULT_EXCLUDED_PATTERNS))
     language_overrides: list[LanguageOverride] = field(default_factory=list)
+    chunkers: list[ChunkerMapping] = field(default_factory=list)
 
 
 # ---------------------------------------------------------------------------
@@ -115,14 +146,103 @@ _SETTINGS_DIR_NAME = ".cocoindex_code"
 _SETTINGS_FILE_NAME = "settings.yml"  # project-level
 _USER_SETTINGS_FILE_NAME = "global_settings.yml"  # user-level
 
+_ENV_DB_PATH_MAPPING = "COCOINDEX_CODE_DB_PATH_MAPPING"
+
+
+@dataclass
+class DbPathMapping:
+    source: Path
+    target: Path
+
+
+_db_path_mapping: list[DbPathMapping] | None = None
+
+
+def _parse_db_path_mapping() -> list[DbPathMapping]:
+    """Parse ``COCOINDEX_CODE_DB_PATH_MAPPING`` env var.
+
+    Format: ``/src1=/dst1,/src2=/dst2``
+    Both source and target must be absolute paths.
+    """
+    raw = os.environ.get(_ENV_DB_PATH_MAPPING, "")
+    if not raw.strip():
+        return []
+
+    mappings: list[DbPathMapping] = []
+    for entry in raw.split(","):
+        entry = entry.strip()
+        if not entry:
+            continue
+        parts = entry.split("=", 1)
+        if len(parts) != 2 or not parts[0] or not parts[1]:
+            raise ValueError(
+                f"{_ENV_DB_PATH_MAPPING}: invalid entry {entry!r}, expected format 'source=target'"
+            )
+        source = Path(parts[0])
+        target = Path(parts[1])
+        if not source.is_absolute():
+            raise ValueError(
+                f"{_ENV_DB_PATH_MAPPING}: source path must be absolute, got {source!r}"
+            )
+        if not target.is_absolute():
+            raise ValueError(
+                f"{_ENV_DB_PATH_MAPPING}: target path must be absolute, got {target!r}"
+            )
+        mappings.append(DbPathMapping(source=source.resolve(), target=target.resolve()))
+    return mappings
+
+
+def resolve_db_dir(project_root: Path) -> Path:
+    """Return the directory for database files given a project root.
+
+    Applies ``COCOINDEX_CODE_DB_PATH_MAPPING`` if set, otherwise falls back
+    to ``project_root / ".cocoindex_code"``.
+    """
+    global _db_path_mapping  # noqa: PLW0603
+    if _db_path_mapping is None:
+        _db_path_mapping = _parse_db_path_mapping()
+
+    resolved = project_root.resolve()
+    for mapping in _db_path_mapping:
+        if resolved == mapping.source or resolved.is_relative_to(mapping.source):
+            rel = resolved.relative_to(mapping.source)
+            return mapping.target / rel
+    return project_root / _SETTINGS_DIR_NAME
+
+
+def get_db_path_mappings() -> list[DbPathMapping]:
+    """Return the parsed DB path mappings from ``COCOINDEX_CODE_DB_PATH_MAPPING``."""
+    global _db_path_mapping  # noqa: PLW0603
+    if _db_path_mapping is None:
+        _db_path_mapping = _parse_db_path_mapping()
+    return list(_db_path_mapping)
+
+
+def _reset_db_path_mapping_cache() -> None:
+    """Reset the cached mapping (for tests)."""
+    global _db_path_mapping  # noqa: PLW0603
+    _db_path_mapping = None
+
+
+_TARGET_SQLITE_DB_NAME = "target_sqlite.db"
+_COCOINDEX_DB_NAME = "cocoindex.db"
+
+
+def target_sqlite_db_path(project_root: Path) -> Path:
+    """Return the path to the vector index SQLite database for a project."""
+    return resolve_db_dir(project_root) / _TARGET_SQLITE_DB_NAME
+
+
+def cocoindex_db_path(project_root: Path) -> Path:
+    """Return the path to the CocoIndex state database for a project."""
+    return resolve_db_dir(project_root) / _COCOINDEX_DB_NAME
+
 
 def user_settings_dir() -> Path:
     """Return ``~/.cocoindex_code/``.
 
     Respects ``COCOINDEX_CODE_DIR`` env var for overriding the base directory.
     """
-    import os
-
     override = os.environ.get("COCOINDEX_CODE_DIR")
     if override:
         return Path(override)
@@ -162,7 +282,7 @@ def find_legacy_project_root(start: Path) -> Path | None:
     """
     current = start.resolve()
     while True:
-        if (current / _SETTINGS_DIR_NAME / "cocoindex.db").exists():
+        if (current / _SETTINGS_DIR_NAME / _COCOINDEX_DB_NAME).exists():
             return current
         parent = current.parent
         if parent == current:
@@ -261,6 +381,8 @@ def _project_settings_to_dict(settings: ProjectSettings) -> dict[str, Any]:
         d["language_overrides"] = [
             {"ext": lo.ext, "lang": lo.lang} for lo in settings.language_overrides
         ]
+    if settings.chunkers:
+        d["chunkers"] = [{"ext": cm.ext, "module": cm.module} for cm in settings.chunkers]
     return d
 
 
@@ -268,10 +390,12 @@ def _project_settings_from_dict(d: dict[str, Any]) -> ProjectSettings:
     overrides = [
         LanguageOverride(ext=lo["ext"], lang=lo["lang"]) for lo in d.get("language_overrides", [])
     ]
+    chunkers = [ChunkerMapping(ext=cm["ext"], module=cm["module"]) for cm in d.get("chunkers", [])]
     return ProjectSettings(
         include_patterns=d.get("include_patterns", list(DEFAULT_INCLUDED_PATTERNS)),
         exclude_patterns=d.get("exclude_patterns", list(DEFAULT_EXCLUDED_PATTERNS)),
         language_overrides=overrides,
+        chunkers=chunkers,
     )
 
 

cocoindex_code-0.2.7/src/cocoindex_code/config.py

@@ -1,142 +0,0 @@
-"""Configuration management for cocoindex-code."""
-
-from __future__ import annotations
-
-import json
-import os
-from dataclasses import dataclass
-from pathlib import Path
-
-_DEFAULT_MODEL = "sbert/sentence-transformers/all-MiniLM-L6-v2"
-
-
-def _find_root_with_marker(start: Path, markers: list[str]) -> Path | None:
-    """Walk up from start, return first directory containing any marker."""
-    current = start
-    while True:
-        if any((current / m).exists() for m in markers):
-            return current
-        parent = current.parent
-        if parent == current:
-            return None
-        current = parent
-
-
-def _discover_codebase_root() -> Path:
-    """Discover the codebase root directory.
-
-    Discovery order:
-    1. Find nearest parent with `.cocoindex_code` directory (re-anchor to previously-indexed tree)
-    2. Find nearest parent with any common project root marker
-    3. Fall back to current working directory
-    """
-    cwd = Path.cwd()
-
-    # First, look for existing .cocoindex_code directory
-    root = _find_root_with_marker(cwd, [".cocoindex_code"])
-    if root is not None:
-        return root
-
-    # Then, look for common project root markers
-    markers = [".git", "pyproject.toml", "package.json", "Cargo.toml", "go.mod"]
-    root = _find_root_with_marker(cwd, markers)
-    return root if root is not None else cwd
-
-
-def _parse_json_string_list_env(var_name: str) -> list[str]:
-    """Parse an environment variable as a JSON array of strings."""
-    raw_value = os.environ.get(var_name, "")
-    if not raw_value.strip():
-        return []
-
-    try:
-        parsed = json.loads(raw_value)
-    except json.JSONDecodeError as exc:
-        raise ValueError(f"{var_name} must be a JSON array of strings, got invalid JSON") from exc
-
-    if not isinstance(parsed, list):
-        raise ValueError(f"{var_name} must be a JSON array of strings")
-
-    result: list[str] = []
-    for item in parsed:
-        if not isinstance(item, str):
-            raise ValueError(f"{var_name} must be a JSON array of strings")
-        item = item.strip()
-        if item:
-            result.append(item)
-
-    return result
-
-
-@dataclass
-class Config:
-    """Configuration loaded from environment variables."""
-
-    codebase_root_path: Path
-    embedding_model: str
-    index_dir: Path
-    device: str | None
-    extra_extensions: dict[str, str | None]
-    excluded_patterns: list[str]
-
-    @classmethod
-    def from_env(cls) -> Config:
-        """Load configuration from environment variables."""
-        # Get root path from env or discover it
-        root_path_str = os.environ.get("COCOINDEX_CODE_ROOT_PATH")
-        if root_path_str:
-            root = Path(root_path_str).resolve()
-        else:
-            root = _discover_codebase_root()
-
-        # Get embedding model
-        # Prefix "sbert/" for SentenceTransformers models, otherwise LiteLLM.
-        embedding_model = os.environ.get(
-            "COCOINDEX_CODE_EMBEDDING_MODEL",
-            _DEFAULT_MODEL,
-        )
-
-        # Index directory is always under the root
-        index_dir = root / ".cocoindex_code"
-
-        # Device: auto-detect CUDA or use env override
-        device = os.environ.get("COCOINDEX_CODE_DEVICE")
-
-        # Extra file extensions (format: "inc:php,yaml,toml" — optional lang after colon)
-        raw_extra = os.environ.get("COCOINDEX_CODE_EXTRA_EXTENSIONS", "")
-        extra_extensions: dict[str, str | None] = {}
-        for token in raw_extra.split(","):
-            token = token.strip()
-            if not token:
-                continue
-            if ":" in token:
-                ext, lang = token.split(":", 1)
-                extra_extensions[f".{ext.strip()}"] = lang.strip() or None
-            else:
-                extra_extensions[f".{token}"] = None
-
-        # Excluded file glob patterns
-        excluded_patterns = _parse_json_string_list_env("COCOINDEX_CODE_EXCLUDED_PATTERNS")
-
-        return cls(
-            codebase_root_path=root,
-            embedding_model=embedding_model,
-            index_dir=index_dir,
-            device=device,
-            extra_extensions=extra_extensions,
-            excluded_patterns=excluded_patterns,
-        )
-
-    @property
-    def target_sqlite_db_path(self) -> Path:
-        """Path to the vector index SQLite database."""
-        return self.index_dir / "target_sqlite.db"
-
-    @property
-    def cocoindex_db_path(self) -> Path:
-        """Path to the CocoIndex state database."""
-        return self.index_dir / "cocoindex.db"
-
-
-# Module-level singleton — imported directly by all modules that need configuration
-config: Config = Config.from_env()