kodit 0.3.4__tar.gz → 0.3.5__tar.gz

{kodit-0.3.4 → kodit-0.3.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: kodit
-Version: 0.3.4
+Version: 0.3.5
 Summary: Code indexing for better AI code generation
 Project-URL: Homepage, https://docs.helixml.tech/kodit/
 Project-URL: Documentation, https://docs.helixml.tech/kodit/
@@ -102,6 +102,7 @@ code. This index is used to build a snippet library, ready for ingestion into an
 - **NEW in 0.3**: Index private repositories via a PAT
 - **NEW in 0.3**: Improved progress monitoring and reporting during indexing
 - **NEW in 0.3**: Advanced code slicing infrastructure with Tree-sitter parsing
+- **NEW in 0.4**: Automatic periodic sync to keep indexes up-to-date
 
 ### MCP Server
 

{kodit-0.3.4 → kodit-0.3.5}/README.md

@@ -49,6 +49,7 @@ code. This index is used to build a snippet library, ready for ingestion into an
 - **NEW in 0.3**: Index private repositories via a PAT
 - **NEW in 0.3**: Improved progress monitoring and reporting during indexing
 - **NEW in 0.3**: Advanced code slicing infrastructure with Tree-sitter parsing
+- **NEW in 0.4**: Automatic periodic sync to keep indexes up-to-date
 
 ### MCP Server
 

{kodit-0.3.4 → kodit-0.3.5}/docs/reference/configuration/index.md

@@ -117,6 +117,34 @@ DB_URL=postgresql+asyncpg://postgres:mysecretpassword@localhost:5432/kodit
 DEFAULT_SEARCH_PROVIDER=vectorchord
 ```
 
+## Configuring Sync
+
+### Periodic Sync Configuration
+
+Kodit can automatically sync all indexed codebases at regular intervals to keep them up-to-date with the latest changes. This is especially useful for server deployments where multiple users are working with the same codebases.
+
+```bash
+# Enable/disable periodic sync (default: true)
+SYNC_PERIODIC_ENABLED=true
+
+# Sync interval in seconds (default: 1800 = 30 minutes)
+SYNC_PERIODIC_INTERVAL_SECONDS=1800
+
+# Number of retry attempts for failed syncs (default: 3)
+SYNC_PERIODIC_RETRY_ATTEMPTS=3
+```
+
+The sync scheduler will:
+- Run automatically in the background when the server starts
+- Sync all existing indexes at the configured interval
+- Handle failures gracefully with retry logic
+- Log detailed progress and results
+- Shut down cleanly when the server stops
+
+{{< info >}}
+**Note**: The sync scheduler only syncs existing indexes. It does not create new indexes for repositories that haven't been indexed yet.
+{{< /info >}}
+
 ## Configuring Enrichment
 
 ### Default Enrichment Provider

{kodit-0.3.4 → kodit-0.3.5}/docs/reference/deployment/docker-compose.yaml

@@ -34,6 +34,11 @@ services:
       AUTO_INDEXING_SOURCES_1_URI: https://github.com/fastapi/fastapi
       AUTO_INDEXING_SOURCES_2_URI: https://github.com/helix-ml/kodit
 
+      # Sync configuration
+      SYNC_PERIODIC_ENABLED: true
+      SYNC_PERIODIC_INTERVAL_SECONDS: 1800  # 30 minutes
+      SYNC_PERIODIC_RETRY_ATTEMPTS: 3
+
   vectorchord:
     image: tensorchord/vchord-suite:pg17-20250601
     environment:

{kodit-0.3.4 → kodit-0.3.5}/docs/reference/deployment/kubernetes.yaml

@@ -81,6 +81,12 @@ spec:
               value: "https://github.com/pydantic/pydantic"
             - name: AUTO_INDEXING_SOURCES_1_URI
               value: "https://github.com/helix-ml/kodit"
+            - name: SYNC_PERIODIC_ENABLED
+              value: "true"
+            - name: SYNC_PERIODIC_INTERVAL_SECONDS
+              value: "1800"  # 30 minutes
+            - name: SYNC_PERIODIC_RETRY_ATTEMPTS
+              value: "3"
           ports:
             - containerPort: 8080
           readinessProbe:

{kodit-0.3.4 → kodit-0.3.5}/docs/reference/indexing/index.md

@@ -332,6 +332,24 @@ Kodit shows progress during indexing operations:
 - Snippet extraction progress
 - Index building progress (BM25, embeddings)
 
+### Automatic Sync
+
+For server deployments, Kodit includes an automatic sync scheduler that keeps your indexes up-to-date:
+
+- **Periodic sync**: Automatically re-indexes all existing sources at configurable intervals
+- **Failure handling**: Gracefully handles sync failures with retry logic
+- **Background operation**: Runs in the background without blocking the MCP server
+- **Configurable timing**: Adjust sync frequency based on your needs
+
+See the [Sync Configuration](/kodit/reference/sync/index.md) documentation for detailed setup instructions.
+
+The sync scheduler is enabled by default and will:
+
+- Start automatically when you run `kodit serve`
+- Sync all existing indexes every 30 minutes by default
+- Log detailed progress and results
+- Retry failed operations up to 3 times by default
+
 ## Privacy and Security
 
 ### Local Processing

kodit-0.3.5/docs/reference/sync/index.md

@@ -0,0 +1,79 @@
+---
+title: Sync
+description: Automatic synchronization of indexed codebases
+weight: 3
+---
+
+Kodit includes an automatic sync scheduler that keeps your indexed codebases up-to-date with the latest changes. This is especially useful for server deployments where multiple users are working with the same codebases.
+
+## How Sync Works
+
+The sync scheduler runs as a background service that:
+
+1. **Periodically checks** all existing indexes at a configurable interval
+2. **Syncs each index** by pulling the latest changes from the source repository
+3. **Updates the search index** with any new or modified code snippets
+4. **Handles failures gracefully** with configurable retry attempts
+5. **Logs detailed progress** for monitoring and debugging
+
+## Configuration
+
+### Environment Variables
+
+Configure the sync scheduler using these environment variables:
+
+```bash
+# Enable/disable periodic sync (default: true)
+SYNC_PERIODIC_ENABLED=true
+
+# Sync interval in seconds (default: 1800 = 30 minutes)
+SYNC_PERIODIC_INTERVAL_SECONDS=1800
+
+# Number of retry attempts for failed syncs (default: 3)
+SYNC_PERIODIC_RETRY_ATTEMPTS=3
+```
+
+### Common Configuration Examples
+
+#### Quick Development Setup
+
+For rapid development with frequent changes:
+
+```bash
+SYNC_PERIODIC_ENABLED=true
+SYNC_PERIODIC_INTERVAL_SECONDS=300  # 5 minutes
+SYNC_PERIODIC_RETRY_ATTEMPTS=1
+```
+
+#### Production Server Setup
+
+For production deployments with stable codebases:
+
+```bash
+SYNC_PERIODIC_ENABLED=true
+SYNC_PERIODIC_INTERVAL_SECONDS=3600  # 1 hour
+SYNC_PERIODIC_RETRY_ATTEMPTS=3
+```
+
+#### Disable Sync
+
+If you prefer to sync manually:
+
+```bash
+SYNC_PERIODIC_ENABLED=false
+```
+
+## Limitations
+
+- **Only syncs existing indexes**: The sync scheduler does not create new indexes for repositories that haven't been indexed yet
+- **Sequential processing**: Indexes are synced one at a time to avoid overwhelming the system
+- **No conflict resolution**: If there are conflicts during sync, the operation may fail and require manual intervention
+
+## Manual Sync
+
+You can also manually sync indexes using the CLI:
+
+```bash
+# Sync all indexes
+kodit index --sync
+```

{kodit-0.3.4 → kodit-0.3.5}/src/kodit/_version.py

@@ -17,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.3.4'
-__version_tuple__ = version_tuple = (0, 3, 4)
+__version__ = version = '0.3.5'
+__version_tuple__ = version_tuple = (0, 3, 5)

{kodit-0.3.4 → kodit-0.3.5}/src/kodit/app.py

@@ -6,28 +6,47 @@ from contextlib import asynccontextmanager
 from asgi_correlation_id import CorrelationIdMiddleware
 from fastapi import FastAPI
 
+from kodit.application.services.sync_scheduler import SyncSchedulerService
 from kodit.config import AppContext
 from kodit.infrastructure.indexing.auto_indexing_service import AutoIndexingService
 from kodit.mcp import mcp
 from kodit.middleware import ASGICancelledErrorMiddleware, logging_middleware
 
-# Global auto-indexing service
+# Global services
 _auto_indexing_service: AutoIndexingService | None = None
+_sync_scheduler_service: SyncSchedulerService | None = None
 
 
 @asynccontextmanager
 async def app_lifespan(_: FastAPI) -> AsyncIterator[None]:
-    """Manage application lifespan for auto-indexing."""
-    global _auto_indexing_service  # noqa: PLW0603
-    # Start auto-indexing service
+    """Manage application lifespan for auto-indexing and sync."""
+    global _auto_indexing_service, _sync_scheduler_service  # noqa: PLW0603
+
     app_context = AppContext()
     db = await app_context.get_db()
+
+    # Start auto-indexing service
     _auto_indexing_service = AutoIndexingService(
         app_context=app_context,
         session_factory=db.session_factory,
     )
     await _auto_indexing_service.start_background_indexing()
+
+    # Start sync scheduler service
+    if app_context.sync.enabled:
+        _sync_scheduler_service = SyncSchedulerService(
+            app_context=app_context,
+            session_factory=db.session_factory,
+        )
+        _sync_scheduler_service.start_periodic_sync(
+            interval_seconds=app_context.sync.interval_seconds
+        )
+
     yield
+
+    # Stop services
+    if _sync_scheduler_service:
+        await _sync_scheduler_service.stop_periodic_sync()
     if _auto_indexing_service:
         await _auto_indexing_service.stop()
 

kodit-0.3.5/src/kodit/application/services/sync_scheduler.py

@@ -0,0 +1,128 @@
+"""Service for scheduling periodic sync operations."""
+
+import asyncio
+from collections.abc import Callable
+from contextlib import suppress
+
+import structlog
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from kodit.application.factories.code_indexing_factory import (
+    create_code_indexing_application_service,
+)
+from kodit.config import AppContext
+from kodit.domain.services.index_query_service import IndexQueryService
+from kodit.infrastructure.indexing.fusion_service import ReciprocalRankFusionService
+from kodit.infrastructure.sqlalchemy.index_repository import SqlAlchemyIndexRepository
+
+
+class SyncSchedulerService:
+    """Service for scheduling periodic sync operations."""
+
+    def __init__(
+        self,
+        app_context: AppContext,
+        session_factory: Callable[[], AsyncSession],
+    ) -> None:
+        """Initialize the sync scheduler service."""
+        self.app_context = app_context
+        self.session_factory = session_factory
+        self.log = structlog.get_logger(__name__)
+        self._sync_task: asyncio.Task | None = None
+        self._shutdown_event = asyncio.Event()
+
+    def start_periodic_sync(self, interval_seconds: float = 1800) -> None:
+        """Start periodic sync of all indexes."""
+        self.log.info("Starting periodic sync", interval_seconds=interval_seconds)
+
+        self._sync_task = asyncio.create_task(self._sync_loop(interval_seconds))
+
+    async def stop_periodic_sync(self) -> None:
+        """Stop the periodic sync task."""
+        self.log.info("Stopping periodic sync")
+        self._shutdown_event.set()
+
+        if self._sync_task and not self._sync_task.done():
+            self._sync_task.cancel()
+            with suppress(asyncio.CancelledError):
+                await self._sync_task
+
+    async def _sync_loop(self, interval_seconds: float) -> None:
+        """Run the sync loop at the specified interval."""
+        while not self._shutdown_event.is_set():
+            try:
+                await self._perform_sync()
+            except Exception as e:
+                self.log.exception("Sync operation failed", error=e)
+
+            # Wait for the interval or until shutdown
+            try:
+                await asyncio.wait_for(
+                    self._shutdown_event.wait(), timeout=interval_seconds
+                )
+                # If we reach here, shutdown was requested
+                break
+            except TimeoutError:
+                # Continue to next sync cycle
+                continue
+
+    async def _perform_sync(self) -> None:
+        """Perform a sync operation on all indexes."""
+        self.log.info("Starting sync operation")
+
+        async with self.session_factory() as session:
+            # Create services
+            service = create_code_indexing_application_service(
+                app_context=self.app_context,
+                session=session,
+            )
+            index_query_service = IndexQueryService(
+                index_repository=SqlAlchemyIndexRepository(session=session),
+                fusion_service=ReciprocalRankFusionService(),
+            )
+
+            # Get all existing indexes
+            all_indexes = await index_query_service.list_indexes()
+
+            if not all_indexes:
+                self.log.info("No indexes found to sync")
+                return
+
+            self.log.info("Syncing indexes", count=len(all_indexes))
+
+            success_count = 0
+            failure_count = 0
+
+            # Sync each index
+            for index in all_indexes:
+                try:
+                    self.log.info(
+                        "Syncing index",
+                        index_id=index.id,
+                        source=str(index.source.working_copy.remote_uri),
+                    )
+
+                    await service.run_index(index, progress_callback=None)
+                    success_count += 1
+
+                    self.log.info(
+                        "Index sync completed",
+                        index_id=index.id,
+                        source=str(index.source.working_copy.remote_uri),
+                    )
+
+                except Exception as e:
+                    failure_count += 1
+                    self.log.exception(
+                        "Index sync failed",
+                        index_id=index.id,
+                        source=str(index.source.working_copy.remote_uri),
+                        error=e,
+                    )
+
+            self.log.info(
+                "Sync operation completed",
+                total=len(all_indexes),
+                success=success_count,
+                failures=failure_count,
+            )

{kodit-0.3.4 → kodit-0.3.5}/src/kodit/cli.py

@@ -63,11 +63,105 @@ def cli(
     ctx.obj = config
 
 
+async def _handle_auto_index(
+    app_context: AppContext,
+    sources: list[str],  # noqa: ARG001
+) -> list[str]:
+    """Handle auto-index option and return sources to process."""
+    log = structlog.get_logger(__name__)
+    log.info("Auto-indexing configuration", config=app_context.auto_indexing)
+    if not app_context.auto_indexing or not app_context.auto_indexing.sources:
+        click.echo("No auto-index sources configured.")
+        return []
+    auto_sources = app_context.auto_indexing.sources
+    click.echo(f"Auto-indexing {len(auto_sources)} configured sources...")
+    return [source.uri for source in auto_sources]
+
+
+async def _handle_sync(
+    service: Any,
+    index_query_service: IndexQueryService,
+    sources: list[str],
+) -> None:
+    """Handle sync operation."""
+    log = structlog.get_logger(__name__)
+    log_event("kodit.cli.index.sync")
+
+    # Get all existing indexes
+    all_indexes = await index_query_service.list_indexes()
+
+    if not all_indexes:
+        click.echo("No existing indexes found to sync.")
+        return
+
+    # Filter indexes if specific sources are provided
+    indexes_to_sync = all_indexes
+    if sources:
+        # Filter indexes that match the provided sources
+        source_uris = set(sources)
+        indexes_to_sync = [
+            index for index in all_indexes
+            if str(index.source.working_copy.remote_uri) in source_uris
+        ]
+
+        if not indexes_to_sync:
+            click.echo(
+                f"No indexes found for the specified sources: {', '.join(sources)}"
+            )
+            return
+
+    click.echo(f"Syncing {len(indexes_to_sync)} indexes...")
+
+    # Sync each index
+    for index in indexes_to_sync:
+        click.echo(f"Syncing: {index.source.working_copy.remote_uri}")
+
+        # Create progress callback for this sync operation
+        progress_callback = create_multi_stage_progress_callback()
+
+        try:
+            await service.run_index(index, progress_callback)
+            click.echo(f"✓ Sync completed: {index.source.working_copy.remote_uri}")
+        except Exception as e:
+            log.exception("Sync failed", index_id=index.id, error=e)
+            click.echo(
+                f"✗ Sync failed: {index.source.working_copy.remote_uri} - {e}"
+            )
+
+
+async def _handle_list_indexes(index_query_service: IndexQueryService) -> None:
+    """Handle listing all indexes."""
+    log_event("kodit.cli.index.list")
+    # No source specified, list all indexes
+    indexes = await index_query_service.list_indexes()
+    headers: list[str | Cell] = [
+        "ID",
+        "Created At",
+        "Updated At",
+        "Source",
+        "Num Snippets",
+    ]
+    data = [
+        [
+            index.id,
+            index.created_at,
+            index.updated_at,
+            index.source.working_copy.remote_uri,
+            len(index.source.working_copy.files),
+        ]
+        for index in indexes
+    ]
+    click.echo(Table(headers=headers, data=data))
+
+
 @cli.command()
 @click.argument("sources", nargs=-1)
 @click.option(
     "--auto-index", is_flag=True, help="Index all configured auto-index sources"
 )
+@click.option(
+    "--sync", is_flag=True, help="Sync existing indexes with their remotes"
+)
 @with_app_context
 @with_session
 async def index(
@@ -76,8 +170,9 @@ async def index(
     sources: list[str],
     *,  # Force keyword-only arguments
     auto_index: bool,
+    sync: bool,
 ) -> None:
-    """List indexes, or index data sources."""
+    """List indexes, index data sources, or sync existing indexes."""
     log = structlog.get_logger(__name__)
     service = create_code_indexing_application_service(
         app_context=app_context,
@@ -89,36 +184,16 @@ async def index(
     )
 
     if auto_index:
-        log.info("Auto-indexing configuration", config=app_context.auto_indexing)
-        if not app_context.auto_indexing or not app_context.auto_indexing.sources:
-            click.echo("No auto-index sources configured.")
+        sources = await _handle_auto_index(app_context, sources)
+        if not sources:
             return
-        auto_sources = app_context.auto_indexing.sources
-        click.echo(f"Auto-indexing {len(auto_sources)} configured sources...")
-        sources = [source.uri for source in auto_sources]
+
+    if sync:
+        await _handle_sync(service, index_query_service, sources)
+        return
 
     if not sources:
-        log_event("kodit.cli.index.list")
-        # No source specified, list all indexes
-        indexes = await index_query_service.list_indexes()
-        headers: list[str | Cell] = [
-            "ID",
-            "Created At",
-            "Updated At",
-            "Source",
-            "Num Snippets",
-        ]
-        data = [
-            [
-                index.id,
-                index.created_at,
-                index.updated_at,
-                index.source.working_copy.remote_uri,
-                len(index.source.working_copy.files),
-            ]
-            for index in indexes
-        ]
-        click.echo(Table(headers=headers, data=data))
+        await _handle_list_indexes(index_query_service)
         return
     # Handle source indexing
     for source in sources:

{kodit-0.3.4 → kodit-0.3.5}/src/kodit/config.py

@@ -81,6 +81,18 @@ class AutoIndexingConfig(BaseModel):
         return v
 
 
+class PeriodicSyncConfig(BaseModel):
+    """Configuration for periodic/scheduled syncing."""
+
+    enabled: bool = Field(default=True, description="Enable periodic sync")
+    interval_seconds: float = Field(
+        default=1800, description="Interval between automatic syncs in seconds"
+    )
+    retry_attempts: int = Field(
+        default=3, description="Number of retry attempts for failed syncs"
+    )
+
+
 class CustomAutoIndexingEnvSource(EnvSettingsSource):
     """Custom environment source for parsing AutoIndexingConfig."""
 
@@ -173,6 +185,9 @@ class AppContext(BaseSettings):
     auto_indexing: AutoIndexingConfig | None = Field(
         default=AutoIndexingConfig(), description="Auto-indexing configuration"
     )
+    periodic_sync: PeriodicSyncConfig = Field(
+        default=PeriodicSyncConfig(), description="Periodic sync configuration"
+    )
     _db: Database | None = None
 
     def model_post_init(self, _: Any) -> None:

kodit-0.3.5/tests/kodit/application/services/__init__.py

@@ -0,0 +1 @@
+"""Test modules for application services."""