ducta 0.1.0__py3-none-any.whl

ducta/__init__.py

@@ -0,0 +1,39 @@
+"""
+Copyright (C) 2024-2026 Faustino Lopez Ramos
+
+This file is part of ducta.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License. You may obtain a copy
+of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+License for the specific language governing permissions and limitations
+under the License.
+
+SPDX-License-Identifier: Apache-2.0
+"""
+
+from __future__ import annotations
+
+from importlib.metadata import PackageNotFoundError, version
+
+from loguru import logger
+
+try:
+    __version__ = version("ducta")
+except PackageNotFoundError:  # pragma: no cover - not installed (e.g. running from source)
+    __version__ = "0.0.0+unknown"
+
+# Silence the library by default (standard loguru pattern for libraries).
+# Ducta emits no log records unless the consuming application explicitly turns
+# them on with ``logger.enable("ducta")``. The CLI and API server do this when
+# they configure their own handlers. This guarantees Ducta's internal module
+# paths (its core) are never exposed to end users through stray import-time logs.
+logger.disable("ducta")
+
+__all__ = ["__version__"]

ducta/api/README.md

@@ -0,0 +1,146 @@
+# Ducta API
+
+> **The REST + WebSocket server of Ducta.** Built on **FastAPI**, it exposes pipeline configuration, execution, git history, data quality, MLOps, and workspace management over HTTP — and serves the bundled web UI. Sources (local paths or Git URLs) are provided per request, so one server can drive many projects.
+
+## At a glance
+
+| | |
+|---|---|
+| **Purpose** | Expose Ducta over HTTP/WS for tools, automation, and the web UI |
+| **Layer** | Interface (service) |
+| **Depends on** | `core`, `setting`, `console`; optional SQLAlchemy persistence |
+| **Used by** | `ui` and any external HTTP client |
+| **Key entry points** | `create_app()` / `ducta.api.main:app`, routers under `/api`, `ducta server start` |
+| **Install** | `pip install "ducta[api]"` |
+
+## Where it fits
+
+```mermaid
+flowchart TD
+    console["console · CLI"] --> core
+    api["api · REST / WebSocket"] --> core
+    ui["ui · Web UI"] --> api
+    setting["setting · Config foundation"] --> core["core · Orchestration"]
+    core --> gate["gate · Data I/O"]
+    core --> stream["stream · Streaming"]
+    core --> check["check · Data quality"]
+    core --> mlrun["mlrun · MLOps"]
+    style api fill:#4f46e5,stroke:#312e81,color:#fff
+```
+
+---
+
+## 1. Overview
+
+### For Non-Technical Users
+Ducta API is the **backend brain** behind the web app and any automation. It lets tools and teammates — not just one person at a terminal — start pipelines, watch them run live, browse configuration, review data-quality results, and manage models, all through a standard web interface with optional login.
+
+### For Technical Users
+Ducta API is the service layer, implementing:
+*   **App factory & lifespan**: `create_app()` wires middleware, routes, exception handlers, optional DB persistence + migrations, the execution manager, and a config cache; it warns when bound to a non-local address without auth (`main.py`).
+*   **Routers under `/api`**: auth, execution (+ WebSocket log streaming), configs, environments, git, repository, projects, nodes, mlops, quality, ingestion, templates, workspace, certificates, connect, plus a `/ws` terminal and unauthenticated health probes (`routes/`).
+*   **Optional JWT auth**: `AuthService` (bcrypt password hashing, jose-signed HS256 tokens with type/expiry checks); production settings refuse the default secret and require `auth_enabled` (`auth/`, `config.py`).
+*   **Execution engine**: an in-memory `ExecutionManager` with a bounded queue, concurrent-run limits, log buffering, and optional DB-backed persistence layered transparently (`execution/`).
+*   **Source resolution with SSRF defense**: per-request local paths are security-validated; Git clones are host-allow-listed and reject internal/non-routable targets (loopback, private, link-local metadata endpoint) (`source/resolver.py`).
+*   **Persistence & repositories**: optional SQLAlchemy-async engine, Alembic-style migrations, and repository/store abstractions for projects, nodes, and execution history (`db/`, `repositories/`, `repository/`).
+*   **Middleware**: request-ID tagging, CORS, and configurable per-IP rate limiting (in-memory or Redis) (`middleware/`).
+
+---
+
+## 2. Configuration
+
+Settings come from environment variables (Pydantic `Settings`, `config.py`). Key variables:
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `HOST` / `PORT` | `127.0.0.1` / `8000` | Bind address (non-local without auth triggers a warning) |
+| `ENVIRONMENT` | `development` | `production` enforces auth + a non-default JWT secret |
+| `AUTH_ENABLED` | `false` | Require JWT authentication |
+| `JWT_SECRET_KEY` | `change-me-in-production` | HS256 signing key (must be overridden in prod) |
+| `JWT_EXPIRATION_HOURS` | `24` | Access-token lifetime |
+| `DATABASE_URL` | `""` (in-memory) | `sqlite+aiosqlite:///...` or `postgresql+asyncpg://...` |
+| `RATE_LIMIT_ENABLED` | `false` | Per-IP rate limiting (`RATE_LIMIT_REQUESTS`, `..._WINDOW_SECONDS`, `RATE_LIMIT_REDIS_URL`) |
+| `MAX_CONCURRENT_EXECUTIONS` | `5` | Parallel pipeline runs |
+| `CORS_ORIGINS` | `["*"]` | Allowed origins |
+| `GIT_CLONE_ALLOWED_HOSTS` | `[]` | Allow-list for Git clone hosts (empty = public hosts only, internal blocked) |
+
+**Production checklist**: set `ENVIRONMENT=production`, `AUTH_ENABLED=true`, a strong `JWT_SECRET_KEY`, and either bind to `127.0.0.1` or place the server behind an authenticating proxy.
+
+---
+
+## 3. Running the Server
+
+```bash
+# Via the CLI (launches this API + serves the UI)
+ducta server start --host 127.0.0.1 --port 8000
+ducta ui --port 8000            # same server, opens a browser
+
+# Directly with uvicorn
+uvicorn ducta.api.main:app --host 127.0.0.1 --port 8000
+
+# Production-style env
+export ENVIRONMENT=production
+export AUTH_ENABLED=true
+export JWT_SECRET_KEY="$(openssl rand -hex 32)"
+export DATABASE_URL="postgresql+asyncpg://user:pass@localhost:5432/ducta"
+uvicorn ducta.api.main:app --host 0.0.0.0 --port 8000 --workers 4
+```
+
+Interactive docs are served at `/docs` (Swagger) and `/redoc`; the OpenAPI schema at `/openapi.json`.
+
+---
+
+## 4. HTTP Quickstart
+
+All application endpoints are under the `/api` prefix. Sources are passed per request.
+
+### Step 1: Health check (no auth)
+```bash
+curl http://127.0.0.1:8000/health
+```
+
+### Step 2: Authenticate (when AUTH_ENABLED=true)
+```bash
+TOKEN=$(curl -s -X POST http://127.0.0.1:8000/api/auth/login \
+  -H "Content-Type: application/json" \
+  -d '{"username":"admin","password":"admin"}' | jq -r .access_token)
+```
+
+### Step 3: Connect to a source and inspect configs
+```bash
+# Register/resolve a source (local path or Git URL) -> returns a project_id
+curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
+  http://127.0.0.1:8000/api/connect -d '{"source":"/path/to/project"}'
+
+# List configs for an environment
+curl -H "Authorization: Bearer $TOKEN" \
+  "http://127.0.0.1:8000/api/configs/dev?source=/path/to/project"
+```
+
+### Step 4: Execute a pipeline
+```bash
+curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
+  http://127.0.0.1:8000/api/projects/{project_id}/pipelines/sales_daily/execute \
+  -d '{"env":"dev"}'
+```
+
+### Step 5: Manage runs and stream logs
+```bash
+# List / cancel / retry executions
+curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/api/executions
+curl -X POST -H "Authorization: Bearer $TOKEN" \
+  http://127.0.0.1:8000/api/executions/{execution_id}/cancel
+```
+```
+# Live log stream over WebSocket
+ws://127.0.0.1:8000/api/ws/logs/{execution_id}
+```
+
+### Programmatic (ASGI / tests)
+```python
+from fastapi.testclient import TestClient
+from ducta.api.main import app
+
+client = TestClient(app)
+assert client.get("/health").status_code == 200
+```

ducta/api/__init__.py

@@ -0,0 +1,267 @@
+"""
+Copyright (C) 2024-2026 Faustino Lopez Ramos
+
+This file is part of ducta.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License. You may obtain a copy
+of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+License for the specific language governing permissions and limitations
+under the License.
+
+SPDX-License-Identifier: Apache-2.0
+"""
+
+from __future__ import annotations
+
+from ducta import __version__
+
+# ---------------------------------------------------------------------------
+# Auth
+# ---------------------------------------------------------------------------
+from ducta.api.auth import AuthService, UserStore, get_user_store
+
+# ---------------------------------------------------------------------------
+# Configuration
+# ---------------------------------------------------------------------------
+from ducta.api.config import Settings, get_settings
+
+# ---------------------------------------------------------------------------
+# Core background services
+# ---------------------------------------------------------------------------
+from ducta.api.core import ConfigFileWatcher, ConfigLockManager, GitSyncManager
+
+# ---------------------------------------------------------------------------
+# Database
+# ---------------------------------------------------------------------------
+from ducta.api.db import get_db_session, get_engine, is_db_enabled
+
+# ---------------------------------------------------------------------------
+# Dependencies
+# ---------------------------------------------------------------------------
+from ducta.api.dependencies import (
+    WebSocketAuthError,
+    get_config_service,
+    get_current_user,
+    get_execution_manager,
+    get_git_sync_manager,
+    get_node_service,
+    get_source_path,
+    get_workspace_manager,
+    require_permission,
+    resolve_websocket_user,
+)
+
+# ---------------------------------------------------------------------------
+# Exceptions
+# ---------------------------------------------------------------------------
+from ducta.api.exceptions import (
+    AuthenticationError,
+    ConcurrencyError,
+    ConfigFileNotFoundError,
+    ConfigValidationError,
+    ConflictError,
+    DuctaAPIError,
+    ExecutionError,
+    ExecutionNotFoundError,
+    ExpiredTokenError,
+    InvalidTokenError,
+    NodeNotFoundError,
+    NotFoundError,
+    PipelineNotFoundError,
+    ProjectAlreadyExistsError,
+    ProjectNotFoundError,
+    RepositoryAdapterError,
+    SyntaxValidationError,
+    ValidationError,
+    WorkspaceNotFoundError,
+)
+
+# ---------------------------------------------------------------------------
+# Execution
+# ---------------------------------------------------------------------------
+from ducta.api.execution import ExecutionManager
+
+# ---------------------------------------------------------------------------
+# App factory
+# ---------------------------------------------------------------------------
+from ducta.api.main import create_app, warn_if_insecure_exposure
+
+# ---------------------------------------------------------------------------
+# Middleware
+# ---------------------------------------------------------------------------
+from ducta.api.middleware import RateLimitMiddleware, register_middleware
+
+# ---------------------------------------------------------------------------
+# Models
+# ---------------------------------------------------------------------------
+from ducta.api.models import (
+    BlameLine,
+    CommitInfo,
+    ConfigFileResponse,
+    ConfigValidationResponse,
+    ConnectRequest,
+    DiffResponse,
+    ExecutionResponse,
+    ExecutionStatus,
+    GitCommitInfo,
+    GitCommitRequest,
+    GitCommitResponse,
+    GitExternalChangesResponse,
+    GitHistoryResponse,
+    GitStageRequest,
+    GitStageResponse,
+    GitStatusResponse,
+    LoginRequest,
+    PushPullRequest,
+    PushPullResponse,
+    RepositoryConnectRequest,
+    RepositoryInfo,
+    StructureInfo,
+    TokenResponse,
+    User,
+)
+
+# ---------------------------------------------------------------------------
+# Repositories
+# ---------------------------------------------------------------------------
+from ducta.api.repositories import (
+    ConfigRepository,
+    NodeRepository,
+    PipelineRepository,
+    ProjectRepository,
+)
+from ducta.api.repository import (
+    AWSAdapter,
+    AzureAdapter,
+    GitHubAdapter,
+    LocalAdapter,
+    RepositoryAdapter,
+)
+
+# ---------------------------------------------------------------------------
+# Routes
+# ---------------------------------------------------------------------------
+from ducta.api.routes import register_routes
+
+# ---------------------------------------------------------------------------
+# Services
+# ---------------------------------------------------------------------------
+from ducta.api.services import (
+    CachedConfigService,
+    ConfigService,
+    NodeSchemaService,
+    NodeService,
+    ProjectService,
+    get_cached_config_service,
+)
+
+# ---------------------------------------------------------------------------
+# Source resolution
+# ---------------------------------------------------------------------------
+from ducta.api.source import ResolvedSource, SourceInfo, SourceResolver
+
+# ---------------------------------------------------------------------------
+# Workspace
+# ---------------------------------------------------------------------------
+from ducta.api.workspace import WorkspaceManager
+
+__all__ = [
+    "__version__",
+    "Settings",
+    "get_settings",
+    "AuthenticationError",
+    "ConfigFileNotFoundError",
+    "ConfigValidationError",
+    "ConflictError",
+    "ConcurrencyError",
+    "ExecutionError",
+    "ExecutionNotFoundError",
+    "ExpiredTokenError",
+    "InvalidTokenError",
+    "NodeNotFoundError",
+    "NotFoundError",
+    "PipelineNotFoundError",
+    "ProjectAlreadyExistsError",
+    "ProjectNotFoundError",
+    "RepositoryAdapterError",
+    "SyntaxValidationError",
+    "DuctaAPIError",
+    "ValidationError",
+    "WorkspaceNotFoundError",
+    "WebSocketAuthError",
+    "get_config_service",
+    "get_current_user",
+    "get_execution_manager",
+    "get_git_sync_manager",
+    "get_node_service",
+    "get_source_path",
+    "get_workspace_manager",
+    "require_permission",
+    "resolve_websocket_user",
+    "create_app",
+    "warn_if_insecure_exposure",
+    "AuthService",
+    "UserStore",
+    "get_user_store",
+    "ConfigFileWatcher",
+    "ConfigLockManager",
+    "GitSyncManager",
+    "get_db_session",
+    "get_engine",
+    "is_db_enabled",
+    "ExecutionManager",
+    "get_execution_manager",
+    "RateLimitMiddleware",
+    "register_middleware",
+    "BlameLine",
+    "CommitInfo",
+    "ConfigFileResponse",
+    "ConfigValidationResponse",
+    "ConnectRequest",
+    "DiffResponse",
+    "ExecutionResponse",
+    "ExecutionStatus",
+    "GitCommitInfo",
+    "GitCommitRequest",
+    "GitCommitResponse",
+    "GitExternalChangesResponse",
+    "GitHistoryResponse",
+    "GitStageRequest",
+    "GitStageResponse",
+    "GitStatusResponse",
+    "LoginRequest",
+    "PushPullRequest",
+    "PushPullResponse",
+    "RepositoryConnectRequest",
+    "RepositoryInfo",
+    "StructureInfo",
+    "TokenResponse",
+    "User",
+    "ConfigRepository",
+    "NodeRepository",
+    "PipelineRepository",
+    "ProjectRepository",
+    "AWSAdapter",
+    "AzureAdapter",
+    "GitHubAdapter",
+    "LocalAdapter",
+    "RepositoryAdapter",
+    "register_routes",
+    "CachedConfigService",
+    "ConfigService",
+    "NodeSchemaService",
+    "NodeService",
+    "ProjectService",
+    "get_cached_config_service",
+    "ResolvedSource",
+    "SourceInfo",
+    "SourceResolver",
+    "WorkspaceManager",
+]

ducta/api/auth/__init__.py

@@ -0,0 +1,26 @@
+"""
+Copyright (C) 2024-2026 Faustino Lopez Ramos
+
+This file is part of ducta.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License. You may obtain a copy
+of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+License for the specific language governing permissions and limitations
+under the License.
+
+SPDX-License-Identifier: Apache-2.0
+
+Auth package — JWT service and user management.
+"""
+
+from ducta.api.auth.service import AuthService
+from ducta.api.auth.users import UserStore, get_user_store
+
+__all__ = ["AuthService", "UserStore", "get_user_store"]

ducta/api/auth/service.py

@@ -0,0 +1,154 @@
+"""
+Copyright (C) 2024-2026 Faustino Lopez Ramos
+
+This file is part of ducta.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License. You may obtain a copy
+of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+License for the specific language governing permissions and limitations
+under the License.
+
+SPDX-License-Identifier: Apache-2.0
+"""
+
+from __future__ import annotations
+
+import threading
+from datetime import datetime, timedelta, timezone
+from typing import Optional
+from uuid import uuid4
+
+import bcrypt as _bcrypt  # type: ignore
+from jose import ExpiredSignatureError, JWTError, jwt  # type: ignore
+
+from ducta.api.exceptions import ExpiredTokenError, InvalidTokenError
+
+
+class AuthService:
+    """Handles JWT lifecycle and password operations (basic login only)."""
+
+    _TOKEN_TYPE_ACCESS = "access"
+
+    def __init__(
+        self,
+        secret_key: str,
+        algorithm: str = "HS256",
+        access_token_expires_hours: int = 24,
+    ) -> None:
+        self._secret = secret_key
+        self._algorithm = algorithm
+        self._access_ttl = timedelta(hours=access_token_expires_hours)
+        # jti -> expiry. In-memory/per-process (same constraint as the
+        # in-memory rate limiter): a logout on one worker won't revoke the
+        # token on another. Bounded by each entry's own token expiry, not by
+        # size — a revoked jti only needs remembering until it would have
+        # expired anyway; _prune_expired_revocations sweeps stale entries.
+        self._revoked: dict[str, datetime] = {}
+        self._revoked_lock = threading.Lock()
+
+    @staticmethod
+    def _bcrypt_bytes(plain: str) -> bytes:
+        """Encode a password for bcrypt.
+
+        bcrypt only considers the first 72 bytes and raises on longer inputs in
+        recent versions; truncate explicitly so hashing and verification stay
+        consistent and never error on long passphrases.
+        """
+        return plain.encode("utf-8")[:72]
+
+    def hash_password(self, plain: str) -> str:
+        """Return a bcrypt hash of *plain*."""
+        return _bcrypt.hashpw(self._bcrypt_bytes(plain), _bcrypt.gensalt()).decode("utf-8")
+
+    def verify_password(self, plain: str, hashed: str) -> bool:
+        """Return True if *plain* matches the bcrypt *hashed* password."""
+        return _bcrypt.checkpw(self._bcrypt_bytes(plain), hashed.encode("utf-8"))
+
+    def create_access_token(self, user_id: str, username: str, roles: list[str]) -> str:
+        """Return a signed JWT access token."""
+        now = datetime.now(tz=timezone.utc)
+        payload = {
+            "sub": user_id,
+            "username": username,
+            "roles": roles,
+            "type": self._TOKEN_TYPE_ACCESS,
+            "jti": str(uuid4()),
+            "iat": now,
+            "exp": now + self._access_ttl,
+        }
+        return jwt.encode(payload, self._secret, algorithm=self._algorithm)
+
+    @property
+    def access_token_lifetime_seconds(self) -> int:
+        return int(self._access_ttl.total_seconds())
+
+    def decode_access_token(self, token: str) -> dict:
+        """Decode and validate an access token."""
+        try:
+            payload = jwt.decode(token, self._secret, algorithms=[self._algorithm])
+        except ExpiredSignatureError:
+            raise ExpiredTokenError("Token has expired")
+        except JWTError:
+            raise InvalidTokenError("Token is invalid or has been tampered with")
+
+        if payload.get("type") != self._TOKEN_TYPE_ACCESS:
+            raise InvalidTokenError(
+                f"Expected token type '{self._TOKEN_TYPE_ACCESS}', got '{payload.get('type')}'"
+            )
+        if self.is_revoked(payload.get("jti", "")):
+            raise InvalidTokenError("Token has been revoked (logged out)")
+        return payload
+
+    def revoke_token(self, jti: str, expires_at: datetime) -> None:
+        """Revoke a token by its ``jti`` — checked by every subsequent `decode_access_token`.
+
+        No previously-issued JWT can otherwise be invalidated before its own
+        expiry (that's the whole point of a stateless token) — logout only
+        cleared the client's cookie, so a captured/leaked token kept working
+        for its full lifetime regardless of logout.
+        """
+        if not jti:
+            return
+        with self._revoked_lock:
+            self._prune_expired_revocations_locked()
+            self._revoked[jti] = expires_at
+
+    def is_revoked(self, jti: str) -> bool:
+        if not jti:
+            return False
+        with self._revoked_lock:
+            return jti in self._revoked
+
+    def _prune_expired_revocations_locked(self) -> None:
+        now = datetime.now(tz=timezone.utc)
+        expired = [jti for jti, exp in self._revoked.items() if exp <= now]
+        for jti in expired:
+            del self._revoked[jti]
+
+
+_auth_service: Optional[AuthService] = None
+_auth_service_lock = threading.Lock()
+
+
+def get_auth_service() -> AuthService:
+    """Return the module-level AuthService singleton (thread-safe)."""
+    global _auth_service
+    if _auth_service is None:
+        with _auth_service_lock:
+            if _auth_service is None:
+                from ducta.api.config import get_settings
+
+                s = get_settings()
+                _auth_service = AuthService(
+                    secret_key=s.jwt_secret_key,
+                    algorithm=s.jwt_algorithm,
+                    access_token_expires_hours=s.jwt_expiration_hours,
+                )
+    return _auth_service