juniper-data 0.4.2__tar.gz → 0.6.0__tar.gz

{juniper_data-0.4.2 → juniper_data-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: juniper-data
-Version: 0.4.2
+Version: 0.6.0
 Summary: Dataset generation and management service for the Juniper ecosystem
 Author: Paul Calnon
 License: MIT
@@ -25,14 +25,17 @@ Requires-Dist: pytest-benchmark>=4.0.0; extra == "test"
 Requires-Dist: httpx>=0.24.0; extra == "test"
 Requires-Dist: coverage[toml]>=7.0.0; extra == "test"
 Requires-Dist: juniper-data-client>=0.3.0; extra == "test"
+Provides-Extra: observability
+Requires-Dist: prometheus-client>=0.20.0; extra == "observability"
+Requires-Dist: sentry-sdk[fastapi]>=2.0.0; extra == "observability"
 Provides-Extra: dev
 Requires-Dist: ruff>=0.9.0; extra == "dev"
 Requires-Dist: mypy>=1.0.0; extra == "dev"
-Requires-Dist: bandit[sarif]>=1.7.9; extra == "dev"
+Requires-Dist: bandit[sarif]>=1.9.4; extra == "dev"
 Requires-Dist: pip-audit>=2.7.0; extra == "dev"
 Requires-Dist: pre-commit>=3.0.0; extra == "dev"
 Provides-Extra: all
-Requires-Dist: juniper-data[api,arc-agi,dev,test]; extra == "all"
+Requires-Dist: juniper-data[api,arc-agi,dev,observability,test]; extra == "all"
 Dynamic: license-file
 
 # Juniper Data
@@ -52,7 +55,7 @@ Verified compatible versions:
 |---|---|---|---|---|---|
 | 0.4.x | 0.3.x | 0.2.x | >=0.3.1 | >=0.1.0 | >=0.1.0 |
 
-For full-stack Docker deployment and integration tests, see [juniper-deploy](https://github.com/pcalnon/juniper-deploy).
+For full-stack Docker deployment and integration tests, see `juniper-deploy`.
 
 ## Architecture
 
@@ -94,10 +97,20 @@ JuniperData is the **foundational data layer** of the Juniper ecosystem. Juniper
 
 ```bash
 # Full stack with all three services:
-git clone https://github.com/pcalnon/juniper-deploy.git
+git clone https://github.com/pcalnon/juniper-deploy.git  # (private repository)
 cd juniper-deploy && docker compose up --build
 ```
 
+## Dependency Lockfile
+
+The `requirements.lock` file pins exact dependency versions for reproducible Docker builds. The `pyproject.toml` retains flexible `>=` ranges for local development.
+
+**Regenerate after changing dependencies in `pyproject.toml`:**
+
+```bash
+uv pip compile pyproject.toml --extra api --extra observability -o requirements.lock
+```
+
 ## Installation
 
 ### Basic Installation
@@ -143,28 +156,62 @@ uvicorn juniper_data.api.app:app --reload
 
 ## API Endpoints
 
-| Endpoint                        | Method | Description                        |
-| ------------------------------- | ------ | ---------------------------------- |
-| `/v1/health`                    | GET    | Health check endpoint              |
-| `/v1/datasets`                  | GET    | List available datasets            |
-| `/v1/datasets/{id}`             | GET    | Get a specific dataset             |
-| `/v1/generators/spiral`         | POST   | Generate a new spiral dataset      |
-| `/v1/generators/spiral/config`  | GET    | Get spiral generator configuration |
+| Endpoint                              | Method | Description                                          |
+| ------------------------------------- | ------ | ---------------------------------------------------- |
+| `/v1/health`                          | GET    | Health check                                         |
+| `/v1/health/live`                     | GET    | Liveness probe                                       |
+| `/v1/health/ready`                    | GET    | Readiness probe (checks storage)                     |
+| `/v1/generators`                      | GET    | List all generators with schemas                     |
+| `/v1/generators/{name}/schema`        | GET    | Get parameter schema for a generator                 |
+| `/v1/datasets`                        | POST   | Create dataset (or return cached dataset)            |
+| `/v1/datasets`                        | GET    | List dataset IDs                                     |
+| `/v1/datasets/filter`                 | GET    | Filter metadata by generator/tags/date/name/version |
+| `/v1/datasets/stats`                  | GET    | Aggregate dataset statistics                         |
+| `/v1/datasets/versions`               | GET    | List all versions for a logical dataset name         |
+| `/v1/datasets/latest`                 | GET    | Get latest version for a logical dataset name        |
+| `/v1/datasets/batch-create`           | POST   | Create multiple datasets                             |
+| `/v1/datasets/batch-delete`           | POST   | Delete multiple datasets                             |
+| `/v1/datasets/batch-tags`             | PATCH  | Update tags on multiple datasets                    |
+| `/v1/datasets/batch-export`           | POST   | Export multiple datasets as ZIP                     |
+| `/v1/datasets/cleanup-expired`        | POST   | Delete expired datasets                             |
+| `/v1/datasets/{id}`                   | GET    | Get dataset metadata                                 |
+| `/v1/datasets/{id}`                   | DELETE | Delete a dataset                                     |
+| `/v1/datasets/{id}/artifact`          | GET    | Download NPZ artifact                                |
+| `/v1/datasets/{id}/preview`           | GET    | Preview first N samples as JSON                      |
+| `/v1/datasets/{id}/tags`              | PATCH  | Add/remove tags on one dataset                       |
+
+See [docs/api/JUNIPER_DATA_API.md](docs/api/JUNIPER_DATA_API.md) for full endpoint documentation including filtering, batch operations, and tagging.
+
+### Named Dataset Versioning
+
+`POST /v1/datasets` supports logical names for versioned datasets:
+
+- Set `name` to group related datasets into a version series.
+- Persisted creates with the same `name` auto-increment `meta.dataset_version` (`1`, `2`, `3`, ...).
+- Repeating an identical request returns the cached dataset and keeps its existing version.
+- Use `GET /v1/datasets/versions?name=<dataset_name>` to view history and `GET /v1/datasets/latest?name=<dataset_name>` to resolve the latest.
 
 ## Project Structure
 
 ```bash
-JuniperData/
+juniper-data/
 ├── juniper_data/
 │   ├── core/           # Core functionality and base classes
-│   ├── generators/     # Dataset generators
-│   │   └── spiral/     # Spiral dataset generator
+│   ├── generators/     # Dataset generators (8 types)
+│   │   ├── spiral/     # Multi-spiral classification
+│   │   ├── xor/        # XOR classification
+│   │   ├── gaussian/   # Mixture of Gaussians
+│   │   ├── circles/    # Concentric circles
+│   │   ├── checkerboard/ # 2D checkerboard pattern
+│   │   ├── csv_import/ # CSV/JSON file import
+│   │   ├── mnist/      # MNIST / Fashion-MNIST
+│   │   └── arc_agi/    # ARC-AGI visual reasoning
 │   ├── storage/        # Dataset persistence layer
-│   └── api/            # FastAPI application
-│       └── routes/     # API route handlers
-├── tests/
-│   ├── unit/           # Unit tests
-│   └── integration/    # Integration tests
+│   ├── api/            # FastAPI application
+│   │   └── routes/     # API route handlers
+│   └── tests/          # Test suite
+│       ├── unit/       # Unit tests
+│       └── integration/ # Integration tests
 ├── pyproject.toml      # Project configuration
 └── README.md           # This file
 ```
@@ -186,8 +233,8 @@ pytest --cov=juniper_data --cov-report=html
 ### Code Formatting
 
 ```bash
-black juniper_data tests
-isort juniper_data tests
+ruff format juniper_data tests
+ruff check --fix juniper_data tests
 ```
 
 ### Type Checking

{juniper_data-0.4.2 → juniper_data-0.6.0}/README.md

@@ -15,7 +15,7 @@ Verified compatible versions:
 |---|---|---|---|---|---|
 | 0.4.x | 0.3.x | 0.2.x | >=0.3.1 | >=0.1.0 | >=0.1.0 |
 
-For full-stack Docker deployment and integration tests, see [juniper-deploy](https://github.com/pcalnon/juniper-deploy).
+For full-stack Docker deployment and integration tests, see `juniper-deploy`.
 
 ## Architecture
 
@@ -57,10 +57,20 @@ JuniperData is the **foundational data layer** of the Juniper ecosystem. Juniper
 
 ```bash
 # Full stack with all three services:
-git clone https://github.com/pcalnon/juniper-deploy.git
+git clone https://github.com/pcalnon/juniper-deploy.git  # (private repository)
 cd juniper-deploy && docker compose up --build
 ```
 
+## Dependency Lockfile
+
+The `requirements.lock` file pins exact dependency versions for reproducible Docker builds. The `pyproject.toml` retains flexible `>=` ranges for local development.
+
+**Regenerate after changing dependencies in `pyproject.toml`:**
+
+```bash
+uv pip compile pyproject.toml --extra api --extra observability -o requirements.lock
+```
+
 ## Installation
 
 ### Basic Installation
@@ -106,28 +116,62 @@ uvicorn juniper_data.api.app:app --reload
 
 ## API Endpoints
 
-| Endpoint                        | Method | Description                        |
-| ------------------------------- | ------ | ---------------------------------- |
-| `/v1/health`                    | GET    | Health check endpoint              |
-| `/v1/datasets`                  | GET    | List available datasets            |
-| `/v1/datasets/{id}`             | GET    | Get a specific dataset             |
-| `/v1/generators/spiral`         | POST   | Generate a new spiral dataset      |
-| `/v1/generators/spiral/config`  | GET    | Get spiral generator configuration |
+| Endpoint                              | Method | Description                                          |
+| ------------------------------------- | ------ | ---------------------------------------------------- |
+| `/v1/health`                          | GET    | Health check                                         |
+| `/v1/health/live`                     | GET    | Liveness probe                                       |
+| `/v1/health/ready`                    | GET    | Readiness probe (checks storage)                     |
+| `/v1/generators`                      | GET    | List all generators with schemas                     |
+| `/v1/generators/{name}/schema`        | GET    | Get parameter schema for a generator                 |
+| `/v1/datasets`                        | POST   | Create dataset (or return cached dataset)            |
+| `/v1/datasets`                        | GET    | List dataset IDs                                     |
+| `/v1/datasets/filter`                 | GET    | Filter metadata by generator/tags/date/name/version |
+| `/v1/datasets/stats`                  | GET    | Aggregate dataset statistics                         |
+| `/v1/datasets/versions`               | GET    | List all versions for a logical dataset name         |
+| `/v1/datasets/latest`                 | GET    | Get latest version for a logical dataset name        |
+| `/v1/datasets/batch-create`           | POST   | Create multiple datasets                             |
+| `/v1/datasets/batch-delete`           | POST   | Delete multiple datasets                             |
+| `/v1/datasets/batch-tags`             | PATCH  | Update tags on multiple datasets                    |
+| `/v1/datasets/batch-export`           | POST   | Export multiple datasets as ZIP                     |
+| `/v1/datasets/cleanup-expired`        | POST   | Delete expired datasets                             |
+| `/v1/datasets/{id}`                   | GET    | Get dataset metadata                                 |
+| `/v1/datasets/{id}`                   | DELETE | Delete a dataset                                     |
+| `/v1/datasets/{id}/artifact`          | GET    | Download NPZ artifact                                |
+| `/v1/datasets/{id}/preview`           | GET    | Preview first N samples as JSON                      |
+| `/v1/datasets/{id}/tags`              | PATCH  | Add/remove tags on one dataset                       |
+
+See [docs/api/JUNIPER_DATA_API.md](docs/api/JUNIPER_DATA_API.md) for full endpoint documentation including filtering, batch operations, and tagging.
+
+### Named Dataset Versioning
+
+`POST /v1/datasets` supports logical names for versioned datasets:
+
+- Set `name` to group related datasets into a version series.
+- Persisted creates with the same `name` auto-increment `meta.dataset_version` (`1`, `2`, `3`, ...).
+- Repeating an identical request returns the cached dataset and keeps its existing version.
+- Use `GET /v1/datasets/versions?name=<dataset_name>` to view history and `GET /v1/datasets/latest?name=<dataset_name>` to resolve the latest.
 
 ## Project Structure
 
 ```bash
-JuniperData/
+juniper-data/
 ├── juniper_data/
 │   ├── core/           # Core functionality and base classes
-│   ├── generators/     # Dataset generators
-│   │   └── spiral/     # Spiral dataset generator
+│   ├── generators/     # Dataset generators (8 types)
+│   │   ├── spiral/     # Multi-spiral classification
+│   │   ├── xor/        # XOR classification
+│   │   ├── gaussian/   # Mixture of Gaussians
+│   │   ├── circles/    # Concentric circles
+│   │   ├── checkerboard/ # 2D checkerboard pattern
+│   │   ├── csv_import/ # CSV/JSON file import
+│   │   ├── mnist/      # MNIST / Fashion-MNIST
+│   │   └── arc_agi/    # ARC-AGI visual reasoning
 │   ├── storage/        # Dataset persistence layer
-│   └── api/            # FastAPI application
-│       └── routes/     # API route handlers
-├── tests/
-│   ├── unit/           # Unit tests
-│   └── integration/    # Integration tests
+│   ├── api/            # FastAPI application
+│   │   └── routes/     # API route handlers
+│   └── tests/          # Test suite
+│       ├── unit/       # Unit tests
+│       └── integration/ # Integration tests
 ├── pyproject.toml      # Project configuration
 └── README.md           # This file
 ```
@@ -149,8 +193,8 @@ pytest --cov=juniper_data --cov-report=html
 ### Code Formatting
 
 ```bash
-black juniper_data tests
-isort juniper_data tests
+ruff format juniper_data tests
+ruff check --fix juniper_data tests
 ```
 
 ### Type Checking

{juniper_data-0.4.2 → juniper_data-0.6.0}/juniper_data/__init__.py

@@ -14,7 +14,7 @@ except ImportError:
     ARC_AGI_AVAILABLE = False
     arc_agi = None  # type: ignore[assignment]
 
-__version__ = "0.4.2"
+__version__ = "0.6.0"
 __author__ = "Paul Calnon"
 
 

{juniper_data-0.4.2 → juniper_data-0.6.0}/juniper_data/api/app.py

@@ -12,7 +12,15 @@ from fastapi.responses import JSONResponse
 from juniper_data import __version__
 from juniper_data.storage import LocalFSDatasetStore
 
-from .middleware import SecurityMiddleware
+from .middleware import RequestBodyLimitMiddleware, SecurityHeadersMiddleware, SecurityMiddleware
+from .observability import (
+    PrometheusMiddleware,
+    RequestIdMiddleware,
+    configure_logging,
+    configure_sentry,
+    get_prometheus_app,
+    set_build_info,
+)
 from .routes import datasets, generators, health
 from .security import APIKeyAuth, RateLimiter
 from .settings import Settings, get_settings
@@ -26,10 +34,11 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
     store = LocalFSDatasetStore(storage_path)
     datasets.set_store(store)
 
-    logging.basicConfig(
-        level=getattr(logging, settings.log_level.upper(), logging.INFO),
-        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
-    )
+    configure_logging(settings.log_level, settings.log_format, "juniper-data")
+    configure_sentry(settings.sentry_dsn, "juniper-data", __version__, send_pii=settings.sentry_send_pii, traces_sample_rate=settings.sentry_traces_sample_rate)
+    if settings.metrics_enabled:
+        set_build_info("juniper_data", __version__)
+
     logger = logging.getLogger("juniper_data")
     logger.info(f"JuniperData API v{__version__} starting")
     logger.info(f"Storage path: {storage_path.absolute()}")
@@ -52,28 +61,37 @@ def create_app(settings: Settings | None = None) -> FastAPI:
     if settings is None:
         settings = get_settings()
 
+    # Disable interactive API docs when authentication is enabled (production).
+    docs_enabled = not settings.api_keys
     app = FastAPI(
         title="Juniper Data API",
         description="Dataset generation and management service for the Juniper ecosystem",
         version=__version__,
         lifespan=lifespan,
+        docs_url="/docs" if docs_enabled else None,
+        redoc_url="/redoc" if docs_enabled else None,
+        openapi_url="/openapi.json" if docs_enabled else None,
     )
 
     app.state.settings = settings
 
-    # Only allow credentialed CORS requests when origins are explicitly specified.
-    # Browsers do not permit Access-Control-Allow-Credentials: true with a wildcard
-    # origin (Access-Control-Allow-Origin: "*"), so the default ["*"] intentionally
-    # disables credentials unless concrete origins are configured.
+    # CORS: only enable when origins are explicitly configured.
     allow_credentials = bool(settings.cors_origins) and "*" not in settings.cors_origins
 
-    app.add_middleware(
-        CORSMiddleware,
-        allow_origins=settings.cors_origins,
-        allow_credentials=allow_credentials,
-        allow_methods=["*"],
-        allow_headers=["*"],
-    )
+    if settings.cors_origins:
+        app.add_middleware(
+            CORSMiddleware,
+            allow_origins=settings.cors_origins,
+            allow_credentials=allow_credentials,
+            allow_methods=["*"],
+            allow_headers=["*"],
+        )
+
+    # Request body size limit
+    app.add_middleware(RequestBodyLimitMiddleware)
+
+    # Security headers (outermost — runs on every response)
+    app.add_middleware(SecurityHeadersMiddleware)
 
     api_key_auth = APIKeyAuth(settings.api_keys)
     rate_limiter = RateLimiter(
@@ -86,15 +104,27 @@ def create_app(settings: Settings | None = None) -> FastAPI:
         rate_limiter=rate_limiter,
     )
 
+    # Observability middleware (added after SecurityMiddleware, before CORS)
+    # Middleware execution is LIFO: last added runs first.
+    # Order: RequestIdMiddleware → PrometheusMiddleware → SecurityMiddleware → SecurityHeaders → CORS
+    if settings.metrics_enabled:
+        app.add_middleware(PrometheusMiddleware, service_name="juniper-data", namespace="juniper_data")
+    app.add_middleware(RequestIdMiddleware)
+
     app.include_router(health.router, prefix="/v1")
     app.include_router(generators.router, prefix="/v1")
     app.include_router(datasets.router, prefix="/v1")
 
+    # Mount Prometheus metrics endpoint
+    if settings.metrics_enabled:
+        app.mount("/metrics", get_prometheus_app())
+
     @app.exception_handler(ValueError)
     async def value_error_handler(request: Request, exc: ValueError) -> JSONResponse:
+        logging.getLogger("juniper_data").debug("Validation error: %s", exc)
         return JSONResponse(
             status_code=400,
-            content={"detail": str(exc)},
+            content={"detail": "Invalid request parameters"},
         )
 
     @app.exception_handler(Exception)

{juniper_data-0.4.2 → juniper_data-0.6.0}/juniper_data/api/middleware.py

@@ -16,6 +16,54 @@ EXEMPT_PATHS = {
     "/redoc",
 }
 
+# Default Content-Security-Policy for API-only services.
+_DEFAULT_CSP = "default-src 'none'; frame-ancestors 'none'"
+
+
+class SecurityHeadersMiddleware(BaseHTTPMiddleware):
+    """Add security headers to all responses.
+
+    Injects standard security headers (X-Content-Type-Options, X-Frame-Options,
+    Referrer-Policy, Permissions-Policy, CSP, and conditional HSTS) into every
+    HTTP response.
+    """
+
+    def __init__(self, app: ASGIApp, content_security_policy: str = _DEFAULT_CSP) -> None:
+        super().__init__(app)
+        self._csp = content_security_policy
+
+    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
+        response = await call_next(request)
+
+        response.headers["X-Content-Type-Options"] = "nosniff"
+        response.headers["X-Frame-Options"] = "DENY"
+        response.headers["Referrer-Policy"] = "strict-origin-when-cross-origin"
+        response.headers["Permissions-Policy"] = "camera=(), microphone=(), geolocation=()"
+        response.headers["Content-Security-Policy"] = self._csp
+
+        # Only add HSTS when the request arrived over TLS (via reverse proxy)
+        if request.headers.get("X-Forwarded-Proto") == "https":
+            response.headers["Strict-Transport-Security"] = "max-age=31536000; includeSubDomains"
+
+        return response
+
+
+_MAX_REQUEST_BODY_BYTES = 10 * 1024 * 1024  # 10 MB
+
+
+class RequestBodyLimitMiddleware(BaseHTTPMiddleware):
+    """Reject requests whose Content-Length exceeds a configurable limit."""
+
+    def __init__(self, app: ASGIApp, max_bytes: int = _MAX_REQUEST_BODY_BYTES) -> None:
+        super().__init__(app)
+        self._max_bytes = max_bytes
+
+    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
+        content_length = request.headers.get("content-length")
+        if content_length is not None and int(content_length) > self._max_bytes:
+            return JSONResponse(status_code=413, content={"detail": "Request body too large"})
+        return await call_next(request)
+
 
 class SecurityMiddleware(BaseHTTPMiddleware):
     """Middleware for API key authentication and rate limiting.

juniper_data-0.6.0/juniper_data/api/models/__init__.py

@@ -0,0 +1 @@
+"""API response models."""

juniper_data-0.6.0/juniper_data/api/models/health.py

@@ -0,0 +1,26 @@
+"""Health check response models for standardized readiness reporting."""
+
+from datetime import datetime
+from typing import Literal
+
+from pydantic import BaseModel, Field
+
+
+class DependencyStatus(BaseModel):
+    """Health status of a single dependency."""
+
+    name: str
+    status: Literal["healthy", "unhealthy", "degraded", "not_configured"]
+    latency_ms: float | None = None
+    message: str | None = None
+
+
+class ReadinessResponse(BaseModel):
+    """Standard /v1/health/ready response for all Juniper services."""
+
+    status: Literal["ready", "degraded", "not_ready"]
+    version: str
+    service: str
+    timestamp: float = Field(default_factory=lambda: datetime.now().timestamp())
+    dependencies: dict[str, DependencyStatus] = {}
+    details: dict[str, object] = {}