juniper-data 0.4.2__tar.gz

juniper_data-0.4.2/LICENSE

@@ -0,0 +1,9 @@
+# Juniper Canopy project
+
+Copyright 2024, 2025 Paul Calnon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

juniper_data-0.4.2/PKG-INFO

@@ -0,0 +1,216 @@
+Metadata-Version: 2.4
+Name: juniper-data
+Version: 0.4.2
+Summary: Dataset generation and management service for the Juniper ecosystem
+Author: Paul Calnon
+License: MIT
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Provides-Extra: arc-agi
+Requires-Dist: arc-agi>=0.9.0; extra == "arc-agi"
+Provides-Extra: api
+Requires-Dist: fastapi>=0.100.0; extra == "api"
+Requires-Dist: uvicorn[standard]>=0.23.0; extra == "api"
+Requires-Dist: pydantic-settings>=2.0.0; extra == "api"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+Requires-Dist: pytest-timeout>=2.2.0; extra == "test"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "test"
+Requires-Dist: pytest-benchmark>=4.0.0; extra == "test"
+Requires-Dist: httpx>=0.24.0; extra == "test"
+Requires-Dist: coverage[toml]>=7.0.0; extra == "test"
+Requires-Dist: juniper-data-client>=0.3.0; extra == "test"
+Provides-Extra: dev
+Requires-Dist: ruff>=0.9.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: bandit[sarif]>=1.7.9; extra == "dev"
+Requires-Dist: pip-audit>=2.7.0; extra == "dev"
+Requires-Dist: pre-commit>=3.0.0; extra == "dev"
+Provides-Extra: all
+Requires-Dist: juniper-data[api,arc-agi,dev,test]; extra == "all"
+Dynamic: license-file
+
+# Juniper Data
+
+Dataset generation and management service for the Juniper ecosystem.
+
+## Overview
+
+Juniper Data provides a centralized service for generating, storing, and serving datasets used by the Juniper neural network projects. It supports various dataset types including the classic two-spiral classification problem.
+
+## Ecosystem Compatibility
+
+This service is part of the [Juniper](https://github.com/pcalnon/juniper-ml) ecosystem.
+Verified compatible versions:
+
+| juniper-data | juniper-cascor | juniper-canopy | data-client | cascor-client | cascor-worker |
+|---|---|---|---|---|---|
+| 0.4.x | 0.3.x | 0.2.x | >=0.3.1 | >=0.1.0 | >=0.1.0 |
+
+For full-stack Docker deployment and integration tests, see [juniper-deploy](https://github.com/pcalnon/juniper-deploy).
+
+## Architecture
+
+JuniperData is the **foundational data layer** of the Juniper ecosystem. JuniperCascor and juniper-canopy both call JuniperData to generate and retrieve datasets.
+
+```
+┌─────────────────────┐     REST+WS      ┌──────────────────────┐
+│   juniper-canopy     │ ◄──────────────► │    JuniperCascor     │
+│   Dashboard         │                  │    Training Svc      │
+│   Port 8050         │                  │    Port 8200         │
+└──────────┬──────────┘                  └──────────┬───────────┘
+           │ REST                                    │ REST
+           ▼                                         ▼
+┌──────────────────────────────────────────────────────────────┐
+│                      JuniperData  ◄── (this service)          │
+│                   Dataset Service  ·  Port 8100               │
+└──────────────────────────────────────────────────────────────┘
+```
+
+**Data contract**: datasets are served as NPZ archives with keys `X_train`, `y_train`, `X_test`, `y_test`, `X_full`, `y_full` (all `float32`).
+
+## Related Services
+
+| Service | Relationship | Environment Variable |
+|---------|-------------|---------------------|
+| [juniper-cascor](https://github.com/pcalnon/juniper-cascor) | Consumes JuniperData for training datasets | `JUNIPER_DATA_URL=http://localhost:8100` |
+| [juniper-canopy](https://github.com/pcalnon/juniper-canopy) | Consumes JuniperData for visualization data | `JUNIPER_DATA_URL=http://localhost:8100` |
+| [juniper-data-client](https://github.com/pcalnon/juniper-data-client) | PyPI client library for this service | `pip install juniper-data-client` |
+
+### Service Configuration
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `JUNIPER_DATA_HOST` | `0.0.0.0` | Listen address |
+| `JUNIPER_DATA_PORT` | `8100` | Service port |
+| `JUNIPER_DATA_LOG_LEVEL` | `INFO` | Log verbosity |
+
+### Docker Deployment
+
+```bash
+# Full stack with all three services:
+git clone https://github.com/pcalnon/juniper-deploy.git
+cd juniper-deploy && docker compose up --build
+```
+
+## Installation
+
+### Basic Installation
+
+```bash
+pip install -e .
+```
+
+### With API Support
+
+```bash
+pip install -e ".[api]"
+```
+
+### Development Installation
+
+```bash
+pip install -e ".[dev]"
+```
+
+### Full Installation
+
+```bash
+pip install -e ".[all]"
+```
+
+## Quick Start
+
+### Generate a Spiral Dataset
+
+```python
+from juniper_data.generators.spiral import SpiralGenerator
+
+generator = SpiralGenerator()
+dataset = generator.generate(n_points=100, n_spirals=2, noise=0.1)
+```
+
+### Start the API Server
+
+```bash
+uvicorn juniper_data.api.app:app --reload
+```
+
+## API Endpoints
+
+| Endpoint                        | Method | Description                        |
+| ------------------------------- | ------ | ---------------------------------- |
+| `/v1/health`                    | GET    | Health check endpoint              |
+| `/v1/datasets`                  | GET    | List available datasets            |
+| `/v1/datasets/{id}`             | GET    | Get a specific dataset             |
+| `/v1/generators/spiral`         | POST   | Generate a new spiral dataset      |
+| `/v1/generators/spiral/config`  | GET    | Get spiral generator configuration |
+
+## Project Structure
+
+```bash
+JuniperData/
+├── juniper_data/
+│   ├── core/           # Core functionality and base classes
+│   ├── generators/     # Dataset generators
+│   │   └── spiral/     # Spiral dataset generator
+│   ├── storage/        # Dataset persistence layer
+│   └── api/            # FastAPI application
+│       └── routes/     # API route handlers
+├── tests/
+│   ├── unit/           # Unit tests
+│   └── integration/    # Integration tests
+├── pyproject.toml      # Project configuration
+└── README.md           # This file
+```
+
+## Development
+
+### Running Tests
+
+```bash
+pytest
+```
+
+### Running Tests with Coverage
+
+```bash
+pytest --cov=juniper_data --cov-report=html
+```
+
+### Code Formatting
+
+```bash
+black juniper_data tests
+isort juniper_data tests
+```
+
+### Type Checking
+
+```bash
+mypy juniper_data
+```
+
+## Juniper Ecosystem
+
+| Repository | Description |
+|-----------|-------------|
+| [juniper-data](https://github.com/pcalnon/juniper-data) | Dataset generation service (this repo) |
+| [juniper-cascor](https://github.com/pcalnon/juniper-cascor) | CasCor neural network training service |
+| [juniper-canopy](https://github.com/pcalnon/juniper-canopy) | Real-time monitoring dashboard |
+| [juniper-data-client](https://github.com/pcalnon/juniper-data-client) | PyPI: `juniper-data-client` |
+| [juniper-cascor-client](https://github.com/pcalnon/juniper-cascor-client) | PyPI: `juniper-cascor-client` |
+| [juniper-cascor-worker](https://github.com/pcalnon/juniper-cascor-worker) | PyPI: `juniper-cascor-worker` |
+
+## License
+
+MIT License - Copyright (c) 2024-2026 Paul Calnon
+
+## Git Leaks
+
+![gitleaks badge](https://img.shields.io/badge/protected%20by-gitleaks-blue)

juniper_data-0.4.2/README.md

@@ -0,0 +1,179 @@
+# Juniper Data
+
+Dataset generation and management service for the Juniper ecosystem.
+
+## Overview
+
+Juniper Data provides a centralized service for generating, storing, and serving datasets used by the Juniper neural network projects. It supports various dataset types including the classic two-spiral classification problem.
+
+## Ecosystem Compatibility
+
+This service is part of the [Juniper](https://github.com/pcalnon/juniper-ml) ecosystem.
+Verified compatible versions:
+
+| juniper-data | juniper-cascor | juniper-canopy | data-client | cascor-client | cascor-worker |
+|---|---|---|---|---|---|
+| 0.4.x | 0.3.x | 0.2.x | >=0.3.1 | >=0.1.0 | >=0.1.0 |
+
+For full-stack Docker deployment and integration tests, see [juniper-deploy](https://github.com/pcalnon/juniper-deploy).
+
+## Architecture
+
+JuniperData is the **foundational data layer** of the Juniper ecosystem. JuniperCascor and juniper-canopy both call JuniperData to generate and retrieve datasets.
+
+```
+┌─────────────────────┐     REST+WS      ┌──────────────────────┐
+│   juniper-canopy     │ ◄──────────────► │    JuniperCascor     │
+│   Dashboard         │                  │    Training Svc      │
+│   Port 8050         │                  │    Port 8200         │
+└──────────┬──────────┘                  └──────────┬───────────┘
+           │ REST                                    │ REST
+           ▼                                         ▼
+┌──────────────────────────────────────────────────────────────┐
+│                      JuniperData  ◄── (this service)          │
+│                   Dataset Service  ·  Port 8100               │
+└──────────────────────────────────────────────────────────────┘
+```
+
+**Data contract**: datasets are served as NPZ archives with keys `X_train`, `y_train`, `X_test`, `y_test`, `X_full`, `y_full` (all `float32`).
+
+## Related Services
+
+| Service | Relationship | Environment Variable |
+|---------|-------------|---------------------|
+| [juniper-cascor](https://github.com/pcalnon/juniper-cascor) | Consumes JuniperData for training datasets | `JUNIPER_DATA_URL=http://localhost:8100` |
+| [juniper-canopy](https://github.com/pcalnon/juniper-canopy) | Consumes JuniperData for visualization data | `JUNIPER_DATA_URL=http://localhost:8100` |
+| [juniper-data-client](https://github.com/pcalnon/juniper-data-client) | PyPI client library for this service | `pip install juniper-data-client` |
+
+### Service Configuration
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `JUNIPER_DATA_HOST` | `0.0.0.0` | Listen address |
+| `JUNIPER_DATA_PORT` | `8100` | Service port |
+| `JUNIPER_DATA_LOG_LEVEL` | `INFO` | Log verbosity |
+
+### Docker Deployment
+
+```bash
+# Full stack with all three services:
+git clone https://github.com/pcalnon/juniper-deploy.git
+cd juniper-deploy && docker compose up --build
+```
+
+## Installation
+
+### Basic Installation
+
+```bash
+pip install -e .
+```
+
+### With API Support
+
+```bash
+pip install -e ".[api]"
+```
+
+### Development Installation
+
+```bash
+pip install -e ".[dev]"
+```
+
+### Full Installation
+
+```bash
+pip install -e ".[all]"
+```
+
+## Quick Start
+
+### Generate a Spiral Dataset
+
+```python
+from juniper_data.generators.spiral import SpiralGenerator
+
+generator = SpiralGenerator()
+dataset = generator.generate(n_points=100, n_spirals=2, noise=0.1)
+```
+
+### Start the API Server
+
+```bash
+uvicorn juniper_data.api.app:app --reload
+```
+
+## API Endpoints
+
+| Endpoint                        | Method | Description                        |
+| ------------------------------- | ------ | ---------------------------------- |
+| `/v1/health`                    | GET    | Health check endpoint              |
+| `/v1/datasets`                  | GET    | List available datasets            |
+| `/v1/datasets/{id}`             | GET    | Get a specific dataset             |
+| `/v1/generators/spiral`         | POST   | Generate a new spiral dataset      |
+| `/v1/generators/spiral/config`  | GET    | Get spiral generator configuration |
+
+## Project Structure
+
+```bash
+JuniperData/
+├── juniper_data/
+│   ├── core/           # Core functionality and base classes
+│   ├── generators/     # Dataset generators
+│   │   └── spiral/     # Spiral dataset generator
+│   ├── storage/        # Dataset persistence layer
+│   └── api/            # FastAPI application
+│       └── routes/     # API route handlers
+├── tests/
+│   ├── unit/           # Unit tests
+│   └── integration/    # Integration tests
+├── pyproject.toml      # Project configuration
+└── README.md           # This file
+```
+
+## Development
+
+### Running Tests
+
+```bash
+pytest
+```
+
+### Running Tests with Coverage
+
+```bash
+pytest --cov=juniper_data --cov-report=html
+```
+
+### Code Formatting
+
+```bash
+black juniper_data tests
+isort juniper_data tests
+```
+
+### Type Checking
+
+```bash
+mypy juniper_data
+```
+
+## Juniper Ecosystem
+
+| Repository | Description |
+|-----------|-------------|
+| [juniper-data](https://github.com/pcalnon/juniper-data) | Dataset generation service (this repo) |
+| [juniper-cascor](https://github.com/pcalnon/juniper-cascor) | CasCor neural network training service |
+| [juniper-canopy](https://github.com/pcalnon/juniper-canopy) | Real-time monitoring dashboard |
+| [juniper-data-client](https://github.com/pcalnon/juniper-data-client) | PyPI: `juniper-data-client` |
+| [juniper-cascor-client](https://github.com/pcalnon/juniper-cascor-client) | PyPI: `juniper-cascor-client` |
+| [juniper-cascor-worker](https://github.com/pcalnon/juniper-cascor-worker) | PyPI: `juniper-cascor-worker` |
+
+## License
+
+MIT License - Copyright (c) 2024-2026 Paul Calnon
+
+## Git Leaks
+
+![gitleaks badge](https://img.shields.io/badge/protected%20by-gitleaks-blue)

juniper_data-0.4.2/juniper_data/__init__.py

@@ -0,0 +1,88 @@
+"""
+Juniper Data - Dataset generation and management service for the Juniper ecosystem.
+"""
+
+import os
+
+from dotenv import load_dotenv
+
+try:
+    import arc_agi
+
+    ARC_AGI_AVAILABLE = True
+except ImportError:
+    ARC_AGI_AVAILABLE = False
+    arc_agi = None  # type: ignore[assignment]
+
+__version__ = "0.4.2"
+__author__ = "Paul Calnon"
+
+
+def get_arc_agi_env() -> bool:
+    """
+    Ensure ARC_AGI_ENV is available by loading environment variables if needed.
+
+    This function attempts to load environment variables from a `.env` file and then
+    returns whether the `ARC_AGI_ENV` environment variable is set.
+
+    Returns:
+        bool: True if ARC_AGI_ENV is set after loading, otherwise False.
+    """
+    # Attempt to load variables from a .env file, but base the result solely on
+    # whether ARC_AGI_ENV is present afterwards to provide consistent semantics.
+    load_dotenv()
+    return bool(os.getenv("ARC_AGI_ENV"))
+
+
+def reload_arc_agi_env() -> bool:
+    """
+    Reloads all of the Environment Variables from local OS env whether already loaded or not.
+
+    Returns:
+        bool: True if environment variables were loaded from a .env file, False otherwise.
+    """
+    return bool(load_dotenv())
+
+
+def get_arc_api_key() -> str | None:
+    """
+    Return the current value of the ARC_API_KEY environment variable as a string.
+    """
+    return os.getenv("ARC_API_KEY") or None
+
+
+def get_arc_agi_api_url() -> str | None:
+    """
+    Return the current value of the ARC_AGI_API as a URL/endpoint string.
+
+    Reading the environment at call time avoids import-time side effects
+    and makes it easier to adjust configuration in tests.
+    """
+    return os.getenv("ARC_AGI_API") or None
+
+
+def get_arc_agi_arcade() -> "arc_agi.Arcade | None":
+    """
+    Create and return an :class:`arc_agi.Arcade` instance configured from environment variables.
+
+    The API key is read from the environment via :func:`get_arc_api_key`, avoiding import-time
+    side effects and making it easier to adjust configuration in tests.
+
+    Raises:
+        ImportError: If the ``arc-agi`` package is not installed.
+    """
+    if not ARC_AGI_AVAILABLE:
+        raise ImportError("arc-agi package not installed. Install with: pip install 'juniper-data[arc-agi]'")
+    # Automatically uses ARC_API_KEY from environment:  arc = arc_agi.Arcade(), Or pass the API key explicitly
+    return arc_agi.Arcade(arc_api_key=get_arc_api_key()) or None
+
+
+# Deprecated
+def get_arc_agi_api() -> str | None:
+    """
+    Deprecated alias for :func:`get_arc_agi_api_url`.
+
+    This function returns the same value as :func:`get_arc_agi_api_url` and will be
+    removed in a future release. Use :func:`get_arc_agi_api_url` instead.
+    """
+    return get_arc_agi_api_url()

juniper_data-0.4.2/juniper_data/__main__.py

@@ -0,0 +1,78 @@
+"""Entry point for running the JuniperData API with uvicorn."""
+
+import argparse
+import sys
+
+
+def main() -> int:
+    """Run the JuniperData API server."""
+    try:
+        import uvicorn
+    except ImportError:
+        print("Error: uvicorn not installed. Install with: pip install 'juniper-data[api]'")
+        return 1
+
+    from juniper_data.api.settings import Settings
+
+    parser = argparse.ArgumentParser(
+        description="Run the JuniperData API server",
+        prog="python -m juniper_data",
+    )
+    parser.add_argument(
+        "--host",
+        type=str,
+        default=None,
+        help="Host to bind to (default: from settings, which default to 127.0.0.1)",
+    )
+    parser.add_argument(
+        "--port",
+        type=int,
+        default=None,
+        help="Port to bind to (default: from settings or 8100)",
+    )
+    parser.add_argument(
+        "--storage-path",
+        type=str,
+        default=None,
+        help="Path to dataset storage directory",
+    )
+    parser.add_argument(
+        "--log-level",
+        type=str,
+        choices=["TRACE", "VERBOSE", "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL", "FATAL"],
+        default=None,
+        help="Logging level",
+    )
+    parser.add_argument(
+        "--reload",
+        action="store_true",
+        help="Enable auto-reload for development",
+    )
+
+    args = parser.parse_args()
+
+    settings = Settings()
+
+    host = args.host if args.host is not None else settings.host
+    port = args.port if args.port is not None else settings.port
+    log_level_source = args.log_level if args.log_level is not None else settings.log_level
+    log_level = log_level_source.lower()
+
+    if args.storage_path is not None:
+        import os
+
+        os.environ["JUNIPER_DATA_STORAGE_PATH"] = args.storage_path
+
+    uvicorn.run(
+        "juniper_data.api.app:app",
+        host=host,
+        port=port,
+        log_level=log_level,
+        reload=args.reload,
+    )
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

juniper_data-0.4.2/juniper_data/api/__init__.py

@@ -0,0 +1,10 @@
+"""API module for Juniper Data service."""
+
+from .app import create_app
+from .settings import Settings, get_settings
+
+__all__ = [
+    "create_app",
+    "Settings",
+    "get_settings",
+]

juniper_data-0.4.2/juniper_data/api/app.py

@@ -0,0 +1,111 @@
+"""FastAPI application factory and configuration."""
+
+import logging
+from collections.abc import AsyncGenerator
+from contextlib import asynccontextmanager
+from pathlib import Path
+
+from fastapi import FastAPI, Request
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import JSONResponse
+
+from juniper_data import __version__
+from juniper_data.storage import LocalFSDatasetStore
+
+from .middleware import SecurityMiddleware
+from .routes import datasets, generators, health
+from .security import APIKeyAuth, RateLimiter
+from .settings import Settings, get_settings
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
+    """Application lifespan handler for startup/shutdown."""
+    settings: Settings = app.state.settings
+    storage_path = Path(settings.storage_path)
+    store = LocalFSDatasetStore(storage_path)
+    datasets.set_store(store)
+
+    logging.basicConfig(
+        level=getattr(logging, settings.log_level.upper(), logging.INFO),
+        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
+    )
+    logger = logging.getLogger("juniper_data")
+    logger.info(f"JuniperData API v{__version__} starting")
+    logger.info(f"Storage path: {storage_path.absolute()}")
+
+    yield
+
+    logger.info("JuniperData API shutting down")
+
+
+def create_app(settings: Settings | None = None) -> FastAPI:
+    """Create and configure the FastAPI application.
+
+    Args:
+        settings: Optional settings override. If not provided,
+                  settings are loaded from environment variables.
+
+    Returns:
+        Configured FastAPI application instance.
+    """
+    if settings is None:
+        settings = get_settings()
+
+    app = FastAPI(
+        title="Juniper Data API",
+        description="Dataset generation and management service for the Juniper ecosystem",
+        version=__version__,
+        lifespan=lifespan,
+    )
+
+    app.state.settings = settings
+
+    # Only allow credentialed CORS requests when origins are explicitly specified.
+    # Browsers do not permit Access-Control-Allow-Credentials: true with a wildcard
+    # origin (Access-Control-Allow-Origin: "*"), so the default ["*"] intentionally
+    # disables credentials unless concrete origins are configured.
+    allow_credentials = bool(settings.cors_origins) and "*" not in settings.cors_origins
+
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=settings.cors_origins,
+        allow_credentials=allow_credentials,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+
+    api_key_auth = APIKeyAuth(settings.api_keys)
+    rate_limiter = RateLimiter(
+        requests_per_minute=settings.rate_limit_requests_per_minute,
+        enabled=settings.rate_limit_enabled,
+    )
+    app.add_middleware(
+        SecurityMiddleware,
+        api_key_auth=api_key_auth,
+        rate_limiter=rate_limiter,
+    )
+
+    app.include_router(health.router, prefix="/v1")
+    app.include_router(generators.router, prefix="/v1")
+    app.include_router(datasets.router, prefix="/v1")
+
+    @app.exception_handler(ValueError)
+    async def value_error_handler(request: Request, exc: ValueError) -> JSONResponse:
+        return JSONResponse(
+            status_code=400,
+            content={"detail": str(exc)},
+        )
+
+    @app.exception_handler(Exception)
+    async def general_exception_handler(request: Request, exc: Exception) -> JSONResponse:
+        logging.getLogger("juniper_data").exception("Unhandled exception")
+        return JSONResponse(
+            status_code=500,
+            content={"detail": "Internal server error"},
+        )
+
+    return app
+
+
+app = create_app()