juniper-data 0.6.0__tar.gz → 0.7.0__tar.gz

juniper_data-0.7.0/PKG-INFO

@@ -0,0 +1,243 @@
+Metadata-Version: 2.4
+Name: juniper-data
+Version: 0.7.0
+Summary: Dataset generation and management service for the Juniper ecosystem
+Author: Paul Calnon
+License: MIT
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: cachetools>=5.3.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Provides-Extra: arc-agi
+Requires-Dist: arc-agi>=0.9.0; extra == "arc-agi"
+Provides-Extra: equities
+Requires-Dist: yfinance>=0.2.40; extra == "equities"
+Requires-Dist: pandas>=2.0.0; extra == "equities"
+Provides-Extra: api
+Requires-Dist: fastapi>=0.100.0; extra == "api"
+Requires-Dist: starlette>=1.0.1; extra == "api"
+Requires-Dist: uvicorn[standard]>=0.23.0; extra == "api"
+Requires-Dist: pydantic-settings>=2.0.0; extra == "api"
+Requires-Dist: juniper-observability>=0.4.0; extra == "api"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+Requires-Dist: pytest-timeout>=2.2.0; extra == "test"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "test"
+Requires-Dist: pytest-benchmark>=4.0.0; extra == "test"
+Requires-Dist: hypothesis>=6.0.0; extra == "test"
+Requires-Dist: httpx>=0.24.0; extra == "test"
+Requires-Dist: coverage[toml]>=7.0.0; extra == "test"
+Requires-Dist: juniper-data-client>=0.3.0; extra == "test"
+Requires-Dist: PyYAML>=6.0; extra == "test"
+Provides-Extra: observability
+Requires-Dist: prometheus-client>=0.20.0; extra == "observability"
+Requires-Dist: sentry-sdk[fastapi]>=2.0.0; extra == "observability"
+Provides-Extra: dev
+Requires-Dist: ruff>=0.9.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: bandit[sarif]>=1.9.4; extra == "dev"
+Requires-Dist: pip-audit>=2.7.0; extra == "dev"
+Requires-Dist: pre-commit>=3.0.0; extra == "dev"
+Provides-Extra: all
+Requires-Dist: juniper-data[api,arc-agi,dev,equities,observability,test]; extra == "all"
+Dynamic: license-file
+
+<!-- markdownlint-disable MD013 MD033 MD041 -->
+<!--
+  MD013 (line-length): README contains prose paragraphs that intentionally
+                       exceed the 512-char ecosystem limit. Disabled file-wide
+                       since wrapping mid-sentence harms PyPI rendering.
+  MD033 (no-inline-html): The right-aligned logo + spacing rely on HTML.
+  MD041 (first-line-heading): The HTML logo is the first line by design.
+-->
+<div align="right" width="150px" height="150px" align="right" valign="top"> <img src="images/Juniper_Logo_150px.png" alt="Juniper" align="right" valign="top" width="150px" /></div>
+<br /> <br /> <br /> <br />
+
+# Juniper: Dynamic Neural Network Research Platform
+
+Juniper is an AI/ML research platform for investigating dynamic neural network architectures and novel learning paradigms.  The project emphasizes ground-up implementations from primary literature, enabling a more transparent exploration of fundamental algorithms.
+
+## Juniper Data
+
+`juniper-data` is the **dataset-generation service** of the Juniper platform. It is a FastAPI service that produces NPZ-formatted datasets from a catalogue of generators — including the classic two-spiral and concentric-circles problems, XOR and Gaussian mixtures, a CSV/JSON import path, MNIST/Fashion-MNIST, and the ARC-AGI visual-reasoning task families — and serves them through a REST surface that supports a named-version registry, batch creation and export, tag-based filtering, and per-dataset preview. `juniper-data` is the upstream of both `juniper-cascor` (training) and `juniper-canopy` (visualisation): the dataset identifiers it returns are the substrate on which the rest of the platform conducts comparative work.
+
+## Distribution
+
+`juniper-data` is published on PyPI as **[`juniper-data`](https://pypi.org/project/juniper-data/)**.
+The package is also surfaced through the platform meta-distribution
+**[`juniper-ml`](https://pypi.org/project/juniper-ml/)**, which installs
+the full client stack via `pip install juniper-ml[all]`.
+
+```bash
+pip install juniper-data
+```
+
+## Ecosystem Compatibility
+
+This service is part of the [Juniper](https://github.com/pcalnon/juniper-ml) ecosystem.
+Verified compatible versions:
+
+| juniper-data | juniper-cascor | juniper-canopy | data-client | cascor-client | cascor-worker |
+|--------------|----------------|----------------|-------------|---------------|---------------|
+| 0.6.x        | 0.5.x          | 0.5.x          | >=0.4.1     | >=0.4.0       | >=0.4.0       |
+
+For full-stack Docker deployment and integration tests, see [`juniper-deploy`](https://github.com/pcalnon/juniper-deploy).
+
+## Architecture
+
+`juniper-data` is the **foundational data layer** of the Juniper ecosystem. Both `juniper-cascor` and `juniper-canopy` call `juniper-data` to generate, version, and retrieve datasets.
+
+```text
+┌─────────────────────┐     REST+WS      ┌──────────────────────┐
+│   juniper-canopy    │ ◄──────────────► │    juniper-cascor    │
+│   Dashboard         │                  │    Training Svc      │
+│   Port 8050         │                  │    Port 8200         │
+└──────────┬──────────┘                  └──────────┬───────────┘
+           │ REST                                   │ REST
+           ▼                                        ▼
+┌──────────────────────────────────────────────────────────────┐
+│                  juniper-data  ◄── (this service)            │
+│                  Dataset Service · Port 8100                 │
+└──────────────────────────────────────────────────────────────┘
+```
+
+**Data contract**: datasets are served as NPZ archives with the keys `X_train`, `y_train`, `X_test`, `y_test`, `X_full`, `y_full`, all of dtype `float32`.
+
+## Related Services
+
+| Service | Relationship | Notes |
+|---------|-------------|-------|
+| [juniper-cascor](https://github.com/pcalnon/juniper-cascor) | Consumes `juniper-data` for training datasets | Set `JUNIPER_DATA_URL` |
+| [juniper-canopy](https://github.com/pcalnon/juniper-canopy) | Consumes `juniper-data` for visualisation data | Set `JUNIPER_DATA_URL` |
+| [juniper-data-client](https://github.com/pcalnon/juniper-data-client) | Python HTTP client for this service | `pip install juniper-data-client` |
+
+## Service Configuration
+
+Configuration is sourced from `juniper_data/api/settings.py` (Pydantic `BaseSettings`, `env_prefix="JUNIPER_DATA_"`). The complete env-var surface is listed below.
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `JUNIPER_DATA_HOST` | No | `127.0.0.1` | Bind address (override to `0.0.0.0` for Docker) |
+| `JUNIPER_DATA_PORT` | No | `8100` | Service port |
+| `JUNIPER_DATA_STORAGE_PATH` | No | `./data/datasets` | Filesystem path for persisted dataset artifacts |
+| `JUNIPER_DATA_IMPORT_DIR` | No | `/data/imports` | Filesystem path for CSV/JSON imports |
+| `JUNIPER_DATA_LOG_LEVEL` | No | `INFO` | Log verbosity (`DEBUG`, `INFO`, `WARNING`, `ERROR`) |
+| `JUNIPER_DATA_LOG_FORMAT` | No | `text` | `text` or `json` (structured logging) |
+| `JUNIPER_DATA_CORS_ORIGINS` | No | `[]` | Allowed CORS origins |
+| `JUNIPER_DATA_API_KEYS` | No | `None` | Comma-separated or JSON-array API keys; authentication disabled when unset; Docker-secrets file path supported via the implicit `*_FILE` convention |
+| `JUNIPER_DATA_RATE_LIMIT_ENABLED` | No | `true` | Enforce per-IP request rate limiting |
+| `JUNIPER_DATA_RATE_LIMIT_REQUESTS_PER_MINUTE` | No | `60` | Per-IP rate limit |
+| `JUNIPER_DATA_SENTRY_DSN` | No | `None` | Sentry DSN for error tracking |
+| `JUNIPER_DATA_SENTRY_SEND_PII` | No | `false` | Whether Sentry should send personally identifiable information |
+| `JUNIPER_DATA_SENTRY_TRACES_SAMPLE_RATE` | No | `0.1` | Sentry tracing sample rate |
+| `JUNIPER_DATA_METRICS_ENABLED` | No | `false` | Expose `/metrics` for Prometheus scraping |
+| `JUNIPER_DATA_METRICS_TRUSTED_IPS` | No | `["127.0.0.1", "::1"]` | IPs allowed to scrape `/metrics` |
+
+## Docker Deployment
+
+```bash
+# Full stack (recommended) — see juniper-deploy:
+git clone https://github.com/pcalnon/juniper-deploy.git  # (private repository)
+cd juniper-deploy && docker compose up --build
+
+# Standalone:
+docker build -t juniper-data:latest .
+docker run --rm -p 8100:8100 -e JUNIPER_DATA_HOST=0.0.0.0 juniper-data:latest
+```
+
+The Dockerfile is multi-stage (Python 3.14-slim builder + runtime). Container health is probed against `/v1/health/ready`.
+
+## Dependency Lockfile
+
+The `requirements.lock` file pins exact dependency versions for reproducible Docker builds. The `pyproject.toml` retains flexible `>=` ranges for local development.
+
+Regenerate after changing dependencies in `pyproject.toml`:
+
+```bash
+uv pip compile pyproject.toml --extra api --extra observability -o requirements.lock
+```
+
+The ecosystem-wide lockfile-freshness gate enforces regeneration on every PR that touches `pyproject.toml`; if regeneration triggers the self-pin trap of `uv pip compile -o requirements.lock` reading the existing file, compile to `/tmp/requirements.lock` and `mv` into place.
+
+## Active Research Components
+
+`juniper-data` contributes three research components to the Juniper platform: the **ARC-AGI dataset families** (ARC-AGI-1 and ARC-AGI-2), loadable from the Hugging Face Hub or from local copies and exposed through the same NPZ-artifact contract as the simpler generators, which makes them directly usable as the substrate for comparative architecture-growth experiments; the **named-version dataset registry** (`POST /v1/datasets` with a `name` parameter auto-increments `meta.dataset_version`; `GET /v1/datasets/versions` and `/v1/datasets/latest` resolve the history), which gives experiments reproducible dataset references rather than opaque UUIDs; and the **dataset-API surface** itself — preview, filtering by tags, batch operations, and tag-based metadata queries — which together comprise the operational interface through which platform users compose and curate dataset corpora. The implementation of these surfaces is engineering rather than research; the **availability** of curated datasets and stable versioned references is itself the research artifact.
+
+## Quick Start Guide
+
+### Prerequisites
+
+- Python ≥ 3.12 (Docker image uses 3.14)
+- Conda environment `JuniperData`
+- For ARC-AGI loading from the Hub: internet access at first load; subsequent loads are cached
+
+### Installation
+
+```bash
+git clone https://github.com/pcalnon/juniper-data.git
+cd juniper-data
+conda activate JuniperData
+pip install -e ".[all]"
+```
+
+The PyPI release is installable via `pip install juniper-data`; the editable-clone form above is the standard for active development. The optional-dependency extras are `api`, `arc-agi`, `observability`, `test`, `dev`, and `all`.
+
+### Verification
+
+Start the service:
+
+```bash
+uvicorn --factory juniper_data.api.app:get_app --reload
+```
+
+Confirm the service responds:
+
+```bash
+curl http://localhost:8100/v1/health
+curl http://localhost:8100/v1/health/ready
+curl http://localhost:8100/v1/generators
+```
+
+Generate a small dataset directly from Python:
+
+```python
+from juniper_data.generators.spiral import SpiralGenerator
+
+generator = SpiralGenerator()
+dataset = generator.generate(n_points=100, n_spirals=2, noise=0.1)
+```
+
+### Next Steps
+
+- [`docs/QUICK_START.md`](docs/QUICK_START.md) — complete installation guide
+- [`docs/USER_MANUAL.md`](docs/USER_MANUAL.md) — comprehensive usage guide
+- [`docs/api/JUNIPER_DATA_API.md`](docs/api/JUNIPER_DATA_API.md) — full REST endpoint reference (filtering, batch operations, tagging, versioning)
+- [`juniper-deploy`](https://github.com/pcalnon/juniper-deploy) — Docker Compose orchestration for the full-stack platform
+- [`juniper-ml`](https://pypi.org/project/juniper-ml/) — platform meta-package on PyPI
+
+## Research Philosophy
+
+The Juniper platform exists to study learning algorithms whose network architecture is not fixed in advance. Its initial anchor is the Cascade-Correlation algorithm of Fahlman and Lebiere (1990), implemented from the primary literature without recourse to higher-level abstractions that elide the algorithm's operational detail. The organising commitment is that algorithm implementations remain inspectable at the level at which they were originally specified: candidate units, correlation objectives, weight-freezing semantics, and the structural events that grow the network are first-class artifacts of the codebase rather than internal details of a library wrapper. This permits comparative work — across algorithms, datasets, and hyperparameter regimes — to be conducted on a known and reproducible substrate.
+
+The current platform comprises a Cascade-Correlation training service exposing a REST and WebSocket interface, a dataset-generation service with a named-version registry that includes the ARC-AGI families, a real-time monitoring dashboard for inspecting training dynamics as they occur, and a distributed worker that parallelises candidate-unit training across hosts. Near-term work extends the architectural-growth catalogue beyond Cascade-Correlation, introduces multi-network orchestration for comparative experiments at the level of network populations rather than individual runs, and tightens the dataset–training–monitoring loop into a reproducible research workbench. The longer-term direction is the systematic empirical study of constructive and architecture-growing learning algorithms, with first-class infrastructure for the ablation, comparison, and replication that such a study requires.
+
+## Documentation
+
+| Document | Purpose |
+|----------|---------|
+| [`docs/DOCUMENTATION_OVERVIEW.md`](docs/DOCUMENTATION_OVERVIEW.md) | Navigation index for all `juniper-data` documentation |
+| [`docs/QUICK_START.md`](docs/QUICK_START.md) | Get running in five minutes |
+| [`docs/USER_MANUAL.md`](docs/USER_MANUAL.md) | Comprehensive usage guide |
+| [`docs/REFERENCE.md`](docs/REFERENCE.md) | Configuration, environment variables, and operational reference |
+| [`docs/ENVIRONMENT_SETUP.md`](docs/ENVIRONMENT_SETUP.md) | Conda environment and editable-install setup |
+| [`docs/DEVELOPER_CHEATSHEET.md`](docs/DEVELOPER_CHEATSHEET.md) | Quick-reference card for development tasks |
+| [`docs/api/JUNIPER_DATA_API.md`](docs/api/JUNIPER_DATA_API.md) | Complete REST endpoint reference |
+| [`CHANGELOG.md`](CHANGELOG.md) | Version history |
+
+## License
+
+MIT License — Copyright (c) 2024-2026 Paul Calnon

juniper_data-0.7.0/README.md

@@ -0,0 +1,195 @@
+<!-- markdownlint-disable MD013 MD033 MD041 -->
+<!--
+  MD013 (line-length): README contains prose paragraphs that intentionally
+                       exceed the 512-char ecosystem limit. Disabled file-wide
+                       since wrapping mid-sentence harms PyPI rendering.
+  MD033 (no-inline-html): The right-aligned logo + spacing rely on HTML.
+  MD041 (first-line-heading): The HTML logo is the first line by design.
+-->
+<div align="right" width="150px" height="150px" align="right" valign="top"> <img src="images/Juniper_Logo_150px.png" alt="Juniper" align="right" valign="top" width="150px" /></div>
+<br /> <br /> <br /> <br />
+
+# Juniper: Dynamic Neural Network Research Platform
+
+Juniper is an AI/ML research platform for investigating dynamic neural network architectures and novel learning paradigms.  The project emphasizes ground-up implementations from primary literature, enabling a more transparent exploration of fundamental algorithms.
+
+## Juniper Data
+
+`juniper-data` is the **dataset-generation service** of the Juniper platform. It is a FastAPI service that produces NPZ-formatted datasets from a catalogue of generators — including the classic two-spiral and concentric-circles problems, XOR and Gaussian mixtures, a CSV/JSON import path, MNIST/Fashion-MNIST, and the ARC-AGI visual-reasoning task families — and serves them through a REST surface that supports a named-version registry, batch creation and export, tag-based filtering, and per-dataset preview. `juniper-data` is the upstream of both `juniper-cascor` (training) and `juniper-canopy` (visualisation): the dataset identifiers it returns are the substrate on which the rest of the platform conducts comparative work.
+
+## Distribution
+
+`juniper-data` is published on PyPI as **[`juniper-data`](https://pypi.org/project/juniper-data/)**.
+The package is also surfaced through the platform meta-distribution
+**[`juniper-ml`](https://pypi.org/project/juniper-ml/)**, which installs
+the full client stack via `pip install juniper-ml[all]`.
+
+```bash
+pip install juniper-data
+```
+
+## Ecosystem Compatibility
+
+This service is part of the [Juniper](https://github.com/pcalnon/juniper-ml) ecosystem.
+Verified compatible versions:
+
+| juniper-data | juniper-cascor | juniper-canopy | data-client | cascor-client | cascor-worker |
+|--------------|----------------|----------------|-------------|---------------|---------------|
+| 0.6.x        | 0.5.x          | 0.5.x          | >=0.4.1     | >=0.4.0       | >=0.4.0       |
+
+For full-stack Docker deployment and integration tests, see [`juniper-deploy`](https://github.com/pcalnon/juniper-deploy).
+
+## Architecture
+
+`juniper-data` is the **foundational data layer** of the Juniper ecosystem. Both `juniper-cascor` and `juniper-canopy` call `juniper-data` to generate, version, and retrieve datasets.
+
+```text
+┌─────────────────────┐     REST+WS      ┌──────────────────────┐
+│   juniper-canopy    │ ◄──────────────► │    juniper-cascor    │
+│   Dashboard         │                  │    Training Svc      │
+│   Port 8050         │                  │    Port 8200         │
+└──────────┬──────────┘                  └──────────┬───────────┘
+           │ REST                                   │ REST
+           ▼                                        ▼
+┌──────────────────────────────────────────────────────────────┐
+│                  juniper-data  ◄── (this service)            │
+│                  Dataset Service · Port 8100                 │
+└──────────────────────────────────────────────────────────────┘
+```
+
+**Data contract**: datasets are served as NPZ archives with the keys `X_train`, `y_train`, `X_test`, `y_test`, `X_full`, `y_full`, all of dtype `float32`.
+
+## Related Services
+
+| Service | Relationship | Notes |
+|---------|-------------|-------|
+| [juniper-cascor](https://github.com/pcalnon/juniper-cascor) | Consumes `juniper-data` for training datasets | Set `JUNIPER_DATA_URL` |
+| [juniper-canopy](https://github.com/pcalnon/juniper-canopy) | Consumes `juniper-data` for visualisation data | Set `JUNIPER_DATA_URL` |
+| [juniper-data-client](https://github.com/pcalnon/juniper-data-client) | Python HTTP client for this service | `pip install juniper-data-client` |
+
+## Service Configuration
+
+Configuration is sourced from `juniper_data/api/settings.py` (Pydantic `BaseSettings`, `env_prefix="JUNIPER_DATA_"`). The complete env-var surface is listed below.
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `JUNIPER_DATA_HOST` | No | `127.0.0.1` | Bind address (override to `0.0.0.0` for Docker) |
+| `JUNIPER_DATA_PORT` | No | `8100` | Service port |
+| `JUNIPER_DATA_STORAGE_PATH` | No | `./data/datasets` | Filesystem path for persisted dataset artifacts |
+| `JUNIPER_DATA_IMPORT_DIR` | No | `/data/imports` | Filesystem path for CSV/JSON imports |
+| `JUNIPER_DATA_LOG_LEVEL` | No | `INFO` | Log verbosity (`DEBUG`, `INFO`, `WARNING`, `ERROR`) |
+| `JUNIPER_DATA_LOG_FORMAT` | No | `text` | `text` or `json` (structured logging) |
+| `JUNIPER_DATA_CORS_ORIGINS` | No | `[]` | Allowed CORS origins |
+| `JUNIPER_DATA_API_KEYS` | No | `None` | Comma-separated or JSON-array API keys; authentication disabled when unset; Docker-secrets file path supported via the implicit `*_FILE` convention |
+| `JUNIPER_DATA_RATE_LIMIT_ENABLED` | No | `true` | Enforce per-IP request rate limiting |
+| `JUNIPER_DATA_RATE_LIMIT_REQUESTS_PER_MINUTE` | No | `60` | Per-IP rate limit |
+| `JUNIPER_DATA_SENTRY_DSN` | No | `None` | Sentry DSN for error tracking |
+| `JUNIPER_DATA_SENTRY_SEND_PII` | No | `false` | Whether Sentry should send personally identifiable information |
+| `JUNIPER_DATA_SENTRY_TRACES_SAMPLE_RATE` | No | `0.1` | Sentry tracing sample rate |
+| `JUNIPER_DATA_METRICS_ENABLED` | No | `false` | Expose `/metrics` for Prometheus scraping |
+| `JUNIPER_DATA_METRICS_TRUSTED_IPS` | No | `["127.0.0.1", "::1"]` | IPs allowed to scrape `/metrics` |
+
+## Docker Deployment
+
+```bash
+# Full stack (recommended) — see juniper-deploy:
+git clone https://github.com/pcalnon/juniper-deploy.git  # (private repository)
+cd juniper-deploy && docker compose up --build
+
+# Standalone:
+docker build -t juniper-data:latest .
+docker run --rm -p 8100:8100 -e JUNIPER_DATA_HOST=0.0.0.0 juniper-data:latest
+```
+
+The Dockerfile is multi-stage (Python 3.14-slim builder + runtime). Container health is probed against `/v1/health/ready`.
+
+## Dependency Lockfile
+
+The `requirements.lock` file pins exact dependency versions for reproducible Docker builds. The `pyproject.toml` retains flexible `>=` ranges for local development.
+
+Regenerate after changing dependencies in `pyproject.toml`:
+
+```bash
+uv pip compile pyproject.toml --extra api --extra observability -o requirements.lock
+```
+
+The ecosystem-wide lockfile-freshness gate enforces regeneration on every PR that touches `pyproject.toml`; if regeneration triggers the self-pin trap of `uv pip compile -o requirements.lock` reading the existing file, compile to `/tmp/requirements.lock` and `mv` into place.
+
+## Active Research Components
+
+`juniper-data` contributes three research components to the Juniper platform: the **ARC-AGI dataset families** (ARC-AGI-1 and ARC-AGI-2), loadable from the Hugging Face Hub or from local copies and exposed through the same NPZ-artifact contract as the simpler generators, which makes them directly usable as the substrate for comparative architecture-growth experiments; the **named-version dataset registry** (`POST /v1/datasets` with a `name` parameter auto-increments `meta.dataset_version`; `GET /v1/datasets/versions` and `/v1/datasets/latest` resolve the history), which gives experiments reproducible dataset references rather than opaque UUIDs; and the **dataset-API surface** itself — preview, filtering by tags, batch operations, and tag-based metadata queries — which together comprise the operational interface through which platform users compose and curate dataset corpora. The implementation of these surfaces is engineering rather than research; the **availability** of curated datasets and stable versioned references is itself the research artifact.
+
+## Quick Start Guide
+
+### Prerequisites
+
+- Python ≥ 3.12 (Docker image uses 3.14)
+- Conda environment `JuniperData`
+- For ARC-AGI loading from the Hub: internet access at first load; subsequent loads are cached
+
+### Installation
+
+```bash
+git clone https://github.com/pcalnon/juniper-data.git
+cd juniper-data
+conda activate JuniperData
+pip install -e ".[all]"
+```
+
+The PyPI release is installable via `pip install juniper-data`; the editable-clone form above is the standard for active development. The optional-dependency extras are `api`, `arc-agi`, `observability`, `test`, `dev`, and `all`.
+
+### Verification
+
+Start the service:
+
+```bash
+uvicorn --factory juniper_data.api.app:get_app --reload
+```
+
+Confirm the service responds:
+
+```bash
+curl http://localhost:8100/v1/health
+curl http://localhost:8100/v1/health/ready
+curl http://localhost:8100/v1/generators
+```
+
+Generate a small dataset directly from Python:
+
+```python
+from juniper_data.generators.spiral import SpiralGenerator
+
+generator = SpiralGenerator()
+dataset = generator.generate(n_points=100, n_spirals=2, noise=0.1)
+```
+
+### Next Steps
+
+- [`docs/QUICK_START.md`](docs/QUICK_START.md) — complete installation guide
+- [`docs/USER_MANUAL.md`](docs/USER_MANUAL.md) — comprehensive usage guide
+- [`docs/api/JUNIPER_DATA_API.md`](docs/api/JUNIPER_DATA_API.md) — full REST endpoint reference (filtering, batch operations, tagging, versioning)
+- [`juniper-deploy`](https://github.com/pcalnon/juniper-deploy) — Docker Compose orchestration for the full-stack platform
+- [`juniper-ml`](https://pypi.org/project/juniper-ml/) — platform meta-package on PyPI
+
+## Research Philosophy
+
+The Juniper platform exists to study learning algorithms whose network architecture is not fixed in advance. Its initial anchor is the Cascade-Correlation algorithm of Fahlman and Lebiere (1990), implemented from the primary literature without recourse to higher-level abstractions that elide the algorithm's operational detail. The organising commitment is that algorithm implementations remain inspectable at the level at which they were originally specified: candidate units, correlation objectives, weight-freezing semantics, and the structural events that grow the network are first-class artifacts of the codebase rather than internal details of a library wrapper. This permits comparative work — across algorithms, datasets, and hyperparameter regimes — to be conducted on a known and reproducible substrate.
+
+The current platform comprises a Cascade-Correlation training service exposing a REST and WebSocket interface, a dataset-generation service with a named-version registry that includes the ARC-AGI families, a real-time monitoring dashboard for inspecting training dynamics as they occur, and a distributed worker that parallelises candidate-unit training across hosts. Near-term work extends the architectural-growth catalogue beyond Cascade-Correlation, introduces multi-network orchestration for comparative experiments at the level of network populations rather than individual runs, and tightens the dataset–training–monitoring loop into a reproducible research workbench. The longer-term direction is the systematic empirical study of constructive and architecture-growing learning algorithms, with first-class infrastructure for the ablation, comparison, and replication that such a study requires.
+
+## Documentation
+
+| Document | Purpose |
+|----------|---------|
+| [`docs/DOCUMENTATION_OVERVIEW.md`](docs/DOCUMENTATION_OVERVIEW.md) | Navigation index for all `juniper-data` documentation |
+| [`docs/QUICK_START.md`](docs/QUICK_START.md) | Get running in five minutes |
+| [`docs/USER_MANUAL.md`](docs/USER_MANUAL.md) | Comprehensive usage guide |
+| [`docs/REFERENCE.md`](docs/REFERENCE.md) | Configuration, environment variables, and operational reference |
+| [`docs/ENVIRONMENT_SETUP.md`](docs/ENVIRONMENT_SETUP.md) | Conda environment and editable-install setup |
+| [`docs/DEVELOPER_CHEATSHEET.md`](docs/DEVELOPER_CHEATSHEET.md) | Quick-reference card for development tasks |
+| [`docs/api/JUNIPER_DATA_API.md`](docs/api/JUNIPER_DATA_API.md) | Complete REST endpoint reference |
+| [`CHANGELOG.md`](CHANGELOG.md) | Version history |
+
+## License
+
+MIT License — Copyright (c) 2024-2026 Paul Calnon

{juniper_data-0.6.0 → juniper_data-0.7.0}/juniper_data/__init__.py

@@ -2,6 +2,7 @@
 Juniper Data - Dataset generation and management service for the Juniper ecosystem.
 """
 
+import importlib.metadata
 import os
 
 from dotenv import load_dotenv
@@ -14,7 +15,14 @@ except ImportError:
     ARC_AGI_AVAILABLE = False
     arc_agi = None  # type: ignore[assignment]
 
-__version__ = "0.6.0"
+# Single source of truth: the installed distribution's metadata (OQ-1 of the
+# build-provenance effort — juniper-ml notes/BUILD_PROVENANCE_DESIGN_2026-06-14.md).
+# Falls back to the literal only in a bare source checkout where the package is
+# not installed, so this constant can no longer drift from pyproject's version.
+try:
+    __version__ = importlib.metadata.version("juniper-data")
+except importlib.metadata.PackageNotFoundError:  # pragma: no cover - source checkout
+    __version__ = "0.7.0"
 __author__ = "Paul Calnon"
 
 

{juniper_data-0.6.0 → juniper_data-0.7.0}/juniper_data/__main__.py

@@ -64,7 +64,8 @@ def main() -> int:
         os.environ["JUNIPER_DATA_STORAGE_PATH"] = args.storage_path
 
     uvicorn.run(
-        "juniper_data.api.app:app",
+        "juniper_data.api.app:get_app",
+        factory=True,
         host=host,
         port=port,
         log_level=log_level,

{juniper_data-0.6.0 → juniper_data-0.7.0}/juniper_data/api/app.py

@@ -1,5 +1,6 @@
 """FastAPI application factory and configuration."""
 
+import functools
 import logging
 from collections.abc import AsyncGenerator
 from contextlib import asynccontextmanager
@@ -8,12 +9,14 @@ from pathlib import Path
 from fastapi import FastAPI, Request
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse
+from starlette import status
 
-from juniper_data import __version__
+from juniper_data import __version__, provenance
 from juniper_data.storage import LocalFSDatasetStore
 
 from .middleware import RequestBodyLimitMiddleware, SecurityHeadersMiddleware, SecurityMiddleware
 from .observability import (
+    MetricsAuthMiddleware,
     PrometheusMiddleware,
     RequestIdMiddleware,
     configure_logging,
@@ -37,11 +40,17 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
     configure_logging(settings.log_level, settings.log_format, "juniper-data")
     configure_sentry(settings.sentry_dsn, "juniper-data", __version__, send_pii=settings.sentry_send_pii, traces_sample_rate=settings.sentry_traces_sample_rate)
     if settings.metrics_enabled:
-        set_build_info("juniper_data", __version__)
+        set_build_info("juniper_data", __version__, git_sha=provenance.git_sha(), build_date=provenance.build_date())
 
     logger = logging.getLogger("juniper_data")
     logger.info(f"JuniperData API v{__version__} starting")
-    logger.info(f"Storage path: {storage_path.absolute()}")
+    # ``Path.absolute()`` is pure path manipulation (no I/O); the
+    # ASYNC240 rule is over-conservative here and flags every
+    # ``pathlib.Path`` method without distinguishing stat-bound ones
+    # from text-only ones. Lifespan startup is also a one-shot
+    # event, not a request handler — even if there were I/O it
+    # wouldn't block per-request latency.
+    logger.info(f"Storage path: {storage_path.absolute()}")  # noqa: ASYNC240
 
     yield
 
@@ -115,15 +124,19 @@ def create_app(settings: Settings | None = None) -> FastAPI:
     app.include_router(generators.router, prefix="/v1")
     app.include_router(datasets.router, prefix="/v1")
 
-    # Mount Prometheus metrics endpoint
+    # Mount Prometheus metrics endpoint (SEC-16: wrap with trusted-IP
+    # auth because ASGI sub-app mounts bypass SecurityMiddleware).
     if settings.metrics_enabled:
-        app.mount("/metrics", get_prometheus_app())
+        app.mount(
+            "/metrics",
+            MetricsAuthMiddleware(get_prometheus_app(), settings.metrics_trusted_ips),
+        )
 
     @app.exception_handler(ValueError)
     async def value_error_handler(request: Request, exc: ValueError) -> JSONResponse:
         logging.getLogger("juniper_data").debug("Validation error: %s", exc)
         return JSONResponse(
-            status_code=400,
+            status_code=status.HTTP_400_BAD_REQUEST,
             content={"detail": "Invalid request parameters"},
         )
 
@@ -131,11 +144,31 @@ def create_app(settings: Settings | None = None) -> FastAPI:
     async def general_exception_handler(request: Request, exc: Exception) -> JSONResponse:
         logging.getLogger("juniper_data").exception("Unhandled exception")
         return JSONResponse(
-            status_code=500,
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
             content={"detail": "Internal server error"},
         )
 
     return app
 
 
-app = create_app()
+@functools.lru_cache(maxsize=1)
+def get_app() -> FastAPI:
+    """Return the singleton FastAPI app instance (lazy factory).
+
+    Use with uvicorn's factory mode::
+
+        uvicorn --factory juniper_data.api.app:get_app
+
+    or programmatically::
+
+        uvicorn.run("juniper_data.api.app:get_app", factory=True)
+
+    The first call builds the app via :func:`create_app` with default
+    settings; subsequent calls return the same instance from
+    ``functools.lru_cache``. Replaces the previous module-level
+    ``app = create_app()`` (CLN-JD-03), which read environment variables
+    and registered middleware at import time. Tests that need a fresh
+    instance with overridden settings should continue to call
+    :func:`create_app` directly with explicit ``Settings``.
+    """
+    return create_app()