lumilake 0.1.0.dev1__tar.gz → 0.1.2__tar.gz

{lumilake-0.1.0.dev1 → lumilake-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lumilake
-Version: 0.1.0.dev1
+Version: 0.1.2
 Summary: Data analytics engine for agentic workflows
 Author: Lumilake contributors
 License-Expression: Apache-2.0
@@ -20,19 +20,19 @@ Requires-Python: <3.13,>=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Provides-Extra: sdk
-Requires-Dist: lumilake-sdk==0.1.0.dev1; extra == "sdk"
+Requires-Dist: lumilake-sdk==0.1.2; extra == "sdk"
 Provides-Extra: cli
-Requires-Dist: lumilake-cli==0.1.0.dev1; extra == "cli"
-Requires-Dist: lumilake-deploy==0.1.0.dev1; extra == "cli"
+Requires-Dist: lumilake-cli==0.1.2; extra == "cli"
+Requires-Dist: lumilake-deploy==0.1.2; extra == "cli"
 Provides-Extra: deploy
-Requires-Dist: lumilake-deploy==0.1.0.dev1; extra == "deploy"
+Requires-Dist: lumilake-deploy==0.1.2; extra == "deploy"
 Provides-Extra: hook
-Requires-Dist: lumilake-hook==0.1.0.dev1; extra == "hook"
+Requires-Dist: lumilake-hook==0.1.2; extra == "hook"
 Provides-Extra: all
-Requires-Dist: lumilake-sdk==0.1.0.dev1; extra == "all"
-Requires-Dist: lumilake-cli==0.1.0.dev1; extra == "all"
-Requires-Dist: lumilake-deploy==0.1.0.dev1; extra == "all"
-Requires-Dist: lumilake-hook==0.1.0.dev1; extra == "all"
+Requires-Dist: lumilake-sdk==0.1.2; extra == "all"
+Requires-Dist: lumilake-cli==0.1.2; extra == "all"
+Requires-Dist: lumilake-deploy==0.1.2; extra == "all"
+Requires-Dist: lumilake-hook==0.1.2; extra == "all"
 Dynamic: license-file
 
 # Lumilake
@@ -87,12 +87,50 @@ lumilake deploy -C ~/lumilake-deploy init --flowmesh   # ~/lumilake-deploy/.env
 $EDITOR ~/lumilake-deploy/.env                          # fill in DATABASE_URL / S3 / model keys
 lumilake deploy -C ~/lumilake-deploy pull               # fetch ghcr.io/mlsys-io/lumilake_server:<tag>
 lumilake deploy -C ~/lumilake-deploy up                 # bring the stack up via docker compose
+lumilake deploy -C ~/lumilake-deploy purge dev --dry-run # preview cleanup of one local image tag
 ```
 
 `LUMILAKE_DEPLOY_DIR=~/lumilake-deploy` is an equivalent override. The deployment directory only needs to hold your `.env` files (and any local state docker compose creates) — the compose file and server image are resolved from the installed `lumilake-deploy` package and GHCR. The server listens on `http://127.0.0.1:9000` by default — open `/docs` for the API browser.
 
 Note: a real workflow run also requires running PostgreSQL and S3-compatible storage; agent-style retrievals (`DataRetrievalOp` with `type: agent`) additionally require `LUMID_DATA_URL`. See `docs/ENV.md` for the env contract. If you don't have your own data plane, the repo ships a bundled Postgres + MinIO at `scripts/dev/compose.data-plane.yml` — see `docs/E2E_DEMO.md` for the full three-step demo flow (data plane → load demo data → run a workflow).
 
+### Hello world
+
+The repo ships a `hello-world.yaml` template — `FormatOp` →
+`LambdaOp` → `LLMChatOp` — that is the smallest copy-paste starting
+point for a Lumilake YAML workflow. Submit it once the stack is up and
+`S3_URL` points at reachable S3-compatible storage. From a source
+checkout, start the bundled data plane first:
+
+```bash
+docker compose -f scripts/dev/compose.data-plane.yml up -d
+```
+
+PyPI installs do not include `scripts/dev/`; use your own reachable S3
+endpoint, or download the repo's data-plane compose file alongside the
+workflow template before running the local-only example.
+
+```bash
+# From a source checkout:
+uv run lumilake job submit examples/templates/yaml/hello-world.yaml \
+    --format yaml --input 'Name=world' --output-prefix demo/hello-world
+
+# From a PyPI install (download the template alongside lumilake):
+curl -O https://raw.githubusercontent.com/mlsys-io/lumilake_OSS/main/examples/templates/yaml/hello-world.yaml
+lumilake job submit hello-world.yaml \
+    --format yaml --input 'Name=world' --output-prefix demo/hello-world
+lumilake job watch <job_id>
+lumilake job result <job_id>
+```
+
+The template uses `Qwen/Qwen3-8B`, which is the bundled text-demo model.
+You do not need to pre-populate or inspect `cached_models` before the
+first run; it can be empty until after a worker serves a job. Only edit
+`config.model` if your FlowMesh stack is configured for a different
+model or the job fails with a missing-model / worker-placement error.
+
+### Real workflows
+
 Submit and inspect a workflow. From a source checkout the example
 workflow file is at `examples/templates/yaml/trading-agent.yaml`;
 PyPI installs do not ship the templates, so pass an absolute path to a
@@ -100,12 +138,10 @@ workflow file you have locally:
 
 ```bash
 # From a source checkout:
-uv run lumilake login http://127.0.0.1:9000
 uv run lumilake job submit examples/templates/yaml/trading-agent.yaml \
     --format yaml --input 'Stock=NVDA,AAPL,MSFT' --output-prefix demo/trading-agent
 
 # From a PyPI install (lumilake on PATH; supply your own workflow file):
-lumilake login http://127.0.0.1:9000
 lumilake job submit /path/to/your/workflow.yaml \
     --format yaml --input 'Stock=NVDA,AAPL,MSFT' --output-prefix demo/trading-agent
 
@@ -113,6 +149,10 @@ lumilake job list
 lumilake job watch <job_id>
 ```
 
+`lumilake deploy up` writes `~/.lumilake/config.toml` so subsequent
+calls find the local server automatically. For remote / hosted servers,
+set `LUMILAKE_BASE_URL` instead.
+
 See `docs/E2E_DEMO.md` for a full reproduction using the bundled demo
 workflows and dataset.
 
@@ -140,6 +180,14 @@ Generate both Lumilake and bundled FlowMesh env files:
 lumilake deploy init --flowmesh
 ```
 
+If another FlowMesh stack is already running on the same host, check
+ports before `deploy up`. Common co-tenant FlowMesh defaults are HTTP
+`8000`, gRPC `50051`, Redis control `6379`, and Redis telemetry `6380`.
+The bundled stack reads `SERVER_HTTP_PORT`, `SERVER_GRPC_PORT`,
+`REDIS_CONTROL_PORT`, and `REDIS_TELEMETRY_PORT` from `.env.flowmesh`;
+change them to free ports and keep `LUMILAKE_RUNTIME_ORCHESTRATOR_URL`
+in `.env` aligned with `SERVER_HTTP_PORT`.
+
 Common deployment commands:
 
 ```bash
@@ -153,7 +201,7 @@ lumilake deploy down
 lumilake deploy clean
 ```
 
-Use `deploy down` to stop services while keeping data volumes. Use `deploy clean` or `deploy reset` only when you want to remove local stack state.
+Use `deploy down` to stop services while keeping data volumes (non-destructive). Use `deploy clean` or `deploy reset` only when you want to remove local stack state; both delete every Lumilake-managed volume, and `reset` prompts for confirmation by default (`--yes` skips the prompt).
 
 ## Python SDK
 
@@ -221,7 +269,7 @@ A minimal in-memory plugin is available under `examples/plugins/simple_plugin/`.
 uv sync --group lint --group test --extra cli
 uv run pre-commit install --install-hooks -t pre-commit -t prepare-commit-msg -t commit-msg
 uv run pre-commit run --all-files
-uv run pytest tests/ --ignore=tests/server
+uv run pytest tests/
 ```
 
 After changing dependencies, run:

lumilake-0.1.0.dev1/lumilake.egg-info/PKG-INFO → lumilake-0.1.2/README.md

@@ -1,40 +1,3 @@
-Metadata-Version: 2.4
-Name: lumilake
-Version: 0.1.0.dev1
-Summary: Data analytics engine for agentic workflows
-Author: Lumilake contributors
-License-Expression: Apache-2.0
-Project-URL: Homepage, https://github.com/mlsys-io/lumilake_OSS
-Project-URL: Repository, https://github.com/mlsys-io/lumilake_OSS
-Project-URL: Issues, https://github.com/mlsys-io/lumilake_OSS/issues
-Keywords: agentic-workflows,data-analytics,lakehouse,workflow-orchestration
-Classifier: Development Status :: 3 - Alpha
-Classifier: Environment :: Console
-Classifier: Framework :: FastAPI
-Classifier: Intended Audience :: Developers
-Classifier: Intended Audience :: Science/Research
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Typing :: Typed
-Requires-Python: <3.13,>=3.12
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Provides-Extra: sdk
-Requires-Dist: lumilake-sdk==0.1.0.dev1; extra == "sdk"
-Provides-Extra: cli
-Requires-Dist: lumilake-cli==0.1.0.dev1; extra == "cli"
-Requires-Dist: lumilake-deploy==0.1.0.dev1; extra == "cli"
-Provides-Extra: deploy
-Requires-Dist: lumilake-deploy==0.1.0.dev1; extra == "deploy"
-Provides-Extra: hook
-Requires-Dist: lumilake-hook==0.1.0.dev1; extra == "hook"
-Provides-Extra: all
-Requires-Dist: lumilake-sdk==0.1.0.dev1; extra == "all"
-Requires-Dist: lumilake-cli==0.1.0.dev1; extra == "all"
-Requires-Dist: lumilake-deploy==0.1.0.dev1; extra == "all"
-Requires-Dist: lumilake-hook==0.1.0.dev1; extra == "all"
-Dynamic: license-file
-
 # Lumilake
 
 [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)
@@ -87,12 +50,50 @@ lumilake deploy -C ~/lumilake-deploy init --flowmesh   # ~/lumilake-deploy/.env
 $EDITOR ~/lumilake-deploy/.env                          # fill in DATABASE_URL / S3 / model keys
 lumilake deploy -C ~/lumilake-deploy pull               # fetch ghcr.io/mlsys-io/lumilake_server:<tag>
 lumilake deploy -C ~/lumilake-deploy up                 # bring the stack up via docker compose
+lumilake deploy -C ~/lumilake-deploy purge dev --dry-run # preview cleanup of one local image tag
 ```
 
 `LUMILAKE_DEPLOY_DIR=~/lumilake-deploy` is an equivalent override. The deployment directory only needs to hold your `.env` files (and any local state docker compose creates) — the compose file and server image are resolved from the installed `lumilake-deploy` package and GHCR. The server listens on `http://127.0.0.1:9000` by default — open `/docs` for the API browser.
 
 Note: a real workflow run also requires running PostgreSQL and S3-compatible storage; agent-style retrievals (`DataRetrievalOp` with `type: agent`) additionally require `LUMID_DATA_URL`. See `docs/ENV.md` for the env contract. If you don't have your own data plane, the repo ships a bundled Postgres + MinIO at `scripts/dev/compose.data-plane.yml` — see `docs/E2E_DEMO.md` for the full three-step demo flow (data plane → load demo data → run a workflow).
 
+### Hello world
+
+The repo ships a `hello-world.yaml` template — `FormatOp` →
+`LambdaOp` → `LLMChatOp` — that is the smallest copy-paste starting
+point for a Lumilake YAML workflow. Submit it once the stack is up and
+`S3_URL` points at reachable S3-compatible storage. From a source
+checkout, start the bundled data plane first:
+
+```bash
+docker compose -f scripts/dev/compose.data-plane.yml up -d
+```
+
+PyPI installs do not include `scripts/dev/`; use your own reachable S3
+endpoint, or download the repo's data-plane compose file alongside the
+workflow template before running the local-only example.
+
+```bash
+# From a source checkout:
+uv run lumilake job submit examples/templates/yaml/hello-world.yaml \
+    --format yaml --input 'Name=world' --output-prefix demo/hello-world
+
+# From a PyPI install (download the template alongside lumilake):
+curl -O https://raw.githubusercontent.com/mlsys-io/lumilake_OSS/main/examples/templates/yaml/hello-world.yaml
+lumilake job submit hello-world.yaml \
+    --format yaml --input 'Name=world' --output-prefix demo/hello-world
+lumilake job watch <job_id>
+lumilake job result <job_id>
+```
+
+The template uses `Qwen/Qwen3-8B`, which is the bundled text-demo model.
+You do not need to pre-populate or inspect `cached_models` before the
+first run; it can be empty until after a worker serves a job. Only edit
+`config.model` if your FlowMesh stack is configured for a different
+model or the job fails with a missing-model / worker-placement error.
+
+### Real workflows
+
 Submit and inspect a workflow. From a source checkout the example
 workflow file is at `examples/templates/yaml/trading-agent.yaml`;
 PyPI installs do not ship the templates, so pass an absolute path to a
@@ -100,12 +101,10 @@ workflow file you have locally:
 
 ```bash
 # From a source checkout:
-uv run lumilake login http://127.0.0.1:9000
 uv run lumilake job submit examples/templates/yaml/trading-agent.yaml \
     --format yaml --input 'Stock=NVDA,AAPL,MSFT' --output-prefix demo/trading-agent
 
 # From a PyPI install (lumilake on PATH; supply your own workflow file):
-lumilake login http://127.0.0.1:9000
 lumilake job submit /path/to/your/workflow.yaml \
     --format yaml --input 'Stock=NVDA,AAPL,MSFT' --output-prefix demo/trading-agent
 
@@ -113,6 +112,10 @@ lumilake job list
 lumilake job watch <job_id>
 ```
 
+`lumilake deploy up` writes `~/.lumilake/config.toml` so subsequent
+calls find the local server automatically. For remote / hosted servers,
+set `LUMILAKE_BASE_URL` instead.
+
 See `docs/E2E_DEMO.md` for a full reproduction using the bundled demo
 workflows and dataset.
 
@@ -140,6 +143,14 @@ Generate both Lumilake and bundled FlowMesh env files:
 lumilake deploy init --flowmesh
 ```
 
+If another FlowMesh stack is already running on the same host, check
+ports before `deploy up`. Common co-tenant FlowMesh defaults are HTTP
+`8000`, gRPC `50051`, Redis control `6379`, and Redis telemetry `6380`.
+The bundled stack reads `SERVER_HTTP_PORT`, `SERVER_GRPC_PORT`,
+`REDIS_CONTROL_PORT`, and `REDIS_TELEMETRY_PORT` from `.env.flowmesh`;
+change them to free ports and keep `LUMILAKE_RUNTIME_ORCHESTRATOR_URL`
+in `.env` aligned with `SERVER_HTTP_PORT`.
+
 Common deployment commands:
 
 ```bash
@@ -153,7 +164,7 @@ lumilake deploy down
 lumilake deploy clean
 ```
 
-Use `deploy down` to stop services while keeping data volumes. Use `deploy clean` or `deploy reset` only when you want to remove local stack state.
+Use `deploy down` to stop services while keeping data volumes (non-destructive). Use `deploy clean` or `deploy reset` only when you want to remove local stack state; both delete every Lumilake-managed volume, and `reset` prompts for confirmation by default (`--yes` skips the prompt).
 
 ## Python SDK
 
@@ -221,7 +232,7 @@ A minimal in-memory plugin is available under `examples/plugins/simple_plugin/`.
 uv sync --group lint --group test --extra cli
 uv run pre-commit install --install-hooks -t pre-commit -t prepare-commit-msg -t commit-msg
 uv run pre-commit run --all-files
-uv run pytest tests/ --ignore=tests/server
+uv run pytest tests/
 ```
 
 After changing dependencies, run:

{lumilake-0.1.0.dev1 → lumilake-0.1.2}/docs/API.md

@@ -49,3 +49,13 @@ Most API responses use an envelope:
 ```
 
 Errors use normal HTTP status codes with a JSON error body. The Python SDK unwraps successful envelopes and raises typed exceptions for HTTP failures.
+
+### Error Status Codes
+
+- `400 Bad Request` — malformed request syntax (the JSON or YAML body could not be parsed).
+- `404 Not Found` — the resource is missing.
+- `409 Conflict` — the resource is in a state that cannot satisfy the request; the cancel endpoint includes the job's terminal `status` (`completed` / `failed` / `cancelled`) in `detail`.
+- `422 Unprocessable Entity` — the payload was syntactically valid but failed validation (schema, header value, batch-size, input shape, unresolved location, duplicate graph name, graph compilation, ...).
+- `5xx` — server-side faults (DB unavailable, runtime backend error).
+
+The `X-Request-ID` response header echoes the request trace id (either the inbound `X-Request-ID` header or a freshly minted `req-<uid>` token); the same id is attached to every server log record produced under that request.

lumilake-0.1.2/docs/CLI.md

@@ -0,0 +1,125 @@
+# CLI Reference
+
+Install the CLI extra or use `uv run` from a source checkout:
+
+```bash
+pip install "lumilake[cli]"     # full CLI + deploy lifecycle
+lumilake --help
+```
+
+```bash
+uv run lumilake --help
+```
+
+`pip install lumilake-cli` (no extra) installs a thin CLI that omits the
+deploy lifecycle — useful for users that only need
+`job` / `worker` / `trace` against a remote server. Running
+`lumilake deploy` from a thin install prints an install hint pointing
+back at `lumilake[cli]` or `lumilake-cli[deploy]`.
+
+## Configuration
+
+The CLI resolves the target server URL in this order:
+
+1. `LUMILAKE_BASE_URL` environment variable.
+2. `~/.lumilake/config.toml` (written by `lumilake deploy up`).
+3. `http://127.0.0.1:9000` (the local deploy default).
+
+| Command | Purpose |
+|---------|---------|
+| `lumilake config` | Show the resolved base URL and where it came from. |
+| `lumilake info` | Query server health metadata. |
+| `lumilake health` | Query server health. |
+
+`lumilake deploy up` writes the resolved local server URL to `~/.lumilake/config.toml` after the stack is healthy (preserving any saved `api_key`), so subsequent CLI and SDK calls find it automatically. Remote / hosted users should set `LUMILAKE_BASE_URL` instead.
+
+### Authentication
+
+When the server runs an `IdentityProvider` plugin (e.g. `lumid_lumilake_plugin`), the CLI presents a bearer token via `Authorization: Bearer <api_key>`. Three commands manage it:
+
+| Command | Purpose |
+|---------|---------|
+| `lumilake init <url> --api-key <key> [--config <path>] [--force]` | Save `base_url` + `api_key` to `~/.lumilake/config.toml`. Confirms overwrite unless `--force`. |
+| `lumilake deinit [--config <path>]` | Delete the config file. |
+| `lumilake config [--source auto\|file\|env] [--show-api-key]` | Show the resolved config. `api_key` is redacted (first 4 + last 4) unless `--show-api-key`. |
+
+Resolution order matches the SDK: explicit args > `LUMILAKE_BASE_URL` / `LUMILAKE_API_KEY` env > `~/.lumilake/config.toml` > local defaults. The CLI sends no `Authorization` header when `api_key` resolves to nothing — fine for local-mode deploys, rejected by deploys running with `LUMILAKE_REQUIRE_IDENTITY_PROVIDER=1`.
+
+## Deploy
+
+Every `lumilake deploy` command reads `.env` (and optionally `.env.flowmesh`) from `--project-dir` (`-C <path>`) or, when not given, the current working directory. `LUMILAKE_DEPLOY_DIR` is an equivalent override.
+
+```bash
+mkdir -p ~/lumilake-deploy
+lumilake deploy -C ~/lumilake-deploy init   # writes ~/lumilake-deploy/.env
+# or:
+LUMILAKE_DEPLOY_DIR=~/lumilake-deploy lumilake deploy init
+```
+
+`init` previews the file (or a unified diff against an existing one) before asking for confirmation; pass `--force` to skip the preview and prompt.
+
+The compose file and image are resolved from the installed `lumilake-deploy` package and GHCR; the deployment directory only needs to hold your `.env` files (and any local state docker compose creates).
+
+If another FlowMesh stack is already running on the host, resolve port
+collisions before `deploy up`. Common co-tenant defaults are FlowMesh
+HTTP `8000`, gRPC `50051`, Redis control `6379`, and Redis telemetry
+`6380`; the bundled stack reads the override keys documented in
+`docs/ENV.md` from `.env.flowmesh`.
+
+| Command | Purpose |
+|---------|---------|
+| `lumilake deploy init [--flowmesh] [--force]` | Create `.env` (preview + confirm); optionally also `.env.flowmesh`. |
+| `lumilake deploy doctor [--flowmesh]` | Validate deployment env files. |
+| `lumilake deploy build` | Build the Lumilake server Docker image from source. |
+| `lumilake deploy pull` | Pull the published server image from the registry (`$LUMILAKE_REGISTRY`). |
+| `lumilake deploy up` | Start the local stack and persist `base_url` to `~/.lumilake/config.toml` (image must already be present — run `pull` or `build` first). |
+| `lumilake deploy down [--wipe-archive]` | Stop the stack but keep data volumes. Non-destructive. |
+| `lumilake deploy clean` | Stop the stack and delete volumes. Destructive. |
+| `lumilake deploy purge <image_tag> [--dry-run] [--force]` | Remove one local Lumilake server image tag for the configured registry. |
+| `lumilake deploy reset [--yes]` | Wipe every volume, then start the stack again. Destructive. |
+| `lumilake deploy status` | Show known stack container state. |
+| `lumilake deploy restart [service]` | Restart one service or the full stack. |
+| `lumilake deploy logs [service]` | Stream or tail service logs. |
+| `lumilake deploy update-flowmesh` | Re-lock and install the latest FlowMesh packages. |
+
+### `deploy down` vs `deploy reset`
+
+| Command | Stops services | Removes data volumes |
+|---------|----------------|----------------------|
+| `lumilake deploy down` | yes | no — archive (job records, run artifacts), compute Postgres, and MinIO corpus data survive, so `deploy up` resumes against the same state. `--wipe-archive` additionally wipes compute Postgres and FlowMesh runtime-state volumes while preserving MinIO corpus data. |
+| `lumilake deploy clean` | yes | yes (every Lumilake-managed volume) |
+| `lumilake deploy reset` | yes | yes (every Lumilake-managed volume), then re-runs `deploy up`. Prompts for confirmation; pass `--yes` to skip the prompt in scripts. |
+
+Use `deploy down` between sessions. Reach for `clean` / `reset` only when you intentionally want to drop the local stack state.
+Use `deploy purge <image_tag> --dry-run` to preview image cleanup; it removes
+local server image tags only, not containers or volumes.
+
+## Jobs
+
+| Command | Purpose |
+|---------|---------|
+| `lumilake job submit <workflow>` | Submit a workflow spec. |
+| `lumilake job preview <workflow>` | Compile and schedule a workflow without dispatching runtime work. |
+| `lumilake job list` | List jobs. |
+| `lumilake job info <job_id>` | Show one job. |
+| `lumilake job progress <job_id>` | Show detailed progress. |
+| `lumilake job result <job_id>` | Show the final result. |
+| `lumilake job inputs <job_id>` | Show resolved job inputs. |
+| `lumilake job watch <job_id>` | Watch status until completion or failure. |
+| `lumilake job cancel <job_id>` | Cancel a job. |
+| `lumilake job artifact <job_id> --path <path> --output <file>` | Download a stored job artifact. |
+
+Run `uv run lumilake job submit --help` for format, input, and output flags.
+
+## Workers and Traces
+
+| Command | Purpose |
+|---------|---------|
+| `lumilake worker list` | List runtime workers. |
+| `lumilake worker get <worker_id>` | Show one worker. |
+| `lumilake trace list` | List execution traces. |
+| `lumilake trace get <exec_id>` | Show a trace summary, JSON payload, or Mermaid graph. |
+
+## `--json` flag
+
+Most query commands (`lumilake job list/info/progress/result/inputs`, `lumilake worker list/get`, `lumilake trace list`) emit the server's JSON envelope by default, so scripts can parse them directly. `lumilake trace get` defaults to a human-readable summary table; pass `--json` (a shortcut for `--format json`) for parse-friendly output.

{lumilake-0.1.0.dev1 → lumilake-0.1.2}/docs/CODE_STYLE.md

@@ -53,8 +53,7 @@ This guide collects rules agents and contributors should apply before editing so
 
 ## Tests
 
-- Default unit validation is `uv run pytest tests/ --ignore=tests/server`.
-- `tests/server` exercises live-server behavior and is reserved for pre-release validation.
+- Default unit validation is `uv run pytest tests/`. CI also collects coverage via `pytest-cov` against the SDK, CLI, deploy, hook, and server packages.
 - When changing runtime behavior, add or update a focused test in the same PR.
 - Do not ignore failing checks. A red test on `main` is a CI gap to close, not a reason to skip validation.
 

{lumilake-0.1.0.dev1 → lumilake-0.1.2}/docs/E2E_DEMO.md

@@ -98,9 +98,9 @@ docker compose -f scripts/dev/compose.data-plane.yml down -v
 ```
 
 **Skip this step entirely** if you have your own Postgres and S3 — just
-set `DATABASE_URL`, `S3_URL`, and `S3_USER_DATA_PREFIX` in your `.env`
-to your own endpoints. Nothing in step 2 or 3 reads the bundled
-defaults if you've overridden them.
+set `DATABASE_URL` and `S3_URL` in your `.env` to your own endpoints.
+Nothing in step 2 or 3 reads the bundled defaults if you've overridden
+them.
 
 ---
 
@@ -205,8 +205,7 @@ The shipped `.env.example` is **pre-pointed at step 1's data plane**
 - Set a model provider key (`OPENAI_API_KEY`, etc.) — only if you
   author workflows that call hosted providers; the bundled demos run
   on local open-weight models.
-- Point at your own Postgres / S3 (override `DATABASE_URL`, `S3_URL`,
-  `S3_USER_DATA_PREFIX`).
+- Point at your own Postgres / S3 (override `DATABASE_URL`, `S3_URL`).
 - Enable agent retrievals (`LUMID_DATA_URL=http://127.0.0.1:9102`).
 - Set `LUMILAKE_GPU_DEVICES` to one or more free GPU indices on your
   host (default is empty — no GPU workers). On a shared host, pick an

lumilake-0.1.2/docs/ENV.md

@@ -0,0 +1,154 @@
+# Environment Reference
+
+Lumilake reads configuration from `.env` for local deployment and from process environment variables for direct server runs. Generate the template with:
+
+```bash
+lumilake deploy -C <deploy-dir> init
+```
+
+`-C / --project-dir` (or `LUMILAKE_DEPLOY_DIR`) selects the directory the env files are written to and later read from. Workspace-checkout users may prefix every command with `uv run`; from a PyPI install (`pip install 'lumilake[cli]'`) invoke `lumilake` directly.
+
+To also generate a bundled FlowMesh stack env file:
+
+```bash
+lumilake deploy -C <deploy-dir> init --flowmesh
+```
+
+Run `lumilake deploy -C <deploy-dir> doctor` after editing `.env`.
+
+## Required Keys
+
+| Key | Purpose |
+|-----|---------|
+| `LUMILAKE_SERVER_HOST` | Host the FastAPI server binds to. |
+| `LUMILAKE_SERVER_PORT` | Port the FastAPI server binds to. |
+| `LUMILAKE_RUNTIME_ORCHESTRATOR_URL` | FlowMesh server URL used for workflow dispatch. |
+| `S3_ARCHIVE_PREFIX` | `bucket/prefix` where job records and runtime artifacts are archived. |
+| `LUMILAKE_IMAGE_TAG` | Docker image tag used by local deployment. Defaults to `dev` (the rolling main-built image). Pin to a `vX.Y.Z` semver tag for production. |
+| `LUMILAKE_REGISTRY` | Container registry the deploy CLI pulls the server image from. Defaults to `ghcr.io/mlsys-io`. **Trust-bearing**: setting this points `lumilake deploy pull` at a different host, so only override it to a registry you control. |
+
+Compute data plane (required for every direct SQL/S3 op):
+
+| Key | Purpose |
+|-----|---------|
+| `DATABASE_URL` | PostgreSQL connection string used by every SQL `DataRetrievalOp`. |
+| `S3_URL` | S3-compatible endpoint and credentials used by every S3 `DataRetrievalOp` and the archive (`S3_ARCHIVE_PREFIX`). Its path (`bucket/prefix`) is also the scan root for data-profile listing. |
+| `S3_CERT_FILE` | Path to an S3 TLS cert bundle. Optional; the bundled compose mounts the file when set. |
+
+## Lumid.data routing
+
+Only `DataRetrievalOp` with `type: agent` routes through lumid.data — SQL and S3 retrievals always go direct against `DATABASE_URL` / `S3_URL` regardless of `LUMID_DATA_URL`.
+
+Agent-retrieval keys:
+
+| Key | Purpose |
+|-----|---------|
+| `LUMID_DATA_URL` | Base URL for lumid.data's `/agent/v1` endpoint. Required when a workflow contains agent retrievals. |
+| `LUMID_DATA_TOKEN` | Bearer token sent to lumid.data. |
+| `LUMID_DATA_TIMEOUT_SECONDS` | HTTP timeout for lumid.data calls. Defaults to `30`. |
+
+## Server and Logging
+
+| Key | Purpose |
+|-----|---------|
+| `LUMILAKE_LOG_LEVEL` | Server log level. Defaults to `INFO`. |
+| `LUMILAKE_SKIP_DOTENV_CHECK` | Set to `1` when the server's env is injected directly (Docker) so startup does not require a `.env` file. |
+| `LUMILAKE_REQUIRE_IDENTITY_PROVIDER` | When truthy, rejects requests with 503 unless at least one `IdentityProvider` plugin is registered. Recommended for cloud deploys; leave unset locally. |
+| `LUMILAKE_RUNTIME_TOKEN` | Scheduler-internal credential for FlowMesh control-plane reads (worker enumeration, profile fetches). Never used to authorize HTTP request handlers. Leave empty for local; in cloud, set to a token whose FlowMesh principal can read worker metadata. Required when `LUMILAKE_REQUIRE_IDENTITY_PROVIDER` is set. |
+| `LUMILAKE_RECOVER_IN_FLIGHT_JOBS` | Whether server startup marks pending/running jobs as failed. Default `1` (recommended for single-instance deploys). Set to `0` in HA / multi-instance deploys so a starting standby doesn't clobber jobs the active instance is still running. |
+| `LUMILAKE_RUNTIME_MANAGER_TYPE` | Runtime dispatch backend. `default` or `flowmesh`. |
+| `LUMILAKE_JOB_MANAGER_TYPE` | Job manager implementation. Currently only `priority`. |
+| `LUMILAKE_HTTP_TIMEOUT_SECONDS` | Outbound HTTP timeout for server-side calls. Defaults to `300`. |
+
+## Scheduler and Runtime Tuning
+
+| Key | Purpose |
+|-----|---------|
+| `LUMILAKE_OPTIMIZER_TYPE` | Optimizer implementation. Defaults to `halo`. |
+| `LUMILAKE_OPTIMIZER_BATCH_SIZE` | Maximum batch size used by the optimizer loop. |
+| `LUMILAKE_BATCH_ACCUMULATION_SECONDS` | Wait window before forming the first optimization batch. |
+| `LUMILAKE_STARVATION_LIMIT` | Starvation threshold; `0` means immediate override. |
+| `LUMILAKE_OPTIMIZER_SUBPROCESS_TIMEOUT_SECONDS` | Timeout for optimizer subprocess execution. |
+| `LUMILAKE_QUEUE_QUANTUM_HIGH` | High-priority queue quantum. Defaults to `20`. |
+| `LUMILAKE_QUEUE_QUANTUM_MEDIUM` | Medium-priority queue quantum. Defaults to `10`. |
+| `LUMILAKE_QUEUE_QUANTUM_LOW` | Low-priority queue quantum. Defaults to `5`. |
+| `LUMILAKE_POLL_TIMEOUT_SECONDS` | Overall timeout for runtime polling. Defaults to `inf`. |
+| `LUMILAKE_POLL_INTERVAL_SECONDS` | Interval between runtime status polls. Defaults to `5`. |
+| `LUMILAKE_FLOWMESH_OUTPUT_DESTINATION` | FlowMesh result delivery mode. `local` (default) or `http`. |
+
+## Worker Groups
+
+| Key | Purpose |
+|-----|---------|
+| `LUMILAKE_CPU_WORKER_GROUP_SIZE` | Number of CPU workers FlowMesh provisions. |
+| `LUMILAKE_GPU_WORKER_GROUP_SIZE` | Number of GPU workers FlowMesh provisions. At least one of CPU / GPU size must be `> 0`. |
+| `LUMILAKE_GPU_DEVICES` | GPU device ids Lumilake assigns to FlowMesh workers (one per device). Specific index (`0`), comma-separated subset (`0,2`), or `all` to cover every `nvidia-smi` device. Leave blank to skip GPU worker creation. |
+
+## Plugins
+
+| Key | Purpose |
+|-----|---------|
+| `LUMILAKE_PLUGINS` | Comma-separated list of plugin module names the server imports at startup. See `docs/PLUGINS.md`. |
+| `LUMILAKE_PLUGIN_DIR` | Host directory bind-mounted to `/app/plugins` (read-only) and added to `PYTHONPATH`. Each `*.py` placed here is importable by the names listed in `LUMILAKE_PLUGINS`. Defaults to `./plugins` relative to the deploy directory. |
+
+## Data Profiling
+
+| Key | Purpose |
+|-----|---------|
+| `LUMILAKE_DATA_PROFILE_NUM_TEST_QUERIES` | Number of test queries per data profile. Defaults to `1`. |
+| `LUMILAKE_S3_PROFILE_COST_PER_FILE` | Cost-model coefficient (per file) for S3 profile estimates. |
+| `LUMILAKE_S3_PROFILE_COST_PER_MIB` | Cost-model coefficient (per MiB) for S3 profile estimates. |
+| `LUMILAKE_LOCAL_DATA_PROFILE_PLAN_VARIANTS` | Comma-separated planner variants used for local data profiles. Defaults to `default,prefer_index,prefer_seq,prefer_nestloop`. |
+| `LUMILAKE_DISABLE_DATA_PROFILE` | When truthy (`1`/`true`/`yes`/`on`), the server skips inline data-profile task build/run, skips `collect_data_profile` at batch dispatch, and the HALO optimizer falls back to its static cost model (any supplied profile results are dropped). Defaults to off. |
+
+## vLLM Backend
+
+| Key | Purpose |
+|-----|---------|
+| `LUMILAKE_VLLM_MAX_NUM_BATCHED_TOKENS` | vLLM `max_num_batched_tokens`. Defaults to `2048`. |
+| `LUMILAKE_VLLM_MAX_CUDAGRAPH_CAPTURE_SIZE` | vLLM CUDA-graph capture size. Defaults to `64`. |
+| `LUMILAKE_VLLM_GPU_MEMORY_UTILIZATION` | vLLM GPU memory utilization fraction. Defaults to `0.9`. |
+
+## Hardware Requirements
+
+Used by FlowMesh worker placement.
+
+| Key | Purpose |
+|-----|---------|
+| `HARDWARE_CPU_REQUIREMENT` | CPU cores per worker. Defaults to `8`. |
+| `HARDWARE_MEMORY_REQUIREMENT` | Memory per worker. Defaults to `16Gi`. |
+| `HARDWARE_GPU_REQUIREMENT` | GPUs per worker. Defaults to `1`. |
+| `HARDWARE_GPU_MEMORY_REQUIREMENT` | GPU memory per worker. Defaults to `8Gi`. |
+
+## FlowMesh TLS and Worker Config
+
+Picked up by the bundled FlowMesh deploy helpers.
+
+| Key | Purpose |
+|-----|---------|
+| `SERVER_HTTP_PORT` | FlowMesh HTTP API port in `.env.flowmesh`. Keep `LUMILAKE_RUNTIME_ORCHESTRATOR_URL` in `.env` and `FLOWMESH_BASE_URL` in `.env.flowmesh` aligned with this port. |
+| `SERVER_GRPC_PORT` | FlowMesh gRPC port in `.env.flowmesh`. Change it when a co-tenant FlowMesh stack already owns `50051`. |
+| `REDIS_CONTROL_PORT` | FlowMesh control Redis port in `.env.flowmesh`. Change it when another stack already owns `6379`. |
+| `REDIS_TELEMETRY_PORT` | FlowMesh telemetry Redis port in `.env.flowmesh`. Change it when another stack already owns `6380`. |
+| `FLOWMESH_BASE_URL` | FlowMesh HTTP base URL consumed by FlowMesh workers; should point at `SERVER_HTTP_PORT`. |
+| `REDIS_TLS_DIR` | Directory holding Redis TLS certs. Empty disables TLS bind-mounts. |
+| `SERVER_TLS_DIR` | Directory holding FlowMesh server gRPC TLS certs. Empty disables TLS. |
+| `SERVER_WORKER_CONFIG` | Path to a FlowMesh worker config override. |
+
+`lumilake deploy init --flowmesh` writes `.env.flowmesh`. On hosts that
+also run another FlowMesh stack, check the common defaults before
+`deploy up`: HTTP `8000`, gRPC `50051`, Redis control `6379`, and Redis
+telemetry `6380`.
+
+## SDK / CLI Client
+
+Consumed by the SDK and deploy CLI helpers.
+
+| Key | Purpose |
+|-----|---------|
+| `LUMILAKE_BASE_URL` | SDK / CLI override for the saved server URL (see `docs/SDK.md` and `docs/CLI.md` for resolution order). |
+| `LUMILAKE_API_KEY` | Bearer token sent as `Authorization: Bearer <key>` by the SDK / CLI. Required when the server runs an `IdentityProvider` plugin with `LUMILAKE_REQUIRE_IDENTITY_PROVIDER=1`. |
+| `LUMILAKE_TIMEOUT` | SDK / CLI HTTP timeout override in seconds. Values `<= 0` are ignored. |
+| `LUMILAKE_DEPLOY_DIR` | Default `--project-dir` for `lumilake deploy`. |
+
+See `.env.example` for the deploy-time template and defaults.