lightcone-cli 0.2.0__py3-none-any.whl

lightcone/cli/__init__.py

@@ -0,0 +1,16 @@
+"""lightcone-cli — the command-line surface for the Lightcone toolchain."""
+
+from __future__ import annotations
+
+try:
+    from importlib.metadata import version
+
+    __version__ = version("lightcone-cli")
+except Exception:
+    __version__ = "0.0.0.dev"
+
+
+def main() -> None:
+    from lightcone.cli.commands import main as _main
+
+    _main()

lightcone/cli/claude/lightcone/agents/lc-extractor.md

@@ -0,0 +1,114 @@
+---
+name: lc-extractor
+description: Extract prior insights from scientific papers for ASTRA analyses. Reads PDFs, identifies claims relevant to target decisions, extracts verbatim quotes, and verifies them. Use for literature extraction during /lc-new.
+tools: Read, Bash
+---
+
+You are an ASTRA prior insight extraction agent with self-validation capability. Your task is to extract prior insights from a single paper and format them for an ASTRA analysis. Prior insights are knowledge from literature that informs analysis decisions — they go in the `prior_insights:` section of astra.yaml.
+
+## Analysis Context
+
+[ANALYSIS_CONTEXT -- paste the analysis summary: problem statement, relevant decisions needing evidence, and what kind of support would be most useful]
+
+## Your Paper
+
+- DOI: [DOI]
+- Version: [VERSION -- include only for arXiv papers, omit this line otherwise]
+- PDF Path: [PDF_PATH -- absolute path from `astra paper path`]
+- Target decisions: [TARGET_DECISIONS -- list each decision ID, its label, and its options with descriptions]
+
+## Instructions
+
+1. Read the PDF at the path above using the Read tool.
+2. Identify claims relevant to the target decisions.
+3. For each relevant claim, extract:
+   - A clear claim (1-2 sentences stating what we learned)
+   - An exact quote from the paper (verbatim, 1-3 sentences)
+   - The page number where the quote appears (as a hint)
+   - Prefix and suffix context (~20-100 chars each) for robust matching
+4. Validate all quotes using batch verification (see below).
+5. Return ONLY verified prior insights as YAML.
+
+## Batch Verification Loop
+
+After extracting all quotes from the paper:
+
+1. Build a JSON object with all quotes:
+   ```json
+   {"quotes": [
+     {"text": "exact quote 1", "page": 5, "prefix": "context before", "suffix": "context after"},
+     {"text": "exact quote 2", "page": 12, "prefix": "context before", "suffix": "context after"}
+   ]}
+   ```
+
+2. Run batch verification (extracts PDF text once, verifies all quotes):
+   ```bash
+   echo '<json>' | astra paper verify-quotes "[DOI]" [--version N]
+   ```
+
+3. Parse the JSON response. Check each result's `status`: "verified" or "not_found".
+
+4. For any "not_found" quotes: re-read the relevant PDF section, correct the quote text, prefix, and suffix.
+
+5. Repeat batch verification with corrected quotes (max 3 iterations).
+
+6. If still failing after 3 attempts, drop those quotes and note which ones could not be verified.
+
+## Output Format
+
+Return ONLY this YAML structure. Do not include any other text outside the YAML block.
+
+```yaml
+prior_insights:
+  <insight_id>:
+    id: <insight_id>
+    claim: "<What we learned from this paper>"
+    created_at: "[TIMESTAMP]"
+    evidence:
+      - id: ev1
+        doi: "[DOI]"
+        version: <version if arXiv, omit otherwise>
+        quote:
+          type: TextQuoteSelector
+          exact: "<VERIFIED exact quote from paper>"
+          prefix: "<~20-100 chars BEFORE the quote>"
+          suffix: "<~20-100 chars AFTER the quote>"
+        location:
+          type: FragmentSelector
+          page: <page number hint>
+    scope: "<when this applies -- optional, include only if the claim has limited applicability>"
+
+decision_links:
+  <decision_id>:
+    <option_id>:
+      - <insight_id>
+
+verification_summary:
+  total_quotes: <N>
+  verified: <N>
+  failed: <N>
+  failed_details: "<description of any quotes that could not be verified, or 'none'>"
+```
+
+## Rules
+
+- Use lowercase_with_underscores for insight IDs
+- Quotes must be EXACT -- copy verbatim from the PDF
+- One claim per insight -- do not combine multiple claims
+- Only extract insights relevant to the target decisions
+- Only include insights whose quotes passed verification
+- If no relevant insights found, return `prior_insights: {}`
+- prefix and suffix are REQUIRED for every TextQuoteSelector
+- For arXiv papers, always include the version field in evidence
+
+## Troubleshooting: Verification Failures
+
+| Failure | Cause | Fix |
+|---------|-------|-----|
+| `Quote not found` | Paraphrased or introduced typos | Re-read the PDF page, copy the exact text, re-verify |
+| `Paper not in cache` | Paper was not downloaded before validation | Run `astra paper add <doi>` |
+| `Wrong page` | Page number is incorrect (quote exists elsewhere) | Check `found_pages` in JSON output, update page number |
+| `prefix/suffix mismatch` | Context text does not match surrounding text | Re-read the area around the quote, copy exact surrounding text |
+| Persistent `not_found` | OCR artifacts, ligatures, or Unicode differences | Try shorter quote avoiding problem characters; increase prefix/suffix |
+
+**Recovery**: Re-read the failing page, copy the exact text, update prefix/suffix, verify with `astra paper verify-quote`, then run `astra validate astra.yaml --verify-evidence`.

lightcone/cli/claude/lightcone/guides/astra-reference.md

@@ -0,0 +1,290 @@
+# ASTRA Reference
+
+## What an ASTRA Analysis Is
+
+An ASTRA analysis is a structured layer between the code and the paper. It surfaces the inputs a computation depends on, the outputs it produces, and -- critically -- every methodological decision that could plausibly affect the results. The goal is to make the full decision space explicit and machine-readable, so that alternative defensible choices can be systematically explored rather than silently baked in.
+
+An `astra.yaml` spec captures this for a single unit of work. The structure is **self-similar**: a top-level analysis and a nested sub-analysis have exactly the same shape. Everything in this reference applies equally to both.
+
+## astra.yaml Structure
+
+Fields: `name`, `description`, `version`, `authors`, `tags`, `inputs`, `outputs`, `decisions`, `prior_insights`, `findings`, `analyses`, `container`.
+
+```yaml
+# Simple analysis -- everything at top level
+version: "1.0"
+name: "My Analysis"
+description: "What this analysis investigates."
+inputs:
+  - id: training_data
+    type: data
+    source: "data/train.csv"
+decisions:
+  scaling:
+    label: "Feature Scaling"
+    tags: [preprocessing]        # optional freeform tags for grouping
+    rationale: "Affects convergence"
+    default: standard
+    options:
+      standard: { label: "StandardScaler" }
+      minmax: { label: "MinMaxScaler" }
+  use_pca:
+    label: "Use PCA"
+    default: "no"
+    options:
+      "yes": { label: "Yes" }
+      "no": { label: "No" }
+  n_components:
+    label: "PCA Components"
+    default: "50"
+    options:
+      "50": { label: "50 components" }
+      "100": { label: "100 components" }
+outputs:
+  - id: accuracy
+    type: metric
+    recipe:
+      command: python scripts/evaluate.py
+container: Containerfile
+```
+
+## Decisions
+
+A decision is a methodological choice where a different defensible option could plausibly produce a different numerical result. Include it if changing the choice could shift a quantitative outcome -- even modestly. Many small decisions can compound. When in doubt, include it.
+
+**Not decisions -- skip these:**
+
+- **Tooling choices** that produce identical numerical results: programming language, library/framework (PyTorch vs TensorFlow), file format, parallelization strategy, plotting style.
+- **Fixed constraints** with no degrees of freedom: "use the data that exists," "satisfy the grant requirements."
+- **What to produce** -- decisions control *how* something is computed, not *what* outputs exist. Outputs are fixed by the analysis structure.
+
+**These ARE decisions -- do not skip:**
+
+- Algorithmic choices (MCMC vs optimization, KDE vs histogram, smoothing method)
+- Numerical parameters and thresholds (sigma clipping level, bin width, convergence criterion, iteration count)
+- Statistical method choices (bootstrap vs analytic errors, Bayesian vs frequentist)
+- Data selection criteria (quality cuts, magnitude limits, spatial boundaries)
+- Correction and calibration choices (which reddening law, which zero-point, which prior)
+
+### Parameterization
+
+**Every decision must be parameterized in code** -- never hardcode a decision value. Accept all decisions as CLI args.
+
+**Underscore convention:** IDs use underscores in `astra.yaml` (`prior_range`). lightcone-cli passes `--prior_range wide`. Scripts must match: `parser.add_argument('--prior_range')`, **not** `--prior-range`.
+
+### Constraints
+
+- `when: "decision.option"` -- decision only exists given an upstream choice (e.g., `svm_kernel` only exists `when: model.svm`)
+- `incompatible_with: ["decision.option"]` -- cannot coexist in a universe
+- `requires: ["decision.option"]` -- must be selected together
+- `excluded: true` + `excluded_reason: "..."` -- option considered but rejected (cannot be default or selected)
+
+## Writing Results
+
+Convention path: `results/<universe_id>/<output_id>.<ext>` -- no `path` field needed.
+
+- `metric` -- JSON (`{"value": 0.95}`)
+- `figure` -- PNG
+- `table` -- CSV
+- `data` -- Parquet/HDF5
+- `report` -- Markdown
+
+## Recipe Format
+
+Inline on outputs. Fields: `command` (required), `inputs`, `container`, `resources`.
+
+```yaml
+outputs:
+  - id: accuracy
+    type: metric
+    recipe:
+      command: python scripts/evaluate.py
+      inputs: [trained_model]               # Dependency on other output
+      container: ghcr.io/proj/ml:latest     # Overrides analysis-level default
+      resources: { cpus: 4, memory: "32GB", gpus: 1, time_limit: "2h" }
+```
+
+Set `container:` at analysis level (all recipes inherit); per-recipe `container:` overrides. Pass either a container image name (e.g., `python:3.12-slim`, `ghcr.io/org/img:latest`) or a path to a Containerfile (e.g., `Containerfile`, `containers/Dockerfile`). The runtime figures out whether to pull or build.
+
+### Conditional Outputs
+
+Outputs can have `when` conditions -- the output only exists when the condition is met for a given universe. Uses the same syntax as decision `when` (negation with `~`, lists AND'd).
+
+```yaml
+outputs:
+  - id: faint_metrics
+    type: metric
+    when: "~training_sample.bright_only"          # Only when NOT bright_only
+    recipe: { command: python scripts/evaluate.py }
+  - id: combined_report
+    type: report
+    when: ["~training_sample.bright_only", model.svm]  # AND: both must be true
+    recipe: { command: python scripts/combo.py }
+```
+
+## Universe Management
+
+A universe selects one option per decision -- a defensible alternative analysis path. Bug fixes and refactors are normal commits, not universes.
+
+```bash
+astra universe generate -n experiment1 -d "Testing hypothesis X"
+# Edit universes/experiment1.yaml, then: lc run --universe experiment1
+```
+
+**Adding a new decision:** (1) add to `astra.yaml` with options/default/rationale, (2) add parameter to code, (3) add to all existing universe files with default, (4) create new universe, (5) `astra validate astra.yaml`.
+
+## Prior Insights and Findings
+
+Two kinds of insight, distinguished by direction:
+
+- **Prior insights** (`prior_insights:`) — knowledge from outside the analysis that informs decisions. From literature (by DOI) or artifacts from a prior/parent analysis.
+- **Findings** (`findings:`) — conclusions from the analysis itself, backed by its own output artifacts.
+
+Both use the same model (id, claim, created_at, evidence). Placement determines direction.
+
+```yaml
+prior_insights:
+  layer_norm_stability:
+    id: layer_norm_stability
+    claim: "Layer normalization improves training stability"
+    created_at: "2025-01-15T10:30:00"
+    evidence:
+      - id: e1
+        doi: "10.48550/arXiv.1607.06450"
+        quote: { type: TextQuoteSelector, exact: "Exact text", prefix: "~20-100 chars before", suffix: "~20-100 chars after" }
+        location: { type: FragmentSelector, page: 5 }
+      - id: e2
+        doi: "10.48550/arXiv.1607.06450"
+        figure: { type: FigureSelector, label: "Figure 3a", caption: "..." }
+    scope: "Context where this applies (optional)"
+
+findings:
+  scaling_result:
+    id: scaling_result
+    claim: "StandardScaler achieves 97% accuracy vs 91% for MinMaxScaler"
+    created_at: "2025-02-01T14:00:00"
+    evidence:
+      - id: e1
+        artifact: "accuracy"            # Content selectors optional for artifacts
+      - id: e2
+        artifact: "model_comparison"
+        quote: { type: TextQuoteSelector, exact: "StandardScaler achieved 97% accuracy vs 91% for MinMaxScaler" }
+```
+
+Link prior insights to decisions: `options: { layer_norm: { insights: [layer_norm_stability] } }`
+
+Artifact references are validated against declared outputs — `astra validate` flags any `artifact:` that doesn't match an output ID. Literature evidence requires at least one content selector (quote, figure, or table); artifact evidence does not.
+
+**Sub-analysis findings as prior insights:** When a sub-analysis explores a specific question (calibration study, simulation validation, sensitivity test), its findings can inform decisions elsewhere. The parent or sibling references the sub-analysis output as artifact evidence in its own `prior_insights`, e.g. `artifact: "build_mocks.noise_diagnostics"`. This creates a traceable chain from sub-analysis conclusion to downstream decision.
+
+## Sub-Analyses
+
+### What a Sub-Analysis Is
+
+Each `astra.yaml` -- root or nested -- represents a **unit of work**: meaningful inputs, methodological decisions, meaningful outputs. A sub-analysis is one of these units nested inside a larger analysis. It can be understood, executed, and evaluated on its own terms.
+
+### When to Split
+
+Default to a **single analysis**. Split into sub-analyses only when:
+
+- **Decision ownership** -- the stage has its own decisions that could meaningfully vary, clearly scoped to that stage rather than the broader analysis. Shared decisions live at the parent (`from: ../`); stage-specific decisions live in the sub-analysis. If you can't cleanly assign decisions to levels, the split is probably wrong.
+- **Reusability** -- someone working on a different paper could use this stage's output as-is (a cleaned catalog, a trained emulator, a set of mocks).
+- **Side quests** -- independent investigations (diagnostics, calibrations, simulation studies) that have different inputs/outputs/code from the main analysis are sub-analyses, not universes. Universes are different parameter choices on the same pipeline.
+- **If boundaries are unclear**, start flat and split later when they become explicit: separate stage outputs, explicit `from` links, clear decision ownership per level.
+
+### Worked Examples
+
+#### Two-Stage Pipeline (DAG Split)
+
+A paper builds mock galaxy catalogs, then trains a neural network on them for photometric redshift estimation. Natural split:
+- **`build_mocks`**: simulation inputs + survey properties, decisions about noise model and selection function. Produces mock catalogs.
+- **`photo_z`**: mocks (from sibling) + real survey data, decisions about network architecture and training. Produces redshift estimates.
+
+The mock-building decisions are independent from training decisions. Someone could reuse the mocks for a different estimator.
+
+#### When NOT to Split
+
+A paper downloads galaxies, applies quality cuts, corrects for extinction, computes luminosity functions, fits a Schechter function. Five steps -- but one objective, shared decisions, one end product. 
+
+### Anti-Patterns
+
+- **Splitting by script** rather than by analytical unit.
+- **Zero-decision sub-analyses** that just pass data through -- make these output recipes in the parent.
+- **Premature splitting.** Start flat, split when boundaries become explicit. Easier to split a working flat analysis than merge a broken hierarchical one.
+- **Forcing a linear DAG.** Independent stages don't need to be wired in sequence just because the paper presents them that way.
+
+### Composition Mechanics
+
+Each sub-analysis lives in its own directory with its own `astra.yaml`. The parent lists them with `path:` references:
+
+```yaml
+# Root astra.yaml
+inputs:
+  - id: survey_catalog
+    type: data
+    source: "data/survey.parquet"
+decisions:
+  cosmology_model:               # Shared across stages
+    label: "Cosmological Model"
+    tags: [physics]
+    default: flat_lcdm
+    options:
+      flat_lcdm: { label: "Flat LCDM" }
+      wcdm: { label: "wCDM" }
+outputs:
+  - id: trained_model
+    type: data
+    from: train_network.trained_model   # Alias -- produced by sub-analysis
+analyses:
+  build_mocks:
+    path: ./analyses/build_mocks
+  train_network:
+    path: ./analyses/train_network
+```
+
+Inside each sub-analysis's own `astra.yaml`, `from:` wires inputs and decisions to the parent or siblings:
+
+```yaml
+# analyses/train_network/astra.yaml
+inputs:
+  - id: training_data
+    type: data
+    from: build_mocks.mock_catalog       # Sibling output
+outputs:
+  - id: trained_model
+    type: data
+    recipe: { command: python src/train.py, resources: { gpus: 1, memory: "32GB" } }
+decisions:
+  cosmology_model:
+    from: ../cosmology_model             # Inherit parent decision
+  noise_model:
+    label: "Noise Model"
+    default: heteroscedastic
+    options:
+      homoscedastic: { label: "Homoscedastic" }
+      heteroscedastic: { label: "Heteroscedastic" }
+```
+
+**Wiring patterns:**
+- **Input `from:`** -- `from: parent_input_id` (parent input) or `from: sibling_id.output_id` (sibling output).
+- **Decision `from: ../parent_id`** -- inherits a parent decision. The sub-analysis uses the parent's value; do not set it in the sub-analysis universe.
+- **Output `from: sub.output`** at root level creates an alias to a sub-analysis output.
+- **`universe:` field** in universe files selects which sub-analysis universe to load: `build_mocks: { universe: baseline }` loads `./analyses/build_mocks/universes/baseline.yaml`.
+
+## CLI Reference (astra)
+
+```bash
+astra validate astra.yaml                       # Validate (run after every change)
+astra validate astra.yaml --verify-evidence     # + verify insight quotes against PDFs
+astra info [--decisions]                      # Analysis summary / decision details
+astra universe generate -n NAME [-d "desc"]   # Generate universe from defaults
+astra universe check universes/x.yaml         # Check universe constraints
+astra viz                                     # Visualize decision space
+astra schema show analysis                    # Show JSON schema
+```
+
+## Validation
+
+Run `astra validate astra.yaml` after **every** spec change. Additional checks:
+- Universe files: `astra universe check universes/<name>.yaml`
+- Evidence quotes: `astra validate astra.yaml --verify-evidence`

lightcone/cli/claude/lightcone/guides/lightcone-cli-reference.md

@@ -0,0 +1,75 @@
+# lightcone-cli Reference
+
+Reference for lightcone-cli execution: CLI commands, development workflow, status interpretation, HPC execution, and failure diagnosis. For astra.yaml spec syntax, see `astra-reference.md`.
+
+## CLI Reference
+
+```bash
+lc init [DIR]                            # Scaffold a new ASTRA project
+lc init NAME --sub-analysis              # Scaffold sub-analysis and wire into parent
+lc run [OUTPUT] [--universe NAME]        # Execute recipes via Dagster (auto-builds)
+lc run --partition gpu --qos shared      # Unknown flags passed through to SLURM
+lc run --no-build                        # Skip automatic container builds
+lc build [--force] [--runtime docker]    # Build container images from specs
+lc status [--universe NAME]              # Materialization + container status
+lc dev [--port 3000]                     # Dagster webserver UI
+lc target [--set NAME] [--list]          # Manage execution targets
+lc setup                                 # Interactive target setup wizard
+```
+
+## Creating Sub-Analyses
+
+`lc init NAME --sub-analysis` scaffolds a sub-analysis and wires it into the parent project. It:
+
+1. Creates `analyses/<name>/` with its own `astra.yaml`, `CLAUDE.md`, `scripts/`, `universes/baseline.yaml`, and `results/`
+2. Adds a `path:` entry to the parent `astra.yaml` under `analyses:`
+3. Adds a `universe: baseline` entry to all existing parent universe files
+
+After scaffolding, populate the sub-analysis's `astra.yaml` with inputs, outputs, and decisions. Use `from:` references to wire inputs and decisions to the parent or siblings — see `astra-reference.md` under "Composition Mechanics."
+
+## Development Workflow
+
+Three overlapping phases:
+
+1. **Write & Debug** -- Run scripts directly (`python scripts/compute.py`) to iterate. Write them recipe-ready from the start: parameterize decisions, write to convention paths, one script per output.
+2. **Integrate** -- Add `recipe:` blocks to outputs in `astra.yaml`. Track with `lc status` (`no recipe` / `pending` / `ok`). Set `container:` at analysis level or per-recipe — pass an image name (e.g., `python:3.12-slim`) or a path to a Containerfile (e.g., `Containerfile`).
+3. **Materialize** -- `lc run` executes via Dagster in containers (Docker or SLURM). Falls back to local execution if Docker is unavailable. Done when `lc status` shows all `ok`.
+
+**An output is not done until `lc run` produces it.** Running scripts directly is for debugging only — final results must always come from `lc run` so they are reproducible inside containers.
+
+### Spec-Code Invariant
+
+**`astra.yaml` must always reflect the code and vice versa.** When you change one, update the other immediately:
+- Add a decision to code? Add it to `astra.yaml` and all universe files.
+- Add an output or change a script? Update the `recipe:` block in `astra.yaml`.
+- Remove or rename something? Update both sides and run `astra validate astra.yaml`.
+
+## Status Interpretation
+
+`lc status` shows outputs vs universes. **Progression:** `no recipe` --> `pending` --> `ok`
+
+- `ok` -- Recipe exists, results on disk. Done.
+- `pending` -- Recipe exists, not materialized. Run `lc run`.
+- `no recipe` -- No `recipe:` block yet. Still in Write & Debug phase.
+
+Container status: `prebuilt: image`, `build: Containerfile (built)`, or `(not built)` (needs `lc build`).
+
+## Failure Diagnosis
+
+- **Script arg not recognized** -- Use underscores in argparse to match decision IDs
+- **Recipe input not found** -- Materialize upstream outputs first
+
+After failure: fix, then `lc run <output_id> --universe <name>`.
+
+## HPC / Interactive Execution
+
+If `SLURM_JOB_ID` is set in the environment, `lc run` automatically uses `srun` (instant) instead of `sbatch` (queued). This means the user is inside an interactive allocation and execution will be fast.
+
+If `SLURM_JOB_ID` is not set and the target is SLURM, `lc run` submits via `sbatch`. Any unknown flags are passed through as SLURM directives:
+```bash
+lc run --qos shared --constraint gpu      # NERSC
+lc run --partition gpu-a100               # TACC, SDSC, etc.
+lc run --gres=gpu:4 --partition batch     # any cluster
+```
+
+Recipe `resources` (gpus, memory, cpus, time_limit) are portable and used for batch scheduling. They are ignored in interactive mode since the allocation already provides them.

lightcone/cli/claude/lightcone/guides/ui-brand.md

@@ -0,0 +1,86 @@
+# Lightcone UI Brand
+
+Visual patterns for user-facing lightcone-cli output. Skills reference this file for consistent formatting.
+
+The principle: use markdown idiomatically. Minimal Unicode decoration. Let the content breathe.
+
+## Stage Banners
+
+Use for major phase transitions. A markdown `###` heading.
+
+```
+### Research Question
+```
+
+**Stage names:**
+- `Research Question`
+- `Analysis Structure`
+- `Deep Dive — [Section]`
+- `Finalizing`
+- `Specification Complete`
+- `Verify — <universe_id>`
+- `Verification Report — <universe_id>`
+- `Project Discovery`
+- `Inventory`
+- `Scaffolding`
+- `Migration Plan`
+- `Migration Complete`
+
+---
+
+## Status Symbols
+
+```
+✓  Complete / Passed
+○  Pending
+✗  Failed
+```
+
+---
+
+## Action Prompts
+
+When user input is needed, use bold text:
+
+```
+**What are you trying to learn? Describe the research question in your own words.**
+```
+
+No boxes, no rules, no blockquotes (those render grey/dimmed).
+
+---
+
+## Next Up Block
+
+At end of major completions:
+
+```
+---
+**Next up**
+
+**{Description}** — {one-line detail}
+
+`{copy-paste command}`
+
+Also available: `/lc-verify`, ...
+```
+
+---
+
+## Tables
+
+```
+| Section | Decisions | Outputs |
+|---------|-----------|---------|
+| main    | 3         | 2       |
+```
+
+---
+
+## Anti-Patterns
+
+- Heavy Unicode box-drawing (━, ─── rules, ► arrows)
+- Boxing prompts in horizontal rules
+- Random emoji
+- Walls of decoration that distract from content
+- Missing Next Up block after completions