extropy-run 0.2.2__tar.gz

extropy_run-0.2.2/.claude/skills/extropy/SKILL.md

@@ -0,0 +1,152 @@
+---
+name: extropy
+description: Help run the extropy pipeline — population creation, scenario compilation, simulation, debugging validation errors, and interpreting results. Use when the user asks about running extropy commands, fixing spec issues, understanding results, or needs guidance on what extropy can simulate.
+allowed-tools: Read, Grep, Glob, Bash, Edit, Write
+argument-hint: "[command or question]"
+---
+
+# Extropy Assistant
+
+You are an expert assistant for the Extropy predictive intelligence framework. You help users run the full pipeline, debug issues, interpret results, and understand what Extropy can and cannot do.
+
+## What Extropy Is
+
+Extropy simulates how real human populations respond to scenarios. It creates synthetic populations grounded in real-world statistical data, enriches them with LLM-extrapolated psychographic attributes, connects them via social networks, and runs agent-based simulations where each agent reasons individually via LLM calls. The output is distributional predictions — not a poll, but a simulation of emergent collective behavior.
+
+## The Pipeline
+
+```
+extropy spec → extropy extend → extropy sample → extropy network → extropy persona → extropy scenario → extropy simulate → extropy results
+```
+
+| Step | Command                                                                               | What It Does                                     |
+| ---- | ------------------------------------------------------------------------------------- | ------------------------------------------------ |
+| 1    | `extropy spec "<description>" -o base.yaml`                                           | Build base population spec from natural language |
+| 2    | `extropy extend base.yaml -s "<scenario>" -o population.yaml`                         | Add scenario-specific attributes                 |
+| 3    | `extropy sample population.yaml -o agents.json --seed 42`                             | Sample concrete agents from distributions        |
+| 4    | `extropy network agents.json -o network.json -p population.yaml --seed 42`            | Generate social network graph                    |
+| 5    | `extropy persona population.yaml --agents agents.json`                                | Generate persona rendering config                |
+| 6    | `extropy scenario -p population.yaml -a agents.json -n network.json -o scenario.yaml` | Compile executable scenario spec                 |
+| 7    | `extropy simulate scenario.yaml -o results/ --seed 42`                                | Run the simulation                               |
+| 8    | `extropy results results/`                                                            | View outcomes                                    |
+
+Add `-y` / `--yes` to skip confirmation prompts for scripting.
+
+## Configuration
+
+Extropy uses a two-zone config system:
+
+- **Pipeline zone** (steps 1-6): `extropy config set pipeline.provider claude`
+- **Simulation zone** (step 7): `extropy config set simulation.provider openai`
+
+```bash
+extropy config show          # View current config
+extropy config set <key> <value>  # Set a value
+extropy config reset         # Reset to defaults
+```
+
+API keys are always env vars: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`.
+
+## When Helping Users Run Commands
+
+1. **Check prerequisites first** — does the user have the input files the command needs? Read the directory to verify.
+2. **Use `--seed 42`** on sample, network, and simulate for reproducibility unless the user specifies otherwise.
+3. **Show what happened** — after each command, briefly explain the output (e.g., "Sampled 500 agents with 38 attributes each").
+4. **For `extropy spec` and `extropy extend`** — these are interactive and make LLM calls. Warn the user about API costs for large populations. Use `-y` only if the user asks for non-interactive mode.
+
+## Debugging Validation Errors
+
+When `extropy validate` fails, common issues and fixes:
+
+### Structural Errors (must fix)
+
+- **Circular dependency** — Check the `depends_on` fields. Use `extropy validate <spec> --strict` for full details. The error message includes the cycle path.
+- **Invalid distribution params** — e.g., `std: 0` or `min > max`. Fix in the YAML directly.
+- **Missing dependency reference** — A formula or condition references an attribute that doesn't exist. Check spelling.
+- **Duplicate attribute names** — Two attributes with the same `name` field.
+- **Invalid modifier conditions** — Condition syntax uses restricted Python. Only these builtins are allowed: abs, min, max, round, int, float, str, len, sum, all, any, bool.
+
+### Semantic Warnings (should fix)
+
+- **No-op modifiers** — A modifier that doesn't actually change anything (e.g., `multiply: 1.0, add: 0`).
+- **Categorical option mismatch** — Modifier references an option that doesn't exist in the attribute's options list. Use `extropy fix <spec>` to auto-fix fuzzy matches.
+- **Modifier stacking** — Multiple modifiers that could conflict.
+
+### Common Fix Commands
+
+```bash
+extropy validate population.yaml              # Check for issues
+extropy validate population.yaml --strict     # Treat warnings as errors
+extropy fix population.yaml --dry-run         # Preview auto-fixes
+extropy fix population.yaml                   # Apply auto-fixes
+```
+
+## Interpreting Results
+
+When the user asks about simulation results:
+
+1. **Read the key files:**
+
+   - `meta.json` — run config, timing, rate limiter stats
+   - `outcome_distributions.json` — final aggregate outcomes
+   - `by_timestep.json` — how outcomes evolved over time
+   - `agent_states.json` — per-agent final states (for deep dives)
+
+2. **Highlight interesting patterns:**
+
+   - Convergence speed (how many timesteps until exposure saturated)
+   - Distribution shape (is it polarized or clustered?)
+   - Sentiment vs. position mismatches (agents who chose "comply" but have negative sentiment)
+   - Conviction levels (high conviction = stable opinions, low = could shift with more exposure)
+
+3. **Suggest segment analysis:**
+
+   ```bash
+   extropy results results/ --segment income
+   extropy results results/ --segment age
+   ```
+
+4. **For per-agent deep dives:**
+   ```bash
+   extropy results results/ --agent agent_042
+   ```
+
+## What Extropy CAN Simulate
+
+Extropy is built for **population-level behavioral prediction** with heterogeneous agents:
+
+- **Product/service adoption** — How will surgeons respond to a new AI diagnostic tool? Who adopts early vs. resists, and why?
+- **Policy compliance** — Congestion tax, vaccine mandates, zoning changes. Identifies inequity and friction points across demographics.
+- **Information spread** — How rumors, news, or messaging propagates through social networks. Which groups are most susceptible?
+- **Collective action** — Strike propensity, boycott participation, protest. Predicts "silent" effects like early retirement waves.
+- **Message testing** — Synthetic focus groups at scale. Test political messaging, marketing campaigns, or crisis communications on granular population segments.
+- **Pricing changes** — Subscription price hikes, fee introductions. Predicts churn, downgrade, and complaint patterns by segment.
+
+## What Extropy CANNOT Simulate
+
+- **Organizational hierarchies** — No org charts, reporting lines, or workflow dependencies. Agents are peers in a social network.
+- **Physical/spatial logistics** — No coordinates, distances, collision, or capacity. Geography is a semantic label, not a map.
+- **High-frequency quantitative models** — LLM reasoning is semantic and probabilistic, not mathematically precise or sub-second.
+- **Multi-event cascades** — Currently single-event scenarios only. Sequential reactive events (event A triggers event B) require running separate simulations.
+
+## Simulation Tips
+
+- **Population size vs. cost**: Each agent makes 1-2 LLM calls per reasoning round. A 500-agent, 50-timestep simulation could make ~5,000-25,000 API calls. Start small (100 agents, 2-5 timesteps) to validate, then scale up.
+- **Two-pass reasoning**: Pass 1 (pivotal model, e.g., gpt-5) does freeform role-play. Pass 2 (routine model, e.g., gpt-5-mini) classifies into outcomes. This is more accurate but uses 2x the calls.
+- **Seed exposure rules**: The scenario's `seed_exposure.rules` control who learns about the event when. If exposure rate is low after simulation, check these rules — the conditions might be too restrictive.
+- **Multi-touch threshold**: Default is 3 — agents re-reason after 3 new exposures since their last reasoning. Lower it for faster opinion evolution, raise it for more stable opinions.
+- **Stopping conditions**: Simulations stop at max_timesteps OR when stop_conditions are met (e.g., `exposure_rate > 0.95 and no_state_changes_for > 5`).
+
+## File Formats Quick Reference
+
+| File                                 | Format     | Key Fields                                                                |
+| ------------------------------------ | ---------- | ------------------------------------------------------------------------- |
+| `population.yaml`                    | YAML       | meta, attributes, sampling_order, grounding                               |
+| `agents.json`                        | JSON array | Each object has `_id` + all attribute values                              |
+| `network.json`                       | JSON       | meta, nodes, edges (bidirectional, typed)                                 |
+| `*.persona.yaml`                     | YAML       | intro_template, treatments, groups, phrasings, population_stats           |
+| `scenario.yaml`                      | YAML       | meta, event, seed_exposure, interaction, spread, outcomes, simulation     |
+| `results/meta.json`                  | JSON       | scenario_name, population_size, model, completed_at                       |
+| `results/outcome_distributions.json` | JSON       | Per-outcome aggregate distributions                                       |
+| `results/by_timestep.json`           | JSON array | Per-timestep: exposure_rate, position_distribution, sentiment, conviction |
+| `results/agent_states.json`          | JSON array | Per-agent: position, sentiment, conviction, public_statement, memory      |

extropy_run-0.2.2/.env.example

@@ -0,0 +1,9 @@
+# Extropy API Keys
+# These are the only settings that belong in .env (secrets only).
+# For provider/model config, use: extropy config set <key> <value>
+
+# OpenAI (required if using openai as provider)
+OPENAI_API_KEY=sk-...
+
+# Anthropic (from https://console.anthropic.com/settings/keys)
+ANTHROPIC_API_KEY=sk-ant-...

extropy_run-0.2.2/.github/workflows/claude-code-review.yml

@@ -0,0 +1,45 @@
+name: Claude Code Review
+
+on:
+  workflow_dispatch:
+  pull_request:
+    types: [opened, synchronize, ready_for_review, reopened]
+    # Optional: Only run on specific file changes
+    # paths:
+    #   - "src/**/*.ts"
+    #   - "src/**/*.tsx"
+    #   - "src/**/*.js"
+    #   - "src/**/*.jsx"
+
+jobs:
+  claude-review:
+    # Optional: Filter by PR author
+    # if: |
+    #   github.event.pull_request.user.login == 'external-contributor' ||
+    #   github.event.pull_request.user.login == 'new-developer' ||
+    #   github.event.pull_request.author_association == 'FIRST_TIME_CONTRIBUTOR'
+
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: read
+      issues: read
+      id-token: write
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Run Claude Code Review
+        id: claude-review
+        uses: anthropics/claude-code-action@v1
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+          plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
+          plugins: 'code-review@claude-code-plugins'
+          prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}'
+          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
+          # or https://code.claude.com/docs/en/cli-reference for available options
+

extropy_run-0.2.2/.github/workflows/claude.yml

@@ -0,0 +1,50 @@
+name: Claude Code
+
+on:
+  issue_comment:
+    types: [created]
+  pull_request_review_comment:
+    types: [created]
+  issues:
+    types: [opened, assigned]
+  pull_request_review:
+    types: [submitted]
+
+jobs:
+  claude:
+    if: |
+      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
+      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
+      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
+      (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: read
+      issues: read
+      id-token: write
+      actions: read # Required for Claude to read CI results on PRs
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Run Claude Code
+        id: claude
+        uses: anthropics/claude-code-action@v1
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+
+          # This is an optional setting that allows Claude to read CI results on PRs
+          additional_permissions: |
+            actions: read
+
+          # Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
+          # prompt: 'Update the pull request description to include a summary of changes.'
+
+          # Optional: Add claude_args to customize behavior and configuration
+          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
+          # or https://code.claude.com/docs/en/cli-reference for available options
+          # claude_args: '--allowed-tools Bash(gh pr:*)'
+

extropy_run-0.2.2/.github/workflows/publish.yml

@@ -0,0 +1,28 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Build package
+        run: |
+          pip install build
+          python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          skip-existing: true

extropy_run-0.2.2/.github/workflows/test.yml

@@ -0,0 +1,35 @@
+name: Tests
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [main, dev]
+  pull_request:
+    branches: [main, dev]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v4
+        with:
+          enable-cache: true
+      - run: uv python install 3.12
+      - run: uv sync --group dev
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v4
+        with:
+          enable-cache: true
+      - run: uv python install ${{ matrix.python-version }}
+      - run: uv sync --group dev
+      - run: uv run pytest --tb=short -q

extropy_run-0.2.2/.gitignore

@@ -0,0 +1,73 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments
+env/
+venv/
+.venv/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# Environment
+.env
+
+# Storage (database and populations)
+storage/
+
+# Simulation results
+results/
+
+# Cache
+data/cache/
+
+# Logs
+logs/
+
+# OS
+.DS_Store
+Thumbs.db
+
+
+austin/
+examples/
+
+# Simulation artifacts
+*.sqlite
+*.db
+*.jsonl
+
+# Tool caches
+__pypackages__/
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+
+# Claude Code
+.claude/agents/
+
+# Private (competitive intel, internal planning, todos)
+private/
+todo/

extropy_run-0.2.2/CLAUDE.md

@@ -0,0 +1,196 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What Extropy Is
+
+Extropy is a predictive intelligence framework that simulates how real human populations respond to scenarios. It creates synthetic populations grounded in real-world statistical data, enriches them with LLM-extrapolated psychographic attributes, connects them via social networks, and runs agent-based simulations where each agent reasons individually via LLM calls. The output is not a poll — it's a simulation of emergent collective behavior.
+
+Competitor reference: [Aaru](https://aaru.com) operates in the same space (multi-agent population simulation for predictive intelligence). Extropy differentiates through its grounding pipeline — every attribute distribution is researched from real-world sources with citations, not just LLM-generated.
+
+## Commands
+
+```bash
+pip install -e ".[dev]"      # Install with dev deps
+
+# Set API keys (secrets only — in .env or env vars)
+export ANTHROPIC_API_KEY=sk-ant-...
+export OPENAI_API_KEY=sk-...
+
+# Configure providers and models
+extropy config set pipeline.provider claude        # Use Claude for population/scenario building
+extropy config set simulation.provider openai      # Use OpenAI for agent reasoning
+extropy config set simulation.model gpt-5-mini     # Override simulation model
+extropy config set simulation.routine_model gpt-5-mini  # Cheap model for Pass 2 classification
+extropy config set simulation.rate_tier 2          # Rate limit tier (1-4)
+extropy config show                                # View current config
+
+pytest                       # Run all tests
+pytest tests/test_sampler.py # Single test file
+pytest -k "test_name"        # Single test by name
+
+ruff check .                 # Lint
+ruff format .                # Format
+```
+
+CLI entry point: `extropy` (defined in `pyproject.toml` → `extropy.cli:app`). Python >=3.11.
+
+## Pipeline (7 sequential commands)
+
+```
+extropy spec → extropy extend → extropy sample → extropy network → extropy persona → extropy scenario → extropy simulate
+ │
+ extropy results
+ extropy estimate
+```
+
+Each command produces a file consumed by the next. `extropy validate` is a utility runnable at any point. `extropy results` is a viewer for simulation output. `extropy estimate` predicts simulation cost (LLM calls, tokens, USD) without requiring API keys.
+
+## Architecture
+
+Three phases, each mapping to a package under `extropy/`:
+
+### Phase 1: Population Creation (`extropy/population/`)
+
+**The validity pipeline.** This is where predictive accuracy is won or lost.
+
+1. **Sufficiency check** (`spec_builder/sufficiency.py`) — LLM validates the description has enough context (who, how many, where).
+
+2. **Attribute selection** (`spec_builder/selector.py`) — LLM discovers 25-40 attributes across 4 categories: `universal` (age, gender), `population_specific` (specialty, seniority), `context_specific` (scenario-relevant), `personality` (Big Five). Each attribute gets a type (`int`/`float`/`categorical`/`boolean`) and sampling strategy (`independent`/`derived`/`conditional`).
+
+3. **Hydration** (`spec_builder/hydrator.py` → `hydrators/`) — The most important step. Four sub-steps, each using different LLM tiers:
+   - **2a: Independent** (`hydrators/independent.py`) — `agentic_research()` with web search finds real-world distributions with source URLs. This is the grounding layer.
+   - **2b: Derived** (`hydrators/derived.py`) — `reasoning_call()` specifies deterministic formulas (e.g., `years_experience = age - 26`).
+   - **2c: Conditional base** (`hydrators/conditional.py`) — `agentic_research()` finds base distributions for attributes that depend on others.
+   - **2d: Conditional modifiers** (`hydrators/conditional.py`) — `reasoning_call()` specifies how attribute values shift based on other attributes. Type-specific: numeric gets `multiply`/`add`, categorical gets `weight_overrides`, boolean gets `probability_override`.
+
+4. **Constraint binding** (`spec_builder/binder.py`) — Topological sort (Kahn's algorithm, `utils/graphs.py`) resolves attribute dependencies into a valid sampling order. Raises `CircularDependencyError` with cycle path.
+
+5. **Sampling** (`sampler/core.py`) — Iterates through `sampling_order`, routing each attribute by strategy. Supports 6 distribution types: normal, lognormal, uniform, beta, categorical, boolean. Hard constraints (min/max) are clamped post-sampling. Formula parameters evaluated via `utils/eval_safe.py` (restricted Python eval, whitelisted builtins only).
+
+6. **Network generation** (`network/`) — Hybrid algorithm: similarity-based edge probability with degree correction, calibrated via binary search to hit target avg_degree, then Watts-Strogatz rewiring (5%) for small-world properties. Edge probability: `base_rate * sigmoid(similarity) * degree_factor_a * degree_factor_b`. All network behavior is data-driven via `NetworkConfig`: attribute weights, edge type rules (evaluated by priority via `_eval_edge_condition()`), influence factors (ordinal/boolean/numeric), and degree multipliers. `NetworkConfig` can be generated from a `PopulationSpec` via LLM (`config_generator.py`), loaded from YAML (`--network-config`), or auto-detected as `{population_stem}.network-config.yaml`. Empty config (no `-p` or `-c`) produces a flat network.
+
+### Phase 2: Scenario Compilation (`extropy/scenario/`)
+
+**Compiler** (`compiler.py`) orchestrates 5 steps: parse event → generate exposure rules → determine interaction model → define outcomes → assemble spec.
+
+- **Event types**: product_launch, policy_change, pricing_change, technology_release, organizational_change, market_event, crisis_event
+- **Exposure channels**: broadcast, targeted, organic — with per-timestep rules containing conditions and probabilities
+- **Outcomes**: categorical (enum options), boolean, float (with range), open_ended
+- Auto-configures simulation parameters based on population size (<500: 50 timesteps, ≤5000: 100, >5000: 168)
+
+### Phase 3: Simulation (`extropy/simulation/`)
+
+**Engine** (`engine.py`) runs per-timestep loop, decomposed into sub-functions:
+1. **`_apply_exposures(timestep)`** — Apply seed exposures from scenario rules (`propagation.py`), then propagate through network via conviction-gated sharing (very_uncertain agents don't share). Uses pre-built adjacency list for O(1) neighbor lookups.
+2. **`_reason_agents(timestep)`** — Select agents to reason (first exposure OR multi-touch threshold exceeded, default: 3 new exposures since last reasoning), filter out already-processed agents (resume support), split into chunks of `chunk_size` (default 50), run two-pass async LLM reasoning per chunk (`reasoning.py`, rate-limiter-controlled), commit per chunk:
+   - **Pass 1 (role-play)**: Agent reasons in first person with no categorical enums. Produces reasoning, public_statement, sentiment, conviction (0-100 integer, bucketed post-hoc), will_share. Memory trace (last 3 reasoning summaries) is fed back for re-reasoning agents.
+   - **Pass 2 (classification)**: A cheap model classifies the free-text reasoning into scenario-defined categorical/boolean/float outcomes. Position is extracted here — it is output-only, never used in peer influence.
+3. **`_process_reasoning_chunk(timestep, results, old_states)`** — Process a chunk of reasoning results: conviction-based flip resistance (firm+ agents reject flips unless new conviction is moderate+), conviction-gated sharing, state persistence. State updates are batched via `batch_update_states()`.
+4. Compute timestep summary, check stopping conditions (`stopping.py`) — Compound conditions like `"exposure_rate > 0.95 and no_state_changes_for > 10"`, convergence detection via sentiment variance.
+
+**Semantic peer influence**: Agents see peers' `public_statement` + sentiment tone, NOT position labels.
+
+**Adjacency list**: Built at engine init from network edges (both directions). Stored as `dict[str, list[tuple[str, dict]]]`. Passed to `propagate_through_network()` and used in `_get_peer_opinions()` for O(1) neighbor lookups instead of O(E) scans.
+
+**Two-pass reasoning rationale**: Single-pass reasoning caused 83% of agents to pick safe middle options (central tendency bias). Splitting role-play from classification fixes this — agents reason naturally in Pass 1, then a cheap model maps to categories in Pass 2.
+
+**Checkpointing** (`state.py`, `engine.py`): Each timestep phase has its own transaction — exposures, each reasoning chunk, and summary are committed separately. The `simulation_metadata` table stores checkpoint state: `mark_timestep_started()` records which timestep is in progress, `mark_timestep_completed()` clears it. On resume (`_get_resume_timestep()`), the engine detects crashed-mid-timestep (checkpoint set) or last-completed-timestep and picks up from there. Already-processed agents within a partial timestep are skipped via `get_agents_already_reasoned_this_timestep()`. Metadata uses immediate commits (not wrapped in `transaction()`). Setup methods (`_create_schema`, `initialize_agents`) also retain their own commits. CLI: `--chunk-size` (default 50).
+
+**Live progress** (`progress.py`, `cli/commands/simulate.py`): `SimulationProgress` is a thread-safe dataclass shared between the simulation thread and CLI display thread. The engine calls `record_agent_done(position, sentiment, conviction)` per agent via an `on_agent_done` callback passed to `batch_reason_agents()`. The CLI renders a `Rich.Live` display at 4 Hz showing timestep/agent progress, exposure rate, elapsed time, and unicode distribution bars for each position. Position counts are cumulative across all timesteps. Verbose mode logs periodic summary blocks every 50 agents via `_log_verbose_summary()`.
+
+**Conviction system**: Agents output a 0-100 integer score on a free scale (with descriptive anchors: 0=no idea, 25=leaning, 50=clear opinion, 75=quite sure, 100=certain). `score_to_conviction_float()` buckets it immediately: 0-15→0.1 (very_uncertain), 16-35→0.3 (leaning), 36-60→0.5 (moderate), 61-85→0.7 (firm), 86-100→0.9 (absolute). Agents never see categorical labels or float values — only the 0-100 scale. On re-reasoning, memory traces show the bucketed label (e.g. "you felt *moderate* about this") via `float_to_conviction()`. Engine conviction thresholds reference `CONVICTION_MAP[ConvictionLevel.*]` constants, not hardcoded floats.
+
+**Cost estimation** (`simulation/estimator.py`): `extropy estimate` runs a simplified SIR-like propagation model to predict LLM calls per timestep without making any API calls. Token counts estimated from persona size + scenario content. Pricing from `core/pricing.py` model database. Supports `--verbose` for per-timestep breakdown.
+
+**Rate limiter** (`core/rate_limiter.py`): Token bucket with dual RPM + TPM buckets. Provider-aware defaults from `core/rate_limits.py` (Anthropic/OpenAI, tiers 1-4). Replaces the old hardcoded `Semaphore(50)`. CLI flags: `--rate-tier`, config: `simulation.rate_tier`, `simulation.rpm_override`, `simulation.tpm_override`.
+
+**Persona system** (`population/persona/` + `simulation/persona.py`): The `extropy persona` command generates a `PersonaConfig` via 5-step LLM pipeline (structure → boolean → categorical → relative → concrete phrasings). At simulation time, agents are rendered computationally using this config — no per-agent LLM calls. Relative attributes (personality, attitudes) are positioned against population stats via z-scores ("I'm much more price-sensitive than most people"). Concrete attributes use format specs for proper number/time rendering. **Trait salience**: If `decision_relevant_attributes` is set on `OutcomeConfig`, those attributes are grouped first under "Most Relevant to This Decision" in the persona.
+
+## LLM Integration (`extropy/core/llm.py`)
+
+All LLM calls go through this file — never call providers directly elsewhere. Two-zone routing:
+
+**Pipeline zone** (phases 1-2: spec, extend, persona, scenario) — configured via `extropy config set pipeline.*`:
+
+| Function | Default Model | Tools | Use |
+|----------|--------------|-------|-----|
+| `simple_call()` | provider default (haiku/gpt-5-mini) | none | Sufficiency checks, simple extractions |
+| `reasoning_call()` | provider default (sonnet/gpt-5) | none | Attribute selection, hydration, scenario compilation. Supports validator callback + retry |
+| `agentic_research()` | provider default (sonnet/gpt-5) | web_search | Distribution hydration with real-world data. Extracts source URLs |
+
+**Simulation zone** (phase 3: agent reasoning) — configured via `extropy config set simulation.*`:
+
+| Function | Default Model | Tools | Use |
+|----------|--------------|-------|-----|
+| `simple_call_async()` | provider default | none | Pass 1 role-play reasoning + Pass 2 classification (async) |
+
+Two-pass model routing: Pass 1 uses `simulation.model` (pivotal reasoning). Pass 2 uses `simulation.routine_model` (cheap classification). Both default to provider default if not set. CLI: `--model`, `--pivotal-model`, `--routine-model`. Standard inference only — no thinking/extended models (no o1, o3, extended thinking).
+
+**Provider abstraction** (`extropy/core/providers/`): `LLMProvider` base class with `OpenAIProvider` and `ClaudeProvider` implementations. Factory functions `get_pipeline_provider()` and `get_simulation_provider()` read from `ExtropyConfig`. Base class provides `_retry_with_validation()` — shared validation-retry loop used by both providers' `reasoning_call()` and `agentic_research()`. Both providers implement `_with_retry()` / `_with_retry_async()` for transient API errors (APIConnectionError, InternalServerError, RateLimitError) with exponential backoff (`2^attempt + random(0,1)` seconds, max 3 retries).
+
+**Config** (`extropy/config.py`): `ExtropyConfig` with `PipelineConfig` and `SimZoneConfig` zones. Resolution order: env vars > config file (`~/.config/extropy/config.json`) > defaults. API keys always from env vars (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`). For package use: `from extropy.config import configure, ExtropyConfig`.
+
+**Default zones**: Pipeline = Claude (population/scenario building). Simulation = OpenAI (agent reasoning). `SimZoneConfig` fields: `provider`, `model`, `pivotal_model`, `routine_model`, `max_concurrent`, `rate_tier`, `rpm_override`, `tpm_override`.
+
+All calls use structured output (`response_format: json_schema`). Failed validations are fed back as "PREVIOUS ATTEMPT FAILED" prompts for self-correction.
+
+## Data Models (`extropy/core/models/`)
+
+All Pydantic v2. Key hierarchy:
+
+- `population.py`: `PopulationSpec` → `AttributeSpec` → `SamplingConfig` → `Distribution` / `Modifier` / `Constraint`
+- `scenario.py`: `ScenarioSpec` → `Event`, `SeedExposure` (channels + rules), `InteractionConfig`, `SpreadConfig`, `OutcomeConfig`
+- `simulation.py`: `ConvictionLevel`, `MemoryEntry`, `AgentState` (conviction, public_statement), `PeerOpinion` (public_statement, credibility), `ReasoningContext` (memory_trace), `ReasoningResponse` (conviction, public_statement, reasoning_summary), `SimulationRunConfig` (pivotal_model, routine_model, chunk_size), `TimestepSummary` (average_conviction, sentiment_variance)
+- `network.py`: `Edge`, `NodeMetrics`, `NetworkMetrics`
+- `validation.py`: `ValidationIssue`, `ValidationResult`
+
+YAML serialization via `to_yaml()`/`from_yaml()` on `PopulationSpec` and `ScenarioSpec`.
+
+## Validation (`extropy/population/validator/`)
+
+Two layers for population specs:
+- **Structural** (`structural.py`): ERROR-level — type/modifier compatibility, range violations, distribution params, dependency cycles, condition syntax, formula references, duplicates, strategy consistency
+- **Semantic** (`semantic.py`): WARNING-level — no-op detection, modifier stacking, categorical option reference validity
+
+Scenario validation (`extropy/scenario/validator.py`): attribute reference validity, edge type references, probability ranges.
+
+## Key Conventions
+
+- Conditions and formulas use restricted Python syntax via `eval_safe()` — whitelisted builtins only (abs, min, max, round, int, float, str, len, sum, all, any, bool)
+- Agent IDs use the `_id` field from agent JSON, falling back to string index
+- Network edges are bidirectional (stored as source/target, traversed both ways)
+- Exposure credibility: `event_credibility * channel_credibility` for seed, fixed 0.85 for peer
+- "Position" = first required categorical outcome (extracted in Pass 2, used for aggregation/output only — never used in peer influence)
+- Peer influence is semantic: agents see neighbors' `public_statement` + sentiment tone, not position labels
+- Conviction: agents output 0-100 integer, bucketed to 5 float levels (0.1/0.3/0.5/0.7/0.9) via `score_to_conviction_float()`. Agents see only the 0-100 scale; memory traces show categorical labels. Engine thresholds reference `CONVICTION_MAP[ConvictionLevel.*]`
+- Memory traces: 3-entry sliding window per agent, fed back into reasoning prompts for re-reasoning
+- Progress callbacks use typed `Protocol` classes from `extropy/utils/callbacks.py` (`StepProgressCallback`, `TimestepProgressCallback`, `ItemProgressCallback`, `HydrationProgressCallback`, `NetworkProgressCallback`) — structurally compatible with plain callables via duck typing
+- The `persona` command generates detailed persona configs; `extend` still generates a simpler `persona_template` for backwards compatibility
+- Simulation auto-detects `{population_stem}.persona.yaml` and uses the new rendering if present
+- Network config is data-driven via `NetworkConfig` (YAML-serializable). No built-in reference preset is shipped. CLI: `extropy network -p population.yaml` (LLM-generated), `-c config.yaml` (manual), `--save-config` (export)
+
+## Tests
+
+pytest + pytest-asyncio. Fixtures in `tests/conftest.py` include seeded RNG (`Random(42)`), minimal/complex population specs, and sample agents. 580+ tests across 15+ test files:
+
+- `test_models.py`, `test_network.py`, `test_sampler.py`, `test_scenario.py`, `test_simulation.py`, `test_validator.py` — core logic
+- `test_engine.py` — mock-based engine integration (seed exposure, flip resistance, conviction-gated sharing, chunked reasoning, checkpointing, resume logic, metadata lifecycle, progress state wiring)
+- `test_progress.py` — SimulationProgress thread-safe state (begin_timestep, record_agent_done, position counts, running averages, snapshot isolation)
+- `test_compiler.py` — scenario compiler pipeline with mocked LLM calls, auto-configuration
+- `test_providers.py` — provider response extraction, transient error retry, validation-retry exhaustion, source URL extraction (mocked HTTP)
+- `test_rate_limiter.py` — token bucket, dual-bucket rate limiter, `for_provider` factory
+- `test_cli.py` — CLI smoke tests (`config show/set`, `validate`, `--version`)
+- `test_estimator.py` — cost estimation, pricing lookup, token estimation
+
+CI: `.github/workflows/test.yml` — lint (ruff check + format) and test (pytest, matrix: Python 3.11/3.12/3.13) via `astral-sh/setup-uv@v4`.
+
+## File Formats
+
+- Population/scenario specs: YAML
+- Network config: YAML (`NetworkConfig.to_yaml()`/`from_yaml()`) — attribute weights, edge type rules, influence factors, degree multipliers
+- Agents: JSON (array of objects with `_id`)
+- Network: JSON (`{meta, nodes, edges}`)
+- Simulation state: SQLite (tables: agent_states, exposures, memory_traces, timeline, timestep_summaries, shared_to, simulation_metadata)
+- Timeline: JSONL (streaming, crash-safe)
+- Results: JSON files in output directory (agent_states.json, by_timestep.json, outcome_distributions.json, meta.json)

extropy_run-0.2.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025-2026 Devesh Paragiri
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.