hypermind 0.23.0__tar.gz → 0.24.0__tar.gz

{hypermind-0.23.0 → hypermind-0.24.0}/CHANGELOG.md

@@ -14,7 +14,44 @@ changes only on major-version bumps.
 
 ## [Unreleased]
 
-_Current trunk; new work after v0.11.0._
+_Current trunk._
+
+## [0.23.0] — 2026-07-08
+
+### Added
+- **Distributed, Locust-style execution** — `hypermind.orgs.distributed`. A master coordinator shards a swarm across workers (other machines or local processes) over a TCP control plane (length-prefixed CBOR frames); each worker runs its shard as `OrgSim(lightweight=True)` against the **shared Relata namespace** and returns bounded aggregates, merged into one `OrgReport` (#177).
+  - `Coordinator` / `Worker`, `run_distributed(...)` (single-host convenience), and a `master` / `worker` CLI (`python -m hypermind.orgs.distributed`).
+  - Capacity-weighted shard planning; a worker that drops before returning is recorded as a failed shard — the run completes and notes it, never hangs.
+  - `OrgSim.export_aggregates()` for the merge; `OrgModel` wire (de)serialisation.
+
+## [0.22.0] — 2026-07-08
+
+### Added
+- **Batched Relata writes** — `RelataBackend(write_batch_size=N)` buffers claim rows and flushes via one bulk `batch_upsert` instead of a per-claim round-trip; `OrgSim` defaults to 64. ~batch× fewer round-trips at scale (#175).
+- **Bounded / streaming report** — `OrgSim(lightweight=True)` caps memory to O(members + windows), not O(records): per-member streaming aggregates + a rolling claim window, exact scorecard totals. `metrics.scorecard_from_aggregates(...)` (#176).
+- Parallel keypair generation at `testbed()` spin-up (thread pool; the crypto releases the GIL) — part of #177.
+
+## [0.21.0] — 2026-07-08
+
+### Added
+- **Native response-quality rubrics** — `hypermind.orgs.quality`: `sim.assess(question)` → `QualityReport`. Four rubric groups, all on by default, switchable via `disable`: native/deterministic (provenance coverage, dispute exposure, reputation support, consensus, abstention — verifiable from the signed ledger), judge-based faithfulness (grounding, hallucination rate, answer relevance, context precision), calibration (Brier, self-consistency), and context recall. Missing deps degrade to informational notes. `llm_judge(...)` factory.
+
+## [0.20.0] — 2026-07-08
+
+### Added
+- **Hybrid retrieval** — `reciprocal_rank_fusion(...)` and `hybrid_retriever(*retrievers, reranker=None)` fuse dense + entity-graph + lexical signals (RRF) with an optional cross-encoder rerank, over our own Relata data. Plugs into the existing `OrgSim(retriever=...)` seam.
+
+## [0.12.0] – [0.19.0] — 2026-06 … 2026-07
+
+### Added
+- **`hypermind.orgs` — the living-simulation engine.** Turn a declarative `OrgModel` / `Unit` / `Seat` into a running multi-agent simulation (research → publish → cite → dispute → resolve → learn → evolve) persisted to Relata, driven by pluggable `Interaction` strategies (`cooperative` / `adversarial` / `crowd`).
+  - **Template library**: `university`, `hospital`, `city_at_war`, `foreign_policy_cabinet`, `un_security_council`, `internet_crowd`, `research_lab`, … via `list_templates()` / `get_template()` — the core never imports them.
+  - **Per-agent models** (per-seat provider/model or a router) and **group-scoped native tools** (declarative or random grants; ungranted tools are absent, not refused).
+  - **Study → ask**: `sim.study(topic)` then `sim.ask(question)` — a transparent, queryable swarm that answers from what it learned, weighted by earned standing.
+  - **Proper RAG** for `ask`: lexical → semantic embeddings (`embedder=`, `openai_embedder`) → claim↔entity graph (`use_graph_retrieval`, `MentionGraph`) → real NER hooks (`llm_ner_extractor`, `chat_ner_extractor`, `pioneer_ner_extractor`) → Relata-backed retrieval (`relata_retriever`).
+  - **Scorecard & analysis**: reputation ranking, citation DAG, dispute ledger, Brier calibration (#151), convergence metrics (#152), memory-RAG (#153), deliberator mode (#154), tool-trace capture (#155), adversarial dynamics (#156), pluggable ground truth (#157), experiment/variant harness (#158), interventions (#159), visualization snapshots (#160), verifiable receipts (#161).
+- **Scalability hardening** across the engine: bounded peer window, O(1) member indices, top-k cross-pollination, concurrent think phase (`asyncio.gather` + semaphore) and concurrent blocking I/O (`asyncio.to_thread`).
+- `examples/org_university.py` and an expanded examples index.
 
 ## [0.11.0] — 2026-05-10
 

{hypermind-0.23.0 → hypermind-0.24.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypermind
-Version: 0.23.0
+Version: 0.24.0
 Summary: Federated agent intelligence — reference SDK for the HyperMind SCITT contested-claims profile
 Project-URL: Homepage, https://github.com/ZySec-AI/hypermind
 Project-URL: Documentation, https://github.com/ZySec-AI/hypermind/tree/main/docs
@@ -102,7 +102,7 @@ Agents share what they learn. Dispute what they doubt. Build trust over time.
 [![PyPI](https://img.shields.io/pypi/v/hypermind?color=4c8cff&label=pip%20install%20hypermind)](https://pypi.org/project/hypermind/)
 [![Python](https://img.shields.io/pypi/pyversions/hypermind?color=4c8cff)](https://pypi.org/project/hypermind/)
 [![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-1753%20passing-brightgreen)](#testing--quality)
+[![Tests](https://img.shields.io/badge/tests-2133%20passing-brightgreen)](#testing--quality)
 [![Coverage](https://img.shields.io/badge/coverage-95%25-brightgreen)](#testing--quality)
 [![Status](https://img.shields.io/badge/status-v0.10-blue)](#project-status)
 
@@ -147,7 +147,7 @@ Open **http://127.0.0.1:8765** and you'll see:
 ```bash
 make run-fast      # skip the webui rebuild (when JS hasn't changed)
 make webui-dev     # Vite dev server with hot-reload on :5173
-make test          # 1753 tests pass, ~80s
+make test          # 2133 tests pass, ~80s
 make demo          # examples/01_hello_world from the CLI
 make demos         # all numbered tutorials + scenarios sequentially
 make help          # show all targets
@@ -232,6 +232,10 @@ report.print()        # reputation ranking · citation graph · dispute ledger
 - **Per-agent models.** Any seat can pin its own provider/model (`Seat("Professor", llm="anthropic", model="claude-opus-4-8")`), or hand `OrgSim` a router for random / cost-aware assignment.
 - **Native tools, group-scoped.** Tools (web search, retrieval, KB query, Relata research) are granted per unit/seat and enforced — an ungranted tool is *absent* from that agent's toolset. Grants can be declarative or randomised; register your own with `hypermind.orgs.register_tool(...)`.
 - **Relata end-to-end.** Organisation → tenant; member → private memory session (agent kid); claims / disputes / reputation → tenant-scoped ledger rows; chain of custody → the Merkle log. Restart replays the whole org.
+- **Study → ask (a queryable swarm).** Let the swarm learn a topic, then question it: `await sim.study(topic)` then `await sim.ask("...")` — every member recalls what it learned, reasons, and the answers are aggregated weighted by earned standing. `sim.ask` studies first automatically if you skip the explicit step.
+- **Proper RAG.** Retrieval for `ask` is pluggable: lexical (default), semantic embeddings (`embedder=`), the claim↔entity graph (`use_graph_retrieval=True`), or `hybrid_retriever(...)` which fuses dense + graph + lexical via Reciprocal Rank Fusion with an optional cross-encoder rerank. Relata *is* the vector store + graph (`relata_retriever(..., "search_knowledge" / "find_connections")`).
+- **Answer quality, natively.** `await sim.assess(question)` returns a `QualityReport` scoring grounding, hallucination rate, calibration (Brier), self-consistency, and hm-native rubrics that are *verifiable from the signed ledger* — provenance coverage, dispute exposure, reputation-weighted support. All rubrics on by default; missing deps (judge / ground-truth) degrade to informational notes.
+- **Scales like Locust.** `OrgSim(lightweight=True)` caps RAM to O(members), not O(records); Relata writes are batched. For horizontal scale, `hypermind.orgs.distributed` runs a **master/worker** control plane — shard a swarm across processes or machines against the shared Relata namespace, merged into one report (`python -m hypermind.orgs.distributed master/worker`, or `run_distributed(...)`).
 
 Worked example: [`examples/org_university.py`](examples/org_university.py). Discover all presets via `hypermind.orgs.templates.list_templates()`.
 

{hypermind-0.23.0 → hypermind-0.24.0}/README.md

@@ -14,7 +14,7 @@ Agents share what they learn. Dispute what they doubt. Build trust over time.
 [![PyPI](https://img.shields.io/pypi/v/hypermind?color=4c8cff&label=pip%20install%20hypermind)](https://pypi.org/project/hypermind/)
 [![Python](https://img.shields.io/pypi/pyversions/hypermind?color=4c8cff)](https://pypi.org/project/hypermind/)
 [![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-1753%20passing-brightgreen)](#testing--quality)
+[![Tests](https://img.shields.io/badge/tests-2133%20passing-brightgreen)](#testing--quality)
 [![Coverage](https://img.shields.io/badge/coverage-95%25-brightgreen)](#testing--quality)
 [![Status](https://img.shields.io/badge/status-v0.10-blue)](#project-status)
 
@@ -59,7 +59,7 @@ Open **http://127.0.0.1:8765** and you'll see:
 ```bash
 make run-fast      # skip the webui rebuild (when JS hasn't changed)
 make webui-dev     # Vite dev server with hot-reload on :5173
-make test          # 1753 tests pass, ~80s
+make test          # 2133 tests pass, ~80s
 make demo          # examples/01_hello_world from the CLI
 make demos         # all numbered tutorials + scenarios sequentially
 make help          # show all targets
@@ -144,6 +144,10 @@ report.print()        # reputation ranking · citation graph · dispute ledger
 - **Per-agent models.** Any seat can pin its own provider/model (`Seat("Professor", llm="anthropic", model="claude-opus-4-8")`), or hand `OrgSim` a router for random / cost-aware assignment.
 - **Native tools, group-scoped.** Tools (web search, retrieval, KB query, Relata research) are granted per unit/seat and enforced — an ungranted tool is *absent* from that agent's toolset. Grants can be declarative or randomised; register your own with `hypermind.orgs.register_tool(...)`.
 - **Relata end-to-end.** Organisation → tenant; member → private memory session (agent kid); claims / disputes / reputation → tenant-scoped ledger rows; chain of custody → the Merkle log. Restart replays the whole org.
+- **Study → ask (a queryable swarm).** Let the swarm learn a topic, then question it: `await sim.study(topic)` then `await sim.ask("...")` — every member recalls what it learned, reasons, and the answers are aggregated weighted by earned standing. `sim.ask` studies first automatically if you skip the explicit step.
+- **Proper RAG.** Retrieval for `ask` is pluggable: lexical (default), semantic embeddings (`embedder=`), the claim↔entity graph (`use_graph_retrieval=True`), or `hybrid_retriever(...)` which fuses dense + graph + lexical via Reciprocal Rank Fusion with an optional cross-encoder rerank. Relata *is* the vector store + graph (`relata_retriever(..., "search_knowledge" / "find_connections")`).
+- **Answer quality, natively.** `await sim.assess(question)` returns a `QualityReport` scoring grounding, hallucination rate, calibration (Brier), self-consistency, and hm-native rubrics that are *verifiable from the signed ledger* — provenance coverage, dispute exposure, reputation-weighted support. All rubrics on by default; missing deps (judge / ground-truth) degrade to informational notes.
+- **Scales like Locust.** `OrgSim(lightweight=True)` caps RAM to O(members), not O(records); Relata writes are batched. For horizontal scale, `hypermind.orgs.distributed` runs a **master/worker** control plane — shard a swarm across processes or machines against the shared Relata namespace, merged into one report (`python -m hypermind.orgs.distributed master/worker`, or `run_distributed(...)`).
 
 Worked example: [`examples/org_university.py`](examples/org_university.py). Discover all presets via `hypermind.orgs.templates.list_templates()`.
 

{hypermind-0.23.0 → hypermind-0.24.0}/examples/README.md

@@ -87,6 +87,7 @@ Pick one that matches your domain.
 | File | Domain | What it shows |
 |---|---|---|
 | [`org_university.py`](org_university.py) | Any network | Spin up a templated org (university / city / UNO / crowd), run research → cite → dispute → resolve → evolve, print a reputation + citation report. Per-agent models + group-scoped tools, persisted to Relata. |
+| [`org_distributed.py`](org_distributed.py) | Scale-out | Locust-style distributed run: a master coordinator shards the swarm across workers (processes or machines) against the shared Relata namespace, merged into one report (`hypermind.orgs.distributed`). Single-host convenience + multi-machine CLI. |
 
 ### Pilot-shaped demonstrations
 

hypermind-0.24.0/examples/org_distributed.py

@@ -0,0 +1,74 @@
+"""THE STORY — one swarm, many workers: scale a simulation like Locust.
+
+The single-process engine (``examples/org_university.py``) is great to ~1000s of
+agents. Beyond that you scale *out*: ``hypermind.orgs.distributed`` runs a
+**master coordinator** that shards the swarm across **workers** — processes on
+this box or on other machines — each running its slice as
+``OrgSim(lightweight=True)`` against the **shared Relata namespace** (the source
+of truth). The coordinator merges every worker's bounded aggregates into one
+``OrgReport``.
+
+This script shows the single-host convenience path (``run_distributed`` starts a
+coordinator plus in-process workers over loopback) — the *same* code path as
+multi-machine. For true multi-machine, use the CLI instead::
+
+    # on the master host
+    python -m hypermind.orgs.distributed master --template university \
+        --topic "..." --members 10000 --rounds 8 --expected-workers 4 \
+        --host 0.0.0.0 --port 8765 --llm openai --llm-model openai/gpt-4o-mini
+
+    # on each worker host
+    python -m hypermind.orgs.distributed worker \
+        --master-host <master-ip> --port 8765 --capacity 2500
+
+PREREQUISITES
+  * A running Relata instance:   docker compose -f deploy/docker-compose.yml up -d relata
+  * export RELATA_URL=http://localhost:9090   RELATA_TOKEN=...
+  * export OPENAI_API_KEY=...   (workers build their own client from env)
+
+Run with:   python -m examples.org_distributed
+"""
+
+from __future__ import annotations
+
+import asyncio
+import os
+
+
+async def main() -> None:
+    from hypermind.orgs.distributed import run_distributed
+    from hypermind.orgs.templates import university
+
+    relata_url = os.environ.get("RELATA_URL")
+    if not relata_url:
+        raise SystemExit(
+            "Set RELATA_URL (and RELATA_TOKEN) to a running Relata instance. "
+            "See deploy/docker-compose.yml."
+        )
+
+    model = university()  # the shape; run_distributed scales it across workers
+    report = await run_distributed(
+        model,
+        topic="What limits the accuracy of ab-initio protein folding?",
+        members=48,  # sharded across the workers below
+        rounds=4,
+        workers=4,  # 4 in-process workers over loopback (each ~12 members)
+        config={
+            "llm": "openai",
+            "llm_model": "openai/gpt-4o-mini",
+            "base_url": os.environ.get("OPENAI_BASE_URL"),  # e.g. OpenRouter
+            "relata_url": relata_url,
+            "relata_token": os.environ.get("RELATA_TOKEN"),
+        },
+    )
+
+    report.print()
+    sc = report.scorecard
+    print(
+        f"\nmerged across workers → {sc['claims']} claims · {sc['citations']} citations · "
+        f"{len(report.standings)} members"
+    )
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

{hypermind-0.23.0 → hypermind-0.24.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hypermind"
-version = "0.23.0"
+version = "0.24.0"
 description = "Federated agent intelligence — reference SDK for the HyperMind SCITT contested-claims profile"
 readme = "README.md"
 requires-python = ">=3.11,<3.14"

{hypermind-0.23.0 → hypermind-0.24.0}/src/hypermind/__init__.py

@@ -73,7 +73,7 @@ open = HyperMindAgent.open
 #: is held until the §External & human gates clear (IETF I-D, IANA,
 #: third-party audit, OpenSSF acceptance, ≥3 pilots) — at that point we
 #: cut ``1.0.0`` GA. See ROADMAP.md §Status snapshot.
-__version__ = "0.23.0"
+__version__ = "0.24.0"
 
 #: SLSA build-provenance level committed for every release artefact.
 #: Cosign keyless signing + slsa-github-generator + PEP 740 attestations

{hypermind-0.23.0 → hypermind-0.24.0}/src/hypermind/orgs/distributed/__init__.py

@@ -29,24 +29,41 @@ Single host (convenience — coordinator + in-process workers over loopback)::
 from __future__ import annotations
 
 import asyncio
+import logging
 from typing import Any
 
 from hypermind.orgs.distributed.coordinator import Coordinator
-from hypermind.orgs.distributed.planning import merge_results, plan_shards
+from hypermind.orgs.distributed.planning import (
+    merge_results,
+    plan_shards,
+    seat_count,
+    validate_shardable,
+)
+from hypermind.orgs.distributed.provision import provision_hm_types
 from hypermind.orgs.distributed.worker import Worker, default_shard_runner
 from hypermind.orgs.model import OrgModel
 from hypermind.orgs.report import OrgReport
 
+logger = logging.getLogger("hypermind.orgs.distributed")
+
 __all__ = [
     "Coordinator",
+    "DistributedError",
     "Worker",
     "default_shard_runner",
     "merge_results",
     "plan_shards",
+    "provision_hm_types",
     "run_distributed",
+    "seat_count",
+    "validate_shardable",
 ]
 
 
+class DistributedError(RuntimeError):
+    """Raised when a distributed run cannot produce any result (all shards failed)."""
+
+
 async def run_distributed(
     model: OrgModel,
     topic: str,
@@ -66,7 +83,13 @@ async def run_distributed(
     Same code path as multi-machine — the only difference is the workers live in
     this process instead of on remote hosts. Pass ``shard_runner`` to stub the
     per-shard work (tests); omit it for the real ``OrgSim`` runner.
+
+    Raises ``ValueError`` up front if the shard plan isn't viable (#190) and
+    ``DistributedError`` if every shard fails (#191).
     """
+    # Pre-flight: fail fast with a clear message if a shard would be too small.
+    validate_shardable(model, members, [capacity_each] * workers)
+
     coord = Coordinator(
         model,
         topic,
@@ -100,4 +123,13 @@ async def run_distributed(
     ]
     report = await coord_task
     await asyncio.gather(*worker_tasks, return_exceptions=True)
+
+    # Surface shard failures loudly (#191).
+    failed = next((e for e in report.events if e["event"] == "shards_failed"), None)
+    succeeded = next((e["shards"] for e in report.events if e["event"] == "distributed"), 0)
+    if failed:
+        msg = f"{failed['count']} shard(s) failed: {failed['detail']}"
+        if succeeded == 0:
+            raise DistributedError(f"distributed run produced no results — {msg}")
+        logger.warning("distributed run completed with failures — %s", msg)
     return report

hypermind-0.24.0/src/hypermind/orgs/distributed/coordinator.py

@@ -0,0 +1,251 @@
+"""The master: a resilient, Locust-style coordinator over a TCP control plane (#177).
+
+Drives the whole run: provisions the shared Relata state (#192), gathers workers,
+plans shards, and dispatches them from a **work queue**. A shard whose worker
+errors or disconnects is **requeued and reassigned** to another available worker
+(bounded retries, #191/resilience) — the run only fails a shard after exhausting
+retries, and never hangs. Results merge into one :class:`OrgReport`.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import logging
+from collections import deque
+from typing import Any
+
+from hypermind.orgs.distributed import protocol as p
+from hypermind.orgs.distributed.planning import merge_results, validate_shardable
+from hypermind.orgs.distributed.provision import provision_hm_types
+from hypermind.orgs.distributed.serde import model_to_dict
+from hypermind.orgs.model import OrgModel
+from hypermind.orgs.report import OrgReport
+
+logger = logging.getLogger("hypermind.orgs.distributed")
+
+
+class Coordinator:
+    """Master coordinator. Provision → gather workers → dispatch (with retry) → merge."""
+
+    def __init__(
+        self,
+        model: OrgModel,
+        topic: str,
+        *,
+        members: int,
+        rounds: int,
+        expected_workers: int,
+        host: str = "127.0.0.1",
+        port: int = 0,
+        config: dict[str, Any] | None = None,
+        worker_timeout_s: float = 600.0,
+        max_retries: int = 2,
+        provision: bool = True,
+    ) -> None:
+        self.model = model
+        self.topic = topic
+        self.members = members
+        self.rounds = rounds
+        self.expected_workers = expected_workers
+        self.host = host
+        self.port = port
+        self.config = dict(config or {})
+        self.worker_timeout_s = worker_timeout_s
+        self.max_retries = max(0, max_retries)
+        self.provision = provision
+
+        self._writers: dict[str, asyncio.StreamWriter] = {}
+        self._order: list[str] = []
+        self._capacity: dict[str, int] = {}
+        self._alive: set[str] = set()
+        self._idle: deque[str] = deque()
+
+        self._shard_members: dict[int, int] = {}  # shard_id → member count
+        self._pending: deque[int] = deque()  # shard_ids awaiting a worker
+        self._inflight: dict[str, int] = {}  # worker_id → shard_id
+        self._attempts: dict[int, int] = {}
+        self._results: dict[int, dict[str, Any]] = {}
+        self._failed: dict[int, str] = {}
+        self._n_shards = 0
+
+        self._enough_workers = asyncio.Event()
+        self._complete = asyncio.Event()
+        self._planned = False
+        self._lock = asyncio.Lock()
+
+    async def run(self) -> OrgReport:
+        if self.provision and self.config.get("relata_url"):
+            # Master drives Relata state: register the ledger types up front.
+            result = await asyncio.to_thread(
+                provision_hm_types,
+                self.config["relata_url"],
+                namespace=self.model.namespace,
+                token=self.config.get("relata_token"),
+            )
+            logger.info("provisioned Hm types: %s", result)
+
+        server = await asyncio.start_server(self._on_conn, self.host, self.port)
+        self.port = server.sockets[0].getsockname()[1]
+        async with server:
+            try:
+                await asyncio.wait_for(self._enough_workers.wait(), self.worker_timeout_s)
+                await self._plan_and_dispatch()
+                await asyncio.wait_for(self._complete.wait(), self.worker_timeout_s)
+            finally:
+                await self._quit_all()
+        return self._merge()
+
+    # ---- connection lifecycle ------------------------------------------
+    async def _on_conn(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
+        worker_id: str | None = None
+        try:
+            hello = await p.read_frame(reader)
+            if hello.get("type") != p.HELLO:
+                writer.close()
+                return
+            worker_id = str(hello["worker_id"])
+            async with self._lock:
+                self._writers[worker_id] = writer
+                self._capacity[worker_id] = max(0, int(hello.get("capacity", 1)))
+                self._alive.add(worker_id)
+                self._order.append(worker_id)
+                self._idle.append(worker_id)
+                if len(self._order) >= self.expected_workers:
+                    self._enough_workers.set()
+            if self._planned:
+                await self._dispatch()  # a late worker can pick up pending/retry work
+            await self._read_loop(worker_id, reader)
+        except (asyncio.IncompleteReadError, ConnectionError, RuntimeError):
+            pass
+        finally:
+            if worker_id is not None:
+                await self._on_worker_gone(worker_id)
+
+    async def _read_loop(self, worker_id: str, reader: asyncio.StreamReader) -> None:
+        while True:
+            msg = await p.read_frame(reader)
+            mtype = msg.get("type")
+            if mtype == p.RESULT:
+                await self._on_result(worker_id, int(msg["shard_id"]), msg["aggregates"])
+            elif mtype == p.ERROR:
+                await self._on_shard_error(worker_id, int(msg["shard_id"]), str(msg.get("message", "worker error")))
+            # HEARTBEAT / unknown: keep listening.
+
+    # ---- scheduling ----------------------------------------------------
+    async def _plan_and_dispatch(self) -> None:
+        async with self._lock:
+            if self._planned:
+                return
+            caps = [self._capacity[w] for w in self._order]
+            counts = validate_shardable(self.model, self.members, caps)  # raises on bad plan (#190)
+            for n in counts:
+                if n > 0:
+                    sid = self._n_shards
+                    self._shard_members[sid] = n
+                    self._pending.append(sid)
+                    self._n_shards += 1
+            self._planned = True
+            if self._n_shards == 0:
+                self._complete.set()
+        await self._dispatch()
+
+    async def _dispatch(self) -> None:
+        async with self._lock:
+            model_d = model_to_dict(self.model)
+            while self._idle and self._pending:
+                worker_id = self._idle.popleft()
+                if worker_id not in self._alive:
+                    continue
+                sid = self._pending.popleft()
+                self._inflight[worker_id] = sid
+                self._attempts[sid] = self._attempts.get(sid, 0) + 1
+                await self._send(
+                    worker_id,
+                    {
+                        "type": p.ASSIGN,
+                        "shard_id": sid,
+                        "model": model_d,
+                        "topic": self.topic,
+                        "rounds": self.rounds,
+                        "members": self._shard_members[sid],
+                        "namespace": self.model.namespace,
+                        "config": self.config,
+                    },
+                )
+
+    async def _on_result(self, worker_id: str, sid: int, agg: dict[str, Any]) -> None:
+        async with self._lock:
+            self._results[sid] = agg
+            self._inflight.pop(worker_id, None)
+            if worker_id in self._alive:
+                self._idle.append(worker_id)
+            self._maybe_complete()
+        await self._dispatch()
+
+    async def _on_shard_error(self, worker_id: str, sid: int, reason: str) -> None:
+        async with self._lock:
+            self._inflight.pop(worker_id, None)
+            self._requeue_or_fail(sid, reason)
+            if worker_id in self._alive:
+                self._idle.append(worker_id)
+            self._maybe_complete()
+        await self._dispatch()
+
+    async def _on_worker_gone(self, worker_id: str) -> None:
+        async with self._lock:
+            self._alive.discard(worker_id)
+            with contextlib.suppress(ValueError):
+                # deque has no remove-by-value guarantee across versions; rebuild
+                self._idle = deque(w for w in self._idle if w != worker_id)
+            sid = self._inflight.pop(worker_id, None)
+            if sid is not None and sid not in self._results:
+                self._requeue_or_fail(sid, "worker disconnected before returning a result")
+            self._maybe_complete()
+        await self._dispatch()
+
+    def _requeue_or_fail(self, sid: int, reason: str) -> None:
+        if sid in self._results:
+            return
+        if self._attempts.get(sid, 0) <= self.max_retries:
+            logger.warning("shard %d failed (%s) — requeuing (attempt %d)", sid, reason, self._attempts.get(sid, 0))
+            self._pending.append(sid)
+        else:
+            logger.error("shard %d permanently failed after %d attempts: %s", sid, self._attempts.get(sid, 0), reason)
+            self._failed[sid] = reason
+
+    def _maybe_complete(self) -> None:
+        if not self._planned:
+            return
+        # Done when every shard is resolved (result or permanent fail) and nothing
+        # is left to try or in flight.
+        settled = len(self._results) + len(self._failed)
+        if settled >= self._n_shards and not self._pending and not self._inflight:
+            self._complete.set()
+
+    async def _send(self, worker_id: str, msg: dict[str, Any]) -> None:
+        writer = self._writers.get(worker_id)
+        if writer is not None:
+            with contextlib.suppress(Exception):
+                await p.write_frame(writer, msg)
+
+    async def _quit_all(self) -> None:
+        for writer in list(self._writers.values()):
+            with contextlib.suppress(Exception):
+                await p.write_frame(writer, {"type": p.QUIT})
+                writer.close()
+
+    def _merge(self) -> OrgReport:
+        results = [{"shard_id": sid, **agg} for sid, agg in sorted(self._results.items())]
+        report = merge_results(
+            org=self.model.name,
+            topic=self.topic,
+            rounds=self.rounds,
+            interaction=self.model.dynamics.interaction,
+            results=results,
+        )
+        if self._failed:
+            report.events.append(
+                {"event": "shards_failed", "count": len(self._failed), "detail": dict(self._failed)}
+            )
+        return report

{hypermind-0.23.0 → hypermind-0.24.0}/src/hypermind/orgs/distributed/planning.py

@@ -12,9 +12,40 @@ from __future__ import annotations
 from typing import Any
 
 from hypermind.orgs.metrics import scorecard_from_aggregates
+from hypermind.orgs.model import OrgModel
 from hypermind.orgs.report import MemberStanding, OrgReport
 
 
+def seat_count(model: OrgModel) -> int:
+    """Distinct seats in the model — the minimum members a shard can run."""
+    return sum(len(u.seats) for u in model.units)
+
+
+def validate_shardable(model: OrgModel, members: int, capacities: list[int]) -> list[int]:
+    """Plan the shards and ensure each is viable (#190).
+
+    Every worker runs the *whole* model scaled to its shard, so a shard needs at
+    least ``seat_count(model)`` members. Raises a clear ``ValueError`` up front
+    instead of letting each worker fail silently. Returns the shard plan.
+    """
+    seats = seat_count(model)
+    if members < seats:
+        raise ValueError(
+            f"members={members} is fewer than the model's {seats} seats — "
+            f"even a single shard can't be built. Use members ≥ {seats}."
+        )
+    counts = plan_shards(members, capacities)
+    active = sum(1 for c in counts if c > 0)
+    smallest = min((c for c in counts if c > 0), default=0)
+    if smallest and smallest < seats:
+        raise ValueError(
+            f"{members} members across {active} workers ≈ {smallest}/shard, but the model "
+            f"has {seats} seats; each shard needs ≥{seats} members. "
+            f"Use members ≥ workers×{seats} (≥{active * seats}) or fewer workers."
+        )
+    return counts
+
+
 def plan_shards(members: int, capacities: list[int]) -> list[int]:
     """Split ``members`` across workers proportional to ``capacities``.
 

hypermind-0.24.0/src/hypermind/orgs/distributed/provision.py

@@ -0,0 +1,57 @@
+"""Master-driven Relata provisioning (#192).
+
+The governed Relata write path validates object types against the ontology, so
+the four HyperMind ledger types must be registered before any worker writes. The
+master calls :func:`provision_hm_types` on startup ("drive the state"), so a run
+against a fresh Relata just works. Registration is idempotent and persists on the
+server (replayed on restart).
+
+Note: ontology *registration* is runtime (this call). Granting the unauthenticated
+``api-user`` read+write on those types is a **boot-time** server setting in strict
+deployments: ``RELATA_ACL_GRANT=HmClaim:read+write,HmDispute:read+write,``
+``HmReputation:read+write,HmTranscript:read+write`` (or ``RELATA_ORG_MODE=open``
+for dev). We can't set server env from here, so we surface a clear hint if a
+write is later refused.
+"""
+
+from __future__ import annotations
+
+import json
+import urllib.error
+import urllib.request
+
+HM_TYPES = ("HmClaim", "HmDispute", "HmReputation", "HmTranscript")
+
+
+def provision_hm_types(
+    relata_url: str,
+    *,
+    namespace: str,
+    token: str | None = None,
+    types: tuple[str, ...] = HM_TYPES,
+    timeout_s: float = 10.0,
+) -> dict[str, str]:
+    """Register the Hm* ledger types on the Relata server (idempotent).
+
+    Returns ``{type_name: "registered" | "exists" | "error: ..."}``. Never
+    raises for an already-registered type; a transport error on one type is
+    recorded in the result rather than aborting the others.
+    """
+    out: dict[str, str] = {}
+    base = relata_url.rstrip("/")
+    for t in types:
+        body = json.dumps({"name": t}).encode()
+        headers = {"Content-Type": "application/json", "X-Organization-Id": namespace}
+        if token:
+            headers["Authorization"] = f"Bearer {token}"
+        req = urllib.request.Request(f"{base}/types", data=body, headers=headers, method="POST")
+        try:
+            with urllib.request.urlopen(req, timeout=timeout_s) as resp:
+                out[t] = "registered" if resp.status in (200, 201) else f"http {resp.status}"
+        except urllib.error.HTTPError as e:
+            # 409/400 "already registered" is success for our purposes.
+            detail = e.read().decode(errors="replace")[:120]
+            out[t] = "exists" if ("registered" in detail or e.code in (409, 422)) else f"error: http {e.code} {detail}"
+        except Exception as e:
+            out[t] = f"error: {type(e).__name__}: {e}"
+    return out