hypermind 0.16.0__tar.gz → 0.18.0__tar.gz

{hypermind-0.16.0 → hypermind-0.18.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypermind
-Version: 0.16.0
+Version: 0.18.0
 Summary: Federated agent intelligence — reference SDK for the HyperMind SCITT contested-claims profile
 Project-URL: Homepage, https://github.com/ZySec-AI/hypermind
 Project-URL: Documentation, https://github.com/ZySec-AI/hypermind/tree/main/docs

hypermind-0.18.0/RELATA_STORAGE.md

@@ -0,0 +1,192 @@
+# How `hypermind.orgs` stores information in RelataDB — a deep dive
+
+This documents exactly what a simulation writes to RelataDB, in what shape, and
+how it's partitioned — grounded in the code (`src/hypermind/storage/relata.py`,
+`src/hypermind/mind/memory_store.py`) and verified against a **live 30-agent
+run** (the `sa3-*` tenants).
+
+---
+
+## 1. Two storage surfaces
+
+A running swarm writes to Relata through **two independent surfaces**:
+
+| Surface | SDK class | What it holds | Signed? | Scope |
+|---|---|---|---|---|
+| **Ledger** | `RelataBackend` | claims, disputes, reputation, transcripts, Merkle log | ✅ yes (COSE) | shared per **org** |
+| **Memory** | `RelataMemoryStore` | each agent's private notes ("what it learned") | ❌ no | private per **agent** |
+
+```
+Organisation (namespace)  ──►  Relata TENANT      (X-Organization-Id header)
+    │
+    ├─ shared LEDGER  (purpose = "hm_ledger")
+    │     HmClaim · HmDispute · HmReputation · HmTranscript · Merkle log
+    │
+    └─ each MEMBER's private MEMORY  (purpose = "agent_notes")
+          session_id = agent kid   ← per-agent partition (isolation-enforced)
+```
+
+Three keys pin every write:
+- **tenant** = the org namespace (`X-Organization-Id`) — one tenant per org.
+- **session** = the agent's Ed25519 key id (`kid`) — the per-agent memory partition.
+- **purpose** = a governance token (`hm_ledger` / `agent_notes`) Relata scopes access by.
+
+---
+
+## 2. The shared ledger — `RelataBackend`
+
+Four custom Relata object types plus the transparency log. `purpose = "hm_ledger"`.
+
+### `HmClaim` — every sealed statement
+One row per published claim, synthesis (citation), **and** oracle resolution.
+
+| field | value |
+|---|---|
+| `object_id` | the claim **kid** (hex) — a content hash of the signed statement |
+| `seq` | monotonic append counter (survives restart via high-water read) |
+| `wire_b64` | **base64 of the full COSE_Sign1 signed envelope** (the CBOR wire bytes) |
+| `created_ms` | write timestamp |
+
+`append_claim` (`relata.py:122`) does `upsert("HmClaim", kid.hex(), {...})`;
+`load_claims` reads them back ordered by `seq` and re-verifies via
+`SignedStatement.from_bytes`.
+
+**Anatomy of one stored claim** (decoded from the live run):
+```
+COSE_Sign1 = [ protected_headers, unprotected, payload, signature ]   # a 4-element CBOR array
+                                        │
+        claim body (CBOR map) ──────────┘
+        { kind, ... , payload: {          ← the user dict the engine published
+            by:   "physics/Professor#1",
+            role: "oracle",
+            round: 0,
+            topic: "...",
+            confidence: 0.85,
+            finding: "The accuracy of ab-initio protein prediction is limited by …",  ← what the agent LEARNED
+            tools_used: []
+        } }
+```
+So **the agent's reasoning is stored inside the signed, content-addressed
+claim** — not in a loose column. The `kid` (object_id) is the hash of the whole
+envelope, so referencing a claim by kid inherently binds to signed content
+(this is what makes the SCITT receipt tamper-evident). One signed claim in the
+live run was **5,724 bytes** on the wire.
+
+The protected headers carry the load-bearing discriminators (`record_type`,
+`namespace_id`, `hlc`, `wire_version`, `signature_alg`, `envelope_schema_hash`,
+`pq_required`) — the **A-ALG-BOUND** property; `from_bytes()` re-encodes and
+checks them *before* verifying the signature.
+
+### `HmDispute` — bonded challenges
+| field | value |
+|---|---|
+| `object_id` / `claim_kid` | the disputed claim's kid (hex) |
+| `payload_b64` | base64 of the pickled `Dispute` (variable-length FSM state) |
+| `updated_ms` | timestamp |
+
+### `HmReputation` — standing, as a Beta posterior
+One row per **(agent, topic)** — topic is the unit/room. Written via
+`batch_upsert` (`relata.py:169`).
+
+| field | value |
+|---|---|
+| `id` | `"{agent_hex}:{topic}"` |
+| `agent_hex`, `topic` | who / where |
+| `alpha`, `beta` | Beta(α, β) parameters; score = α / (α+β) |
+| `last_update_ms` | for lazy time-decay |
+
+Reputation isn't a scalar — it's a **calibrated belief distribution** that
+rises on correct outcomes and is smoothly slashed on wrong ones.
+
+### `HmTranscript` — FROST nonce-reuse defence
+`id` = transcript id (hex); a durable test-and-set so a restart can't reopen
+the threshold-signing nonce window.
+
+### The Merkle log — chain of custody
+RFC 6962 transparency log via `LogClient.append(leaf)` (`relata.py:194`), giving
+a witness-cosigned `(root, tree_size)`. *(In the in-process `orgs` sandbox the
+transparency service runs in-memory, so Relata's log shows 0 leaves for these
+runs; production/federated mode routes leaves here.)*
+
+---
+
+## 3. Per-agent memory — `RelataMemoryStore`
+
+Each member gets a **private notebook** — this is where "what it learned"
+persists between rounds and across sessions. `purpose = "agent_notes"`.
+
+- `session_id = agent_kid.hex()` — a per-agent partition (`memory_store.py:72`).
+- `X-Organization-Id = namespace` — tenant header (`memory_store.py:73`).
+- `remember(content, confidence, memory_class)` → `POST /memory/remember`.
+- `recall(query, top_k, as_of)` → semantic search, ranked by
+  relevance × recency × confidence; `as_of` enables **bi-temporal** recall
+  ("what did this agent know at time T?").
+
+During a run, each agent calls `remember(...)` after publishing (storing its
+finding) and `recall(topic)` before reasoning again — the memory-grounded loop.
+Unlike the ledger, memory is **unsigned** and **not enumerable across agents**
+by design (isolation).
+
+---
+
+## 4. Isolation model (important)
+
+| boundary | enforced? | notes |
+|---|---|---|
+| **per-agent** (`session_id = kid`) | ✅ **yes** — verified, zero cross-agent leaks | the core "1000 agents, private knowledge" property |
+| **per-org** (tenant) on typed ledger rows | ✅ yes | `HmClaim`/`HmDispute`/`HmReputation` are tenant-scoped |
+| **per-org** (tenant) on the **memory** path | ⚠️ **not** a hard boundary in the tested build | same `session_id` across orgs can leak |
+| **Merkle log** | ⚠️ **global** on a shared instance | not tenant-partitioned |
+
+**Operational rule:** run **one Relata instance per org/mission** for a hard
+security boundary, and never reuse a `session_id` across orgs. (See
+`RELATA_COMPATIBILITY.md` for the empirical validation.)
+
+---
+
+## 5. What the live 30-agent run actually stored
+
+Queried directly from Relata (`purpose=hm_ledger`), no hypermind in the loop:
+
+| tenant (org) | HmClaim | of which SEAL / RECEIPT | HmDispute | HmReputation |
+|---|--:|--:|--:|--:|
+| `sa3-university` (cooperative) | 134 | 116 / 18 | 0 | 120 |
+| `sa3-city_at_war` (adversarial) | 159 | — | **23** | 120 |
+| `sa3-foreign_policy_cabinet` | 113 | — | 2 | 120 |
+
+Reading the story from the data: the **cooperative** university produced claims
++ resolutions and **no disputes**; the **adversarial** city-at-war produced
+**23 disputes** (rival factions attacking each other). `HmReputation` = 120 rows
+= 30 agents × up to 4 (topic, unit) posteriors each. `record_type` splits into
+`SEAL` (research claims + citations) and `RECEIPT` (oracle resolutions).
+
+---
+
+## 6. Query it yourself
+
+```python
+from relata import RelataClient
+c = RelataClient("http://localhost:9464", tenant="sa3-university", purpose="hm_ledger")
+
+c.query("SELECT object_id FROM HmClaim")                         # every claim kid
+c.query("SELECT agent_hex, topic, alpha, beta FROM HmReputation")  # standings
+
+# decode a claim end-to-end:
+import base64, cbor2
+from hypermind.wire import SignedStatement
+row  = c.query("SELECT wire_b64, seq FROM HmClaim").rows[0]
+stmt = SignedStatement.from_bytes(base64.b64decode(row["wire_b64"]))   # re-verifies A-ALG-BOUND
+body = cbor2.loads(stmt.payload)
+print(body["payload"]["finding"])   # what that agent concluded
+```
+
+---
+
+## 7. Scalability note
+
+At millions of records the current write path (`upsert` per claim) and the
+in-memory report retention become the bottlenecks — tracked as **#175** (batch
+writes), **#176** (streaming/bounded report), and **#177** (distributed
+multi-process worlds). The 0.17.0 concurrency + O(n²)-removal work makes the
+*compute* side scale to ~1000s of agents; those three issues cover the
+*storage/throughput* side for genuine 10k-agent scale.

{hypermind-0.16.0 → hypermind-0.18.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hypermind"
-version = "0.16.0"
+version = "0.18.0"
 description = "Federated agent intelligence — reference SDK for the HyperMind SCITT contested-claims profile"
 readme = "README.md"
 requires-python = ">=3.11,<3.14"

{hypermind-0.16.0 → hypermind-0.18.0}/src/hypermind/__init__.py

@@ -73,7 +73,7 @@ open = HyperMindAgent.open
 #: is held until the §External & human gates clear (IETF I-D, IANA,
 #: third-party audit, OpenSSF acceptance, ≥3 pilots) — at that point we
 #: cut ``1.0.0`` GA. See ROADMAP.md §Status snapshot.
-__version__ = "0.16.0"
+__version__ = "0.18.0"
 
 #: SLSA build-provenance level committed for every release artefact.
 #: Cosign keyless signing + slsa-github-generator + PEP 740 attestations

{hypermind-0.16.0 → hypermind-0.18.0}/src/hypermind/orgs/__init__.py

@@ -62,6 +62,12 @@ from hypermind.orgs.model import (
 from hypermind.orgs.query import Answer, Contributor, aggregate_answer
 from hypermind.orgs.receipt import VerifyResult, build_receipt, verify_receipt
 from hypermind.orgs.report import MemberStanding, OrgReport
+from hypermind.orgs.retrieval import (
+    MentionGraph,
+    extract_mentions,
+    graph_retriever,
+    relata_retriever,
+)
 from hypermind.orgs.sim import OrgSim
 from hypermind.orgs.stream import graph_snapshot
 from hypermind.orgs.tools import (
@@ -91,6 +97,7 @@ __all__ = [
     "Intervention",
     "LLMJudgeGroundTruth",
     "MemberStanding",
+    "MentionGraph",
     "OrgModel",
     "OrgReport",
     "OrgSim",
@@ -110,12 +117,15 @@ __all__ = [
     "compute_scorecard",
     "declarative_grants",
     "default_context_builder",
+    "extract_mentions",
     "get_interaction",
+    "graph_retriever",
     "graph_snapshot",
     "list_tools",
     "policies_for_role",
     "random_grants",
     "register_tool",
+    "relata_retriever",
     "unregister_tool",
     "verify_receipt",
 ]

hypermind-0.18.0/src/hypermind/orgs/retrieval.py

@@ -0,0 +1,175 @@
+"""Graph- and identity-aware retrieval (entity mentions).
+
+Plain relevance ranking retrieves claims whose *text* overlaps the question.
+That misses the graph: a claim about "ML-DSA" is relevant to a question about
+"post-quantum signatures" even with no word overlap, if the two are linked by
+a shared entity elsewhere in the ledger.
+
+This module extracts **entity mentions** from findings, builds a
+claim↔entity **mention graph**, and offers a graph-aware retriever that:
+  1. pulls entities from the question,
+  2. expands them by co-mention (1-hop graph neighbours),
+  3. returns claims mentioning any of those entities, ranked by match count.
+
+The entity extractor is injectable (swap in an LLM/NER or Relata identity
+resolution); the default is a dependency-free heuristic. A Relata-backed
+retriever adapter (``relata_retriever``) uses the server's real
+``search_knowledge`` / ``find_connections`` graph.
+"""
+
+from __future__ import annotations
+
+import re
+from collections import defaultdict
+from collections.abc import Callable
+from typing import Any
+
+# Capitalised phrases (proper nouns), ALLCAPS acronyms, and hyphenated
+# technical tokens — a cheap entity-mention heuristic with no NLP dependency.
+_PHRASE = re.compile(r"\b([A-Z][A-Za-z0-9]+(?:[- ][A-Z][A-Za-z0-9]+)*)\b")
+_ACRONYM = re.compile(r"\b([A-Z]{2,}(?:-[A-Z0-9]+)*)\b")
+_TECH = re.compile(r"\b([a-z]+(?:-[a-z0-9]+)+)\b")  # e.g. ab-initio, post-quantum
+_STOP = {
+    "The",
+    "A",
+    "An",
+    "This",
+    "That",
+    "It",
+    "We",
+    "I",
+    "In",
+    "On",
+    "Of",
+    "To",
+    "For",
+    "And",
+    "But",
+    "Or",
+    "If",
+    "Is",
+    "Are",
+    "Would",
+    "Should",
+    "Will",
+    "Does",
+    "Do",
+    "Can",
+    "Could",
+    "May",
+    "Might",
+    "Question",
+    "Answer",
+}
+
+
+def extract_mentions(text: str, *, max_mentions: int = 24) -> set[str]:
+    """Extract entity mentions from ``text`` (normalised lowercase)."""
+    if not text:
+        return set()
+    out: set[str] = set()
+    for pat in (_ACRONYM, _PHRASE, _TECH):
+        for m in pat.findall(text):
+            token = m.strip()
+            if token in _STOP or len(token) < 3:
+                continue
+            out.add(token.lower())
+            if len(out) >= max_mentions:
+                return out
+    return out
+
+
+class MentionGraph:
+    """A bipartite claim↔entity graph, built incrementally as claims arrive."""
+
+    def __init__(self, extractor: Callable[[str], set[str]] | None = None) -> None:
+        self._extract = extractor or extract_mentions
+        self._claim_entities: dict[str, set[str]] = {}
+        self._entity_claims: dict[str, set[str]] = defaultdict(set)
+
+    def add(self, claim_kid: str, text: str) -> set[str]:
+        ents = self._extract(text)
+        if ents:
+            self._claim_entities[claim_kid] = ents
+            for e in ents:
+                self._entity_claims[e].add(claim_kid)
+        return ents
+
+    def connected(self, entity: str) -> set[str]:
+        """Entities co-mentioned with ``entity`` in some claim (1-hop)."""
+        neighbours: set[str] = set()
+        for kid in self._entity_claims.get(entity, ()):
+            neighbours |= self._claim_entities.get(kid, set())
+        neighbours.discard(entity)
+        return neighbours
+
+    def claims_for(self, entities: set[str]) -> set[str]:
+        kids: set[str] = set()
+        for e in entities:
+            kids |= self._entity_claims.get(e, set())
+        return kids
+
+    def stats(self) -> dict[str, int]:
+        return {
+            "claims": len(self._claim_entities),
+            "entities": len(self._entity_claims),
+            "edges": sum(len(v) for v in self._claim_entities.values()),
+        }
+
+
+def graph_retriever(
+    graph: MentionGraph,
+    claim_lookup: Callable[[str], tuple[str, str] | None],
+    *,
+    hops: int = 1,
+) -> Callable[[str, int], list[tuple[str, str]]]:
+    """A retriever that expands the question's entities through the graph.
+
+    ``claim_lookup(kid)`` returns ``(author_label, finding_text)`` or None.
+    """
+
+    def retrieve(question: str, k: int) -> list[tuple[str, str]]:
+        q_ents = extract_mentions(question)
+        expanded = set(q_ents)
+        for _ in range(max(0, hops)):
+            for e in list(expanded):
+                expanded |= graph.connected(e)
+        candidate_kids = graph.claims_for(expanded)
+        scored: list[tuple[int, str, str]] = []
+        for kid in candidate_kids:
+            hit = claim_lookup(kid)
+            if hit is None:
+                continue
+            label, text = hit
+            matched = len(expanded & extract_mentions(text))
+            scored.append((matched, label, text))
+        scored.sort(key=lambda t: t[0], reverse=True)
+        return [(label, text) for _n, label, text in scored[:k]]
+
+    return retrieve
+
+
+def relata_retriever(
+    provider: Any, *, mode: str = "search_knowledge"
+) -> Callable[[str, int], list[tuple[str, str]]]:
+    """Retriever backed by Relata's real graph/identity via a RelataToolProvider.
+
+    ``mode`` is ``search_knowledge`` (semantic + keyword over the KB) or
+    ``find_connections`` (graph traversal from question entities).
+    """
+
+    def retrieve(question: str, k: int) -> list[tuple[str, str]]:
+        if mode == "find_connections":
+            ent = next(iter(extract_mentions(question)), question)
+            result = provider.call("find_connections", {"entity": ent, "limit": k})
+        else:
+            result = provider.call("search_knowledge", {"query": question})
+        rows = result.get("results") or result.get("rows") or []
+        out: list[tuple[str, str]] = []
+        for r in rows[:k]:
+            label = str(r.get("source") or r.get("entity") or r.get("id") or "relata")
+            text = str(r.get("content") or r.get("text") or r.get("finding") or r)
+            out.append((label, text))
+        return out
+
+    return retrieve