holosphere 2.0.0-alpha1 → 2.0.0-alpha10

package/docs/FOSDEM_PROPOSAL.md

@@ -0,0 +1,388 @@
+# FOSDEM 2025 - Local First Devroom
+## Talk Proposal
+
+**Title:** "HoloSphere: Federated Local-First Spatial Data with Nostr and Capability Tokens"
+
+**Category:** FOSS Frameworks that enable Local First app development
+
+**Speaker:** [Your name/organization]
+
+**Duration:** 20-25 minutes + Q&A
+
+---
+
+## Abstract
+
+What if your local-first app knew *where* it was - and could federate data across geographic boundaries without a central server?
+
+**HoloSphere** is an open-source JavaScript library that combines three powerful ideas: Uber's H3 hexagonal spatial indexing, the Nostr relay protocol for P2P synchronization, and a novel "hologram" reference system for zero-duplication federation.
+
+Every write is signed with your secp256k1 key, cached in memory, persisted to IndexedDB/filesystem, then published to relays in the background. Reads return instantly from local cache. The result: offline writes, instant reads, eventual sync - with no backend infrastructure.
+
+The key innovation is **holograms** - lightweight references (~150 bytes) that point to data in other locations without copying it. When biodiversity data in a local hexagon federates to regional and national views, only references propagate - the source remains authoritative. Cross-project federation uses **capability tokens** with scoped permissions, enabling controlled data sharing between independent HoloSphere instances.
+
+This talk demonstrates a citizen science use case: field researchers collecting biodiversity observations offline, with data federating up through spatial hierarchies and syncing across the Nostr network when connectivity returns.
+
+---
+
+## How Nostr Enables Local-First
+
+### The Storage Model
+
+HoloSphere uses Nostr's **parameterized replaceable events (kind 30000)** as its storage primitive. Every data item becomes a signed event with a d-tag encoding the path:
+
+```
+d-tag: "appname/holonId/lensName/dataId"
+       "biomap/8928308280fffff/biodiversity/obs-001"
+```
+
+### The Write Path (Local-First)
+
+```
+write(holon, lens, data)
+       |
+  Sign with secp256k1 private key
+       |
+  Cache in memory (~1ms)
+       |
+  Persist to IndexedDB/filesystem (~10ms)
+       |
+  Publish to relays (background, fire-and-forget)
+       |
+  Return success (before relay confirmation)
+```
+
+**Key insight:** The write returns after local persistence, not after relay confirmation. This makes writes instant and offline-capable.
+
+### The Read Path (Cache-First)
+
+1. Check memory cache (5-second TTL) - instant return
+2. Check persistent storage - still fast
+3. Query relays only on cache miss
+4. Background refresh for stale data
+
+**Result:** Hot reads are ~1ms. Cold reads hit local storage before network. Network is truly optional for reads.
+
+### What This Gives You
+
+- **No accounts:** Your keypair *is* your identity
+- **No servers:** Connect to any public relay, or run your own
+- **No lock-in:** Standard Nostr protocol, portable keys
+- **Data sovereignty:** You sign everything, you control access
+
+---
+
+## Federation Without Duplication: Holograms
+
+### The Problem
+
+A biodiversity observation recorded in hexagon A should appear in:
+- The local forest view (resolution 9)
+- The regional view (resolution 6)
+- The national aggregation (resolution 3)
+
+Copying data to each level = storage explosion + stale data + no single source of truth.
+
+### The Solution: Holograms
+
+A **hologram** is a lightweight reference (~150 bytes) that points to the original:
+
+```javascript
+// Original observation in local hexagon
+{
+  id: "obs-001",
+  species: "Parus major",
+  timestamp: 1701234567000,
+  confidence: 0.95
+}
+
+// Hologram in regional hexagon (~150 bytes)
+{
+  id: "obs-001",
+  hologram: true,
+  soul: "biomap/8928308280fffff/biodiversity/obs-001",
+  target: { holonId, lensName, dataId },
+  // Local overrides (optional):
+  aggregationWeight: 1.0,
+  verifiedBy: "regional_coordinator"
+}
+```
+
+### Automatic Resolution
+
+When you read from the regional view, holograms resolve automatically:
+1. Detect `hologram: true`
+2. Follow `soul` to source
+3. Verify capability (if cross-project)
+4. Merge source data with local overrides
+5. Return complete object with `_hologram` metadata
+
+**Circular detection:** Tracks visited souls, max depth 10.
+
+**Active hologram tracking:** Sources know all their holograms via `_meta.activeHolograms` - enables cascade updates and cleanup.
+
+---
+
+## Cross-Project Federation: Capabilities
+
+### The Challenge
+
+Different research projects want to share data:
+- Project A: Local birdsong observations
+- Project B: Regional biodiversity aggregation
+- Project C: National conservation database
+
+How do you control who reads what without a central auth server?
+
+### Capability Tokens
+
+```javascript
+{
+  type: "capability",
+  permissions: ["read"],
+  scope: {
+    holonId: "*",           // Any hexagon
+    lensName: "biodiversity" // Only biodiversity lens
+  },
+  recipient: "project_b_pubkey",
+  expires: 1735689600000,   // 30 days
+  // Signed by issuer's private key
+}
+```
+
+### Federation Handshake (Over Nostr)
+
+1. Project A sends federation request (kind 30078) with offered capability
+2. Project B accepts (kind 30079) with reciprocal capability
+3. Both store received capabilities in their federation registry
+4. Holograms can now resolve across project boundaries
+
+### Registry Tracks
+
+- **Federated partners:** Public keys of trusted projects
+- **Inbound capabilities:** What we can read from them
+- **Outbound capabilities:** What they can read from us
+
+---
+
+## Use Case: Citizen Science Biodiversity Monitoring
+
+### The Scenario
+
+Field researchers and citizen scientists mapping biodiversity in remote areas:
+
+1. **Select Region:** Draw polygon on map (forest, watershed, reserve)
+2. **Convert to Hexagons:** Polygon becomes H3 hexagons at chosen resolution
+3. **Collect Offline:** Record observations in the field
+   - Species identified
+   - GPS coordinates (auto-assigned to hexagon)
+   - Audio recordings
+   - Environmental conditions
+4. **Local Storage:** Data persists to device, immediately queryable
+5. **Background Sync:** When connectivity returns, publishes to relays
+6. **Federation:** Regional coordinators see aggregated data via holograms
+7. **Cross-Project:** National database receives capability-gated access
+
+### Why This Architecture Works
+
+- **Poor connectivity:** Field sites often have no network - offline-first is essential
+- **Spatial data:** Observations are inherently geographic - hexagons are natural
+- **Multi-scale:** Local → regional → national aggregation via hologram hierarchy
+- **Data sovereignty:** Researchers keep keys, control their data
+- **No infrastructure:** Public Nostr relays, no servers to maintain
+
+### Technical Example
+
+```javascript
+// Initialize HoloSphere with keypair
+const hs = new HoloSphere({
+  appName: 'biomap',
+  privateKey: researcherPrivateKey,
+  relays: ['wss://relay.damus.io', 'wss://relay.nostr.band']
+});
+
+// Convert GPS to hexagon
+const hexagon = await hs.toHolon(45.5231, -122.6765, 9);
+
+// Record observation (works offline)
+await hs.write(hexagon, 'biodiversity', {
+  species: 'Parus major',
+  timestamp: Date.now(),
+  audio: recordingBlob,
+  confidence: 0.95,
+  observer: hs.publicKey
+});
+
+// Federate to regional hexagon (creates hologram)
+const regionalHex = (await hs.getParents(hexagon))[2]; // Resolution 6
+await hs.federate(hexagon, regionalHex, 'biodiversity', {
+  direction: 'outbound',
+  mode: 'reference'  // Hologram, not copy
+});
+
+// Regional coordinator queries aggregated data
+const regionalData = await hs.getAll(regionalHex, 'biodiversity');
+// Returns: original observations + resolved holograms from child hexagons
+```
+
+---
+
+## Current Capabilities vs Limitations
+
+### What Works Today
+
+| Feature | Status | Notes |
+|---------|--------|-------|
+| Local persistence | Working | IndexedDB (browser), filesystem (Node) |
+| Nostr relay sync | Working | Fire-and-forget publish, background sync |
+| H3 spatial indexing | Working | 16 resolution levels, parent/child navigation |
+| Hologram references | Working | Same-project federation |
+| Hologram resolution | Working | Automatic, circular detection |
+| Local overrides | Working | Position, metadata on holograms |
+| Active hologram tracking | Working | Source tracks all references |
+| Real-time subscriptions | Working | Nostr-based, with throttling |
+| Capability tokens | Working | Scope + permissions + expiration |
+| Cross-project holograms | Working | With capability verification |
+| Federation discovery | Working | Nostr events for request/accept |
+
+### Known Limitations (Honest Assessment)
+
+| Feature | Status | Notes |
+|---------|--------|-------|
+| Capability signature verification | Stubbed | Tokens generated but verification TODO |
+| CRDT/automatic merge | Not implemented | Last-write-wins only |
+| Offline write queue | Partial | Writes persist locally, no retry queue |
+| Conflict resolution | Not implemented | No vector clocks, no causal ordering |
+| Guaranteed delivery | Not implemented | Relay publish is best-effort |
+| Bidirectional sync | Not implemented | Holograms are read-only references |
+
+### Architecture Classification
+
+HoloSphere is currently **"local-caching with remote sync"** rather than pure **"local-first with remote replication"**:
+
+- Writes: Local-first (persist before relay)
+- Reads: Cache-first (local before network)
+- Sync: Eventually consistent via relays
+- Conflicts: Last-write-wins (no merge)
+
+This is sufficient for many use cases but not full local-first as defined by Ink & Switch.
+
+---
+
+## Demo Plan
+
+Live demonstration (5-7 minutes):
+
+1. **Draw polygon** on interactive map (Leaflet.draw)
+2. **Convert to hexagons** at multiple resolutions
+3. **Record biodiversity observations** in hexagons
+4. **Show local persistence** (IndexedDB inspector)
+5. **Disconnect network** - continue adding data
+6. **Query locally** - data remains available
+7. **Federate to regional hexagon** - create holograms
+8. **Show hologram resolution** - ~150 bytes vs full data
+9. **Reconnect** - watch background sync to relays
+10. **Cross-project federation** - exchange capabilities, resolve foreign holograms
+
+---
+
+## Why This Matters for Local-First
+
+- **Spatial local-first:** First framework combining H3 geospatial with offline-capable P2P
+- **Nostr beyond social:** Demonstrates Nostr as general-purpose sync layer
+- **Federation without servers:** Capability tokens enable controlled sharing
+- **Real use case:** Biodiversity monitoring has genuine connectivity constraints
+- **Honest about gaps:** We show what works and what's still TODO
+
+---
+
+## Technical Stack
+
+- **h3-js** - Uber's hexagonal geospatial indexing
+- **nostr-tools** - Nostr protocol implementation
+- **@noble/curves** - secp256k1 cryptography
+- **IndexedDB/Filesystem** - Persistent local storage
+- **OpenAI** (optional) - AI-assisted spatial queries
+
+---
+
+## Repository & Materials
+
+- **Code:** https://github.com/liminalvillage/holosphere2
+- **Documentation:** LOCALFIRST.md (architecture), VIBE.md (vision)
+- **License:** MIT
+- **Live Demo:** [URL TBD]
+
+---
+
+## Target Audience
+
+- Developers building local-first applications
+- Environmental tech / citizen science builders
+- Decentralized app developers (especially Nostr ecosystem)
+- Anyone interested in geospatial + P2P
+
+## Takeaway
+
+Attendees will understand how to:
+1. Use Nostr as a local-first sync layer (not just social media)
+2. Organize data spatially with H3 hexagonal indexing
+3. Federate across boundaries using holograms (zero duplication)
+4. Control access with capability tokens (no central auth)
+
+And honestly assess which local-first properties the current implementation achieves vs. what remains to be built.
+
+---
+
+## Appendix: Data Structures
+
+### Nostr Event (Data Write)
+
+```javascript
+{
+  kind: 30000,
+  pubkey: "author_hex_pubkey",
+  created_at: 1701234567,
+  tags: [["d", "biomap/8928308280fffff/biodiversity/obs-001"]],
+  content: "{\"species\":\"Parus major\",\"confidence\":0.95,...}",
+  sig: "hex_signature"
+}
+```
+
+### Hologram Object
+
+```javascript
+{
+  id: "obs-001",
+  hologram: true,
+  soul: "biomap/8928308280fffff/biodiversity/obs-001",
+  target: {
+    appname: "biomap",
+    holonId: "8928308280fffff",
+    lensName: "biodiversity",
+    dataId: "obs-001",
+    authorPubKey: "hex_pubkey"  // For cross-project
+  },
+  capability: "base64_token",     // Optional
+  crossHolosphere: true,          // If cross-project
+  // Local overrides:
+  verifiedBy: "coordinator_key",
+  aggregationWeight: 1.0
+}
+```
+
+### Capability Token
+
+```javascript
+{
+  type: "capability",
+  permissions: ["read"],
+  scope: { holonId: "*", lensName: "biodiversity" },
+  recipient: "partner_pubkey",
+  issuer: "holosphere",
+  nonce: "unique_random",
+  issued: 1701234567000,
+  expires: 1703826567000
+}
+// Encoded: base64(JSON).signature
+```

package/docs/LOCALFIRST.md

@@ -0,0 +1,266 @@
+# Local-First Architecture
+
+## Overview
+
+HoloSphere implements **true local-first architecture** where persistent storage is the source of truth, and Nostr relays serve as the sync/replication layer. This document describes the architecture and how it achieves local-first principles.
+
+---
+
+## Architecture
+
+### Storage Layer Stack
+
+```
++----------------------------------------------------------+
+|                    HoloSphere API                        |
++----------------------------------------------------------+
+|                   nostr-wrapper.js                       |
+|          (CRUD abstraction: write, read, update)         |
++----------------------------------------------------------+
+|                   nostr-async.js                         |
+|   (LOCAL-FIRST: persistentGet → background relay sync)   |
++----------------------------------------------------------+
+|                   nostr-client.js                        |
+|  (NostrClient: OutboxQueue, SyncService, relay pool)     |
++----------------------------------------------------------+
+|                                                          |
+|   +------------------+      +-------------------------+  |
+|   | Persistent Store |      | OutboxQueue             |  |
+|   | (PRIMARY)        |      | (guaranteed delivery)   |  |
+|   | filesystem/IDB   |      | exponential backoff     |  |
+|   +------------------+      +-------------------------+  |
+|            |                           |                 |
+|            v                           v                 |
+|   +------------------+      +-------------------------+  |
+|   | Memory Cache     |      | SyncService             |  |
+|   | (optimization)   |      | (background retry)      |  |
+|   +------------------+      +-------------------------+  |
+|                                                          |
+|   +--------------------------------------------------+   |
+|   |           Nostr Relays (Replication)             |   |
+|   |   wss://relay.damus.io, wss://relay.nostr.band   |   |
+|   +--------------------------------------------------+   |
++----------------------------------------------------------+
+```
+
+### Data Flow
+
+#### Write Path (Guaranteed Delivery)
+
+```
+write(holon, lens, data)
+       │
+       ▼
+  Sign with secp256k1 private key
+       │
+       ▼
+  Cache in memory (~1ms)
+       │
+       ▼
+  Persist to storage (~10ms) ──► Return success immediately
+       │
+       ▼
+  Enqueue in OutboxQueue (persistent)
+       │
+       ▼
+  Attempt immediate delivery to relays
+       │
+       ├─► Success: Remove from queue
+       │
+       └─► Failure: SyncService retries with exponential backoff
+                    (1s → 2s → 4s → 8s → 16s, max 60s)
+                    Failed events purged after 24 hours
+```
+
+**Key property:** Writes return after local persistence, not after relay confirmation.
+
+#### Read Path (Persistent-First)
+
+```
+read(holon, lens, dataId)
+       │
+       ▼
+  Check persistent storage FIRST
+       │
+       ├─► Found: Return immediately
+       │          │
+       │          └─► Trigger background refresh from relays
+       │
+       └─► Not found: Query relays (fallback)
+                      │
+                      └─► Cache result locally
+```
+
+**Key property:** Reads never block on network if data exists locally.
+
+---
+
+## Local-First Principles Assessment
+
+Based on [Ink & Switch's Local-First Software](https://www.inkandswitch.com/local-first/) principles:
+
+| Principle | Score | Implementation |
+|-----------|-------|----------------|
+| **No spinners** | ✅ Good | Reads return from persistent storage instantly; writes return after local persist |
+| **Works offline** | ✅ Good | Full read/write functionality; OutboxQueue ensures writes sync when online |
+| **Data ownership** | ✅ Good | Keypair-based identity; user signs all data with secp256k1 |
+| **Longevity** | ✅ Good | Data persists locally (IndexedDB/filesystem); relays are backup |
+| **Privacy** | ✅ Good | User controls keys; data can be encrypted |
+| **Collaboration** | ⚠️ Partial | Last-write-wins; no CRDTs (future enhancement) |
+| **No lock-in** | ✅ Good | Standard Nostr protocol; portable keypairs |
+
+---
+
+## Implementation Details
+
+### OutboxQueue (`src/storage/outbox-queue.js`)
+
+Persistent queue for guaranteed relay delivery:
+
+```javascript
+// Events are enqueued with full retry metadata
+{
+  id: 'event-id',
+  event: { /* signed Nostr event */ },
+  relays: ['wss://relay1', 'wss://relay2'],
+  status: 'pending',  // 'pending' | 'failed'
+  retries: 0,
+  createdAt: 1701234567000,
+  nextRetryAt: 1701234567000,
+}
+```
+
+**Features:**
+- Exponential backoff: 1s → 2s → 4s → 8s → 16s (max 60s)
+- Max 5 retry attempts before marking as 'failed'
+- Failed events auto-purge after 24 hours
+- Manual retry available via `retryFailed(eventId)`
+
+### SyncService (`src/storage/sync-service.js`)
+
+Background service for reliable sync:
+
+```javascript
+const syncService = new SyncService(client, {
+  interval: 10000,  // Process queue every 10 seconds
+  autoStart: true,
+});
+```
+
+**Operations:**
+- Processes pending events from OutboxQueue
+- Retries failed deliveries with backoff
+- Purges old failed events (24h TTL)
+- Graceful shutdown on `client.close()`
+
+### Persistent-First Reads (`src/storage/nostr-async.js`)
+
+```javascript
+// nostrGet() - single item
+export async function nostrGet(client, path, kind, options) {
+  // LOCAL-FIRST: Check persistent storage FIRST
+  const persistedEvent = await client.persistentGet(path);
+  if (persistedEvent) {
+    // Trigger background refresh (non-blocking)
+    client.refreshPathInBackground(path, kind, options);
+    return JSON.parse(persistedEvent.content);
+  }
+
+  // Fallback to relay query only if not in local storage
+  return queryRelays(client, path, kind, options);
+}
+
+// nostrGetAll() - collection
+export async function nostrGetAll(client, pathPrefix, kind, options) {
+  // LOCAL-FIRST: Check persistent storage FIRST
+  const persistedEvents = await client.persistentGetAll(pathPrefix);
+  if (persistedEvents.length > 0) {
+    // Trigger background refresh (non-blocking)
+    client.refreshPrefixInBackground(pathPrefix, kind, options);
+    return parseEvents(persistedEvents);
+  }
+
+  // Fallback to relay query
+  return queryRelays(client, pathPrefix, kind, options);
+}
+```
+
+---
+
+## Configuration Options
+
+```javascript
+const hs = new HoloSphere({
+  appName: 'myapp',
+  relays: ['wss://relay.damus.io'],
+  privateKey: 'hex...',
+
+  // Local-first options
+  persistence: true,           // Enable persistent storage (default: true)
+  backgroundSync: true,        // Enable SyncService (default: true)
+  syncInterval: 10000,         // Sync interval in ms (default: 10000)
+  maxRetries: 5,               // Max retry attempts (default: 5)
+  retryBaseDelay: 1000,        // Base delay for backoff (default: 1000ms)
+  retryMaxDelay: 60000,        // Max delay cap (default: 60000ms)
+  failedTTL: 86400000,         // TTL for failed events (default: 24 hours)
+});
+```
+
+---
+
+## Comparison: Before vs After
+
+### Before (Relay-First with Local Caching)
+
+```
+Read:  Memory cache → NETWORK (blocking up to 30s)
+Write: Cache → Fire-and-forget to relays (no retry)
+```
+
+**Problems:**
+- Cold reads block on network
+- Writes can be lost if relay fails
+- No guaranteed delivery
+
+### After (Local-First with Relay Sync)
+
+```
+Read:  Persistent storage → Return immediately → Background refresh
+Write: Cache → Persist → OutboxQueue → Retry until delivered
+```
+
+**Benefits:**
+- Reads never block on network
+- Writes guaranteed to eventually deliver
+- Works fully offline
+- Data survives app restarts
+
+---
+
+## Remaining Limitations
+
+| Feature | Status | Notes |
+|---------|--------|-------|
+| Conflict resolution | Last-write-wins | No CRDTs; timestamps determine winner |
+| Multi-device sync | Via relays | No direct peer-to-peer sync |
+| Real-time collaboration | Limited | No operational transforms |
+
+### Future Enhancements
+
+1. **CRDTs** - Automerge/Yjs for collaborative editing
+2. **Direct sync** - mDNS/local network device discovery
+3. **Conflict UI** - Surface conflicts for user resolution
+
+---
+
+## References
+
+- [Local-First Software (Ink & Switch)](https://www.inkandswitch.com/local-first/)
+- [Nostr Protocol Specification](https://github.com/nostr-protocol/nips)
+- [CRDTs and Local-First](https://crdt.tech/)
+
+---
+
+*Document Version: 2.0*
+*Last Updated: 2025-12*
+*Status: Implemented*