libp2r2p 0.10.21 → 0.10.23

package/README.md

@@ -21,6 +21,7 @@ never decorated with `meta`; relay provenance belongs to its enclosing item:
 { type: 'event', event, relay }
 { type: 'error', relay, error }
 { type: 'eose', relays: [{ relay, status, error }] } // error is optional
+{ type: 'live-progress', relay, epoch, since, until } // live and live-enabled feeds only
 ```
 
 `eose` means the **initial read attempt is complete**, including empty reads and
@@ -101,6 +102,51 @@ content-key, private-channel and NIP-46 helpers unwrap pool results internally
 and retain their higher-level event contracts. Count, publication and disconnect
 return formats are unchanged.
 
+### Live overlap and progress (0.10.22)
+
+Every live subscription opens with `since = max(filter.since ?? 0, now - 600, 0)`,
+including its first connection. This accepts events that arrive after opening
+but carry a timestamp up to ten minutes earlier. Explicit filter boundaries still
+apply, and retained events before EOSE remain excluded from strictly-live output.
+
+Reconnect recovery overlaps each relay's last received timestamp by 600 seconds,
+without going below the caller's `since` or zero. Empty connections use their
+opening time instead. Cursors advance independently even for events deduplicated
+across relays; future timestamps are capped at their local receipt time. Failed
+or interrupted recoveries retain their earlier baseline for the next reconnect.
+
+Live readers automatically enqueue a `live-progress` envelope every 60 seconds
+while ready, including when no events arrive. There is no enabling option.
+`epoch` is a numeric connection-attempt identifier scoped to this iterator;
+compare it together with `relay`, not across separate iterators. `since` is the
+local EOSE time (or the explicit filter lower bound if later), and `until` is the
+local time at the tick, both inclusive Unix seconds. Repeated markers describe
+the same continuous epoch with an increasing `until`.
+
+Progress starts only after that relay's EOSE and successful completion of any
+pending recovery. It pauses on disconnection, does not span connection epochs,
+and stops on cancellation, draining or `until`. History-only and empty-relay
+reads do not emit it. Timers never issue extra network queries.
+
+Markers share the bounded queue with events and follow all earlier received
+items. A feed preserves that ordering behind its initial history and queued live
+events. Consumers must finish persisting preceding events before persisting
+progress, and must not treat a marker as proof of historical completeness or as
+coverage of a disconnected interval. This is client-observed continuity, not a
+relay acknowledgement; delayed relay delivery or clock skew beyond the overlap
+can still leave gaps. A backward local clock cannot regress an emitted boundary.
+
+Every involuntary live close emits an error and still permits automatic recovery.
+A close without an explicit cause uses `error.code === 'RELAY_LIVE_INTERRUPTED'`
+and `category === 'transport'`; a pre-EOSE close retains status `closed` in the
+initial report. Explicit causes are preserved. Caller cancellation, draining and
+expiry of `filter.until` do not manufacture interruption errors.
+
+Process Nostr events only under `item.type === 'event'`. Handle controls you need
+and ignore unknown types; a new control is neither an event nor stream completion.
+The internal query, private-channel and NIP-46 consumers retain their existing
+higher-level event-only contracts.
+
 ### Read admission and bounded snapshots (0.10.21)
 
 Each pool coordinates event reads by normalized connection URL. Constructor
@@ -774,6 +820,24 @@ The NIP-94 extension also carries optional `download` intent; see
 
 ## Reliable IndexedDB queue reservations
 
+`queue.putBy(indexName, item, { existingOnly = false })` atomically inserts or
+replaces an item through a declared unique, non-multiEntry index. Its key is
+extracted from the item's index keyPath, including nested paths and compound
+keys. Existing items retain their position; new keys append to the tail. It
+returns `true` after commit, or `false` without inserting when `existingOnly`
+is true and the key is absent. This allows late progress checkpoints to avoid
+recreating a cancelled record. Values are cloned before asynchronous work.
+
+Unknown, non-unique or multiEntry indexes throw `ValidationError` with
+`QUEUE_PUT_INDEX_INVALID`; missing or invalid IndexedDB keys use
+`QUEUE_PUT_KEY_INVALID`. Cloning, storage and other unique-index failures retain
+their operational errors. Lookup, replacement and byte accounting share one
+transaction and roll back together. Capacity policy matches `setAt` for a
+replacement and `push` for an insertion. Replacing a reserved item invalidates
+the old token: its `ack`, `nack` and `renew` return false. Successful writes wake
+local waiting consumers. Like the other indexed methods, `putBy` is exclusive
+to `idb-queue`; the Web Storage queue's common API is unchanged.
+
 `createQueue({ prefix, evictionPolicy: 'reject', maxBytes })` opts out of
 capacity eviction. Oversized items reject with `QUEUE_ITEM_TOO_LARGE`, full
 queues with `QUEUE_CAPACITY_EXCEEDED`; browser quota errors propagate without

package/idb-queue/index.js

@@ -725,6 +725,34 @@ export async function createQueue ({
     })
   }
 
+  async function putBy (indexName, item, { existingOnly = false } = {}) {
+    const definition = indexDefinitions.find(index => index.name === indexName)
+    if (!definition?.unique || definition.multiEntry) throw new ValidationError('QUEUE_PUT_INDEX_INVALID')
+    // Snapshot before awaiting IndexedDB so the lookup key and stored value
+    // cannot diverge if the caller changes its object during the transaction.
+    const value = structuredClone(item)
+    const readKey = path => path.split('.').reduce((part, key) => part?.[key], value)
+    const key = Array.isArray(definition.keyPath) ? definition.keyPath.map(readKey) : readKey(definition.keyPath)
+    try { indexedDB.cmp(key, key) } catch { throw new ValidationError('QUEUE_PUT_KEY_INVALID') }
+    const requiredBytes = itemForStorage(0, value).byteSize
+    const written = await mutate(async (tx, state) => {
+      const { result: previous } = await run('get', [key], ITEMS_STORE, indexName, { tx })
+      if (previous) {
+        const index = previous.position - state.head
+        await putItem(tx, state, previous.position, value, {
+          direction: evictionDirectionFor('setAt', { index, length: state.tail - state.head }),
+          protectedPositions: new Set([previous.position])
+        })
+      } else {
+        if (existingOnly) return false
+        await pushInTransaction(tx, state, value)
+      }
+      return true
+    }, { requiredBytes })
+    if (written) wake()
+    return written
+  }
+
   async function getBy (indexName, query) {
     // Explicit index operations avoid scanning queue values in JavaScript.
     return snapshot(async tx => {
@@ -899,6 +927,7 @@ export async function createQueue ({
     removeWhere,
     some,
     clear,
+    putBy,
     getBy,
     someBy,
     removeBy,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libp2r2p",
-  "version": "0.10.21",
+  "version": "0.10.23",
   "description": "Peer-to-relay-to-peer",
   "keywords": [
     "p2r2p",

package/relay/services/relay-pool.js

@@ -13,6 +13,8 @@ const COUNT_TIMEOUT_MS = 5000
 const COUNT_TIMEOUT_AFTER_FIRST_COUNT_MS = 500
 const SEND_TIMEOUT_UNTIL_FIRST_FULFILLMENT_MS = 3000
 const SEND_TIMEOUT_MS = 30000
+const LIVE_OVERLAP_SECONDS = 600
+const LIVE_PROGRESS_INTERVAL_MS = 60000
 
 // Returns a function that should be called for each received event (valid or invalid).
 // Calls onSatisfied() and stops counting once the filter is fully satisfied per relay:
@@ -607,8 +609,34 @@ export class RelayPool {
     // Preserve until for forwarding to the live sub filter and the teardown timer
     const filterUntil = filter.until > 0 ? filter.until : null
 
-    // lastSeenAt: the highest created_at received so far; used as since on reconnect gap fill
-    let lastSeenAt = (filter.since > 0) ? filter.since : null
+    // Recovery uses each relay's received timestamps, or its opening time when empty.
+    const lastSeenAt = new Map()
+    const openedAt = new Map()
+    const pendingGaps = new Map()
+    const attempts = new Map()
+    let nextEpoch = 0
+    const sinceFloor = Math.max(0, filter.since ?? 0)
+    const overlappingSince = timestamp => Math.max(sinceFloor, timestamp - LIVE_OVERLAP_SECONDS)
+    const observeEvent = (event, url) => {
+      // Every relay advances independently, including cross-relay duplicates.
+      // Future-dated events cannot hide an offline interval during recovery.
+      const timestamp = Math.min(event.created_at, Math.floor(Date.now() / 1000))
+      if (Number.isFinite(timestamp) && timestamp > (lastSeenAt.get(url) ?? -Infinity)) lastSeenAt.set(url, timestamp)
+    }
+    const scheduleProgress = attempt => {
+      if (isDone || attempt.closed || attempts.get(attempt.url) !== attempt ||
+          attempt.since === null || !attempt.recovered || attempt.timer) return
+      attempt.timer = maybeUnref(setTimeout(() => {
+        attempt.timer = null
+        if (isDone || attempt.closed || attempts.get(attempt.url) !== attempt) return
+        const until = Math.min(filterUntil ?? Infinity, Math.floor(Date.now() / 1000))
+        if (until > attempt.lastProgress && until >= attempt.since) {
+          enqueue({ type: 'live-progress', relay: attempt.url, epoch: attempt.epoch, since: attempt.since, until })
+          attempt.lastProgress = until
+        }
+        scheduleProgress(attempt)
+      }, LIVE_PROGRESS_INTERVAL_MS))
+    }
 
     // Bounded dedup set to handle overlap between reconnect gap fill and live sub.
     const seenIds = new Set()
@@ -661,6 +689,10 @@ export class RelayPool {
       clearTimeout(untilTimer)
       finishReady('closed', false)
       gapAc.abort()
+      for (const attempt of attempts.values()) {
+        clearTimeout(attempt.timer)
+        attempt.stop.abort()
+      }
       for (const timer of retryTimers.values()) clearTimeout(timer)
       retryTimers.clear()
       liveSubs.forEach(sub => sub.close())
@@ -677,7 +709,6 @@ export class RelayPool {
         if (seenIds.size >= 500) seenIds.delete(seenIds.values().next().value) // evict oldest
         seenIds.add(event.id)
       }
-      if (event.created_at > (lastSeenAt ?? 0)) lastSeenAt = event.created_at
       enqueue({ type: 'event', event, relay: url })
     }
 
@@ -714,10 +745,10 @@ export class RelayPool {
         maybeFinishInitialReady()
       }
     }
-    const reportClosed = url => {
+    const reportClosed = (url, error) => {
       if (!isReady && initialPending.delete(url)) {
         clearTimeout(readyTimeouts.get(url))
-        initialOutcomes.set(url, { relay: url, status: 'closed' })
+        initialOutcomes.set(url, { relay: url, status: 'closed', error })
         maybeFinishInitialReady()
       }
     }
@@ -727,7 +758,7 @@ export class RelayPool {
       const nextDelay = Math.min(reconnectDelay * 2, 5 * 60_000)
       const timer = maybeUnref(setTimeout(() => {
         retryTimers.delete(url)
-        subscribeToRelay(url, lastSeenAt, nextDelay)
+        subscribeToRelay(url, pendingGaps.get(url) ?? lastSeenAt.get(url) ?? openedAt.get(url) ?? null, nextDelay)
       }, reconnectDelay))
       retryTimers.set(url, timer)
     }
@@ -740,24 +771,35 @@ export class RelayPool {
 
     // Runs a reconnect gap fill for a single relay and returns a promise that resolves
     // when it completes. now is shared with the live sub so both use the same boundary.
-    const runReconnectGapFill = (url, gapSince, now, slots) => {
+    const runReconnectGapFill = (url, gapSince, now, slots, attempt) => {
       const gapUntil = filterUntil !== null ? Math.min(now, filterUntil) : now
-      const gapFilter = { ...baseFilter, since: gapSince, until: gapUntil }
+      const gapFilter = { ...baseFilter, since: overlappingSince(gapSince), until: gapUntil }
       const gapGen = _gapEventsGenerator(gapFilter, [url], {
         _admissions: new Map([[url, Promise.resolve(slots)]]),
         queueTimeout,
         timeout: timeoutForReconnectGap,
         timeoutAfterFirstEose: timeoutAfterFirstReconnectGapEose,
         signal,
-        _stopSignal: gapAc.signal
+        _stopSignal: AbortSignal.any([gapAc.signal, attempt.stop.signal])
       })
       return (async () => {
+        let completed = false
+        let failed = false
         for await (const item of gapGen) {
-          if (item?.type === 'event') pushEvent(item.event, url, true)
-          else if (item?.type === 'error' && !isDone) enqueue(item)
+          if (item?.type === 'event') {
+            observeEvent(item.event, url)
+            pushEvent(item.event, url, true)
+          } else if (item?.type === 'error') {
+            failed = true
+            if (!isDone) enqueue(item)
+          } else if (item?.type === 'eose') {
+            completed = item.relays?.length === 1 && ['eose', 'satisfied'].includes(item.relays[0].status)
+          }
         }
+        return completed && !failed
       })().catch(err => {
         reportFailure(url, asError(err))
+        return false
       }).finally(() => slots.history.release())
     }
 
@@ -766,7 +808,10 @@ export class RelayPool {
       // Don't reconnect if we're past the until boundary
       if (filterUntil !== null && now >= filterUntil) return
       const reservation = !initialAdmissions.has(url) && _admissions?.get(url)
-      const recovering = gapSince !== null && gapSince > 0
+      const recovering = gapSince !== null
+      if (recovering) pendingGaps.set(url, gapSince)
+      const attempt = { url, epoch: ++nextEpoch, since: null, lastProgress: -Infinity, recovered: !recovering, timer: null, closed: false, stop: new AbortController() }
+      attempts.set(url, attempt)
       let recoveryLease
       initialAdmissions.add(url)
       Promise.resolve(reservation || this.#admission.acquire(normalizeRelayUrl(url), { live: true, history: recovering, signal: gapAc.signal, queueTimeout })).then(async slots => {
@@ -790,20 +835,22 @@ export class RelayPool {
 
         // Buffer post-EOSE live events while a reconnect gap fill is running so
         // the historical gap is yielded before newer socket events.
-        let liveBuffer = (gapSince !== null && gapSince > 0) ? [] : null
+        let liveBuffer = recovering ? [] : null
         let liveEose = false
         let bufferedBytes = 0
 
         // Open the live sub first so the relay starts buffering incoming events
         // before we scan its database for the reconnect gap fill.
         // Forward until to the relay so it can enforce the boundary server-side.
-        const liveFilter = { ...baseFilter, since: now, limit: 0 }
+        if (!openedAt.has(url)) openedAt.set(url, now)
+        const liveFilter = { ...baseFilter, since: overlappingSince(now), limit: 0 }
         if (filterUntil !== null) liveFilter.until = filterUntil
         const liveSub = relay.subscribe([liveFilter], {
           onevent: (event) => {
             // A limit:0 relay may still send retained events before EOSE. Do not
             // expose them from a strictly-live stream.
-            if (isDone || liveSubs.get(url) !== liveSub || !liveEose) return
+            if (isDone || liveSubs.get(url) !== liveSub || attempt.closed || !liveEose) return
+            observeEvent(event, url)
             if (liveBuffer) {
               const bytes = encoder.encode(JSON.stringify(event)).byteLength
               if (liveBuffer.length >= maxBufferedLiveEvents || bufferedBytes + bytes > maxBufferedLiveBytes) {
@@ -815,33 +862,44 @@ export class RelayPool {
             } else pushEvent(event, url)
           },
           onclose: error => {
+            if (attempt.closed || attempts.get(url) !== attempt) return
+            attempt.closed = true
+            clearTimeout(attempt.timer)
+            attempt.stop.abort()
             if (liveSubs.get(url) === liveSub) liveSubs.delete(url)
-            else if (liveSubs.has(url)) return
             lease.release()
             if (liveLeases.get(url) === lease) liveLeases.delete(url)
             readyRelays.delete(url)
-            if (!isDone) {
-              if (error !== undefined) reportFailure(url, asError(error))
-              else if (!liveEose) reportClosed(url)
-            }
             if (isDone) return
+            if (error !== undefined) reportFailure(url, asError(error))
+            else {
+              const interruption = Object.assign(categorizeRelayError(new Error('RELAY_LIVE_INTERRUPTED'), 'transport'), { code: 'RELAY_LIVE_INTERRUPTED' })
+              enqueue({ type: 'error', relay: url, error: interruption })
+              reportClosed(url, interruption)
+            }
             scheduleReconnect(url, reconnectDelay)
           },
           oneose: () => {
-            if (isDone || (liveSubs.has(url) && liveSubs.get(url) !== liveSub)) return
+            if (isDone || attempt.closed || attempts.get(url) !== attempt || liveEose) return
             liveEose = true
+            attempt.since = Math.max(sinceFloor, Math.floor(Date.now() / 1000))
             markInitialEose(url)
+            scheduleProgress(attempt)
           }
         })
         if (isDone) { liveSub.close(); recoveryLease?.release(); return }
         liveSubs.set(url, liveSub)
 
-        if (gapSince !== null && gapSince > 0) {
-          const task = runReconnectGapFill(url, gapSince, now, slots).finally(() => {
+        if (recovering) {
+          const task = runReconnectGapFill(url, gapSince, now, slots, attempt).then(recovered => {
+            attempt.recovered = recovered
+            if (recovered && !attempt.closed && attempts.get(url) === attempt) pendingGaps.delete(url)
+          }).finally(() => {
             const buf = liveBuffer
             liveBuffer = null
             for (const event of buf) pushEvent(event, url, true)
             gapTasks.delete(task)
+            scheduleProgress(attempt)
             p.resolve()
           })
           gapTasks.add(task)
@@ -939,7 +997,7 @@ export class RelayPool {
 
     // limit:0 is a live-only stream and reports that stream's initial window.
     if (filter.limit === 0) {
-      yield * _liveGenerator(filter, relays, options)
+      yield * _liveGenerator(filter, relays, { ...options, maxBufferedLiveEvents, maxBufferedLiveBytes })
       return
     }