querysub 0.462.0 → 0.463.0

package/valid.md

@@ -0,0 +1,205 @@
+# Querysub Validity, Locks, and Function Reruns
+
+This document covers the core system pieces that drive optimistic concurrency in querysub and the bugs we've found chasing rejection / rerun churn. It is written as a working reference, not a clean tutorial — the bugs section is the load-bearing part.
+
+---
+
+## 1. Core data shapes
+
+### PathValue and Time
+
+Every write is a `PathValue` with a `path`, a `value`, a `time`, a `valid` flag, and (sometimes) `locks` and `transactionPaths`.
+
+`Time` is `{ time, version, creatorId }`. The wall-clock `time` field is the primary ordering key; `version` disambiguates writes at the same wall-clock time on the same node; `creatorId` disambiguates across nodes. `compareTime` sorts by `(time, version, creatorId)` in that order.
+
+Two writes can therefore share the same `time.time` and still be totally ordered as long as `(version, creatorId)` differ. This is the basis for the "function rerun at the same `runAtTime`" mechanism — each rerun gets a fresh `version` from `clientWatcher.getFreeWriteTime` which tracks a per-process `lastVersions: Map<time.time, version>`.
+
+### Locks and lockGroups
+
+A `ReadLock` is `{ path, startTime, endTime, readIsTransparent }`. It claims:
+
+> "I read the value at `path` at `startTime`, and I observed no contention in `(startTime, endTime)`."
+
+Every write committed in the same `commitWrites` call shares the same `locks[]` array reference. `byLockGroup` groups by that array identity. The invariant asserted by `ValidStateComputer`:
+
+- All locks in a group share the same `endTime`.
+- All values in the group share a single validity decision.
+
+A lockGroup is **the** unit of validity. Two values in the same lockGroup must either both be valid or both be rejected.
+
+### Authorities and routing
+
+`PathRouter` maps each path to an authority node via a hash prefix. A given node may be the authority on some paths, a subscriber on others. Key calls:
+
+- `isSelfAuthority(path)` — do we own this path?
+- `isSelfParentAuthority(path)` — do we own everything under this path's parent hash prefix?
+
+Critically: the Data path and the Results path for the same call commit go to **different authorities**. Their lockGroup invariant (same validity) is distributed across authorities, and disagreements between authorities have been the root cause of multiple bugs (see § 4.2).
+
+---
+
+## 2. The committing flow (`PathValueProxyWatcher`)
+
+The proxy watcher runs a user function with synchronized reads/writes:
+
+1. `runWatcher` resets per-watcher state and sets:
+   - `watcher.nextWriteTime = options.runAtTime` (undefined for non-rerun proxies)
+   - If `nextWriteTime` is set, bumps it via `clientWatcher.getFreeWriteTime` to get a unique version for this commit
+   - `watcher.currentReadTime = nextWriteTime`, with `version -= 1_000_000` when set (see § 4.1 for why)
+2. The user function runs. Each read goes through `getCallback`:
+   - Calls `authorityStorage.getValueAtOrBeforeTime(path, currentReadTime)`
+   - Keys the read into `watcher.pendingAccesses` by `currentReadTime`
+3. If the function still has unsynced accesses, it re-evaluates (inner try loop). Each try resets `nextWriteTime` and gets a fresh version. The version used by the final committing run is the one allocated last.
+4. `commitWrites` builds locks from `pendingAccesses` (one lock per `(path, time)` read), with `endTime: readTime` (the bumped read time used during the function).
+5. The constraint at the end of `commitWrites` ensures `writeTime` is strictly after any lock `endTime`. In the runAtTime case this is automatic because `endTime = readTime` is `writeTime - 1_000_000` versions.
+
+### Where the read time / write time / lock endTime fit together
+
+- `nextWriteTime`: the actual time we'll write at — has a unique version via `getFreeWriteTime`.
+- `currentReadTime` (runAtTime case): `nextWriteTime` minus 1,000,000 versions. Used both as the read filter (so we never see our own prior versions) and as the lock `endTime` (so contention checks cover the right range).
+- `currentReadTime` (non-runAtTime case): `undefined`. Reads return the latest value; locks fall back to `endTime = writeTime` (via the `if (!readTime) readTime = writeTime` branch in lock construction).
+
+---
+
+## 3. Validity computation (`ValidStateComputer`)
+
+### Entry point: `ingestValuesAndValidStates`
+
+Called by `PathValueCommitter` (and others) when path values arrive. The flow:
+
+1. **Per-batch dedup by `(path, time)`**: collapse multiple values for the same `(path, time)` in one batch to the last-arriving one. (Pre-fix this was the "duplicate-with-stale-storage" bug — see § 4.3.)
+2. **Storage-vs-incoming dedup**: drop values whose `existing.valid` strictly equals incoming `valid`. Uses strict `===` so `undefined !== false`, which lets us deliberately re-ingest with `valid: undefined` to force a re-check. Bypassable via `initialTriggers.values`.
+3. **Split**: ours vs not-ours (`PathRouter.isSelfAuthority`).
+4. **LockWatcher2 subscription**: for our values' locks on non-self paths, register a watch via `watchValueLocks`. This holds for `MAX_CHANGE_AGE * 2` minimum — guarantees `destroyPath` doesn't strip the value during the lock's active window.
+5. **Initial-sync clobber**: for not-our paths that were just initially-synced, mark stale prior values as invalid.
+6. **Ingest** into `authorityStorage`.
+7. **Compute valid states** in a loop. Each iteration runs `computeValidStates(dependenciesChanged, now, prevValidStates)`. Changes feed back into `dependenciesChanged` via `lockWatcher2.getValuePathWatchers`. Deferred values (see § 4.5) accumulate into `deferredAll`.
+8. **Trigger watchers** via `pathWatcher.triggerValuesChanged`.
+9. **Schedule defer retry**: if any lockGroups deferred, `setTimeout` a re-call with the deferred values' paths in `initialTriggers.values` to bypass dedup.
+
+### Per-lockGroup decision: `computeValidStates`
+
+For each lockGroup:
+
+- Verify all locks share an `endTime`.
+- Verify all values are on a self-authority path (we shouldn't be deciding validity for foreign-owned values).
+- If `now - lockTime > MAX_CHANGE_AGE`, skip (validity is frozen by age).
+- Else for each lock: `isLockValid(lock, now)` (tri-state: `true`/`false`/`"defer"`) and then `isLockContentionFree(lock)`. First definitive failure wins; defers yield to a later definitive failure but otherwise propagate up.
+- If valid is `"defer"`, push values into the deferred array and don't touch storage or `pathValue.valid`.
+- Else update `pathValue.valid` and push to changed if the decision differs from the previous state.
+
+### `isLockValid` semantics
+
+```
+if (transparent) return true
+if (value at lock.startTime exists in storage) return value.valid
+if (!isSynced(lock.path)) return true       // never received, can't decide
+if (foreign authority and now - lock.startTime.time < DEFER_LOCK_WINDOW)
+    return "defer"
+return false
+```
+
+The defer branch is the optimistic-race accommodation: when the lock target is on a foreign authority and the missing value is recent, give it `DEFER_LOCK_WINDOW` (2.5s, from `pathValueCore.ts`) before rejecting. Reads that arrived later than the lock's `startTime` plus the window are decided immediately.
+
+### `isLockContentionFree`
+
+Scans `getValuePlusHistory(lock.path)` for any **valid** value with `startTime < value.time < endTime`. Any hit means another write landed inside the read's claimed range — contention, reject.
+
+The endTime being `currentReadTime` (which is bumped-version of nextWriteTime in the rerun case) means the range `(startTime, endTime)` covers our own intermediate write versions and concurrent writes from other creators at the same wall-clock time. That's the right semantic and the basis of OCC here.
+
+---
+
+## 4. Bugs and fixes
+
+A list of things we have to do because of things that go wrong if we don't.
+
+- **Read at `nextWriteTime.version - 1_000_000` on reruns** (not at `nextWriteTime` directly). Otherwise reruns at the same `runAtTime` read their own previous rejected version `.X[N-1]` as a predecessor, increment it, write `.X[N]`, which gets rejected (contention with `.X[N-1]`), and the function runner re-queues until the `MAX_QUEUE_COUNT` cap fires. The decrement only applies when `runAtTime` is set — natural forward proxies don't have own-prior versions to collide with.
+
+- **Per-batch dedup by `(path, time)` at the top of `ingestValuesAndValidStates`**. Otherwise if a value flip-flops on its source authority and both the valid and the invalid update arrive in the same batch, the storage-vs-incoming dedup compares each against the pre-batch storage state independently — the correction (matching stale storage) gets dropped as a duplicate, and the wrong value survives and clobbers storage. End result: receivers stuck on the wrong validity forever.
+
+- **Use strict `===` in the storage-vs-incoming dedup, not `!!` coercion**. Otherwise `valid: undefined` collapses to `false` and a deliberate "please re-evaluate" re-ingest with `undefined` looks like a duplicate of a rejected value and gets filtered. Strict equality lets us use `undefined` as a signal.
+
+- **Defer `isLockValid` rejection for `DEFER_LOCK_WINDOW` after `lock.startTime`** when the missing value is on a foreign authority. Otherwise concurrent dependent writes get spuriously rejected when network ordering delivers the dependent before its predecessor — e.g. function runner commits `.726`, `.7266`, `.727` in parallel with `noWaitForCommit`, `.7266` arrives at the Data authority first, lock check fails because `.726` isn't there yet, `.7266` rejected. Cascade requeues. With the defer window, the value almost always arrives within 2.5s and `LockWatcher2` fires re-eval naturally.
+
+- **Schedule a `setTimeout(DEFER_LOCK_WINDOW)` retry for any deferred values**. Otherwise if the foreign value never arrives, the deferred lockGroup never gets a decision and the value sits in undefined-valid limbo forever. The retry bypasses the storage dedup via `initialTriggers.values` so it actually re-runs `computeValidStates`; by then the lock age check decides "still missing → reject".
+
+- **Lock `endTime` must be the actual `writeTime` (or a time strictly before any concurrent write at the same `time.time`), not the read time**. Otherwise contention detection at the lock-validation authority misses writes that landed between our read and our write. The fallback `if (!readTime) readTime = writeTime` in `commitWrites` already encodes this for the no-readTime case; the runAtTime case relies on the version-decrement trick (above) to keep `endTime = readTime < writeTime` correct.
+
+- **Log validity transitions with their reason** (`lock-invalid on …`, `lock-contention on …`, `all locks valid and contention-free`). Otherwise cross-authority disagreements are undebuggable — the only way to figure out why two authorities reached different decisions about the same lockGroup is to diff their per-decision logs side by side. Without the reason field, the logs only tell you what the answer was, not why.
+
+### Open: cross-authority lockGroup disagreement
+
+Two authorities holding parts of the same lockGroup (e.g. Data and Results for one commit) can reach different validity decisions and broadcast them independently. The Data authority sometimes commits a value as valid before its own contention scan finishes, briefly emits `valid: true`, then re-rejects. The Results authority uses the brief-valid window to validate dependent writes via its lock check, decides true, and broadcasts. The Data authority's correction races against the Results authority's wrong validation reaching downstream subscribers (notably the function runner's rerun gate, which only re-queues when Results is invalid).
+
+The dedup fix and the defer window mitigate this but don't fix it. The real fix needs cross-authority consensus on lockGroup validity — currently every authority decides independently and the system relies on eventual convergence, which doesn't always converge correctly.
+
+---
+
+## 5. Function runner interactions (`PathFunctionRunner`)
+
+### Rerun gate
+
+`findFunctionsToCall` watches `Calls[*]` and `Results[*]`. For each call, the gate is roughly:
+
+```ts
+let result = atomicObjectRead(moduleData.Results[callId]);
+if (result) continue;   // already have a valid result, no rerun
+queueCall(call);
+```
+
+`atomicObjectRead` returns the latest *valid* value. If Results becomes invalid, the gate opens and re-queues. If it's then revalidated incorrectly (as in § 4.2), the gate closes permanently and the call never runs again.
+
+### Per-rerun commit
+
+Each `proxyWatcher.commitFunction` call uses `runAtTime: callSpec.runAtTime`. `runAtTime` is fixed for the lifetime of the call (it's part of the call's identity). Reruns reuse the same `runAtTime`, get bumped `version` via `getFreeWriteTime`. The FINISHED log line shows `call.runAtTime` (always version 0) — it does **not** show the actual writeTime version, which is why "FINISHED ... .X[0]" appearing twice does not mean a version collision.
+
+### `MAX_QUEUE_COUNT` cap
+
+Default 14 (dev) / 100 (prod). The cap catches infinite rerun loops — most commonly the self-read pattern in § 4.1. Hitting the cap produces an explicit error result rather than rerunning forever.
+
+### Skip on already-written
+
+If a rerun executes but finds Results already has a value (because another authority already wrote it, or the cascade re-validated the prior run while the rerun was in flight), the function logs "Skipping function write" and doesn't commit.
+
+---
+
+## 6. Transaction delaying (`TransactionDelayer`)
+
+Separate from the lock-level defer. Each committed write carries `transactionPaths` (the set of paths the same proxy committed at the same writeTime). On read, the proxy compares its observed values' transactionPaths against the times it actually saw — if it has only an older value for a path the transaction wrote at a newer time, it's missing a transaction part.
+
+`waitIfReceivedIncompleteTransaction`:
+
+1. Computes the newest transaction time that's missing.
+2. Waits up to `MISSING_TRANSACTION_PART_TIMEOUT = 15s` for the missing parts.
+3. While waiting, the proxy holds the watcher's commit (via `triggerOnPromiseFinish`).
+4. On timeout, if still missing, logs an error and lets the commit proceed (the proxy will then likely produce a write that gets rejected — but that's user-visible and self-correcting via rerun).
+
+This is the read-side analog of the lock defer: defer is for "we're an authority and the value we need for a lock check hasn't arrived"; transaction delay is for "we're a reader and one of our transaction siblings is older than the rest of the transaction".
+
+---
+
+## 7. Watching mechanics (`LockWatcher2`)
+
+When we ingest a value whose lock targets a foreign path, we register a `pathWatcher.watchPath` for that path under `lockWatcher2`'s node id. The watch is kept alive until `garbageCollectOldWatchers` retires it (after `MAX_CHANGE_AGE * 2`).
+
+`getValuePathWatchers(pathValue)` returns all our deferred / pending pathValues whose lock range contains `pathValue.time`. This is the mechanism by which a value arriving from a foreign authority triggers re-evaluation of local lockGroups — including the natural recovery for deferred locks (§ 4.5).
+
+Critical guarantee: `destroyPath` (called when a path is unwatched) does NOT fire during a lock's active window, because `LockWatcher2` holds the watch. Earlier in this work we suspected `destroyPath` was wiping `isSyncedCache` and triggering wrong "missing + not-synced → valid by default" decisions in `isLockValid`; LockWatcher2's hold-time invariant rules that out for the window we care about.
+
+---
+
+## 8. Key invariants and where they're enforced
+
+| Invariant | Enforced by |
+|---|---|
+| All locks in a lockGroup share the same `endTime` | `ValidStateComputer.computeValidStates` (asserts, errors) |
+| All values in a lockGroup share a single validity decision | `computeValidStates` per-group loop |
+| `writeTime > all lock endTimes` for one commit | `PathValueProxyWatcher.commitWrites` shift-forward branch (debugbreak if it fires) |
+| Lock endTime = writeTime when readTime is undefined | `commitWrites` fallback at the lock construction |
+| Each commit gets a unique `(time.time, version)` per node | `clientWatcher.lastVersions` map in `getFreeWriteTime` |
+| Same `time.time` from different nodes distinguished by `creatorId` | `compareTime` ordering |
+| LockWatcher2 holds remote-path watches for `MAX_CHANGE_AGE * 2` | `garbageCollectOldWatchers` threshold |
+| Per-batch dedup keeps the last-arriving value for `(path, time)` | top of `ingestValuesAndValidStates` |
+| Cross-batch dedup uses strict `===` on `valid` | `ingestValuesAndValidStates` filter |
+| Deferred lockGroup decisions get re-checked within `DEFER_LOCK_WINDOW` | `setTimeout` at end of `ingestValuesAndValidStates` |
+| **Cross-authority lockGroup consensus** | *not enforced* — open issue (§ 4.2) |

package/src/deployManager/LaunchTrackingHeader.tsx

@@ -1,65 +0,0 @@
-module.allowclient = true;
-
-import { qreact } from "../4-dom/qreact";
-import { css } from "typesafecss";
-import { getBrowserUrlNode } from "../-f-node-discovery/NodeDiscovery";
-import { MachineServiceController } from "./machineSchema";
-
-const SINCE_DAYS = 2;
-const CRASH_HUE = 0;
-const OTHER_HUE = 210;
-
-export class LaunchTrackingHeader extends qreact.Component {
-    render() {
-        let summaries = MachineServiceController(getBrowserUrlNode()).getRecentLaunches(SINCE_DAYS);
-        if (!summaries) return undefined;
-        if (summaries.length === 0) return undefined;
-
-        let totalCrashes = 0;
-        let totalOther = 0;
-        let perKey = new Map<string, { crashes: number; other: number }>();
-        for (let s of summaries) {
-            let isCrash = s.reason === "crashed";
-            if (isCrash) totalCrashes++;
-            else totalOther++;
-            let entry = perKey.get(s.serviceKey);
-            if (!entry) {
-                entry = { crashes: 0, other: 0 };
-                perKey.set(s.serviceKey, entry);
-            }
-            if (isCrash) entry.crashes++;
-            else entry.other++;
-        }
-
-        let ranked: { serviceKey: string; crashes: number; other: number }[] = [];
-        for (let [serviceKey, c] of perKey) {
-            ranked.push({ serviceKey, crashes: c.crashes, other: c.other });
-        }
-        ranked.sort((a, b) => {
-            if (b.crashes !== a.crashes) return b.crashes - a.crashes;
-            return (b.crashes + b.other) - (a.crashes + a.other);
-        });
-        let top = ranked[0];
-
-        let title = `Launches in last ${SINCE_DAYS} days: ${totalCrashes} crashed, ${totalOther} other`;
-        for (let r of ranked) {
-            title += `\n${r.serviceKey}: ${r.crashes} crashed, ${r.other} other`;
-        }
-
-        return <div title={title} className={css.hbox(6).colorhsl(0, 0, 20)}>
-            <span>🚀</span>
-            <span>
-                <span className={css.colorhsl(CRASH_HUE, 70, 35).boldStyle}>{totalCrashes}</span>
-                <span className={css.colorhsl(0, 0, 55)}>|</span>
-                <span className={css.colorhsl(OTHER_HUE, 65, 35).boldStyle}>{totalOther}</span>
-            </span>
-            {top && top.crashes > 0 &&
-                <span className={css.colorhsl(0, 0, 40)}>
-                    (<span className={css.colorhsl(CRASH_HUE, 70, 35)}>{top.crashes}</span>
-                    <span className={css.colorhsl(0, 0, 55)}>|</span>
-                    <span className={css.colorhsl(OTHER_HUE, 65, 35)}>{top.other}</span>)
-                </span>
-            }
-        </div>;
-    }
-}

package/src/diagnostics/FunctionCallInfo.tsx

@@ -1,141 +0,0 @@
-import { qreact } from "../4-dom/qreact";
-import { css } from "../4-dom/css";
-import { t } from "../2-proxy/schema2";
-import { callState, ensureSubscribed } from "./FunctionCallInfoState";
-import { formatNumber, formatPercent, formatTime } from "socket-function/src/formatting/format";
-import { measureBlock } from "socket-function/src/profiling/measure";
-import { StatsValue, getStatsTop } from "socket-function/src/profiling/stats";
-import { isCurrentUserSuperUser } from "../user-implementation/userData";
-
-module.hotreload = true;
-
-function formatTimeStats(stats: StatsValue): string {
-    if (stats.count === 0) return formatTime(0);
-
-    let avgTime = stats.sum / stats.count;
-    let top = getStatsTop(stats);
-
-    if (!top.topHeavy) {
-        return formatTime(avgTime);
-    }
-
-    let bottomAvg = (stats.sum - top.value) / (stats.count - top.count) || 0;
-    let topAvg = top.value / top.count;
-    let topPercent = formatPercent(top.countFraction);
-
-    return `${formatTime(bottomAvg)} (top ${topPercent} = ${formatTime(topAvg)})`;
-}
-
-export class FunctionCallInfo extends qreact.Component {
-    showDetailedStats() {
-        if (!isCurrentUserSuperUser()) return undefined;
-        let state = callState();
-
-        let avgInternalReruns = state.totalCalls > 0 && state.totalInternalReruns / state.totalCalls || 0;
-        let avgFullReruns = state.totalCalls > 0 && state.totalFullReruns / state.totalCalls || 0;
-        let percentMultipleInternalRuns = state.totalCalls > 0 && state.callsWithMultipleInternalRuns / state.totalCalls || 0;
-        let percentCascadingRuns = state.totalCalls > 0 && state.callsWithCascadingRuns / state.totalCalls || 0;
-        let avgCascadingReruns = state.callsWithCascadingRuns > 0 && state.totalInternalRerunsForCascading / state.callsWithCascadingRuns || 0;
-        let percentMultipleFullRuns = state.totalCalls > 0 && state.callsWithMultipleFullRuns / state.totalCalls || 0;
-
-        console.log("=== Function Call Statistics ===");
-        console.log([
-            `${formatNumber(state.totalCalls)} ⚡`,
-            [
-                `${formatNumber(avgInternalReruns)} ♻️`,
-                percentMultipleInternalRuns > 0 && `(${formatPercent(percentMultipleInternalRuns)} 🔦)`,
-                percentCascadingRuns > 0 && `(${formatPercent(percentCascadingRuns)} / ${formatNumber(avgCascadingReruns)} / MAX ${state.maxInternalReruns} 📈)`
-            ].filter(v => v).join(" "),
-            percentMultipleFullRuns > 0 && `${formatNumber(avgFullReruns)} (${formatPercent(percentMultipleFullRuns)}) ⚠️`,
-            `${formatTimeStats(state.evalTimeStats)} 💫 / ${formatTimeStats(state.totalTimeStats)} 🕒`
-        ].filter(v => v).join(" | "));
-        console.log("");
-        console.log("=== Per-Function Statistics ===");
-
-        let perFunctionStats = state.perFunctionStats;
-        if (perFunctionStats) {
-            for (let functionId in perFunctionStats) {
-                let stats = perFunctionStats[functionId];
-                let avgInternal = stats.totalInternalReruns / stats.totalCalls;
-                let avgFull = stats.totalFullReruns / stats.totalCalls;
-                let pctInternal = stats.callsWithMultipleInternalRuns / stats.totalCalls;
-                let pctCascading = stats.callsWithCascadingRuns / stats.totalCalls;
-                let avgCascading = stats.callsWithCascadingRuns > 0 && stats.totalInternalRerunsForCascading / stats.callsWithCascadingRuns || 0;
-                let pctFull = stats.callsWithMultipleFullRuns / stats.totalCalls;
-
-                console.log([
-                    `${functionId}: ${formatNumber(stats.totalCalls)} ⚡`,
-                    [
-                        `${formatNumber(avgInternal)} ♻️`,
-                        pctInternal > 0 && `(${formatPercent(pctInternal)} 🔦)`,
-                        pctCascading > 0 && `(${formatPercent(pctCascading)} / ${formatNumber(avgCascading)} / MAX ${stats.maxInternalReruns} 📈)`
-                    ].filter(v => v).join(" "),
-                    pctFull > 0 && `${formatNumber(avgFull)} (${formatPercent(pctFull)}) ⚠️`,
-                    `${formatTimeStats(stats.evalTimeStats)} 💫 / ${formatTimeStats(stats.totalTimeStats)} 🕒`
-                ].filter(v => v).join(" | "));
-            }
-        }
-    }
-
-    render() {
-        ensureSubscribed();
-        let state = callState();
-
-        let avgInternalReruns = state.totalCalls > 0 && state.totalInternalReruns / state.totalCalls || 0;
-        let avgFullReruns = state.totalCalls > 0 && state.totalFullReruns / state.totalCalls || 0;
-        let percentMultipleInternalRuns = state.totalCalls > 0 && state.callsWithMultipleInternalRuns / state.totalCalls || 0;
-        let percentCascadingRuns = state.totalCalls > 0 && state.callsWithCascadingRuns / state.totalCalls || 0;
-        let avgCascadingReruns = state.callsWithCascadingRuns > 0 && state.totalInternalRerunsForCascading / state.callsWithCascadingRuns || 0;
-        let percentMultipleFullRuns = state.totalCalls > 0 && state.callsWithMultipleFullRuns / state.totalCalls || 0;
-
-        return <div className={css.button.vbox(4).pad2(4).alignItems("end")} onClick={() => this.showDetailedStats()}>
-            <div className={css.hbox(10)}>
-                <span title="Committed synced function calls">
-                    {formatNumber(state.totalCalls)}⚡
-                </span>
-                <div className={css.hbox(7)}>
-                    <span title="Number of function evaluations">
-                        {formatNumber(avgInternalReruns)}♻️
-                    </span>
-                    {percentMultipleInternalRuns > 0 && <span title="Cases where the first call did not have all the synced data">
-                        (🔦{formatPercent(percentMultipleInternalRuns)})
-                    </span>}
-                    {percentCascadingRuns > 0 && <span title="The initial sync did not provide all the data, requiring cascading syncing.">
-                        (📈{formatPercent(percentCascadingRuns)} / {formatNumber(avgCascadingReruns)} / MAX {state.maxInternalReruns})
-                    </span>}
-                </div>
-                {percentMultipleFullRuns > 0 && <span title="Rejected calls requiring full reruns. Unless you are changing data another user is actively changing, this is a bug!" className={css.hsl(60, 100, 80).pad2(2, 1).hslcolor(0, 0, 0)}>
-                    {formatNumber(avgFullReruns)} ({formatPercent(percentMultipleFullRuns)}) ⚠️
-                </span>}
-            </div>
-            <div>
-                <span title="Average time to evaluate the application function (excluding proxy overhead)">
-                    {formatTimeStats(state.evalTimeStats)}💫
-                </span>
-                {" / "}
-                <span title="Average time to evaluate from start to finish, including rejections, etc. Top % shows the top % count, and if top % is shown the other amount has the top values removed.">
-                    {formatTimeStats(state.totalTimeStats)}🕒
-                </span>
-            </div>
-            <div className={css.hbox(20)}>
-                <div className={css.hbox(5)}>
-                    <span title="Remote watched paths (syncing from other nodes)">
-                        {formatNumber(state.remotePathCount)}🔥
-                    </span>
-                    {" / "}
-                    <span title="Local watched paths">
-                        {formatNumber(state.localPathCount)}🏠
-                    </span>
-                    {" / "}
-                    <span title="Total values stored in memory">
-                        {formatNumber(state.totalValueCount)}📦
-                    </span>
-                    {" / "}
-                    <span title="Watchers">
-                        {formatNumber(state.proxyWatcherCount)}🕵🏻‍♀️
-                    </span>
-                </div>
-            </div>
-        </div>;
-    }
-}

package/src/diagnostics/PathDistributionInfo.tsx

@@ -1,110 +0,0 @@
-import { qreact } from "../4-dom/qreact";
-import { css } from "../4-dom/css";
-import { sort } from "socket-function/src/misc";
-import { proxyWatcher } from "../2-proxy/PathValueProxyWatcher";
-import { pathWatcher } from "../0-path-value-core/PathWatcher";
-import { PathRouter } from "../0-path-value-core/PathRouter";
-import { QuerysubController, querysubNodeId } from "../4-querysub/QuerysubController";
-import { getSyncedController } from "../library-components/SyncedController";
-import { SocketFunction } from "socket-function/SocketFunction";
-import { isCurrentUserSuperUser } from "../user-implementation/userData";
-import { delay } from "socket-function/src/batching";
-import { remoteWatcher } from "../1-path-client/RemoteWatcher";
-
-module.hotreload = true;
-
-let querysubController = getSyncedController(QuerysubController);
-
-let pathNodeCache = new Map<string, string | "missing">();
-let pendingPaths = new Set<string>();
-let flushScheduled = false;
-
-function scheduleFlush() {
-    if (flushScheduled) return;
-    flushScheduled = true;
-    let promise = Promise.resolve().then(async () => {
-        flushScheduled = false;
-        if (pendingPaths.size === 0) return;
-        let paths = Array.from(pendingPaths);
-        pendingPaths.clear();
-        let nodeId = await querysubNodeId();
-        await remoteWatcher.flushWatchers();
-        if (!nodeId) return;
-        let result = await QuerysubController.nodes[nodeId].debugGetPathNodeIds(paths);
-        for (let path of paths) {
-            pathNodeCache.set(path, result.get(path) ?? "missing");
-        }
-    });
-    proxyWatcher.triggerOnPromiseFinish(promise, { waitReason: "pathDistribution" });
-}
-
-function getNodeForPath(path: string): string | "missing" | undefined {
-    let cached = pathNodeCache.get(path);
-    if (cached !== undefined) return cached;
-    pendingPaths.add(path);
-    scheduleFlush();
-    return undefined;
-}
-
-export class PathDistributionInfo extends qreact.Component {
-    renderBase() {
-        if (!isCurrentUserSuperUser()) return undefined;
-
-        let nodeSpecs = querysubController(SocketFunction.browserNodeId()).debugGetNodeSpecs();
-
-        let paths = pathWatcher.getAllWatchedPaths().filter(p => !PathRouter.isLocalPath(p));
-        let distribution = new Map<string, string[]>();
-        let missingPaths: string[] = [];
-        for (let path of paths) {
-            let nodeId = getNodeForPath(path);
-            if (nodeId === undefined) continue;
-            if (nodeId === "missing") {
-                missingPaths.push(path);
-                continue;
-            }
-            let nodePaths = distribution.get(nodeId);
-            if (!nodePaths) {
-                nodePaths = [];
-                distribution.set(nodeId, nodePaths);
-            }
-            nodePaths.push(path);
-        }
-
-        let entries = Array.from(distribution.entries());
-        sort(entries, ([nodeId, p]) => nodeId);
-        sort(entries, ([nodeId, p]) => (nodeSpecs?.get(nodeId)?.routeStart ?? 0));
-
-        let titleParts = entries.map(([nodeId, p]) => {
-            let spec = nodeSpecs?.get(nodeId);
-            let range = spec ? ` (${spec.routeStart}-${spec.routeEnd})` : "";
-            return `${nodeId}${range}: ${p.length}`;
-        });
-        if (missingPaths.length > 0) titleParts.push(`missing: ${missingPaths.length}`);
-        let title = titleParts.join("\n");
-
-        let parts: preact.ComponentChild[] = [];
-        for (let i = 0; i < entries.length; i++) {
-            let [nodeId, nodePaths] = entries[i];
-            if (i > 0) parts.push(" | ");
-            let spec = nodeSpecs?.get(nodeId);
-            parts.push(<span className={css.button} onClick={() => console.log(`Paths for ${nodeId}:`, nodePaths, spec && { range: `${spec.routeStart}-${spec.routeEnd}` })}>{nodePaths.length}</span>);
-        }
-        if (missingPaths.length > 0) {
-            if (parts.length > 0) parts.push(" | ");
-            parts.push(<span className={css.button.boldStyle.hslcolor(0, 100, 60)} onClick={() => console.log("Missing paths:", missingPaths)}>{missingPaths.length}</span>);
-        }
-
-        return <div className={css.hbox(4)} title={title}>
-            <span>🌐</span>
-            {parts.length > 0 && parts || "..."}
-        </div>;
-    }
-    render() {
-        try {
-            return this.renderBase();
-        } catch (error) {
-            console.error("Error in rendering PathDistributionInfo:", error);
-            return undefined;
-        }
-    }
-}

package/src/diagnostics/ValuePathWarning.tsx

@@ -1,68 +0,0 @@
-import { qreact } from "../4-dom/qreact";
-import { css } from "../4-dom/css";
-import { Querysub, QuerysubController, querysubNodeId } from "../4-querysub/QuerysubController";
-import { isCurrentUserSuperUser } from "../user-implementation/userData";
-import { t } from "../2-proxy/schema2";
-
-module.hotreload = true;
-
-export class ValuePathWarning extends qreact.Component {
-    state = t.state({
-        count: t.number
-    });
-    componentDidMount() {
-        Querysub.onCommitFinished(async () => {
-            let nodeId = await querysubNodeId();
-            if (!nodeId) return;
-            let count = await QuerysubController.nodes[nodeId].debugGetValuePathCount();
-            Querysub.localCommit(() => this.state.count = count);
-        });
-    }
-    renderBase() {
-        if (!isCurrentUserSuperUser()) return undefined;
-        let count = this.state.count;
-        if (!count) return undefined;
-
-        if (count <= 150) {
-            return <div className={css.hbox(4)}>
-                <span>📄</span>
-                <span>{count}</span>
-            </div>;
-        }
-
-        let bgClass = "";
-        let flashing = false;
-        if (count > 500) {
-            bgClass = css.hsl(0, 80, 60);
-            flashing = true;
-        } else if (count > 300) {
-            bgClass = css.hsl(0, 80, 60);
-        } else {
-            bgClass = css.hsl(50, 100, 40);
-        }
-
-        let animClassName = "ValuePathWarning-flash";
-        return <div className={css.hbox(4).pad2(6, 2) + " " + bgClass + (flashing ? " " + animClassName : "")}>
-            <span>📄</span>
-            <span>{count}</span>
-            <span>warning: high number of files, join is likely not running</span>
-            {flashing && <style>{`
-                @keyframes ${animClassName}-anim {
-                    0%, 100% { background-color: hsl(0, 80%, 60%); }
-                    50% { background-color: hsl(0, 80%, 30%); }
-                }
-                .${animClassName} {
-                    animation: ${animClassName}-anim 0.6s infinite;
-                }
-            `}</style>}
-        </div>;
-    }
-    render() {
-        try {
-            return this.renderBase();
-        } catch (error) {
-            console.error("Error in rendering ValuePathWarning:", error);
-            return undefined;
-        }
-    }
-}