diskeyval 1.0.0

package/.github/workflows/test.yml

@@ -0,0 +1,31 @@
+name: Node.js Test Runner
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+      - master
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        node-version: [22.x, 24.x]
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Use Node.js ${{ matrix.node-version }}
+      uses: actions/setup-node@v4
+      with:
+        node-version: ${{ matrix.node-version }}
+        cache: npm
+
+    - run: npm ci
+    - run: npm run build --if-present
+    - run: npm test
+      env:
+        CI: true

package/LICENSE

@@ -0,0 +1,7 @@
+Copyright 2022 Mark Wylde
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/PLAN.md

@@ -0,0 +1,169 @@
+# diskeyval Implementation Plan
+
+## Goal
+Build a fast distributed key/value store with majority-based semantics:
+
+- `set(key, value)` resolves after majority commit.
+- `get(key)` resolves from majority-committed state.
+- Node-to-node trust is established with CA-signed mTLS certificates.
+- Raft logic is isolated into its own module so it can later be extracted as a standalone package.
+- Test surface is intentionally larger than implementation surface.
+
+## Testing-First Policy
+
+- More tests than implementation code until v1 semantics are stable.
+- New behavior must land with tests in `unit` plus `sim` or `e2e`.
+- Every production bug gets a regression test before fix merge.
+- CI blocks merges when required suites fail.
+
+Target ratios (early project):
+
+- Unit test count >= 2x exported Raft/API methods.
+- Total test scenarios >= 90 before first feature-complete milestone.
+- Safety invariants must be asserted in every simulation/e2e run.
+
+## Architecture
+
+### Module A: `raft-core` (isolated consensus module)
+Responsibilities:
+
+- Leader election (follower/candidate/leader roles)
+- Heartbeats and election timeouts
+- Log replication (`AppendEntries`)
+- Vote requests (`RequestVote`)
+- Commit index advancement after majority replication
+- Apply committed entries through a state machine callback
+- Read path support for linearizable reads (leader read barrier / read index)
+
+Public API target:
+
+- `start()`, `stop()`
+- `propose(command): Promise<CommitResult>`
+- `readBarrier(): Promise<void>`
+- events: `leader`, `commit`, `role-change`
+
+Non-goals for v1:
+
+- Dynamic cluster reconfiguration
+- Snapshot streaming
+- Disk-backed WAL compaction
+
+### Module B: `diskeyval` (store + user API)
+Responsibilities:
+
+- Public API (`start`, `set`, `get`, `end`)
+- In-memory materialized map (`state`)
+- Eventing (`change`, `leader`)
+- Command encoding/decoding for the Raft state machine
+- Routing client writes to leader (or forwarding)
+
+### Module C: transport/security
+Responsibilities:
+
+- mTLS transport between nodes
+- Certificate validation against cluster CA
+- Peer identity checks (`nodeId` matches certificate identity)
+- RPC framing and message serialization
+
+## Delivery Phases
+
+## Phase 0: Test Harness First
+- Establish suite layout: `test/unit`, `test/sim`, `test/e2e`, `test/soak`.
+- Define initial scenario matrix and acceptance gates.
+- Build deterministic simulation harness interfaces before protocol implementation.
+
+Exit criteria:
+- Test runners exist for each suite and scenario inventory is committed.
+
+## Phase 1: Local Raft Core Skeleton
+- Create `lib/raft/` module boundaries and types.
+- Implement role machine + timers + term transitions.
+- Add message handlers for `RequestVote` and empty `AppendEntries`.
+- Unit test election safety basics and timer behavior.
+
+Exit criteria:
+- Leader election stabilizes in a 3-node in-memory simulated cluster.
+
+## Phase 2: Replicated Log + Majority Commit
+- Add log entries and replication indexes (`nextIndex`, `matchIndex`).
+- Implement majority commit rule.
+- Apply committed entries in order to state machine callback.
+- Wire `diskeyval.set` to `raft.propose`.
+
+Exit criteria:
+- `set` resolves only after majority commit in 3-node simulation.
+
+## Phase 3: Read Correctness
+- Implement linearizable reads via `readBarrier()` on leader.
+- Wire `diskeyval.get` through read barrier.
+- Add tests for stale-leader prevention.
+
+Exit criteria:
+- `get` never returns uncommitted or stale values after leader changes.
+
+## Phase 4: Real Transport + mTLS
+- Replace in-memory transport with socket RPC transport.
+- Add mTLS handshake and cert validation.
+- Enforce `nodeId`/certificate identity mapping.
+
+Exit criteria:
+- Multi-process local cluster can elect leader and commit writes over mTLS.
+
+## Phase 5: Reliability + Performance
+- Add retry/backoff for replication RPCs.
+- Add batching/pipelining for append entries.
+- Add metrics hooks (latency, commit lag, election count).
+- Start WAL/snapshot design for restart durability.
+
+Exit criteria:
+- Stable behavior under node restarts/network jitter in integration tests.
+
+## Testing Strategy
+
+- Unit tests for term/vote/commit invariants and edge cases.
+- Simulation tests for partitions, recoveries, delays, drops, and reorderings.
+- Integration/e2e tests for mTLS, multi-process cluster behavior, crash/restart.
+- Property tests for log matching and monotonic commit index.
+- Soak/chaos tests for long-running stability and latency/lag budgets.
+
+## Merge Gates
+
+- Protocol code change: requires `test:unit` and `test:sim`.
+- Transport/security change: requires `test:unit` and `test:e2e`.
+- Release candidate: requires `test:all` plus soak run report.
+
+## Key Invariants (must always hold)
+
+- At most one leader per term.
+- Committed entries never roll back.
+- A node applies entries in log order only.
+- `set` success implies majority persistence in memory/log.
+- `get` is served from majority-committed state.
+
+## Milestone Sequence
+
+1. Raft core skeleton in-process simulation
+2. Majority commit + `set` semantics
+3. Linearizable `get`
+4. mTLS transport
+5. Performance passes and durability
+
+## Status
+
+- Phase 0: completed
+- Phase 1: completed
+- Phase 2: completed
+- Phase 3: completed
+- Phase 4: completed
+- Phase 5: completed
+
+Verification:
+
+- Real test suites are executable (no placeholder todo tests).
+- Unit + simulation + mTLS e2e + soak pass in CI command path (`npm test`).
+
+## Notes
+
+- Keep `raft-core` free of app/store concerns so it can become its own package.
+- Prefer small, composable interfaces between raft, transport, and store.
+- Start with in-memory first for speed, then layer network/durability.

package/README.md

@@ -0,0 +1,119 @@
+# diskeyval
+
+## Installation
+```bash
+npm install --save diskeyval
+```
+
+## Overview
+`diskeyval` is a distributed in-memory key/value store with Raft consensus.
+
+- `set(key, value)` resolves only after majority commit.
+- `get(key)` resolves from majority-committed state.
+- Dynamic cluster reconfiguration is implemented via `reconfigure(peers)`.
+- Inter-node communication uses mTLS with CA-signed certs.
+- Implementation style is functional/context-based (no classes or interfaces).
+
+## API Example
+```typescript
+import diskeyval from 'diskeyval';
+
+const node1 = diskeyval({
+  nodeId: 'node-1',
+  host: '127.0.0.1',
+  port: 8050,
+  peers: [
+    { nodeId: 'node-2', host: '127.0.0.1', port: 8051 },
+    { nodeId: 'node-3', host: '127.0.0.1', port: 8052 }
+  ],
+  tls: {
+    cert: process.env.NODE1_CERT_PEM!,
+    key: process.env.NODE1_KEY_PEM!,
+    ca: process.env.CLUSTER_CA_PEM!
+  }
+});
+
+const node2 = diskeyval({
+  nodeId: 'node-2',
+  host: '127.0.0.1',
+  port: 8051,
+  peers: [
+    { nodeId: 'node-1', host: '127.0.0.1', port: 8050 },
+    { nodeId: 'node-3', host: '127.0.0.1', port: 8052 }
+  ],
+  tls: {
+    cert: process.env.NODE2_CERT_PEM!,
+    key: process.env.NODE2_KEY_PEM!,
+    ca: process.env.CLUSTER_CA_PEM!
+  }
+});
+
+await node1.start();
+await node2.start();
+
+await node1.set('testkey', 'testvalue1');
+const value = await node1.get('testkey');
+
+node1.on('change', ({ key, value }) => {
+  console.log('changed', key, value);
+});
+
+await node1.end();
+await node2.end();
+```
+
+## Options
+```typescript
+type DiskeyvalOptions = {
+  nodeId: string;
+
+  // Required for built-in TLS transport:
+  host?: string;
+  port?: number;
+  peers: string[] | Array<{ nodeId: string; host: string; port: number }>;
+  tls?: {
+    cert: string;
+    key: string;
+    ca: string;
+  };
+
+  // Optional custom transport (used by simulation/in-memory tests):
+  transport?: RaftTransport<SetCommand>;
+  persistence?: {
+    dir: string;
+    compactEvery?: number;
+  };
+
+  electionTimeoutMs?: number;
+  heartbeatMs?: number;
+  proposalTimeoutMs?: number;
+  rpcTimeoutMs?: number;
+};
+```
+
+## Methods
+- `start(): Promise<void>`
+- `set(key: string, value: unknown): Promise<void>`
+- `get<T = unknown>(key: string): Promise<T | undefined>`
+- `reconfigure(peers: ClusterPeer[]): Promise<void>`
+- `getPeers(): ClusterPeer[]`
+- `end(): Promise<void>`
+- `isLeader(): boolean`
+- `leaderId(): string | null`
+- `getMetrics(): RaftMetrics`
+
+## Dynamic Membership
+- Reconfiguration is consensus-committed through the Raft log.
+- Nodes apply committed membership updates at the same log index as other commands.
+- New members catch up from the leader after being added.
+
+## Events
+- `change` with payload `{ key: string; value: unknown }`
+- `leader` with payload `{ nodeId: string | null }`
+
+## Guarantees
+- Majority-acknowledged writes.
+- Majority-consistent reads.
+- Single committed global write order through Raft.
+- mTLS-authenticated inter-node RPC with cert identity checks (CN/SAN).
+- File-backed durability via WAL + periodic snapshot compaction when `persistence` is enabled.

package/TESTING.md

@@ -0,0 +1,94 @@
+# Testing Strategy
+
+This project is intentionally test-heavy. The target is to maintain more test code and scenarios than implementation code during early phases.
+
+## Principles
+
+- Test first for every consensus behavior.
+- Prefer deterministic simulation over flaky timing-only checks.
+- Every bug gets a regression test before the fix is merged.
+- No protocol change without unit + simulation + e2e coverage.
+
+## Suite Layers
+
+1. Unit tests (`test/unit`)
+- Pure protocol/state-machine behavior.
+- No real sockets.
+- Deterministic clocks/network fakes.
+
+2. Simulation tests (`test/sim`)
+- Multi-node in-memory cluster.
+- Partition, drop, delay, reordering, and clock-skew scenarios.
+
+3. End-to-end tests (`test/e2e`)
+- Real processes, real sockets, real mTLS certs.
+- Crash/restart and leadership churn.
+
+4. Soak/chaos tests (`test/soak`)
+- Long-running stability and throughput checks.
+- Randomized failures and recoveries.
+
+## Coverage Targets
+
+- Raft invariants: 100% scenario coverage for leader election, log matching, and commit monotonicity.
+- API semantics: 100% coverage for majority-commit `set` and linearizable `get` behavior.
+- Transport/auth: certificate validation, identity binding, and rejection cases.
+
+## Scenario Backlog (High Priority)
+
+### Election and leadership
+- Single leader elected in a stable 3-node cluster.
+- No dual leaders in the same term.
+- Follower timeout causes candidacy.
+- Split vote retries with term bump.
+- Outdated term candidate is rejected.
+- Leader steps down on higher term append.
+
+### Log replication and commit
+- Leader appends and replicates entry to majority.
+- Entry is committed only after majority.
+- Follower log conflict is truncated and corrected.
+- Commit index advances monotonically.
+- Applied index never exceeds commit index.
+- Old leader cannot commit after losing quorum.
+
+### Read correctness
+- Leader read barrier returns committed value.
+- Stale leader read is rejected or redirected.
+- Read after write is linearizable.
+- Read during leadership change preserves linearizability.
+
+### Failure handling
+- Minority partition cannot commit writes.
+- Majority partition continues serving writes.
+- Rejoined node catches up from log.
+- Node crash and restart rejoin behavior.
+- Network delay and reordering resilience.
+
+### Security and transport
+- Node with unknown CA is rejected.
+- Node with invalid cert signature is rejected.
+- Node ID mismatch to cert identity is rejected.
+- Expired cert is rejected.
+- Valid mTLS peers can form cluster.
+
+### API behavior
+- `set` resolves only after majority commit.
+- `set` times out cleanly when quorum unavailable.
+- `get` returns majority-committed state.
+- `change` event emits once per committed update.
+- `leader` event emits on leadership transitions.
+
+## Merge Gates
+
+A change should not merge unless:
+
+- Relevant unit tests are present and passing.
+- At least one simulation or e2e case covers the behavior.
+- Regression tests exist for any fixed defect.
+
+## CI Plan
+
+- PR: `unit + sim` mandatory.
+- Main branch: `unit + sim + e2e` mandatory.
+- Nightly: `soak` and extended chaos scenarios.