@bopen-io/wallet-toolbox 2.14.5 → 2.15.0-bopen.1

package/CHANGELOG.md

@@ -4,8 +4,53 @@ This document captures the history of significant changes to the wallet-toolbox
 The git commit history contains the details but is unable to draw
 attention to changes that materially alter behavior or extend functionality.
 
-## wallet-toolbox 2.14.5
-
+## wallet-toolbox 2.15.0
+
+- `StorageKnex` runs on Postgres through the knex `pg` client. Settings report
+  `dbtype: 'Postgres'`, and `StorageClient` accepts that value from a remote
+  `StorageServer`; older clients reject it. Install `pg` in the application.
+- On a Postgres knex, `StorageKnex` sets an int8 parser on each connection it
+  acquires, so bigint columns and counts come back as numbers, as with mysql2
+  and better-sqlite3. The process-wide `pg.types` defaults are not changed.
+  `usePostgresInt8Numbers(knex)` applies the same parser to a knex directly, and
+  `Setup.createPostgresKnex(connectionJson, database)` creates one with it.
+- Postgres runs at READ COMMITTED. The findOrInsert helpers retry a failed
+  insert as a find in the same transaction; on Postgres the insert now runs in
+  a savepoint so the transaction stays usable. `allocateChangeInput` re-checks
+  action batch reservations after taking the output row lock.
+- On Postgres, migrations that add indexes to existing tables run outside a
+  transaction and use `CREATE INDEX CONCURRENTLY IF NOT EXISTS`; an invalid
+  index left by an interrupted build is dropped and rebuilt. `StorageKnex.migrate`
+  runs Postgres migrations one at a time so every other migration keeps its
+  journal row in its own transaction. SQLite and MySQL migrations are unchanged.
+- `adminStats` and the admin server's proof request review run on MySQL and
+  Postgres. `adminStats` still reports `WERR_NOT_IMPLEMENTED` on SQLite.
+- `MonitorDaemon` accepts `postgresConnection` (JSON node-postgres config), and
+  the operator `monitor-daemon` command accepts `--database-client postgres`.
+- `ChaintracksStorageKnex` rejects a Postgres knex before running migrations.
+- Paged `StorageKnex` find queries and the `get*ForUser` sync queries order rows
+  by key. Without an order, Postgres could return consecutive LIMIT/OFFSET pages
+  in different orders, so `getSyncChunk` repeated some rows and skipped others.
+- `StorageServer.app` is public, so a host can mount the storage server in its
+  own Express application (`hostApp.use('/storage', server.app)`) instead of
+  calling `start()`. `port` is optional and only required by `start()`.
+- `preRpcMiddleware` adds Express handlers on `POST /` after authentication,
+  the authenticated rate limit and payment, immediately before JSON-RPC
+  dispatch. A handler that responds stops the call before dispatch.
+- `publicRoutes: false` removes the unauthenticated `GET /`, `/robots.txt` and
+  `/healthz` routes. Defaults are unchanged.
+
+- Knex storage adds a unique index on `sync_states` (`userId`,
+  `storageIdentityKey`). Concurrent `findOrInsertSyncStateAuth` calls for a new
+  source could each insert a row, after which every sync with that source failed
+  with `Storage identity has conflicting sync states`. The losing insert now
+  re-reads the row that won. The migration deletes existing duplicates, keeping
+  the oldest row; when the duplicates had different storage names, the kept row
+  restarts as a new sync state because its checkpoint may belong to a different
+  source database.
+- BRC-38 import keeps one sync state per storage identity. An export holding
+  several for one identity imports the oldest under the same rule, and merge
+  updates the target's row for that identity even when its storage name differs.
 - `WalletPermissionsManager` retires no-send transaction ownership and reference
   aliases when `createAction` or `signAction` reports them as `sending` or
   `unproven`, including sendWith-only calls and successful members of an

package/README.md

@@ -66,16 +66,23 @@ journal before recovery; do not delete journal rows or wallet data blindly.
 ### Postgres storage
 
 `StorageKnex` runs on Postgres through the knex `pg` client. Install `pg` in the
-application and pass a Postgres knex:
+application. Settings report `dbtype: 'Postgres'`; remote clients older than
+this release reject that value.
 
-```ts
-import { knex as makeKnex } from 'knex'
+node-postgres returns int8 (bigint columns, counts) as strings by default.
+`StorageKnex` sets a parser on each connection its knex acquires so these come
+back as numbers, as with mysql2 and better-sqlite3. The process-wide `pg.types`
+defaults are not changed. `Setup.createPostgresKnex(connectionJson, database)`
+creates a knex with the same parser installed.
 
-const knex = makeKnex({ client: 'pg', connection: { host, port, user, password, database } })
-const storage = new StorageKnex({ ...StorageKnex.defaultOptions(), chain: 'main', knex })
-```
+Postgres migrations run one at a time. Migrations that add indexes to existing
+tables run outside a transaction and use `CREATE INDEX CONCURRENTLY`, so writes
+continue while an index is built. `MonitorDaemon` takes a `postgresConnection`
+(JSON node-postgres config). `ChaintracksStorageKnex` does not support Postgres.
 
-`ChaintracksStorageKnex` and `adminStats` do not support Postgres.
+Set `RUNPOSTGRES=1` and `POSTGRES_CONNECTION` (JSON node-postgres connection
+config) to include Postgres in the Knex test suites. Test databases are created
+on that server as needed.
 
 ## Overview
 
@@ -278,6 +285,10 @@ serialized RPC response exceeds the service ceiling, remote clients retry the
 read-only request with a smaller chunk budget and remember the working limit
 for the rest of the session.
 
+Knex storage keeps one sync state per user and source storage identity, enforced
+by a unique index. Concurrent registrations of the same source return the row the
+first caller inserted.
+
 Output synchronization requires a local mapping for every non-null source basket
 ID. A missing mapping rejects the page so its transaction and checkpoint can roll
 back; retry after transferring the missing basket. Newer source updates apply
@@ -1024,6 +1035,61 @@ Run `StorageKnex.migrate(...)` before constructing the manager during an
 upgrade. `makeAvailable()` validates and loads an already-migrated database; it
 does not apply schema changes.
 
+### Mounting StorageServer in an Express application
+
+`server.app` is the configured Express application. Mount it in a host
+application instead of calling `start()`; `port` and `http` are only used by
+`start()`.
+
+```typescript
+import express from 'express'
+import { SessionManager } from '@bsv/sdk'
+import { AuthRequest, createAuthMiddleware } from '@bsv/auth-express-middleware'
+import { StorageServer } from '@bsv/wallet-toolbox'
+
+// Use KnexSessionManager when the host runs more than one process.
+const sessionManager = new SessionManager()
+
+const storageServer = new StorageServer(storage, {
+  wallet,
+  monetize: false,
+  sessionManager,
+  // Leave GET /, /robots.txt and /healthz to the host.
+  publicRoutes: false,
+  preRpcMiddleware: [
+    async (req, res, next) => {
+      if (await overQuota((req as AuthRequest).auth.identityKey, req.body.method)) {
+        res.status(507).json({ status: 'error', code: 'ERR_QUOTA_EXCEEDED' })
+        return
+      }
+      next()
+    }
+  ]
+})
+
+const app = express()
+app.use('/storage', storageServer.app)
+app.use(express.json())
+app.use(createAuthMiddleware({ wallet, sessionManager }))
+// ...host routes
+app.listen(8080)
+```
+
+Clients connect with `new StorageClient(wallet, 'https://host.example/storage')`.
+`AuthFetch` sends the BRC-104 handshake to the origin's `/.well-known/auth`,
+not to the mount path, so the host must answer it with its own auth middleware
+using the same `wallet` and the same `sessionManager` as the storage server.
+Mount the storage server before the host's auth middleware and body parsers;
+the storage app authenticates and parses its own requests. It keeps its own
+security headers, CORS, concurrency and rate limits, and inherits the host's
+`trust proxy` setting unless `trustProxy` is set.
+
+`preRpcMiddleware` handlers run on JSON-RPC `POST /` requests after
+authentication, the authenticated rate limit and, when `monetize` is set, the
+payment middleware. They see `req.auth` and the parsed body. A handler that
+responds instead of calling `next()` stops the call before dispatch. They do not
+run on the action batch upload routes.
+
 ## Development
 
 ### Overlay identity verification