@ariestools/aries-dapp-core 0.1.7 → 0.1.9

package/README.md

@@ -1,38 +1,332 @@
 # @ariestools/aries-dapp-core
 
-> **Dev only.** This is the compute half of the local `aries dapp` backend, a
-> development fixture with no auth or metrics and memory-only storage. Not for
-> production at this time.
+Dapp backend core: a process-wide provider locator, pluggable reducer, coherent
+generation publication for S3/R2-backed indexes, and the stock `DappActor`
+timer. Designed for **outbound-only** production indexers (no public inbound
+API on the compute process) as well as the local `aries dapp` development
+fixture.
 
-The brains of a local dapp backend. It pairs with the storage server in
-[`@ariestools/aries-dapp-serve`](../dapp-serve): the server holds three buckets
-(`data`, `state`, `index`), and this package derives the mutable buckets from
-the immutable one on an interval.
+## Development vs production
+
+| Surface | Purpose |
+| --- | --- |
+| `aries-dapp-server` bin + `noopReducer` | **Local development only** — embedded s3rver via `@ariestools/aries-dapp-serve`, memory backing, no real derivation |
+| Project-owned entrypoint + real reducer | **Production** — inject an S3/R2 client, bind logical roles to buckets/prefixes, publish immutable generations |
+
+`dapp-serve` and the noop daemon remain explicit local tools. Production usage
+is a **project-owned** executable that composes this package; dapp-core does
+not dynamically load arbitrary reducer modules and does not store application
+authority (signing seeds, Webble controller roots, etc.).
+
+Published state and indexes are **rebuildable projections**, not authority.
+Public buckets must not be assumed safe for plaintext private material.
 
 ## Pieces
 
-- **`AbstractActor`** — a lean, dapp-owned fork of the xl1 `ActorV3` pattern:
-  the `create`/`start`/`stop` lifecycle plus non-overlapping interval timers
-  (`registerTimer`) and a readiness contract. It swaps the chain's
-  `ProviderFactoryLocator`/`LocatorConfig` machinery for the small
-  `DappProviderLocator` seam, so an actor resolves providers by moniker without
-  the chain protocol config model. Intended to graduate into a shared cli/actor
-  toolkit — and, later, xl1-protocol — where it can re-adopt the real locator.
-- **`DappProviderLocator` / `createDappLocator`** — the **process-wide** service
-  locator, built once and shared across every actor (xl1-cli style). It
-  pre-registers an S3-backed `DappBucketStore` per bucket keyed by moniker.
-- **`DappActor`** — resolves the three stores from the locator and drives its
-  `DappReducer` on an interval to keep `state`/`index` derived from `data`.
-- **`DappReducer`** — the pluggable unit of dapp logic (`reduce({data, state,
-  index, logger})`). `noopReducer` is the default; supply a real one to derive.
-- **`bootDappActors`** — creates and starts N actors against one shared locator.
-
-## Daemon
-
-The composed daemon bin (`dist/bin/dappServer.mjs`) starts the storage server
-and, in the same process, the locator and actors. `aries dapp up` spawns it.
-Env config: `DAPP_PORT` (default 8801), `HOST` (default 127.0.0.1),
-`DAPP_BACKING` (default `memory`), `DAPP_PUBLIC_HOST`, `DAPP_TLS_CERT`,
-`DAPP_TLS_KEY`, and `DAPP_REDUCE_INTERVAL_MS` (default 5000). Under
-`--ssl auto` the daemon exports the local CA via `NODE_EXTRA_CA_CERTS` so the
-in-process S3 client trusts the HTTPS origin.
+- **`AbstractActor`** — create/start/stop lifecycle, non-overlapping timers
+  (`firstRunDelay` is the real delay before the first pass), abort signal on
+  stop, optional shutdown bound for in-flight work, readiness contract.
+- **`DappProviderLocator` / `createDappLocator`** — process-wide locator. One
+  shared S3 client across roles, or **per-role clients** for least-privilege
+  principals; logical role → physical bucket + prefix + optional public CDN base.
+  The **data** role is registered read-only by default (`put` throws).
+- **`DappObjectReader` / `DappObjectWriter` / `DappBucketStore`** — capability
+  split: reducers see `data` as read-only; state/index remain writers.
+  get/getWithMeta/stat/put (metadata + conditional writes)/list*/destroy.
+- **`publishGeneration` / `readPublishedHead`** — fenced publication: create-only
+  generation objects, head-last with `conditional-head` CAS or injected lease
+  (or `unfenced` for local s3rver only). Returns `{ ok: true, … }` or
+  `{ ok: false, conflict }`.
+- **`writeIndexerStatus` / `readIndexerStatus`** — durable floor/cursor,
+  completed position, observed source head, generation, success/error
+  timestamps, consecutive failures (CDN-revalidate).
+- **`DappActor`** — drives a `DappReducer` on an interval; optional status
+  publication; optional `maxConsecutiveFailures` for supervisor restarts.
+- **`bootDappActors`** — starts N actors, runs readiness, stops earlier actors
+  if a later one fails.
+
+## Production composition (outbound-only)
+
+```ts
+import { S3Client } from '@aws-sdk/client-s3'
+import {
+  bootDappActors,
+  createDappLocator,
+  publishGeneration,
+  type DappReducer,
+} from '@ariestools/aries-dapp-core'
+
+const client = new S3Client({
+  region: 'auto',
+  endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
+  forcePathStyle: true,
+  credentials: {
+    accessKeyId: process.env.R2_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,
+  },
+})
+
+const locator = createDappLocator({
+  client,
+  ownClient: true,
+  bindings: {
+    data: {
+      bucket: 'my-dapp',
+      prefix: 'data',
+      publicBaseUrl: 'https://cdn.example.com/data',
+    },
+    state: {
+      bucket: 'my-dapp',
+      prefix: 'state',
+      publicBaseUrl: 'https://cdn.example.com/state',
+    },
+    index: {
+      bucket: 'my-dapp',
+      prefix: 'index',
+      publicBaseUrl: 'https://cdn.example.com/index',
+    },
+  },
+})
+
+const reducer: DappReducer = {
+  name: 'my-app-indexer',
+  version: '1',
+  async reduce({ state, index, signal, logger }) {
+    const published = await publishGeneration({
+      state,
+      index,
+      safety: { mode: 'conditional-head' }, // fail closed if provider lacks CAS
+      // expectedHeadEtag: prior?.headEtag,
+      reducer: { name: 'my-app-indexer', version: '1' },
+      source: { cursor: '…', completedPosition: '…', observedHead: '…' },
+      stateObjects: [{ key: 'view.json', body: '{}', contentType: 'application/json' }],
+      indexObjects: [{ key: 'by-id.json', body: '{}', contentType: 'application/json' }],
+      signal,
+    })
+    if (!published.ok) {
+      logger.warn(`publish conflict: ${published.conflict.kind} ${published.conflict.message}`)
+      return // do not advance checkpoint on conflict
+    }
+    logger.info(`published ${published.generation}`)
+    return {
+      cursor: '…',
+      lastCompletedPosition: '…',
+      observedSourceHead: '…',
+      generation: published.generation,
+      generationRoot: published.manifestKey,
+    }
+  },
+}
+
+const actors = await bootDappActors(locator, [{
+  name: 'MyIndexer',
+  reducer,
+  firstRunDelayMs: 1_000,
+  reduceIntervalMs: 15_000,
+  maxConsecutiveFailures: 10,
+  publishStatus: true,
+}])
+```
+
+Clients never contact this process. They read `head.json`, generation objects,
+and `status.json` from the public CDN/base URLs. Credentials stay out of
+published config and logs.
+
+See also `src/examples/productionComposition.ts`.
+
+### Publication layout
+
+```
+state/head.json                          # mutable head (revalidate CDN)
+state/status.json                        # indexer status (revalidate CDN)
+state/generations/<id>/manifest.json     # immutable
+state/generations/<id>/…                 # immutable state objects
+index/generations/<id>/…                 # immutable index objects
+```
+
+Head is written **last**. A crash before head leaves the previous complete
+generation visible. Checkpoints must not advance past a fully published
+generation (the actor only records generation after a successful reduce that
+published head).
+
+## Local daemon (dev only)
+
+The composed bin (`dist/bin/dappServer.mjs`) starts the storage server and a
+DappActor. `aries dapp up` spawns it.
+
+### CLI
+
+```sh
+# Empty spine (noop reducer)
+aries dapp up
+
+# Real project reducer (ESM .mjs or package export — TypeScript not supported)
+aries dapp up --reducer ./dist/indexer.mjs
+aries dapp up --reducer ./dist/indexer.mjs --reducer-export myIndex
+aries dapp up --reducer @myorg/my-dapp/indexer
+
+# Persist buckets across restarts
+aries dapp up --backing disk --data-dir ~/.aries/dapp/data
+```
+
+Example reducer shipped with this package:
+
+```sh
+aries dapp up --reducer ./node_modules/@ariestools/aries-dapp-core/examples/countFactsReducer.mjs
+# monorepo:
+aries dapp up --reducer ./packages/dapp-core/examples/countFactsReducer.mjs
+```
+
+### Reducer module contract
+
+The module must export a `DappReducer`:
+
+- **default export**, or
+- named export `reducer`, or
+- any name via `--reducer-export` / `DAPP_REDUCER_EXPORT`
+
+```js
+// dist/indexer.mjs
+export default {
+  name: 'my-index',
+  version: '1',
+  async reduce({ data, state, index, signal, logger }) {
+    // read data, publishGeneration / publishIncremental, return progress
+  },
+}
+```
+
+Local publication should use `safety: { mode: 'unfenced' }` (or honor
+`DAPP_PUBLICATION_SAFETY`, which the daemon sets to `unfenced` by default).
+s3rver does not support reliable head CAS.
+
+### Env (daemon)
+
+| Variable | Default | Notes |
+| --- | --- | --- |
+| `DAPP_PORT` | `8801` | |
+| `HOST` | `127.0.0.1` | |
+| `DAPP_BACKING` | `memory` | `memory` \| `disk` |
+| `DAPP_DATA_DIR` | `~/.aries/dapp/data` | disk root |
+| `DAPP_REDUCER` | _(noop)_ | path to `.mjs`/`.js` or package export |
+| `DAPP_REDUCER_EXPORT` | default / `reducer` | named export |
+| `DAPP_REDUCE_INTERVAL_MS` | `5000` | |
+| `DAPP_FIRST_RUN_DELAY_MS` | `0` | |
+| `DAPP_PUBLICATION_SAFETY` | `unfenced` | set by daemon for project reducers |
+| `DAPP_PUBLIC_HOST` / `DAPP_TLS_*` | | `--ssl auto` |
+
+Under `--ssl auto` the daemon exports the local CA via `NODE_EXTRA_CA_CERTS`.
+
+## Production entrypoint (not the CLI bin)
+
+Do **not** run `aries-dapp-server` or `aries dapp up` in production. Write a
+project-owned process that:
+
+1. Builds an `S3Client` for R2/S3 (credentials from env/IAM — never logged)
+2. Calls `createDappLocator({ client, bindings, ownClient: true })`
+3. Implements a `DappReducer` (import statically — no dynamic CLI load required)
+4. Calls `bootDappActors` with `safety: { mode: 'conditional-head' }` inside
+   `publishGeneration` / `publishIncremental`
+5. Listens for SIGTERM and stops actors + `locator.destroy()`
+
+See `examples/productionComposition.ts` and the composition section above.
+There is no inbound public listener; clients read head/status/objects from CDN.
+
+## Least-privilege IAM / R2 notes
+
+- **Writer (private API):** state/index need `s3:GetObject`, `s3:PutObject`, and
+  usually `s3:ListBucket` on the role prefix. Data role: `GetObject`/`ListBucket`
+  only (no `PutObject`). Prefer distinct access keys via `createDappLocator({ clients: { data, state, index } })`.
+- **Public CDN:** allow anonymous `GET`/`HEAD` only. **Deny** `s3:ListBucket` and
+  all `Put*`/`Delete*` for the public principal. dapp-core never requires public
+  LIST or PUT.
+- **Credentials** must not appear in published head/manifest/status or logs.
+
+## Publication safety
+
+| Mode | Use |
+| --- | --- |
+| `conditional-head` | Production when the provider supports If-Match / If-None-Match |
+| `lease` | Production with an application-injected lease/fencing adapter |
+| `unfenced` | **Local s3rver / tests only** — no concurrent-writer safety |
+
+## Releases, pins, rollback, GC
+
+```ts
+import {
+  listReleases, pinRelease, rollbackHead,
+  planGarbageCollection, runGarbageCollection,
+} from '@ariestools/aries-dapp-core'
+
+await pinRelease(state, 'gen-a', { reason: 'hotfix' })
+await rollbackHead(state, {
+  targetGeneration: 'gen-a',
+  safety: { mode: 'conditional-head' },
+  expectedHeadEtag: currentEtag,
+})
+
+const plan = await planGarbageCollection({
+  state,
+  index,
+  policy: { mode: 'bounded', retainGenerations: 5, cdnGraceMs: 86_400_000 },
+})
+await runGarbageCollection(plan, { state, index }, { dryRun: true })
+```
+
+Source **data** is never GC'd. Active head, pins, and in-flight generations are protected.
+
+## Incremental high-cardinality publication
+
+For large sparsely-changing indexes use `publishIncremental` (CAS bodies + 256-way
+sharded map). Unchanged keys keep stable `cas/sha256/<hash>` URLs; each pass
+writes only changed objects + touched shards + root + receipt.
+
+```ts
+import { publishIncremental } from '@ariestools/aries-dapp-core'
+
+const result = await publishIncremental({
+  state,
+  index,
+  safety: { mode: 'conditional-head' },
+  expectedHeadEtag: prior?.headEtag,
+  reducer: { name: 'addr-index', version: '1' },
+  indexChanges: [{ key: 'by-source/0xabc', body: frameJson, contentType: 'application/json' }],
+})
+```
+
+## Browser-neutral consumer
+
+```ts
+import {
+  verifyPublishedView, resolveObject,
+} from '@ariestools/aries-dapp-core/consumer'
+
+const view = await verifyPublishedView({
+  fetch,
+  bases: { state: 'https://cdn.example/state', index: 'https://cdn.example/index' },
+})
+const one = await resolveObject({
+  fetch,
+  bases: { state: 'https://cdn.example/state', index: 'https://cdn.example/index' },
+  logicalKey: 'by-source/0xabc',
+  role: 'index',
+})
+```
+
+No Node built-ins or AWS SDK on this subpath.
+
+## Opt-in R2 contract test
+
+Ordinary CI does not require R2 secrets. With credentials:
+
+```sh
+export DAPP_R2_CONTRACT_TEST=1
+export R2_ACCOUNT_ID=…
+export R2_ACCESS_KEY_ID=…
+export R2_SECRET_ACCESS_KEY=…
+export R2_BUCKET=…
+# optional: R2_PUBLIC_BASE_URL=https://pub-….r2.dev
+pnpm xy test @ariestools/aries-dapp-core
+```
+
+See `src/spec/r2.contract.spec.ts`.