hikoutei 0.3.0 → 0.3.1

package/docs/quick-start.md

@@ -1,7 +1,8 @@
 # Hikoutei Quick Start
 
-The project is called Hikoutei and is currently published on npm as
-`hikoutei`.
+Hikoutei is a typed repository and safe write layer for Google Sheets-backed MVPs.
+SQLite is the application authority; Google Sheets is an asynchronous internal
+projection and human input surface.
 
 ## Installation
 
@@ -9,13 +10,11 @@ The project is called Hikoutei and is currently published on npm as
 npm install hikoutei @mikro-orm/core @mikro-orm/sql
 ```
 
-The root API does not expose MikroORM types. The current built-in SQLite
-provider uses those optional peer dependencies internally; a future provider
-can replace it without changing the entity lifecycle API.
+The root API does not expose MikroORM, SQL, Apps Script, or sync-worker types.
+The current built-in SQLite provider uses the optional MikroORM peer
+dependencies internally.
 
-## Define a scalar entity
-
-Entity metadata and environment-specific Sheet routes are separate:
+## Define an entity
 
 ```ts
 import { createTypedSheets, defineTypedSheetsEntity } from "hikoutei";
@@ -33,26 +32,16 @@ const User = defineTypedSheetsEntity({
 const hikoutei = await createTypedSheets({
   dbName: "./hikoutei.sqlite",
   entities: [User],
-  sheets: {
-    spreadsheetId: process.env.GOOGLE_SHEET_ID!,
-    routes: {
-      User: {
-        systemState: { tabName: "Users_System", registeredRange: "A:Z" },
-      },
-    },
-  },
 });
 ```
 
-`createTypedSheets()` validates the descriptors and routes, opens the local
-SQLite authority, registers canonical/outbox state, and prepares the current
-persistence provider. It does not call Google Sheets or mutate a remote tab.
+`createTypedSheets()` validates entity descriptors, opens the local SQLite
+authority, and creates the declared entity tables. It does not contact Google
+Sheets and accepts no Sheet route or gateway option.
 
 ## Entity lifecycle
 
-Use a request-local manager. Reads come from SQLite, never from the remote Sheet.
-The Unit of Work owns identity maps and snapshots, so mutation followed by
-`flush()` has the same meaning regardless of the persistence provider.
+Use a request-local manager. Reads come from SQLite, never from a remote Sheet.
 
 ```ts
 const em = hikoutei.em.fork();
@@ -69,83 +58,56 @@ const loaded = await em.findOne(User, { id: "u1" });
 if (loaded !== null) {
   loaded.name = "Ada Lovelace";
   await em.flush();
-
-  await em.transactional(async (transactionalEm) => {
-    transactionalEm.remove(loaded);
-  });
 }
+
+await em.transactional(async (transactionalEm) => {
+  const target = await transactionalEm.findOne(User, { id: "u1" });
+  if (target !== null) transactionalEm.remove(target);
+});
 ```
 
-The stable lifecycle methods are `fork()`, `create()`, `find()`, `persist()`,
-`remove()`, `flush()`, and `transactional()`. Relations and provider-specific
-query operators are not part of the scalar release.
-
-## Sheet setup and delivery
-
-`createTypedSheets()` opens the local SQLite authority and validates the
-registered routes, but it does not contact Google Sheets. Provisioning is an
-explicit service/deployment operation, and the included
-[`apps-script/gateway/Code.gs`](../apps-script/gateway/Code.gs) must be deployed
-before the server can deliver Sheet effects.
-
-### Deploy the Apps Script gateway
-
-Complete these steps once for the spreadsheet that the service will use:
-
-1. Create or open the target Google Spreadsheet. From **Extensions → Apps
-   Script**, create a **bound** Apps Script project for that spreadsheet.
-2. Copy the complete `apps-script/gateway/Code.gs` file from this repository or
-   from `node_modules/hikoutei/apps-script/gateway/Code.gs` into the Apps Script
-   editor, then save it.
-3. Choose **Deploy → New deployment**, select **Web app**, and deploy it with
-   **Execute as: Me**. Choose the narrowest access audience that can reach the
-   server. A server outside the Workspace normally requires **Anyone**; the
-   signed request secret still authenticates every useful POST.
-4. Copy the deployed URL ending in `/exec`. Do not use the editor-only `/dev`
-   URL. Paste the URL into the constant near the top of `Code.gs` and save:
-
-   ```js
-   var TYPED_SHEETS_GATEWAY_URL =
-     "https://script.google.com/macros/s/DEPLOYMENT_ID/exec";
-   ```
-
-5. In the **bound** Apps Script editor, select `setupSyncGateway` in the
-   function menu and click **Run**. Approve the requested Google permissions.
-   The helper reads the active spreadsheet ID, stores the ID and shared secret
-   in Script Properties, and logs a copyable local environment block. Running
-   it from a standalone Apps Script project cannot identify the spreadsheet.
-6. Open the execution log and copy the printed values into an untracked server
-   environment file or secret store:
-
-   ```dotenv
-   TYPED_SHEETS_GATEWAY_URL="https://script.google.com/macros/s/DEPLOYMENT_ID/exec"
-   TYPED_SHEETS_GATEWAY_SHARED_SECRET="generated-secret"
-   TYPED_SHEETS_GATEWAY_SHEET_ID="spreadsheet-id"
-   ```
-
-   The sheet ID is derived from the bound spreadsheet; it is not a second
-   spreadsheet to configure. Never put the shared secret in browser code or
-   commit it to Git.
-7. If `Code.gs` changes, use **Deploy → Manage deployments**, edit the Web App
-   deployment, and create a new version before relying on the changed gateway.
-   The `/exec` URL stays the same when an existing deployment is updated.
-
-### Provision and deliver Sheets
-
-Provide a server-side, provider-neutral `HikouteiSheetProvisioner` using the
-server credentials, then explicitly provision the registered tabs and headers:
+The stable lifecycle methods are `fork()`, `create()`, `find()`, `findOne()`,
+`persist()`, `remove()`, `flush()`, and `transactional()`. Relations and
+provider-specific query operators are not part of the scalar release.
 
-```ts
-await hikoutei.setupSheets(provisioner);
-```
+## Internal Sheet synchronization
+
+Applications do not call Sheet APIs or choose a write operation for each
+entity change. A service-side internal sync bootstrap owns:
+
+- projection routes and user-owned field configuration
+- Apps Script URL, spreadsheet ID, and shared secret
+- local projection registration and remote provisioning
+- the outbound effect supervisor and durable outbox delivery
+- User_Input polling, evaluation, conflict handling, and reconciliation
+
+When that internal service mode is active, `flush()` commits the entity table,
+canonical sync state, and durable Sheet effect outbox in one SQLite transaction.
+The call still does not wait for the remote Sheet write.
+
+The internal service provisions and validates the registered tabs before it
+starts delivery. Schema drift or gateway setup failure stops service startup
+rather than silently changing a remote Sheet. These service modules are under
+`src/application/sync/service/` and are intentionally not re-exported from the
+package root.
+
+## Apps Script gateway
+
+The deployable gateway is [`apps-script/gateway/Code.gs`](../apps-script/gateway/Code.gs).
+A service deployment supplies its URL and shared secret through a secret store
+or environment configuration. Never place the shared secret in browser code or
+commit it to Git.
+
+Live Google integration is opt-in and requires a deployed gateway, credentials,
+a spreadsheet, and external quota. The normal test suite uses fake gateways and
+SQLite fixtures without credentials.
 
-The operation is idempotent and rejects schema drift instead of silently
-rewriting headers. After setup, run the sync worker that delivers pending
-outbox effects. A successful `flush()` commits the entity table, canonical sync
-state, and durable Sheet effect outbox in one SQLite transaction; it does not
-mean that the remote Sheet has already been updated.
+## Read/write guarantees
 
-If `setupSyncGateway()` reports that it must run from a bound project, reopen
-Apps Script from the spreadsheet's **Extensions → Apps Script** menu. If the
-server receives `gateway_not_configured`, run the setup helper again and make
-sure the three environment variables are available to the service.
+- SQLite is the source of truth for application reads.
+- `flush()` returns after the local SQLite transaction commits.
+- Sheet delivery is asynchronous and at-least-once.
+- Stale or conflicting User_Input edits are recorded in SQLite rather than
+  silently overwriting canonical data.
+- Gateway response loss is recoverable work, not proof of a failed remote write.
+- Projection route/header drift fails explicitly.

package/docs/write-and-synchronization-flow.md

@@ -1,72 +1,103 @@
 # Write and Synchronization Flow
 
 Hikoutei commits local state first and materializes Google Sheets changes
-asynchronously. `createTypedSheets()` is local-only; remote tab/header
-provisioning happens only through the explicit `setupSheets()` boundary.
+asynchronously. The root ORM API is SQLite-only; the internal sync service owns
+all Sheet communication.
 
-## Outbound flow: SQLite to Google Sheets
+## Public ORM flow
 
 ```text
 em.persist(entity) / em.remove(entity)
           │
           ▼
 SQLite transaction
-  ├─ canonical entity state
-  └─ durable outbox effect
+  └─ entity table
           │
           ▼
-background effect worker
+return to application
+```
+
+`createTypedSheets()` never contacts Google Sheets and does not require a Sheet
+route, gateway client, or provisioner. Reads always come from SQLite.
+
+## Internal sync service flow
+
+The service-side bootstrap uses the same SQLite adapter and adds the canonical
+sync state and durable outbox to the flush transaction:
+
+```text
+EntityManager.flush()
+          │
+          ▼
+SQLite transaction
+  ├─ entity table
+  ├─ canonical sync state
+  ├─ projection registry/state
+  └─ durable Sheet effect outbox
+          │
+          ▼
+internal effect supervisor
   ├─ claim with a lease
   ├─ send a signed operation batch
   ├─ retry or recover an uncertain response
   └─ mark the effect applied/failed
           │
           ▼
-Apps Script gateway ── fast range write ──▶ Google Sheets
+Apps Script gateway ──▶ Google Sheets
 ```
 
-The application request does not wait for the remote Sheet write. A successful
-`flush()` means that SQLite accepted the entity change and the corresponding
-effect was durably queued.
+A successful `flush()` means that SQLite accepted the local change and queued
+its projection effect. It does not mean that the remote Sheet write has
+completed.
+
+## Fast append, update, and delete
 
-## Fast append
+New System_State rows use the bounded fast append operation where possible.
+Updates and deletes use guarded effects with expected visible revision/hash
+evidence. The same internal worker handles retries, response-loss recovery, and
+reconciliation. These operation types are implementation details and are not
+methods on the public EntityManager.
 
-The common system-state append path favors one contiguous range write. It avoids
-doing expensive metadata, snapshot, postcondition, and repair work before every
-append. Those checks belong to a separate safety path so the common write can
-remain small and predictable.
+## Inbound User_Input flow
 
-## Update and delete
+The internal service polls registered `User_Input` projections on a bounded
+interval. It reads the visible row, compares it with canonical SQLite state,
+and sends changed fields through the evaluator:
 
-Update and delete effects are still represented as durable outbox work and are
-sent by the same worker. Their safety policy can be stricter than append because
-overwriting or removing an existing row has a larger failure impact.
+```text
+User_Input polling
+  ├─ normalize literal/blank cells
+  ├─ resolve business-key row binding
+  ├─ validate ownership and field revisions
+  ├─ classify accepted, conflict, stale, or quarantine
+  └─ persist accepted observation and entity mutation in SQLite
+```
 
-## Gateway boundary
+Accepted observation writes update canonical state and the application entity in
+the same SQLite transaction. They do not enqueue a duplicate User_Input effect;
+only the required system projection repair/materialization is considered.
+Conflicts, stale writes, duplicate keys, and malformed cells remain visible in
+SQLite evidence tables for later handling.
 
-The Apps Script gateway is intentionally thin:
+## Provisioning and gateway boundary
+
+Projection provisioning is an internal service-start operation. The bootstrap
+generates route registrations and headers from the internal mapping, verifies
+remote schema drift, and starts workers only after provisioning succeeds. The
+Apps Script gateway remains intentionally thin:
 
 1. verify the signed operation envelope
 2. validate the operation contract
 3. execute the allowlisted Sheet operation
-4. return a structured result to the server worker
-
-Entity evaluation, canonical state, retry policy, reconciliation, and effect
-classification remain in the Node/SQLite side of Hikoutei.
-
-## Uncertain remote results
-
-An HTTP timeout or lost response does not prove that the Sheet write failed. The
-worker treats that result as recoverable work, retries according to its policy,
-and relies on idempotent effects and reconciliation to repair drift later.
+4. return a structured result to the internal worker
 
-The design therefore favors at-least-once delivery with a repair safety net over
-trying to prove exactly-once behavior from a response that may never arrive.
+Applications do not import or call the gateway client, protocol, operation
+builders, polling functions, or provisioning interfaces.
 
-## Inbound path: user edits
+## Failure model
 
-User-owned Sheet changes are a separate inbound concern. A future or configured
-`onEdit`/lightweight polling path reads the user-editable values, compares them
-with SQLite state, and sends the resulting observation into the evaluation and
-conflict pipeline. This path is intentionally separate from the fast outbound
-append path.
+An HTTP timeout or lost response does not prove that a Sheet write failed. The
+worker preserves durable work, classifies recoverable failures, and uses
+idempotent effects plus reconciliation to repair drift later. The SQLite commit
+is the application success boundary; remote delivery is at-least-once and
+asynchronous.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "hikoutei",
-  "version": "0.3.0",
+  "version": "0.3.1",
   "description": "Hikoutei — typed repository and safe write layer for Google Sheets-backed MVPs.",
   "repository": {
     "type": "git",