@agentshouse/kit 0.1.0-alpha.26 → 0.1.0-alpha.27

package/README.md

@@ -12,7 +12,7 @@ The Skill set Kit installs is Matt Pocock's
 On Linux `amd64` or `arm64`, run the published, versioned bootstrap once:
 
 ```sh
-bash <(curl -fsSL https://github.com/agentshouse/kit/releases/download/v0.1.0-alpha.26/connect-linux.sh)
+bash <(curl -fsSL https://github.com/agentshouse/kit/releases/download/v0.1.0-alpha.27/connect-linux.sh)
 ```
 
 The bootstrap first establishes Docker Engine from what it observes on the
@@ -51,8 +51,10 @@ the set again.
 Running a newer release's bootstrap the same way is the only update. When the
 installed container runs another image, the bootstrap pulls this release's
 image and asks the running resident Kit to drain with `kit resident drain`. The
-resident stops acquiring Runs, resumptions, setup and agent CLI work, while the
-Sessions it supervises keep their renewal, owner stops, answers and reports.
+resident stops acquiring Runs, resumptions, setup, agent CLI work, Google
+sign-ins and scheduled Google captures, while the Sessions it supervises keep
+their renewal, owner stops, answers and reports and a capture already running
+finishes.
 While any Agent Session is still running or unreported, including one whose Run
 was being acquired when the drain began, the bootstrap names it and leaves the
 container as it is; rerun it once they end. Once nothing remains,
@@ -141,21 +143,17 @@ anything reaches House.
 
 ## Building the package
 
-The repository checkout is the development source. Build the package
-that can be published by supplying House's dedicated installed-application
-Google OAuth client, its id and its client secret:
+The repository checkout is the development source. Build the package that can
+be published:
 
 ```sh
-HOUSE_KIT_GOOGLE_OAUTH_CLIENT_ID=kit-client.apps.googleusercontent.com \
-HOUSE_KIT_GOOGLE_OAUTH_CLIENT_SECRET=GOCSPX-kit-installed-app \
 npm run build
 ```
 
-The command writes the publishable package to `dist/package`. Both values are
-public instance content fixed into that package at build time: Google issues an
-installed application's client secret without treating it as confidential, and
-still requires it in every token request. The installed Kit does not read either
-from its runtime environment and never requests them from House.
+The command writes the publishable package to `dist/package`. It reads no
+environment and ships no provider OAuth client: each Google account signs in
+with the User's own client, as [the Google preset](#the-google-preset)
+describes.
 
 Runtime dependencies stay minimal and exactly pinned in `package.json` and
 `package-lock.json`: House's own `@agentshouse/mdmodel`, `yaml`, which reads
@@ -254,7 +252,8 @@ mode: `"intermittent"` for a laptop or other normally offline placement,
 any other value refuses as `kit_configuration_invalid`; `kit apply` sends the
 declared mode with the rest of the desired state. An
 instance declares one or more services, and under the `custom` preset each
-service owns the command that produces it.
+service owns the command that produces it. The Kit reads only `custom` instances
+and passes over any other entry of `instances`.
 
 `host` declares the Linux host this installation measures for
 [host capacity reporting](#host-capacity). `null` declares that this Kit reports
@@ -360,120 +359,81 @@ refuses as `kit_hook_rejected` instead of leaving whatever was installed before.
 
 ## The Google preset
 
-The published Kit is built with one House-owned Google installed-application
-OAuth client. Its id and client secret are public build-time instance content,
-not House secrets and not values served by House at run time. Google refresh
-and access tokens remain in the installation's local credential store.
+House holds this Environment's Google Sources, and its owner sets them in the
+App. Each entry names an account name the owner chooses, one service -
+`gmail`, `calendar` or `drive` - the Rooms it goes to, and what its first
+capture takes: `{ "new": true }` for new activity only, `{ "days": n }`,
+`{ "items": n }` or `{ "all": true }`. Neither the configuration file nor any
+command on this computer declares one.
+
+Each service captures a fixed corpus under its own read-only scope: Gmail every
+message but Spam and Trash, Calendar the account's primary calendar, and Drive
+the files the account owns.
+
+The resident reads the list from `/kit/sources/desired` each time its Kit
+control stream connects and each time House signals `sources` on it, then
+reconciles:
+
+- a new entry gets its local state and is captured once its account is signed
+  in for its service;
+- a changed entry's Rooms take effect on its next capture;
+- an entry House no longer lists stops, and its checkpoint, health, gaps,
+  dropped items and unsaved originals are deleted;
+- an account left with no entry has each of its grants revoked at Google's
+  revocation endpoint and its credentials, its OAuth client included, deleted.
+  Google's revocation ends every grant the Google user gave the client's
+  project, so when another listed account is signed in to the same Google user,
+  the Kit deletes the local credentials and revokes nothing.
+
+It then reports every entry to `/kit/sources/report`, and reports an entry
+again after each sign-in and each capture: `signed_out` while its
+account holds no grant for its service, `failed` with its cause code when its
+last capture failed, and `ready` otherwise, each with the account's signed-in
+email once the Kit knows it.
+
+The Kit ships no Google OAuth client. Each account signs in with the User's
+own: an OAuth client of type Desktop app in the User's Google Cloud project,
+with the service's API enabled there. An entry whose account holds no grant for
+its service opens a login renewal for integration `google`, instance the
+account name and service the entry's service, and presents its ceremony on
+House's owner-authenticated page, so the owner's browser never has to reach
+this Environment. When the account has no client yet, the page first collects
+the client id and secret. It then links to Google's consent for that service's
+scope only; Google sends the browser to a `http://127.0.0.1:<port>/` address
+that nothing answers, and the owner pastes that whole address into the page.
+The Kit accepts it only when it carries the state its own consent issued,
+exchanges the code with its PKCE verifier and the account's client, and stores
+the grant beside the client in `state/google/credentials/<account>.json`. The
+collected values pass through House's live relay only; neither they nor any
+token enter a report, a log or anything House stores. Adding a
+service to a signed-in account asks only for that service's consent, and each
+service keeps its own grant. An address that answers another consent, or a code
+Google refuses to exchange, stores nothing; a declined consent or a ceremony
+that outlives its ten-minute bound cancels the sign-in, and the Kit tries again
+at the next reconciliation.
+
+The resident captures each signed-in entry every 15 minutes, with no command.
+An occurrence never overlaps its entry's previous one, and one missed while the
+resident was stopped is not replayed: the next resumes from the checkpoint. The
+first capture fixes the initial choice's boundary, so `new` captures nothing
+that existed then and everything that arrives after.
 
-Each Google instance represents one Google account, and one Kit installation
-may declare as many as the User has. Each instance owns its own local Google
-authorization, selectors, Source references, Room routes, checkpoints and
-health under `state/<instance>/`; the installation's one `kit login` covers
-them all, and no instance can read another's provider credential.
-
-```json
-{
-  "availability": "intermittent",
-  "execution_authority": "refuse",
-  "migrators": [],
-  "instances": [
-    {
-      "id": "google-account",
-      "preset": "google",
-      "scopes": ["https://www.googleapis.com/auth/contacts.readonly"],
-      "services": [
-        {
-          "service": "gmail",
-          "source_ref": "src_google_mail",
-          "rooms": ["work-room"],
-          "backfill": { "days": 30 },
-          "selection": { "labels": ["INBOX"] }
-        },
-        {
-          "service": "calendar",
-          "source_ref": "src_google_calendar",
-          "rooms": ["work-room"],
-          "backfill": { "all": true },
-          "selection": { "calendars": ["primary"] }
-        },
-        {
-          "service": "drive",
-          "source_ref": "src_google_drive",
-          "rooms": ["work-room"],
-          "backfill": { "count": 500 },
-          "selection": { "files": ["file-id"], "folders": ["folder-id"] }
-        }
-      ]
-    }
-  ],
-  "clients": [],
-  "host": null,
-  "skills": false
-}
+```sh
+kit source capture work        # one occurrence of every entry of the account now
+kit source capture work gmail  # one occurrence of one entry now
+kit source status work
 ```
 
-Google authorization is incremental. Enabling Gmail requests only Gmail's
-read-only scope; Calendar and Drive request their own read-only scopes only
-when enabled. Each service scope set has its own local provider grant, so a
-later authorization never widens or replaces a sibling service's grant.
-Additional Google API scopes are declared once on the preset instance and
-remain local; a declared service names only scopes that instance already
-holds. Drive requires at least one explicit file or folder selection and never
-defaults to the whole account.
-
-A Google API outside the three curated services may use the closed declared
-service schema. The declaration is data only: no script or executable transform
-is loaded. Its host must be a Google API domain and it runs under scopes the
-instance already holds.
-
-```json
-{
-  "service": "people",
-  "source_ref": "src_google_people",
-  "rooms": ["work-room"],
-  "backfill": { "all": true },
-  "declaration": {
-    "host": "people.googleapis.com",
-    "path": "/v1/people/me/connections",
-    "query": { "personFields": "names,emailAddresses,metadata" },
-    "page_size": "pageSize",
-    "scopes": ["https://www.googleapis.com/auth/contacts.readonly"],
-    "page_token": {
-      "request": "pageToken",
-      "response": "nextPageToken",
-      "checkpoint_request": "syncToken",
-      "checkpoint_response": "nextSyncToken"
-    },
-    "items": "connections",
-    "fields": {
-      "native_id": "resourceName",
-      "native_version": "etag",
-      "provider_modified_at": "metadata.sources.0.updateTime",
-      "observed_at": "metadata.sources.0.updateTime",
-      "body": ["names.0.displayName", "emailAddresses.0.value"],
-      "container": "metadata.sources.0.type"
-    }
-  }
-}
-```
+`kit source capture` and `kit source status` take the account name where a
+custom connector takes its instance; a name House lists as an account is read
+as the account. While the resident runs, it performs the requested occurrence
+itself, after any occurrence of that entry already running.
 
-A Google instance signs in only through [Environment setup](#environment-setup),
-on House's owner-authenticated page, so the owner's browser never has to reach
-this Environment. For each enabled service whose scopes the instance does not
-hold yet, the page links to Google's consent for exactly that service's scopes.
-Google then sends the browser to a `http://127.0.0.1:<port>/` address that
-nothing answers: the owner copies that whole address from the address bar and
-pastes it into the page. The Kit accepts it only when it carries the state its
-own consent issued, exchanges the code with its PKCE verifier and the built
-client, and stores the grant locally. An address that answers another consent,
-or a code Google refuses to exchange, fails the login and stores nothing; a
-declined consent cancels it. Grants already completed for earlier services stay.
-
-`kit source sign-in <instance>` signs in a custom connector instance through
-its own command, described below. It skips an instance whose local
-authorization is already valid and answers `ready`, `skipped`, or
-`needs_attention`. A Google instance refuses it as `kit_connector_login_guided`
-and names House setup.
+A failed occurrence records its cause. A Google 403 for an API disabled in the
+client's project fails as `api_disabled`, and `kit source status` names the API
+to enable; a revoked scope (`google_scope_revoked`), an expired or revoked grant
+(`google_authority_revoked`) and a provider failure
+(`google_provider_unavailable`) keep their own codes.
 
 ## Environment setup
 
@@ -590,8 +550,10 @@ An object whose content changes without its native version changing keeps its
 key and is refused by House as a delivery-key conflict rather than silently
 overwriting the earlier Record.
 
-The same command owns its module authorization. `kit source sign-in` and the
-setup walk's status check send one JSON line
+The same command owns its module authorization. `kit source sign-in <instance>`
+skips an instance whose local authorization is already valid and answers
+`ready`, `skipped`, or `needs_attention`. It and the setup walk's status check
+send one JSON line
 with `action: "authorization"` and `mode: "status"` or `mode: "login"`; the
 command returns exactly one JSON object such as `{ "authorized": true }`.
 
@@ -1209,8 +1171,8 @@ on the door's own deduplication. When House stops a batch early, its answer
 settles only the prefix it reached, and the unsettled tail is reoffered with the
 rest. The loop repeats while the connector still returns items and no Room
 destination is blocked, and every run ends by writing the service's health into
-this installation's own state. House holds no preset, service, route, checkpoint
-or health of yours.
+this installation's own state. House holds no custom connector's service, route,
+checkpoint or health, and no Google entry's checkpoint or health.
 
 A Capture item that declares a `file` attachment answers one Record per Room
 destination: Rooms of one owner share that owner's Record, and a Room of another
@@ -1755,7 +1717,13 @@ $HOUSE_KIT_HOME/
   state/skill-set.json
   state/how-we-work
   state/continuations/<run identity>.json
-  state/<instance>/credentials/google.json
+  state/google/entries/<source>.json
+  state/google/credentials/<account>.json
+  state/google/checkpoints/<source>.json
+  state/google/health/<source>.json
+  state/google/gaps/<source>.json
+  state/google/dropped/<source>.json
+  state/google/originals/<source>.json
   state/<instance>/checkpoints/<service>.json
   state/<instance>/health/<service>.json
   state/<instance>/gaps/<service>.json
@@ -1765,15 +1733,18 @@ $HOUSE_KIT_HOME/
 
 Every file is written atomically and readable only by its owner. A file the Kit
 cannot read is refused rather than treated as a service that never captured.
-Disabling a service or an instance leaves its state in place, so re-enabling
-resumes from the stored cursor instead of importing again.
+A Google entry's state is keyed by its Source reference and lives only while
+House lists the entry. Disabling a custom service or instance leaves its state
+in place, so re-enabling resumes from the stored cursor instead of importing
+again.
 
 ```sh
 kit source reset acme        # every service
 kit source reset acme notes  # one service
 ```
 
-Reset is the only deletion of captured state. It removes the named checkpoints
+Reset serves custom connectors, and is the only deletion of their captured
+state. It removes the named checkpoints
 and their outstanding retained references so the next capture restarts exactly at
 the configured backfill boundary. It takes the same
 Environment control lease a capture takes, so it cannot delete a checkpoint a
@@ -1993,10 +1964,7 @@ workspaces.
 ```sh
 docker compose -f test/compose.yaml build kit
 docker compose -f test/compose.yaml run --rm kit npm test
-docker compose -f test/compose.yaml run --rm \
-  -e HOUSE_KIT_GOOGLE_OAUTH_CLIENT_ID=kit-client.apps.googleusercontent.com \
-  -e HOUSE_KIT_GOOGLE_OAUTH_CLIENT_SECRET=GOCSPX-kit-installed-app \
-  kit npm run build
+docker compose -f test/compose.yaml run --rm kit npm run build
 ```
 
 `test/pair/run.sh` boots real Core, Auth and PostgreSQL and runs the installed

package/bin/kit.js

@@ -5,12 +5,12 @@ import { join, resolve } from "node:path";
 import { connectHouse, loginHouse, storedEnvironment, storedHouse } from "../src/login.js";
 import { apply } from "../src/apply.js";
 import { captureSource, resetSource, retrySourceOriginals, sourceStatus } from "../src/source.js";
+import { accountEntries, accountStatus, captureAccount } from "../src/google/entries.js";
 import { setupEnvironment } from "../src/setup.js";
 import { signInConnector } from "../src/connector-login.js";
 import { attachSession } from "../src/attach.js";
 import { runResident, runWorkers } from "../src/worker.js";
 import { openSession, releaseControl, requestDrain } from "../src/house.js";
-import { GOOGLE_OAUTH_CLIENT } from "../src/instance/google.js";
 import { reportHost } from "../src/host.js";
 import { signInAgent, updateAgents } from "../src/agents.js";
 import { serveWorkspaces } from "../src/workspace.js";
@@ -37,8 +37,8 @@ const AGENTS_USAGE =
   "usage: kit agents update [--config <path>] | kit agents sign-in <kind> [--config <path>]";
 const WORKSPACE_USAGE = "usage: kit workspace host [--config <path>] [--passes <count>]";
 const SOURCE_USAGE =
-  "usage: kit source <capture|reset|retry-files> <instance> [service] [--config <path>]\n" +
-  "       kit source status <instance> [service] [--config <path>]\n" +
+  "usage: kit source <capture|status> <account|instance> [service] [--config <path>]\n" +
+  "       kit source <reset|retry-files> <instance> [service] [--config <path>]\n" +
   "       kit source sign-in <instance> [--config <path>]";
 const SETUP_USAGE = "usage: kit setup [--config <path>]";
 const REPOSITORY_USAGE =
@@ -108,13 +108,6 @@ function write(line) {
   process.stdout.write(`${line}\n`);
 }
 
-function builtGoogleOAuthClient() {
-  if (GOOGLE_OAUTH_CLIENT === null) {
-    throw new KitRefusal("google_oauth_client_unconfigured", "run npm run build");
-  }
-  return GOOGLE_OAUTH_CLIENT;
-}
-
 async function runLogin(args, directory) {
   const { values } = declared(args, ["--house", "--manual", "--host-browser", "--environment"], 0, { flags: ["--host-browser"] });
   if (values["--manual"] && values["--host-browser"]) throw new Error(LOGIN_USAGE);
@@ -188,15 +181,12 @@ async function runSetup(args, directory) {
   process.once("SIGINT", cancel);
   let setup;
   try {
-    setup = await setupEnvironment(
-      {
-        house,
-        directory,
-        configuration: values["--config"] ?? join(directory, "kit.json"),
-        signal: cancellation.signal,
-      },
-      { googleOAuthClient: builtGoogleOAuthClient },
-    );
+    setup = await setupEnvironment({
+      house,
+      directory,
+      configuration: values["--config"] ?? join(directory, "kit.json"),
+      signal: cancellation.signal,
+    });
   } finally {
     process.removeListener("SIGINT", cancel);
   }
@@ -268,22 +258,27 @@ async function runSourceCapture(args, directory) {
   const house = await storedHouse(directory);
   const cancellation = new AbortController();
   process.once("SIGINT", () => cancellation.abort());
-  const result = await captureSource(
-    {
-      house,
-      directory,
-      configuration: values["--config"] ?? join(directory, "kit.json"),
-      instance: rest[0],
-      service: rest[1] ?? null,
-      signal: cancellation.signal,
-    },
-    { googleOAuthClient: builtGoogleOAuthClient },
-  );
+  const entries = await accountEntries(directory, rest[0]);
+  const options = {
+    house,
+    directory,
+    service: rest[1] ?? null,
+    signal: cancellation.signal,
+  };
+  const result =
+    entries.length > 0
+      ? await captureAccount({ ...options, account: rest[0], entries })
+      : await captureSource({
+          ...options,
+          configuration: values["--config"] ?? join(directory, "kit.json"),
+          instance: rest[0],
+        });
   for (const delivered of result.services) {
     const category =
       delivered.health.category === null ? "" : ` (${delivered.health.category})`;
+    const cause = delivered.health.cause === undefined ? "" : `, cause ${delivered.health.cause}`;
     write(
-      `Delivered ${delivered.delivered} items for ${result.instance}/${delivered.service} in ${delivered.batches} batches; health ${delivered.health.state}${category}.`,
+      `Delivered ${delivered.delivered} items for ${result.instance}/${delivered.service} in ${delivered.batches} batches; health ${delivered.health.state}${category}${cause}.`,
     );
     if (delivered.refused !== null) {
       write(
@@ -332,22 +327,32 @@ async function runSourceRetryFiles(args, directory) {
 async function runSourceStatus(args, directory) {
   const { values, rest } = declared(args, ["--config"], 2);
   if (rest.length === 0) throw new Error(SOURCE_USAGE);
-  const result = await sourceStatus({
-    directory,
-    configuration: values["--config"] ?? join(directory, "kit.json"),
-    instance: rest[0],
-    service: rest[1] ?? null,
-  });
+  const entries = await accountEntries(directory, rest[0]);
+  const result =
+    entries.length > 0
+      ? await accountStatus({ directory, account: rest[0], entries, service: rest[1] ?? null })
+      : await sourceStatus({
+          directory,
+          configuration: values["--config"] ?? join(directory, "kit.json"),
+          instance: rest[0],
+          service: rest[1] ?? null,
+        });
   for (const reported of result.services) {
     const category = reported.health.category === null ? "" : ` (${reported.health.category})`;
+    const cause = reported.health.cause === undefined ? "" : `, cause ${reported.health.cause}`;
     const observed =
       reported.health.observed_at === null ? "never observed" : reported.health.observed_at;
+    const signedIn =
+      reported.signed_in === undefined ? "" : reported.signed_in ? "signed in; " : "signed out; ";
     write(
       `${result.instance}/${reported.service} -> ${reported.source_ref} in ${reported.rooms.join(", ")}: ` +
-        `${reported.health.state}${category} at ${observed}; ` +
+        `${signedIn}${reported.health.state}${category}${cause} at ${observed}; ` +
         `cursor ${reported.checkpoint?.cursor ?? "none"}, ${reported.gaps.length} gaps, ` +
         `${reported.dropped.length} dropped items, ${reported.originals.length} unsaved originals.`,
     );
+    if (reported.enable !== null && reported.enable !== undefined) {
+      write(`Enable the ${reported.enable} in the Google Cloud project of ${result.instance}'s OAuth client.`);
+    }
   }
   if (result.services.some((reported) => reported.health.state !== "ok")) process.exitCode = 1;
 }
@@ -425,14 +430,11 @@ function writeConversationFiles(supervised) {
 async function runWorkerRun(args, directory) {
   const { values } = declared(args, ["--config"], 0);
   const house = await storedHouse(directory);
-  const supervised = await runWorkers(
-    {
-      house,
-      directory,
-      configuration: values["--config"] ?? join(directory, "kit.json"),
-    },
-    { googleOAuthClient: builtGoogleOAuthClient },
-  );
+  const supervised = await runWorkers({
+    house,
+    directory,
+    configuration: values["--config"] ?? join(directory, "kit.json"),
+  });
   write(`Worker runtime ready on tmux ${supervised.tmux} in namespace ${supervised.namespace}.`);
   if (supervised.epochReport !== null) {
     write(`House started a new log epoch; reported ${supervised.epochReport.length} held Runs and House settled each.`);

package/bootstrap.json

@@ -1,12 +1,12 @@
 {
   "schemaVersion": 1,
-  "version": "0.1.0-alpha.26",
+  "version": "0.1.0-alpha.27",
   "hosts": {
     "linux": {
-      "command": "bash <(curl -fsSL https://github.com/agentshouse/kit/releases/download/v0.1.0-alpha.26/connect-linux.sh)",
+      "command": "bash <(curl -fsSL https://github.com/agentshouse/kit/releases/download/v0.1.0-alpha.27/connect-linux.sh)",
       "asset": {
         "name": "connect-linux.sh",
-        "url": "https://github.com/agentshouse/kit/releases/download/v0.1.0-alpha.26/connect-linux.sh"
+        "url": "https://github.com/agentshouse/kit/releases/download/v0.1.0-alpha.27/connect-linux.sh"
       },
       "workspace": {
         "argument": "--workspace",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agentshouse/kit",
-  "version": "0.1.0-alpha.26",
+  "version": "0.1.0-alpha.27",
   "license": "MIT",
   "repository": {
     "type": "git",