@uniquedivine/tg-sai 0.0.5 → 0.1.0

package/README.md

@@ -29,6 +29,23 @@ target defaults to the Sai custody account,
 [@SaiTeam_AI](https://t.me/SaiTeam_AI) (user 8274304175), and the bot is
 [@SaiTeam_bot](https://t.me/SaiTeam_bot). `--target-user` overrides the owner.
 
+## Logs
+
+tg-sai writes no log files. Every run records Telegram calls and app log
+rows in the SQLite file `~/.tg-sai/telemetry.db` (`$TG_SAI_DIR` overrides
+`~/.tg-sai`). Read them with `tg-sai events`:
+
+```bash
+tg-sai events logs --since 24h --level warn --subsystem worker
+tg-sai events show <run_id|job_id|request_id>
+tg-sai events runs
+```
+
+Saved Telegram flood waits live in `~/.tg-sai/state.db`. While one is
+active, a CLI command that would send the held call exits with code 75 and
+prints when to retry. `TG_SAI_TELEMETRY=off` turns off `telemetry.db` only;
+`state.db` and its holds still apply.
+
 ## Development
 
 ```bash

package/RUNBOOK.md

@@ -232,10 +232,11 @@ Find Sai-related chats visible to the active Telegram session:
 tg-sai ops sai-find
 ```
 
-Command `tg-sai ops sai-find` always writes a JSONL file under directory
-`~/.tg-sai/logs/`, using the next available filename such as
-`sai-find-00.jsonl` or `sai-find-01.jsonl`. Use flag `--out <path>` to also
-write a copy somewhere else.
+Command `tg-sai ops sai-find` always writes its chat list as a JSONL file
+under directory `~/.tg-sai/logs/`, using the next available filename such as
+`sai-find-00.jsonl` or `sai-find-01.jsonl`. This file is command output for
+`ops sai-bootstrap --file`, not a log. Use flag `--out <path>` to also write a
+copy somewhere else.
 
 Dry-run Sai bot and owner-target updates from a reviewed chat file:
 
@@ -243,13 +244,19 @@ Dry-run Sai bot and owner-target updates from a reviewed chat file:
 tg-sai ops sai-bootstrap --file ~/.tg-sai/logs/sai-find-00.jsonl
 ```
 
-Command `tg-sai ops sai-bootstrap` always writes a JSONL run log under directory
-`~/.tg-sai/logs/`, using the next available filename such as
-`sai-bootstrap-00.jsonl` or `sai-bootstrap-01.jsonl`. The command waits one
-second between chats by default. Use flag `--delay-ms <ms>` to change that
+Command `tg-sai ops sai-bootstrap` prints one JSON event per line on stdout
+and its `telemetry_run_id`; redirect stdout if you want a copy. Its Telegram
+calls are in the telemetry database: `tg-sai events show <run_id>`. The
+command waits one second between chats by default. Use flag `--delay-ms <ms>` to change that
 delay. When environment variable `TELEGRAM_PASSWORD` is available, execution
 mode attempts ownership transfer by default for eligible chats.
 
+The command sleeps through flood waits up to five minutes (environment
+variable `TG_SAI_FLOOD_MAX_WAIT_MS` overrides). A longer `FLOOD_WAIT_X` or any
+`PEER_FLOOD` stops the remaining chats: stdout gets a `flood_stop` event with
+the chat, the remaining count, the code, and `resumeAt`, and the command exits
+with code 1. A `SLOWMODE_WAIT_X` fails only that chat.
+
 Run a live read-only audit:
 
 ```bash
@@ -300,6 +307,58 @@ TELEGRAM_PASSWORD="<2fa password>" tg-sai add-owner \
   --run
 ```
 
+## Logs, Telemetry, and Flood Holds
+
+tg-sai writes no JSONL run logs. Earlier versions wrote
+`~/.tg-sai/logs/<command>-NN.jsonl` and `sai-bootstrap-NN.jsonl`; those files
+stay readable but nothing new is added. Each process records into SQLite file
+`~/.tg-sai/telemetry.db` (`$TG_SAI_DIR/telemetry.db`):
+
+- `run.start` and `run.end` rows with the sanitized argv and exit code.
+- `tg.rpc.call`, `tg.rpc.attempt`, and `tg.flood.wait` rows for Telegram
+  calls, with summarized request and response shapes.
+- App log rows: a `level` (`debug`, `info`, `warn`, `error`), a short fixed
+  `msg`, and summarized `attrs`. Their `event` is `log.<subsystem>` or a
+  dotted name such as `worker.job.held`.
+
+Read it with `tg-sai events`. Output is TOON; add `--json` for JSON.
+
+```bash
+tg-sai events logs --since 24h --level warn --subsystem worker
+tg-sai events logs --run <run_id> --limit 200
+tg-sai events show <run_id|job_id|request_id>   # calls and log rows
+tg-sai events runs
+tg-sai events floods --since 7d
+tg-sai events stats                             # rows, log rows by level
+```
+
+Saved flood waits live in a separate file, `~/.tg-sai/state.db`. Every tg-sai
+process checks it before invite, create, resolve, and send calls. While a
+hold is active, a CLI command exits with code 75 and one stderr line naming
+the held call group and the UTC time to retry. If `state.db` cannot be read,
+invite and create calls are blocked, and the command also exits 75. The
+hold check keys by the account the session's own `getMe` reports, not by the
+profile's saved user ID; until that is known, invite and create calls are
+blocked the same way (`TG_SAI_HOLD_ACCOUNT_UNKNOWN`, exit 75).
+
+`ops sai-bootstrap` stops the batch with a `flood_stop` event on any of
+these refusals and exits 75. A `PEER_FLOOD` or a Telegram flood wait longer
+than the run's threshold also stops it, with exit 1.
+
+Inspect and override holds with `tg-sai holds`. `set` and `clear` are dry
+runs without `--run`; `clear --run` on an active hold also needs `--force`.
+Each write adds a `holds.set` or `holds.clear` row to `telemetry.db`. A
+`--run` refuses a `state.db` owned by another user, so on the custody VM
+run the worker's binary as the worker user:
+
+```bash
+sudo -u sai-custody-worker env TG_SAI_DIR=/var/lib/sai-custody-worker HOME=/var/lib/sai-custody-worker /opt/sai-custody-worker/current/tg-sai-worker holds list
+```
+
+`TG_SAI_TELEMETRY=off` disables `telemetry.db` writes only. Flood holds in
+`state.db` still apply, and `tg-sai events` reads whatever was recorded
+before.
+
 ## File roles
 
 | File | Role |

package/actionExecutor.ts

@@ -1,9 +1,12 @@
+import { HOLD_ACCOUNT_UNKNOWN, holdErrorSeconds } from "./state/guard"
+import { parseWaitSeconds, rpcErrorText } from "./telemetry/middleware"
+
 /**
- * Structured event emitted by the batch action executor.
+ * Structured progress event emitted by the batch action executor.
  *
- * These objects are the source of truth for both test assertions and production
- * JSONL logs. Keep event payloads stable enough that a failed or interrupted
- * operator run can be inspected after the fact.
+ * These events are on-screen CLI progress (and copied to the run log). They do
+ * not retry: mtcute's flood waiter is the only retry layer, and every Telegram
+ * request, attempt, and flood wait is recorded by telemetry instead.
  */
 export type BatchActionEvent =
   | {
@@ -13,22 +16,6 @@ export type BatchActionEvent =
       chatId: number | string
       /** Stable action name, for example `bot_add`. */
       action: string
-      /** One-based attempt number for this action. */
-      attempt: number
-    }
-  | {
-      /** Telegram returned `FLOOD_WAIT_X` and the executor will retry later. */
-      type: "flood_wait"
-      /** Telegram chat ID or peer for the rate-limited action. */
-      chatId: number | string
-      /** Stable action name, for example `bot_promote_admin`. */
-      action: string
-      /** One-based attempt number that received the flood-wait error. */
-      attempt: number
-      /** Seconds Telegram requested in `FLOOD_WAIT_X`. */
-      waitSeconds: number
-      /** Milliseconds the executor sleeps, including configured buffer. */
-      sleepMs: number
     }
   | {
       /** Operation completed successfully. */
@@ -37,20 +24,16 @@ export type BatchActionEvent =
       chatId: number | string
       /** Stable action name, for example `owner_target_add`. */
       action: string
-      /** Number of attempts required before success. */
-      attempts: number
-      /** Elapsed milliseconds from first attempt to success. */
+      /** Elapsed milliseconds from start to success, flood sleeps included. */
       elapsedMs: number
     }
   | {
-      /** Operation failed with a non-retryable error or exhausted retries. */
+      /** Operation failed; mtcute already retried any short flood wait. */
       type: "action_error"
       /** Telegram chat ID or peer for the failed action. */
       chatId: number | string
       /** Stable action name, for example `transfer_owner`. */
       action: string
-      /** One-based attempt number that produced the final error. */
-      attempt: number
       /** Error name normalized for logs. */
       errorName: string
       /** Error message normalized for logs. */
@@ -78,22 +61,14 @@ export interface TelegramActionContext {
 }
 
 /**
- * Runtime dependencies for `withFloodWaitRetry`.
- *
- * Passing these in keeps retry behavior deterministic in tests: sleep, time, and
- * logging can all be faked without importing mtcute or touching Telegram.
+ * Runtime dependencies for `runTelegramAction`. Passing these in keeps event
+ * output deterministic in tests.
  */
 export interface TelegramActionExecutorDeps {
-  /** Event sink for action start, success, error, and flood-wait events. */
+  /** Event sink for action start, success, and error events. */
   logger: BatchLogger
-  /** Sleep function used after Telegram returns `FLOOD_WAIT_X`. */
-  sleep: (ms: number) => Promise<void>
   /** Clock function used to compute elapsed action time. */
   now: () => number
-  /** Number of retries after the first failed attempt. */
-  maxRetries: number
-  /** Extra milliseconds added to Telegram's required flood-wait duration. */
-  bufferMs: number
 }
 
 /**
@@ -110,28 +85,6 @@ export class MemoryBatchLogger implements BatchLogger {
   }
 }
 
-/**
- * Extracts the wait duration from Telegram-style `FLOOD_WAIT_X` errors.
- *
- * mtcute and related libraries may expose the Telegram error string in the error
- * name, message, or both. This helper checks both fields and returns `null` for
- * non-flood errors so callers can distinguish retryable rate limits from normal
- * failures.
- */
-export function parseFloodWaitSeconds(error: unknown): number | null {
-  if (!(error instanceof Error)) {
-    return null
-  }
-
-  const haystack = `${error.name} ${error.message}`
-  const match = haystack.match(/FLOOD_WAIT_(\d+)/)
-  if (!match) {
-    return null
-  }
-
-  return Number(match[1])
-}
-
 /**
  * Returns a stable display name for unknown thrown values.
  */
@@ -153,67 +106,166 @@ export function errorMessage(error: unknown): string {
 }
 
 /**
- * Runs one Telegram operation with `FLOOD_WAIT_X` retry handling.
+ * Runs one Telegram operation once and reports start, success, or error.
  *
- * The operation is passed as a thunk so the executor can call the exact same
- * Telegram action after waiting. Non-flood errors and exhausted retry attempts
- * are logged as `action_error` and then rethrown for the batch runner to record
- * at the chat level.
+ * There is no retry here: short `FLOOD_WAIT_X` waits are slept through by
+ * mtcute's flood waiter, and longer ones surface as errors. Errors are logged
+ * as `action_error` and rethrown for the batch runner to record per chat.
  */
-export async function withFloodWaitRetry<T>(
+export async function runTelegramAction<T>(
   context: TelegramActionContext,
   operation: () => Promise<T>,
   deps: TelegramActionExecutorDeps,
 ): Promise<T> {
   const startedAt = deps.now()
-  const maxAttempts = deps.maxRetries + 1
-
-  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
+  await deps.logger.event({
+    type: "action_start",
+    chatId: context.chatId,
+    action: context.action,
+  })
+  try {
+    const result = await operation()
+    await deps.logger.event({
+      type: "action_success",
+      chatId: context.chatId,
+      action: context.action,
+      elapsedMs: deps.now() - startedAt,
+    })
+    return result
+  } catch (error) {
     await deps.logger.event({
-      type: "action_start",
+      type: "action_error",
       chatId: context.chatId,
       action: context.action,
-      attempt,
+      errorName: errorName(error),
+      errorMessage: errorMessage(error),
     })
+    throw error
+  }
+}
+
+/** Text of `HoldStoreUnavailableError` (state/holds.ts). */
+const HOLD_STATE_UNAVAILABLE = "TG_SAI_HOLD_STATE_UNAVAILABLE"
+
+/** Why a batch stopped early on a Telegram rate limit. */
+export interface FloodStop {
+  /**
+   * Telegram error code, e.g. `FLOOD_WAIT_15382` or `PEER_FLOOD`, or the
+   * local guard's `TG_SAI_HOLD_<s>`, `TG_SAI_HOLD_STATE_UNAVAILABLE`, or
+   * `TG_SAI_HOLD_ACCOUNT_UNKNOWN`.
+   */
+  code: string
+  /** Wait Telegram asked for; `null` for `PEER_FLOOD` and unreadable holds. */
+  waitS: number | null
+  /** When the wait ends (ISO); `null` when no time is known. */
+  resumeAt: string | null
+  /**
+   * The local guard refused before sending (a saved hold, or holds that
+   * cannot be checked). Such stops exit 75; Telegram's own errors exit 1.
+   */
+  hold: boolean
+}
+
+function holdUntilMs(error: unknown): number | null {
+  const untilMs = (error as { untilMs?: unknown } | null)?.untilMs
+  return typeof untilMs === "number" && Number.isFinite(untilMs)
+    ? untilMs
+    : null
+}
+
+/**
+ * Returns a stop when `error` means the rest of a batch would only hit the
+ * same limit: any `PEER_FLOOD`, a `FLOOD_WAIT_X` (and premium/test-phone
+ * variants) longer than `maxWaitMs`, which mtcute did not sleep through, or
+ * a refusal by the local flood guard (`TG_SAI_HOLD_<s>`, unreadable hold
+ * state, unknown session account). `SLOWMODE_WAIT_X` is per chat, so it
+ * returns `null` and the batch goes on.
+ */
+export function floodStopFor(
+  error: unknown,
+  maxWaitMs: number,
+  nowMs: number,
+): FloodStop | null {
+  const code = rpcErrorText(error)
+  if (code === null) return null
+  if (code === HOLD_STATE_UNAVAILABLE || code === HOLD_ACCOUNT_UNKNOWN) {
+    return { code, waitS: null, resumeAt: null, hold: true }
+  }
+  const holdS = holdErrorSeconds(code)
+  if (holdS !== null) {
+    const untilMs = holdUntilMs(error) ?? nowMs + holdS * 1000
+    return {
+      code,
+      waitS: holdS,
+      resumeAt: new Date(untilMs).toISOString(),
+      hold: true,
+    }
+  }
+  if (code === "PEER_FLOOD") {
+    return { code, waitS: null, resumeAt: null, hold: false }
+  }
+  if (code.startsWith("SLOWMODE_WAIT_")) return null
+  const waitS = parseWaitSeconds(code)
+  if (waitS === null || waitS * 1000 <= maxWaitMs) return null
+  return {
+    code,
+    waitS,
+    resumeAt: new Date(nowMs + waitS * 1000).toISOString(),
+    hold: false,
+  }
+}
+
+/** A flood stop plus where in the batch it happened. */
+export interface BatchStop<T> extends FloodStop {
+  item: T
+  /** Zero-based index of the item that hit the limit. */
+  index: number
+  /** Items after it that were not started. */
+  remaining: number
+}
 
+export interface ChatBatchOptions<T> {
+  /** Runs one item; a throw is reported to `onItemError`. */
+  processItem: (item: T, index: number) => Promise<void>
+  /** Records a per-item failure; the batch continues unless it stops. */
+  onItemError: (item: T, error: unknown, index: number) => void | Promise<void>
+  /** Runs between items (e.g. the inter-chat delay), not after the last. */
+  betweenItems?: (item: T, index: number) => void | Promise<void>
+  /** Flood waits longer than this stop the batch. */
+  maxWaitMs: number
+  now?: () => number
+}
+
+/**
+ * Processes `items` in order, isolating per-item errors, and stops at the
+ * first error `floodStopFor` treats as batch-wide.
+ */
+export async function runChatBatch<T>(
+  items: readonly T[],
+  options: ChatBatchOptions<T>,
+): Promise<{ processed: number; floodStop: BatchStop<T> | null }> {
+  const now = options.now ?? Date.now
+  let processed = 0
+  for (const [index, item] of items.entries()) {
+    processed += 1
     try {
-      const result = await operation()
-      await deps.logger.event({
-        type: "action_success",
-        chatId: context.chatId,
-        action: context.action,
-        attempts: attempt,
-        elapsedMs: deps.now() - startedAt,
-      })
-      return result
+      await options.processItem(item, index)
     } catch (error) {
-      const waitSeconds = parseFloodWaitSeconds(error)
-      const canRetry = waitSeconds !== null && attempt < maxAttempts
-
-      if (!canRetry) {
-        await deps.logger.event({
-          type: "action_error",
-          chatId: context.chatId,
-          action: context.action,
-          attempt,
-          errorName: errorName(error),
-          errorMessage: errorMessage(error),
-        })
-        throw error
+      await options.onItemError(item, error, index)
+      const stop = floodStopFor(error, options.maxWaitMs, now())
+      if (stop) {
+        return {
+          processed,
+          floodStop: {
+            ...stop,
+            item,
+            index,
+            remaining: items.length - index - 1,
+          },
+        }
       }
-
-      const sleepMs = waitSeconds * 1000 + deps.bufferMs
-      await deps.logger.event({
-        type: "flood_wait",
-        chatId: context.chatId,
-        action: context.action,
-        attempt,
-        waitSeconds,
-        sleepMs,
-      })
-      await deps.sleep(sleepMs)
     }
+    if (index < items.length - 1) await options.betweenItems?.(item, index)
   }
-
-  throw new Error(`unreachable action executor state: ${context.action}`)
+  return { processed, floodStop: null }
 }