@llblab/pi-telegram 0.24.4 → 0.24.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +2 -1
- package/CHANGELOG.md +9 -0
- package/README.md +3 -3
- package/docs/architecture.md +3 -2
- package/docs/multi-instance-bus.md +2 -0
- package/index.ts +26 -18
- package/lib/bindings.ts +11 -1
- package/lib/bus-follower.ts +26 -0
- package/lib/commands.ts +44 -2
- package/lib/locks.ts +43 -45
- package/lib/recovery.ts +464 -0
- package/package.json +1 -1
package/AGENTS.md
CHANGED
|
@@ -78,7 +78,8 @@ The core product loop is mobile continuation: start or supervise work in the ter
|
|
|
78
78
|
- The bridge is session-local, paired to one allowed Telegram user, and owns a local queue aligned with Pi lifecycle hooks
|
|
79
79
|
- The Telegram lock owns external Telegram transport, not local Pi execution. Local extension state and queue runtime are per Pi instance: losing exact `/telegram-connect` ownership must not clear accepted queue state or stop local Pi dispatch, but preview, final delivery, attachments, menus, and other Bot API mutations fail closed once another runtime owns transport. A replacement owner must never receive stale direct mutations merely to preserve old-runtime Telegram output. Bus leader lock entries carry a stable `leaderEpoch` minted on acquisition and preserved across heartbeat refreshes; leader-owned Thread Reconciler cleanup/provisioning actions stamp that epoch and skip destructive side effects if current leadership differs.
|
|
80
80
|
- Telegram transport ownership lives only in the extension-local `tmp/telegram/owners.json` store under `default` or named profile slots. Every mutation must serialize the complete cross-process read/check/write transaction through `owners.json.transaction` and fail closed on contention; atomic payload replacement alone does not make ownership linearizable. Refresh, release, takeover, and irreversible leader work additionally fence against the exact acquired owner and leader epoch. In classic/private-chat mode, transport ownership is the whole bridge ownership and `/telegram-connect` from another live Pi offers classic takeover confirmation. In Threaded Mode, transport ownership is the bus leader role and another live Pi registers as follower instead of offering takeover while the leader is live. If follower registration is unavailable, fall back to classic takeover.
|
|
81
|
-
- Polling checks exact `owners.json` ownership every second but refreshes the durable owner heartbeat only every two seconds; an eight-second stale threshold preserves bounded recovery headroom without turning every ownership check into an atomic disk rewrite. Ownership transactions publish a fully initialized private guard directory containing one exact generation-specific `owner.<generation>.json` through same-parent atomic rename. The owner filename and payload generation must match so stale observation cannot rename replacement metadata. Release verifies the owner and atomically renames the stable guard away before cleanup. Directory recovery first claims owner metadata inside the still-occupied guard so a crashed recoverer can itself be replaced without exposing the stable path; transient guard moves and rollback both retry, while inactive process-global claim generations remain recoverable after failed restoration. If secondary recovery cleanup fails after publishing a replacement main guard, release that exact replacement before propagating the failure. Guards must not depend on hard links, direct file rename-overwrite, incomplete create-then-write state, or age-only recovery; malformed or unsupported guards fail closed.
|
|
81
|
+
- Polling checks exact `owners.json` ownership every second but refreshes the durable owner heartbeat only every two seconds; an eight-second stale threshold preserves bounded recovery headroom without turning every ownership check into an atomic disk rewrite. Ownership transactions publish a fully initialized private guard directory containing one exact generation-specific `owner.<generation>.json` through same-parent atomic rename. The owner filename and payload generation must match so stale observation cannot rename replacement metadata. Release verifies the owner and atomically renames the stable guard away before cleanup. Directory recovery first claims owner metadata inside the still-occupied guard so a crashed recoverer can itself be replaced without exposing the stable path; transient guard moves and rollback both retry, while inactive process-global claim generations remain recoverable after failed restoration. If secondary recovery cleanup fails after publishing a replacement main guard, release that exact replacement before propagating the failure. Guards must not depend on hard links, direct file rename-overwrite, incomplete create-then-write state, or age-only recovery; malformed or unsupported guards fail closed during ordinary mutation.
|
|
82
|
+
- `/telegram-connect` is the bounded liveness-recovery boundary for unclean-shutdown corruption in disposable `tmp/telegram` state. It may quarantine only classifier-approved malformed `owners.json`, active-profile `state*.json`, and `owners.json.transaction` after serializing recoverers and revalidating under ownership fencing; it must preserve `telegram.json`, diagnostics, unrelated artifacts, and every verifiable live owner/transaction holder. Stale heartbeats older than eight seconds do not gain protection through PID reuse. Require confirmed local polling suspension before quarantine mutation; a suspension failure blocks recovery, while an ownership-release failure may continue only into final guarded live-owner revalidation. Use the established bounded Windows sharing-violation rename retries. Retry polling exactly once after recovery, preserve unrelated startup errors, and replace blocked/repeated recovery failure with one explicit restart instruction rather than broad `tmp/` deletion or an automatic loop.
|
|
82
83
|
- Telegram profiles are parallel bot/runtime realities, not merely alternate credentials. Persist every bot/session field (`botToken`, `botUsername`, `botId`, `allowedUserId`, `lastUpdateId`) under `telegram.json` `profiles.<name>`, including `profiles.default`; keep shared handlers and assistant/voice/time/thread-lifecycle settings top-level. Bare and explicit `default` setup/connect commands address the same profile. Legacy root identity may normalize atomically only when unambiguous; conflicts fail closed without mutation. Any owner slot, IPC endpoint, state snapshot, diagnostics log, thread/bus binding, runtime cache, ownership map, or operator-visible status surface that could mix bot identities must be deliberately classified as global, session-local, or profile-scoped. Use the `default` owner slot for the default profile and validated profile names for named slots; give named profiles isolated observable surfaces unless there is an explicit top-level shared-config reason not to.
|
|
83
84
|
- Telegram private-chat Threaded Mode is the public switch for the local multi-instance bus. There is no separate public `telegram.json` bus switch: classic/private-chat singleton DM behavior is the base product mode, and when Telegram reports Threaded Mode available for the bot the bridge enables leader/follower bus behavior. If Telegram reports Threaded Mode unavailable or thread operations prove it unavailable, the bridge returns to classic singleton behavior as a first-class mode. During a live Threaded Mode downgrade, the current transport leader keeps the singleton classic polling role and followers disconnect instead of taking over or offering takeover while active thread bindings prove this is a downgrade from a live bus organism. Do not inject slot/thread rename guidance in classic mode. Blocked instances should not silently register as followers unless the effective runtime mode is Threaded Mode.
|
|
84
85
|
- In Threaded Mode, exactly one live leader owns `getUpdates` and direct Telegram Bot API calls for one bot token. `/telegram-connect` means join the local Telegram organism: if no live leader exists the instance becomes leader; if a live leader exists the terminal-visible instance registers as follower; ordinary connect must not offer manual takeover while the leader heartbeat is live. Leadership is an ephemeral transport role, not permanent ownership by the first terminal-visible Pi instance. Heartbeat acknowledgements carry the authenticated live follower-slot roster: on leader loss, the lowest observed live slot attempts the atomic lock first, while higher slots defer for one bounded grace and then re-check so a missing lower follower cannot block recovery indefinitely. Followers must be started by the operator as real Pi processes, register over the local bus, heartbeat, receive only their routed inbound updates, and send outbound replies/previews/files/chat-actions through the leader transport; Telegram must not spawn hidden Pi processes, launch follower subprocesses, or expose a slash command that creates a new local Pi instance. If a leader reloads while followers stay alive, followers must treat rejected/unknown/unauthorized heartbeats as a registration-loss signal and optimistically re-register with the currently active leader before considering promotion; after a short grace window without successful re-registration, a live follower may promote only through an atomic expected-owner transition after the exact observed leader lease becomes stale or inactive. An unreachable IPC endpoint does not authorize replacing a still-live lock owner; remain disconnected/retrying rather than creating split-brain polling. Terminal status must not remain `follower` while the leader registry can no longer route that follower's thread. Heartbeat pruning is liveness bookkeeping, not permission to delete, mark offline, or send a Telegram-visible disconnected notice for a follower's thread; preserve active follower tabs/bindings across transient reload/reconnect incidents and reserve destructive teardown/offline transitions for confirmed manual disconnect, graceful Pi quit, or confirmed reconciliation cases. If the leader dies, a live follower should eventually be able to take over through stale-leader election/recovery so the local multi-instance organism keeps running. Explicit `/telegram-disconnect` retains destructive confirmation; graceful Pi `quit` invokes the same teardown without prompting when top-level `threads.automaticCleanup` is enabled (default), while disabling it and `reload`, `new`, `resume`, or `fork` preserve the binding through safe handoff/suspension. A leader deletes its own bound thread before releasing transport ownership, while a follower sends an authenticated exact-registration-generation disconnect envelope and waits for the leader to delete its assigned thread, mark the durable binding offline, and remove live routing authority before unregistering. Cleanup must be confirmed by successful deletion or explicit already-gone evidence; incomplete cleanup preserves binding/routing state, keeps the session open, and returns actionable retry guidance. A promoted leader authorizes inherited follower-thread cleanup by its current leader epoch rather than the historical owner label. Status is reactive to the effective runtime role and bot capability: classic/non-thread mode shows ordinary `telegram connected`, active bus leadership shows `telegram leader`, and registered followers show `telegram follower`; stable multi-instance truth belongs in the bus role (`leader`/`follower`), while lifecycle is reserved for transient recovery state such as `electing` and must not duplicate stable roles with labels like `leader-active`. When leadership or Threaded Mode availability changes at runtime, the status line and `/telegram-status` must converge promptly instead of preserving a stale role label.
|
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,15 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.24.6: README Positioning Hotfix
|
|
6
|
+
|
|
7
|
+
- `README Positioning`: Removed duplicated automatic thread-cleanup detail from the opening and defaults summary while retaining the canonical behavior in the lifecycle controls and configuration documentation. Impact: the package entrypoint stays focused on the companion runtime model without changing graceful-quit behavior or its configurable cleanup policy.
|
|
8
|
+
|
|
9
|
+
## 0.24.5: Unclean-Shutdown Recovery Hotfix
|
|
10
|
+
|
|
11
|
+
- `Unclean-Shutdown Recovery`: `/telegram-connect` now classifies truncated temporary ownership/routing snapshots and unverifiable ownership transaction debris, quarantines only damaged disposable artifacts when no verifiable live owner protects them, and retries startup exactly once. Recovery serializes cross-process contenders, revalidates under ownership fencing, treats eight-second stale heartbeats as replaceable despite PID reuse, preserves `telegram.json` and diagnostics, and converts blocked or repeated failure into one explicit Pi-restart instruction. Impact: abrupt host shutdown no longer leaves the bridge trapped behind raw JSON/transaction errors or requires deleting the whole agent `tmp/` directory.
|
|
12
|
+
- `Composition Root Cooling`: Moved follower active-auth and transient-election mutable state from `index.ts` into a cohesive `bus-follower` control runtime while leaving direct one-to-one runtime wiring in the composition root. Impact: follower control state now has one testable owner without introducing a wrapper whose only purpose is line-count reduction.
|
|
13
|
+
|
|
5
14
|
## 0.24.4: Context And Thread Lifecycle Hotfix
|
|
6
15
|
|
|
7
16
|
- `Disconnect Context Availability`: Reconciled `telegram_attach`, `telegram_message`, `telegram_help`, their active-tool prompt metadata, and the compact bridge suffix against effective direct-owner or registered-follower authority. Successful disconnect or authority loss removes only pi-telegram surfaces for subsequent requests; reconnect/recovery restores only the operator's previously active pi-telegram subset across same-process reload, while execute-time transport guards remain. Impact: disconnected sessions stop spending context on unavailable Telegram capabilities or inviting calls that must fail.
|
package/README.md
CHANGED
|
@@ -10,8 +10,6 @@ It is a **runtime adapter**, not a remote terminal. Start or supervise work in t
|
|
|
10
10
|
|
|
11
11
|
Proactive push is enabled by default. `assistant.proactivePush` projects every completed public assistant text block from local or autonomous work—including visible checkpoints and the final answer—to the authorized Telegram target once and in order; set it explicitly to `false` to disable projection. It never mirrors local prompts, hidden reasoning, tool traffic, token deltas, Telegram-owned turns, or stale-generation work. See [Outbound](docs/outbound.md#proactive-public-output) and the [configuration reference](docs/public-api.md#configuration-api).
|
|
12
12
|
|
|
13
|
-
Automatic thread cleanup is enabled by default. Graceful Pi quit deletes that instance's Threaded Mode tab; use `🧹 Auto thread cleanup` in Telegram Settings or set `threads.automaticCleanup` to `false` to preserve the tab as a restart hint. This option never changes manual `/telegram-disconnect`, which still asks for confirmation and deletes the tab.
|
|
14
|
-
|
|
15
13
|
This repository is an actively maintained fork of [`badlogic/pi-telegram`](https://github.com/badlogic/pi-telegram). It started from upstream commit [`cb34008`](https://github.com/badlogic/pi-telegram/commit/cb34008460b6c1ca036d92322f69d87f626be0fc) and has since diverged substantially.
|
|
16
14
|
|
|
17
15
|
## Install
|
|
@@ -57,6 +55,8 @@ Paste the bot token. If `~/.pi/agent/telegram.json` already contains a saved tok
|
|
|
57
55
|
|
|
58
56
|
The connected Pi instance owns Telegram polling. Use `/telegram-connect <name>` to activate a named profile. Each profile is a parallel bot runtime with isolated polling, diagnostics, Threaded Mode state, and local bus transport; the `default` profile keeps unsuffixed runtime paths. In classic mode each profile uses a singleton lock. When Telegram private-chat Threaded Mode is available, one live instance becomes the profile's leader and later visible Pi instances register as followers.
|
|
59
57
|
|
|
58
|
+
After an unclean computer shutdown, `/telegram-connect` detects truncated or structurally invalid temporary ownership/routing files, quarantines only the damaged files under `tmp/telegram/recovery/`, and retries once. Saved `telegram.json` configuration and runtime diagnostics remain intact. Recovery never replaces a verifiable live owner; if safe automatic recovery cannot complete, the command gives one explicit Pi-restart instruction instead of requiring deletion of the whole `tmp/` directory.
|
|
59
|
+
|
|
60
60
|
### 4. Pair your Telegram account
|
|
61
61
|
|
|
62
62
|
Open the bot DM and send:
|
|
@@ -214,7 +214,7 @@ Most controls live in Pi commands or the Telegram menu. Environment variables re
|
|
|
214
214
|
| Inbound file limit | `PI_TELEGRAM_INBOUND_FILE_MAX_BYTES`, `TELEGRAM_MAX_FILE_SIZE_BYTES` |
|
|
215
215
|
| Outbound attachment limit | `PI_TELEGRAM_OUTBOUND_ATTACHMENT_MAX_BYTES`, `TELEGRAM_MAX_ATTACHMENT_SIZE_BYTES` |
|
|
216
216
|
|
|
217
|
-
Defaults are chosen for ordinary private-bot use: saved config in `~/.pi/agent`, inbound temp files in `~/.pi/agent/tmp/telegram`, `assistant: { rendering: "rich", draftPreviews: false }` for assistant answer output,
|
|
217
|
+
Defaults are chosen for ordinary private-bot use: saved config in `~/.pi/agent`, inbound temp files in `~/.pi/agent/tmp/telegram`, `assistant: { rendering: "rich", draftPreviews: false }` for assistant answer output, and native Telegram active status for long-running turns.
|
|
218
218
|
|
|
219
219
|
## Extension Platform
|
|
220
220
|
|
package/docs/architecture.md
CHANGED
|
@@ -64,7 +64,7 @@ The repository uses a **Flat Domain DAG**:
|
|
|
64
64
|
- `api`: Bot API helpers, retries, uploads/downloads, temp cleanup, byte limits, chat actions, lazy token clients, and API error recording.
|
|
65
65
|
- `config` / `setup`: `telegram.json`, bot token setup, named bot/session profiles, first-user pairing, authorization, env fallback, atomic persistence, effective config views, and live config accessors.
|
|
66
66
|
- `locks` / `polling`: extension-local transport owner storage, exact-owner epoch exposure, process-global reload generations, owner-aware polling lifecycle/takeover/follower registration, and the cohesive classic-vs-Threaded capability state/monitor/observation/polling orchestration. Polling also owns long-poll controller state, offset admission/persistence, and poll-loop wiring.
|
|
67
|
-
- `bus` / `bus-api` / `bus-leader` / `bus-follower` / `ownership` / `target`: Threaded Mode multi-instance bus contracts, profile-scoped process/endpoint identity, local leader/follower IPC, leader-only orchestration, follower-side manual registration/session runtime, follower-routed Bot API calls, live message ownership, and `{ chatId, threadId? }` target identity. `bus` owns shared protocol, process identity, profile-aware local endpoints, and IPC primitives; `bus-leader` owns leader runtime, leader envelope handling, activation scheduling, and leader polling/server/prune orchestration; `bus-follower` owns process-stable manual-follower keys plus this Pi instance's follower-side registration, heartbeat, one-sequence authenticated client assembly, forwarded-update adaptation/receiving, recovery retry defaults, and routed API caller without any process spawning.
|
|
67
|
+
- `bus` / `bus-api` / `bus-leader` / `bus-follower` / `ownership` / `target`: Threaded Mode multi-instance bus contracts, profile-scoped process/endpoint identity, local leader/follower IPC, leader-only orchestration, follower-side manual registration/session runtime, follower-routed Bot API calls, live message ownership, and `{ chatId, threadId? }` target identity. `bus` owns shared protocol, process identity, profile-aware local endpoints, and IPC primitives; `bus-leader` owns leader runtime, leader envelope handling, activation scheduling, and leader polling/server/prune orchestration; `bus-follower` owns process-stable manual-follower keys plus this Pi instance's follower-side registration, active leader-auth/transient-election control state, heartbeat, one-sequence authenticated client assembly, forwarded-update adaptation/receiving, recovery retry defaults, and routed API caller without any process spawning.
|
|
68
68
|
- `sync`: demand-driven Telegram reconciliation, mutable sync-slice state, nested provisioning activity, and local assumption policy. It does not own a complete Telegram bot read-model; Bot API lacks a complete topic/thread listing surface. It owns sync slices, invalidation triggers, config-persist invalidation sequencing, stale-topic API recovery adaptation, observation intake, status/debug freshness, and reconciliation scheduling across bot identity, pairing assumptions, live target bindings, reservations, and transport health after meaningful observable signals. It should call narrower domain primitives rather than letting `index.ts`, `threads`, or `status` accumulate cross-cutting reconciliation policy.
|
|
69
69
|
- `thread-reconciler`: Threaded Mode control-plane planning for Telegram thread/tab lifecycle. It owns the reconciliation state machine (`stable`, `provisioning`, `sync-required`, `cleanup-required`), pure plans, proof-before-delete rules, pending-provision protection, fresh-creation grace windows, leader-epoch checks, and the single policy authority for destructive thread cleanup actions. It excludes live Telegram API calls, inbound routing, menu rendering, and direct persistence.
|
|
70
70
|
- `threads`: Telegram UI thread/tab binding state mapped to Bot API `message_thread_id` / `ForumTopic` transport. Owns leader and current-instance identity state, profile-bound same-process leader session handoff, status projections, slot allocation from the current extension state, baked compact thread-name selection, current binding persistence, and primitive thread provision helpers. It should not persist stale/offline/failed target history, own destructive cleanup policy, grow into the general Telegram synchronization domain, or expose a rename tool.
|
|
@@ -127,6 +127,7 @@ Telegram configuration lives in `~/.pi/agent/telegram.json`. Bot/session identit
|
|
|
127
127
|
- `owners.json` owns only Telegram transport control. Local extension and accepted queue state remain per Pi instance when ownership moves, but previews, final delivery, dispatch transport mutations, and delayed Bot API work fail closed until exact direct or follower authority becomes valid again.
|
|
128
128
|
- Exact ownership remains checked every second, while the durable owner heartbeat refresh runs every two seconds and becomes stale after eight seconds. This keeps replacement detection responsive while halving steady-state atomic `owners.json` rewrites without changing the cross-platform file-transaction authority. Every acquisition, refresh, release, takeover, and stale recovery serializes through the sibling `owners.json.transaction` guard. The guard publishes one private generation-named owner record atomically, validates filename/payload generation agreement, fences stale recovery and delayed release against replacement-owner ABA, and fails closed on malformed state, unverifiable ownership, contention timeout, or unsupported filesystem behavior. The JSON store publishes through a private same-directory temporary file and atomic rename; atomic payload replacement does not replace transaction serialization.
|
|
129
129
|
- `owners.json` is authoritative and private. `state.json` remains an observable snapshot, `logs.jsonl` remains diagnostics, and followers remain authenticated bus registrations rather than ownership-file writers.
|
|
130
|
+
- Ordinary ownership and state mutations fail closed on malformed files. When `/telegram-connect` itself fails and the recovery classifier finds truncated `owners.json`, truncated profile `state*.json`, or an unverifiable `owners.json.transaction`, it may prioritize runtime liveness: a dedicated recovery transaction serializes contenders, the ownership transaction fences a final reread, and only classifier-approved disposable artifacts move atomically into `tmp/telegram/recovery/<timestamp>-<pid>-<generation>/`. A verifiable live owner or transaction holder blocks mutation. Local polling suspension must complete before any quarantine mutation; failure blocks recovery, while a later ownership-release parse failure may proceed only because final guarded classification still protects any live owner. Quarantine renames use the same bounded `EPERM`/`EBUSY`/`EACCES` sharing retries as ownership publication for native Windows. Stale owner heartbeats older than eight seconds remain replaceable even if the operating system reused their PID. Configuration, logs, and unrelated temporary files never enter the recovery candidate set. The command retries startup once; a second failure becomes one explicit restart instruction rather than another recovery loop.
|
|
130
131
|
|
|
131
132
|
### Persistence I/O Baseline
|
|
132
133
|
|
|
@@ -138,7 +139,7 @@ The three runtime files have different authority and write pressure. Preserve th
|
|
|
138
139
|
|
|
139
140
|
This baseline counts write-producing code paths rather than filesystem implementation details that vary between ext4, APFS, NTFS, and network-backed home directories. Optimization evidence should compare these deterministic triggers first, then use platform smoke evidence for rename, named-pipe, crash, and cleanup behavior. Recovery-critical `state.json` fields are `bot`, `identities`, `reservations`, `pendingProvisions`, `syncObservations`, and `threads`; `runtime`, `liveRoster`, `diagnostics`, and `writtenAtMs` are observational and may use bounded coalescing when authority checks remain unchanged.
|
|
140
141
|
|
|
141
|
-
Version `0.24.0` intentionally does not read or migrate the former agent-level `locks.json`; upgrading resets Telegram ownership. Run `/telegram-connect` when a fresh owner is not elected automatically.
|
|
142
|
+
Version `0.24.0` intentionally does not read or migrate the former agent-level `locks.json`; upgrading resets Telegram ownership. Run `/telegram-connect` when a fresh owner is not elected automatically. Current builds automatically quarantine recognized unclean-shutdown corruption when no live owner protects it. Manual removal of `~/.pi/agent/tmp/telegram/owners.json` and its transaction guard is only a last resort after stopping every Pi instance that could own Telegram; never delete the whole agent `tmp/` directory to repair this extension.
|
|
142
143
|
|
|
143
144
|
### Threaded Mode Multi-Instance Bus
|
|
144
145
|
|
|
@@ -355,6 +355,8 @@ Current state under the agent dir:
|
|
|
355
355
|
- `tmp/telegram/state.json`: volatile extension+bot observable/debug snapshot, not routing authority. It writes `source: "snapshot"` and `writtenAtMs` so consumers do not confuse it with an authoritative database. Every process on one Telegram profile reads this shared path, but only the active transport owner may persist it; followers become writers only after promotion. Status-only persistence refreshes disk-backed bindings before serialization so an already-loaded stale view cannot erase newer leader records. It mirrors `/telegram-status`-style projections: top-level `bot` stores bot-wide capability state such as `threadMode: "unknown" | "enabled" | "disabled"`, `runtime` identifies leader/follower role and process status, `liveRoster` mirrors followers/current targets/reservations, `diagnostics` mirrors status/debug signals, `threads` stores current routeable bindings, `bot.lastSlot` stores the compact slot cursor used when all current threads are gone, and `reservations` records short-lived slot collision guards.
|
|
356
356
|
- Local bus endpoints: Unix-like platforms expose stable `tmp/telegram/bus.sock` and `tmp/telegram/followers/*` symlinks backed by private generation sockets; native Windows uses deterministic named pipes under `\\.\pipe\pi-telegram-...`. These are transient IPC endpoints, not durable routing state.
|
|
357
357
|
|
|
358
|
+
If an unclean host shutdown truncates `owners.json`, a profile `state*.json`, or the ownership transaction guard, `/telegram-connect` classifies the damage before recovery. With no verifiable live owner, one cross-process recovery winner quarantines only those damaged disposable artifacts and startup retries once; followers or leaders appearing during the final guarded reread stop the reset. `telegram.json`, `logs*.jsonl`, other profiles' valid state, and unrelated extension data remain untouched. A blocked or failed reset reports which Pi must restart instead of emitting repeated raw parse/transaction errors.
|
|
359
|
+
|
|
358
360
|
The bridge must not keep a separate durable `telegram-targets.json` history. `state.json` retains current stable manual-follower bindings as restart hints, but they never authorize routing without a matching authenticated live registration. Stale/offline/failed observations are not reusable delivery authority. `sync` remains event-driven assumption reconciliation rather than a full Telegram bot-state mirror because Bot API exposes no complete thread listing surface. Non-current routeable thread bindings are pruned during load/persist; old session records must not be retained just to compute the next slot because `bot.lastSlot` is the only durable cursor. Previous-process leader bindings are treated as occupied TTL-bounded reservations until Telegram confirms deletion: reload/startup may close/delete/probe the old thread, known reservations are retried proactively on leader startup, and if Telegram still accepts the old thread id, the new leader should provision the next free slot (`B`, `C`, …) rather than creating a duplicate same-letter tab or blocking startup on Telegram UI convergence. Routing must use live current threads/follower registry, never reservations. The bus leader provisions its own thread during bus startup/connect and provisions follower threads on `follower.register`; registered followers also live in the leader's in-memory registry and communicate over the local bus socket. The live follower registry can resolve a follower by exact `{ chatId, threadId? }`; the leader uses that target ownership to forward message and edited-message updates to followers, and the follower receiver accepts those updates in addition to callbacks and reactions. Terminal status and `[telegram|thread:name]` resolve the matching current-instance identity through the same target-aware path, preferring registered local metadata over stale shared bindings. Media album grouping and split-text coalescing keys include the thread target, queue reaction mutations can scope by chat/thread to avoid cross-target message-id collisions, active-turn target is exposed for lifecycle cleanup and local direct-tool defaults, transport reply dedup is chat/thread-scoped, stored menu state is keyed by chat/message so callback state lookup cannot collide across chats, and generated button turns plus section prompt/open actions preserve the callback thread target. `telegram_message` and immediate `telegram_attach` delivery can also carry an explicit `thread_id` with `chat_id`; when a follower is registered, their default direct-tool target is the assigned thread target and the bus-aware API runtime routes the send through the leader instead of calling Bot API transport locally.
|
|
359
361
|
|
|
360
362
|
All files containing routing, chat ids, thread ids, or process details use private permissions and represent current state rather than historical target caches.
|
package/index.ts
CHANGED
|
@@ -33,6 +33,7 @@ import * as Preview from "./lib/preview.ts";
|
|
|
33
33
|
import * as Prompts from "./lib/prompts.ts";
|
|
34
34
|
import * as PromptTemplates from "./lib/prompt-templates.ts";
|
|
35
35
|
import * as Queue from "./lib/queue.ts";
|
|
36
|
+
import * as Recovery from "./lib/recovery.ts";
|
|
36
37
|
import * as Replies from "./lib/replies.ts";
|
|
37
38
|
import * as Routing from "./lib/routing.ts";
|
|
38
39
|
import * as Runtime from "./lib/runtime.ts";
|
|
@@ -84,10 +85,11 @@ export default function (pi: Pi.ExtensionAPI) {
|
|
|
84
85
|
manualFollowerOwnerId: telegramManualFollowerOwnerId,
|
|
85
86
|
});
|
|
86
87
|
const telegramBusAuthSecret = Bus.createTelegramBusAuthSecret();
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
Status.TelegramBridgeBusLifecyclePhase | undefined;
|
|
88
|
+
const telegramBusFollowerControlState =
|
|
89
|
+
BusFollower.createTelegramBusFollowerControlState();
|
|
90
90
|
const telegramBusFollowerRegistry = Bus.createTelegramBusFollowerRegistry();
|
|
91
|
+
// Late-bound composition port breaks the follower-availability callback cycle;
|
|
92
|
+
// the mutable availability policy itself remains owned by prompts.
|
|
91
93
|
let modelContextAvailabilityRuntime:
|
|
92
94
|
| Prompts.TelegramModelContextAvailabilityRuntime
|
|
93
95
|
| undefined;
|
|
@@ -318,9 +320,7 @@ export default function (pi: Pi.ExtensionAPI) {
|
|
|
318
320
|
getRecentRuntimeEvents: runtimeEvents.getEvents,
|
|
319
321
|
getRuntimeLockState: lockRuntime.getStatusLabel,
|
|
320
322
|
...threadStatusProjectionRuntime,
|
|
321
|
-
getBusLifecyclePhase
|
|
322
|
-
return telegramBusLifecycleOverridePhase;
|
|
323
|
-
},
|
|
323
|
+
getBusLifecyclePhase: telegramBusFollowerControlState.getLifecyclePhase,
|
|
324
324
|
getBotThreadMode() {
|
|
325
325
|
return threadStore.getBotState();
|
|
326
326
|
},
|
|
@@ -363,9 +363,7 @@ export default function (pi: Pi.ExtensionAPI) {
|
|
|
363
363
|
>({
|
|
364
364
|
socketPath: getTelegramBusSocketPath,
|
|
365
365
|
instanceId: telegramInstanceId,
|
|
366
|
-
getApiAuthSecret
|
|
367
|
-
return telegramActiveBusAuthSecret;
|
|
368
|
-
},
|
|
366
|
+
getApiAuthSecret: telegramBusFollowerControlState.getActiveAuthSecret,
|
|
369
367
|
getForwardingAuthSecret() {
|
|
370
368
|
return telegramBusAuthSecret;
|
|
371
369
|
},
|
|
@@ -800,9 +798,7 @@ export default function (pi: Pi.ExtensionAPI) {
|
|
|
800
798
|
socketPath: getTelegramBusFollowerSocketPath,
|
|
801
799
|
instanceId: telegramInstanceId,
|
|
802
800
|
getContext: telegramSessionContextStore.get,
|
|
803
|
-
getAuthSecret
|
|
804
|
-
return telegramActiveBusAuthSecret;
|
|
805
|
-
},
|
|
801
|
+
getAuthSecret: telegramBusFollowerControlState.getActiveAuthSecret,
|
|
806
802
|
...forwardedRouteHandlers,
|
|
807
803
|
prepareForwardedMessage: textGroupRuntime.prepareForwardedMessage,
|
|
808
804
|
recordRuntimeEvent,
|
|
@@ -821,9 +817,7 @@ export default function (pi: Pi.ExtensionAPI) {
|
|
|
821
817
|
recovery: {
|
|
822
818
|
registrationState: telegramBusFollowerRegistrationState,
|
|
823
819
|
getLeaderState: lockRuntime.getState,
|
|
824
|
-
setLifecyclePhase
|
|
825
|
-
telegramBusLifecycleOverridePhase = phase;
|
|
826
|
-
},
|
|
820
|
+
setLifecyclePhase: telegramBusFollowerControlState.setLifecyclePhase,
|
|
827
821
|
updateStatus,
|
|
828
822
|
promoteToLeader: promoteTelegramBusFollowerToLeader,
|
|
829
823
|
getActiveContext: telegramSessionContextStore.get,
|
|
@@ -839,9 +833,8 @@ export default function (pi: Pi.ExtensionAPI) {
|
|
|
839
833
|
getLeaderAuthSecret(owner) {
|
|
840
834
|
return owner.busSecret;
|
|
841
835
|
},
|
|
842
|
-
setActiveAuthSecret
|
|
843
|
-
|
|
844
|
-
},
|
|
836
|
+
setActiveAuthSecret:
|
|
837
|
+
telegramBusFollowerControlState.setActiveAuthSecret,
|
|
845
838
|
getProfileKey: getTelegramManualFollowerProfileKey,
|
|
846
839
|
recordRuntimeEvent,
|
|
847
840
|
},
|
|
@@ -1044,6 +1037,21 @@ export default function (pi: Pi.ExtensionAPI) {
|
|
|
1044
1037
|
activeTurnRuntime,
|
|
1045
1038
|
lockedPollingRuntime,
|
|
1046
1039
|
stopPolling: disconnectTelegramAndDeleteCurrentThread,
|
|
1040
|
+
recoverPollingStart:
|
|
1041
|
+
Recovery.createTelegramPollingStartRecoveryHandler({
|
|
1042
|
+
getOwnersPath: Paths.resolveTelegramOwnersPath,
|
|
1043
|
+
getStatePaths() {
|
|
1044
|
+
return [
|
|
1045
|
+
Threads.getTelegramTopicTargetsPath(
|
|
1046
|
+
undefined,
|
|
1047
|
+
configStore.getActiveProfileName(),
|
|
1048
|
+
),
|
|
1049
|
+
];
|
|
1050
|
+
},
|
|
1051
|
+
suspendPolling: lockedPollingRuntime.suspend,
|
|
1052
|
+
releaseOwnership: lockRuntime.release,
|
|
1053
|
+
recordRuntimeEvent,
|
|
1054
|
+
}),
|
|
1047
1055
|
getDisconnectThreadName() {
|
|
1048
1056
|
const record = findCurrentThreadRecord();
|
|
1049
1057
|
if (!record?.target.threadId) return undefined;
|
package/lib/bindings.ts
CHANGED
|
@@ -99,6 +99,7 @@ interface TelegramCommandsAndToolsBindingDeps {
|
|
|
99
99
|
activeTurnRuntime: Queue.TelegramActiveTurnStore<Queue.PendingTelegramTurn>;
|
|
100
100
|
lockedPollingRuntime: Locks.TelegramLockedPollingRuntime<Pi.ExtensionContext>;
|
|
101
101
|
stopPolling?: () => Promise<void | string>;
|
|
102
|
+
recoverPollingStart?: Commands.TelegramBridgeCommandRegistrationDeps["recoverPollingStart"];
|
|
102
103
|
getDisconnectThreadName?: () => string | undefined;
|
|
103
104
|
onTransportChanged?: () => Promise<void> | void;
|
|
104
105
|
getStatusLines: (
|
|
@@ -127,6 +128,7 @@ export function registerTelegramCommandsAndTools({
|
|
|
127
128
|
activeTurnRuntime,
|
|
128
129
|
lockedPollingRuntime,
|
|
129
130
|
stopPolling,
|
|
131
|
+
recoverPollingStart,
|
|
130
132
|
getDisconnectThreadName,
|
|
131
133
|
onTransportChanged,
|
|
132
134
|
getStatusLines,
|
|
@@ -234,8 +236,16 @@ export function registerTelegramCommandsAndTools({
|
|
|
234
236
|
getStatusLines,
|
|
235
237
|
reloadConfig: configStore.load,
|
|
236
238
|
hasBotToken: configStore.hasBotToken,
|
|
237
|
-
startPolling:
|
|
239
|
+
startPolling: async (ctx, options) => {
|
|
240
|
+
try {
|
|
241
|
+
return await lockedPollingRuntime.start(ctx, options);
|
|
242
|
+
} catch (error) {
|
|
243
|
+
recordRuntimeEvent("recovery", error, { phase: "polling-start" });
|
|
244
|
+
throw error;
|
|
245
|
+
}
|
|
246
|
+
},
|
|
238
247
|
stopPolling: stopPolling ?? lockedPollingRuntime.stop,
|
|
248
|
+
recoverPollingStart,
|
|
239
249
|
getDisconnectThreadName,
|
|
240
250
|
updateStatus,
|
|
241
251
|
getProfileNames: () =>
|
package/lib/bus-follower.ts
CHANGED
|
@@ -138,6 +138,17 @@ export interface TelegramBusFollowerSessionRefreshHookDeps<TContext> {
|
|
|
138
138
|
) => void;
|
|
139
139
|
}
|
|
140
140
|
|
|
141
|
+
export type TelegramBusFollowerControlLifecyclePhase = "electing";
|
|
142
|
+
|
|
143
|
+
export interface TelegramBusFollowerControlState {
|
|
144
|
+
getActiveAuthSecret: () => string | undefined;
|
|
145
|
+
setActiveAuthSecret: (secret: string | undefined) => void;
|
|
146
|
+
getLifecyclePhase: () => TelegramBusFollowerControlLifecyclePhase | undefined;
|
|
147
|
+
setLifecyclePhase: (
|
|
148
|
+
phase: TelegramBusFollowerControlLifecyclePhase | undefined,
|
|
149
|
+
) => void;
|
|
150
|
+
}
|
|
151
|
+
|
|
141
152
|
export interface TelegramBusFollowerRegistrationState {
|
|
142
153
|
isRegistered: () => boolean;
|
|
143
154
|
getTarget: () => TelegramTarget | undefined;
|
|
@@ -841,6 +852,21 @@ export function createTelegramBusFollowerSessionRefreshHook<TContext>(
|
|
|
841
852
|
};
|
|
842
853
|
}
|
|
843
854
|
|
|
855
|
+
export function createTelegramBusFollowerControlState(): TelegramBusFollowerControlState {
|
|
856
|
+
let activeAuthSecret: string | undefined;
|
|
857
|
+
let lifecyclePhase: TelegramBusFollowerControlLifecyclePhase | undefined;
|
|
858
|
+
return {
|
|
859
|
+
getActiveAuthSecret: () => activeAuthSecret,
|
|
860
|
+
setActiveAuthSecret(secret) {
|
|
861
|
+
activeAuthSecret = secret;
|
|
862
|
+
},
|
|
863
|
+
getLifecyclePhase: () => lifecyclePhase,
|
|
864
|
+
setLifecyclePhase(phase) {
|
|
865
|
+
lifecyclePhase = phase;
|
|
866
|
+
},
|
|
867
|
+
};
|
|
868
|
+
}
|
|
869
|
+
|
|
844
870
|
export function createTelegramBusFollowerRegistrationState(
|
|
845
871
|
options: { onAvailabilityChanged?: () => void } = {},
|
|
846
872
|
): TelegramBusFollowerRegistrationState {
|
package/lib/commands.ts
CHANGED
|
@@ -297,6 +297,11 @@ export interface TelegramBridgeCommandStartPollingResult {
|
|
|
297
297
|
owner?: string;
|
|
298
298
|
}
|
|
299
299
|
|
|
300
|
+
export type TelegramPollingStartRecoveryResult =
|
|
301
|
+
| { kind: "unhandled" }
|
|
302
|
+
| { kind: "retry"; message: string }
|
|
303
|
+
| { kind: "blocked"; message: string };
|
|
304
|
+
|
|
300
305
|
export interface TelegramBridgeCommandRegistrationDeps {
|
|
301
306
|
promptForConfig: (ctx: ExtensionCommandContext, profileName?: string) => Promise<void>;
|
|
302
307
|
getStatusLines: (options?: TelegramBridgeStatusLineOptions) => string[];
|
|
@@ -310,6 +315,9 @@ export interface TelegramBridgeCommandRegistrationDeps {
|
|
|
310
315
|
| Promise<void | TelegramBridgeCommandStartPollingResult>
|
|
311
316
|
| TelegramBridgeCommandStartPollingResult;
|
|
312
317
|
stopPolling: () => Promise<void | string>;
|
|
318
|
+
recoverPollingStart?: (
|
|
319
|
+
error: unknown,
|
|
320
|
+
) => Promise<TelegramPollingStartRecoveryResult>;
|
|
313
321
|
getDisconnectThreadName?: () => string | undefined;
|
|
314
322
|
updateStatus: (ctx: ExtensionCommandContext) => void;
|
|
315
323
|
getProfileNames?: () => string[];
|
|
@@ -390,7 +398,41 @@ export function registerTelegramBridgeCommands(
|
|
|
390
398
|
await deps.promptForConfig(ctx, profileName);
|
|
391
399
|
return;
|
|
392
400
|
}
|
|
393
|
-
let
|
|
401
|
+
let recoveryUsed = false;
|
|
402
|
+
const startWithRecovery = async (
|
|
403
|
+
options: TelegramBridgeCommandStartPollingOptions,
|
|
404
|
+
): Promise<void | TelegramBridgeCommandStartPollingResult> => {
|
|
405
|
+
try {
|
|
406
|
+
return await deps.startPolling(ctx, options);
|
|
407
|
+
} catch (error) {
|
|
408
|
+
if (!deps.recoverPollingStart || recoveryUsed) throw error;
|
|
409
|
+
const recovery = await deps.recoverPollingStart(error);
|
|
410
|
+
if (recovery.kind === "unhandled") throw error;
|
|
411
|
+
if (recovery.kind === "blocked") {
|
|
412
|
+
return { ok: false, message: recovery.message };
|
|
413
|
+
}
|
|
414
|
+
recoveryUsed = true;
|
|
415
|
+
try {
|
|
416
|
+
const retry = await deps.startPolling(ctx, options);
|
|
417
|
+
if (!retry) {
|
|
418
|
+
return { ok: true, message: recovery.message };
|
|
419
|
+
}
|
|
420
|
+
return {
|
|
421
|
+
...retry,
|
|
422
|
+
message: retry.ok
|
|
423
|
+
? `${recovery.message} ${retry.message ?? "Telegram bridge connected."}`
|
|
424
|
+
: retry.message,
|
|
425
|
+
};
|
|
426
|
+
} catch {
|
|
427
|
+
return {
|
|
428
|
+
ok: false,
|
|
429
|
+
message:
|
|
430
|
+
"Telegram temporary state was recovered, but the bridge could not restart. Restart this Pi instance and run /telegram-connect again.",
|
|
431
|
+
};
|
|
432
|
+
}
|
|
433
|
+
}
|
|
434
|
+
};
|
|
435
|
+
let result = await startWithRecovery({
|
|
394
436
|
forceFreshLeaderThread: true,
|
|
395
437
|
});
|
|
396
438
|
if (result && !result.ok && result.canTakeover) {
|
|
@@ -403,7 +445,7 @@ export function registerTelegramBridgeCommands(
|
|
|
403
445
|
deps.updateStatus(ctx);
|
|
404
446
|
return;
|
|
405
447
|
}
|
|
406
|
-
result = await
|
|
448
|
+
result = await startWithRecovery({
|
|
407
449
|
force: true,
|
|
408
450
|
forceFreshLeaderThread: true,
|
|
409
451
|
});
|
package/lib/locks.ts
CHANGED
|
@@ -176,6 +176,42 @@ function sleepSync(ms: number): void {
|
|
|
176
176
|
Atomics.wait(new Int32Array(buffer), 0, 0, ms);
|
|
177
177
|
}
|
|
178
178
|
|
|
179
|
+
export interface TelegramRenameRetryOptions {
|
|
180
|
+
rename?: typeof renameSync;
|
|
181
|
+
attempts?: number;
|
|
182
|
+
retryDelayMs?: number;
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
/** Rename one Telegram runtime artifact with bounded Windows sharing retries. */
|
|
186
|
+
export function renameTelegramPathWithRetry(
|
|
187
|
+
sourcePath: string,
|
|
188
|
+
destinationPath: string,
|
|
189
|
+
options: TelegramRenameRetryOptions = {},
|
|
190
|
+
): boolean {
|
|
191
|
+
const rename = options.rename ?? renameSync;
|
|
192
|
+
const attempts = Math.max(
|
|
193
|
+
1,
|
|
194
|
+
options.attempts ?? TELEGRAM_LOCK_WRITE_RETRY_ATTEMPTS,
|
|
195
|
+
);
|
|
196
|
+
const retryDelayMs = Math.max(
|
|
197
|
+
0,
|
|
198
|
+
options.retryDelayMs ?? TELEGRAM_LOCK_WRITE_RETRY_DELAY_MS,
|
|
199
|
+
);
|
|
200
|
+
for (let attempt = 0; attempt < attempts; attempt += 1) {
|
|
201
|
+
try {
|
|
202
|
+
rename(sourcePath, destinationPath);
|
|
203
|
+
return true;
|
|
204
|
+
} catch (error) {
|
|
205
|
+
if ((error as { code?: unknown })?.code === "ENOENT") return false;
|
|
206
|
+
if (!isRetryableLockWriteError(error) || attempt === attempts - 1) {
|
|
207
|
+
throw error;
|
|
208
|
+
}
|
|
209
|
+
sleepSync(retryDelayMs * (attempt + 1));
|
|
210
|
+
}
|
|
211
|
+
}
|
|
212
|
+
return false;
|
|
213
|
+
}
|
|
214
|
+
|
|
179
215
|
interface TelegramLockTransactionOwner {
|
|
180
216
|
pid: number;
|
|
181
217
|
acquiredAtMs: number;
|
|
@@ -307,29 +343,11 @@ function releaseLockTransactionGuard(
|
|
|
307
343
|
);
|
|
308
344
|
}
|
|
309
345
|
const releasedPath = `${path}.released.${randomUUID()}`;
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
try {
|
|
316
|
-
renameSync(path, releasedPath);
|
|
317
|
-
try {
|
|
318
|
-
removeLockTransactionGuard(releasedPath);
|
|
319
|
-
} catch {
|
|
320
|
-
/* released debris cannot retain transaction authority */
|
|
321
|
-
}
|
|
322
|
-
return;
|
|
323
|
-
} catch (error) {
|
|
324
|
-
if ((error as { code?: unknown })?.code === "ENOENT") return;
|
|
325
|
-
if (
|
|
326
|
-
!isRetryableLockWriteError(error) ||
|
|
327
|
-
attempt === TELEGRAM_LOCK_WRITE_RETRY_ATTEMPTS - 1
|
|
328
|
-
) {
|
|
329
|
-
throw error;
|
|
330
|
-
}
|
|
331
|
-
sleepSync(TELEGRAM_LOCK_WRITE_RETRY_DELAY_MS * (attempt + 1));
|
|
332
|
-
}
|
|
346
|
+
if (!renameTelegramPathWithRetry(path, releasedPath)) return;
|
|
347
|
+
try {
|
|
348
|
+
removeLockTransactionGuard(releasedPath);
|
|
349
|
+
} catch {
|
|
350
|
+
/* released debris cannot retain transaction authority */
|
|
333
351
|
}
|
|
334
352
|
}
|
|
335
353
|
|
|
@@ -411,28 +429,8 @@ function reclaimAbandonedDirectoryGuard(
|
|
|
411
429
|
throw error;
|
|
412
430
|
}
|
|
413
431
|
|
|
414
|
-
const renameWithRetry = (fromPath: string, toPath: string): boolean =>
|
|
415
|
-
|
|
416
|
-
let attempt = 0;
|
|
417
|
-
attempt < TELEGRAM_LOCK_WRITE_RETRY_ATTEMPTS;
|
|
418
|
-
attempt += 1
|
|
419
|
-
) {
|
|
420
|
-
try {
|
|
421
|
-
renameRecovery(fromPath, toPath);
|
|
422
|
-
return true;
|
|
423
|
-
} catch (error) {
|
|
424
|
-
if ((error as { code?: unknown })?.code === "ENOENT") return false;
|
|
425
|
-
if (
|
|
426
|
-
!isRetryableLockWriteError(error) ||
|
|
427
|
-
attempt === TELEGRAM_LOCK_WRITE_RETRY_ATTEMPTS - 1
|
|
428
|
-
) {
|
|
429
|
-
throw error;
|
|
430
|
-
}
|
|
431
|
-
sleepSync(TELEGRAM_LOCK_WRITE_RETRY_DELAY_MS * (attempt + 1));
|
|
432
|
-
}
|
|
433
|
-
}
|
|
434
|
-
return false;
|
|
435
|
-
};
|
|
432
|
+
const renameWithRetry = (fromPath: string, toPath: string): boolean =>
|
|
433
|
+
renameTelegramPathWithRetry(fromPath, toPath, { rename: renameRecovery });
|
|
436
434
|
|
|
437
435
|
activeReclaims.add(reclaimGeneration);
|
|
438
436
|
const stalePath = `${path}.stale.${process.pid}.${randomUUID()}`;
|
package/lib/recovery.ts
ADDED
|
@@ -0,0 +1,464 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Telegram disposable runtime recovery classification
|
|
3
|
+
* Zones: filesystem diagnostics, unclean-shutdown recovery
|
|
4
|
+
* Owns fail-safe classification of temporary ownership and routing artifacts
|
|
5
|
+
*/
|
|
6
|
+
|
|
7
|
+
import { randomUUID } from "node:crypto";
|
|
8
|
+
import {
|
|
9
|
+
existsSync,
|
|
10
|
+
lstatSync,
|
|
11
|
+
mkdirSync,
|
|
12
|
+
readFileSync,
|
|
13
|
+
readdirSync,
|
|
14
|
+
renameSync,
|
|
15
|
+
} from "node:fs";
|
|
16
|
+
import { basename, dirname, join } from "node:path";
|
|
17
|
+
|
|
18
|
+
import {
|
|
19
|
+
isProcessAlive as defaultIsProcessAlive,
|
|
20
|
+
parseTelegramLockEntry,
|
|
21
|
+
renameTelegramPathWithRetry,
|
|
22
|
+
TELEGRAM_BUS_LEADER_STALE_HEARTBEAT_MS,
|
|
23
|
+
withTelegramFileTransaction,
|
|
24
|
+
type TelegramFileTransactionOptions,
|
|
25
|
+
} from "./locks.ts";
|
|
26
|
+
|
|
27
|
+
export type TelegramRuntimeArtifactKind = "owners" | "state" | "transaction";
|
|
28
|
+
|
|
29
|
+
export interface TelegramRuntimeCorruptArtifact {
|
|
30
|
+
kind: TelegramRuntimeArtifactKind;
|
|
31
|
+
path: string;
|
|
32
|
+
reason: string;
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
export type TelegramRuntimeRecoveryClassification =
|
|
36
|
+
| { kind: "clean" }
|
|
37
|
+
| {
|
|
38
|
+
kind: "recoverable-corruption";
|
|
39
|
+
artifacts: TelegramRuntimeCorruptArtifact[];
|
|
40
|
+
}
|
|
41
|
+
| {
|
|
42
|
+
kind: "blocked-live-owner";
|
|
43
|
+
artifacts: TelegramRuntimeCorruptArtifact[];
|
|
44
|
+
livePids: number[];
|
|
45
|
+
};
|
|
46
|
+
|
|
47
|
+
export interface TelegramRuntimeRecoveryClassificationOptions {
|
|
48
|
+
ownersPath: string;
|
|
49
|
+
statePaths?: readonly string[];
|
|
50
|
+
transactionPath?: string;
|
|
51
|
+
isProcessAlive?: (pid: number) => boolean;
|
|
52
|
+
nowMs?: number;
|
|
53
|
+
staleHeartbeatMs?: number;
|
|
54
|
+
ignoredTransactionPids?: readonly number[];
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
export type TelegramRuntimeRecoveryResult =
|
|
58
|
+
| { kind: "not-needed" }
|
|
59
|
+
| {
|
|
60
|
+
kind: "blocked-live-owner";
|
|
61
|
+
livePids: number[];
|
|
62
|
+
quarantineDir?: string;
|
|
63
|
+
}
|
|
64
|
+
| {
|
|
65
|
+
kind: "recovered";
|
|
66
|
+
artifacts: TelegramRuntimeCorruptArtifact[];
|
|
67
|
+
quarantineDir: string;
|
|
68
|
+
};
|
|
69
|
+
|
|
70
|
+
export interface TelegramRuntimeRecoveryOptions
|
|
71
|
+
extends TelegramRuntimeRecoveryClassificationOptions {
|
|
72
|
+
recoveryTransactionPath?: string;
|
|
73
|
+
quarantineRoot?: string;
|
|
74
|
+
pid?: number;
|
|
75
|
+
getNowMs?: () => number;
|
|
76
|
+
quarantineRename?: typeof renameSync;
|
|
77
|
+
quarantineRenameRetryDelayMs?: number;
|
|
78
|
+
transactionOptions?: TelegramFileTransactionOptions;
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
export type TelegramPollingStartRecoveryDecision =
|
|
82
|
+
| { kind: "unhandled" }
|
|
83
|
+
| { kind: "retry"; message: string }
|
|
84
|
+
| { kind: "blocked"; message: string };
|
|
85
|
+
|
|
86
|
+
export interface TelegramPollingStartRecoveryHandlerDeps {
|
|
87
|
+
getOwnersPath: () => string;
|
|
88
|
+
getStatePaths: () => readonly string[];
|
|
89
|
+
suspendPolling: () => Promise<unknown>;
|
|
90
|
+
releaseOwnership?: () => unknown | Promise<unknown>;
|
|
91
|
+
recordRuntimeEvent?: (
|
|
92
|
+
category: string,
|
|
93
|
+
error: unknown,
|
|
94
|
+
details?: Record<string, unknown>,
|
|
95
|
+
) => void;
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
interface ArtifactInspection {
|
|
99
|
+
source: TelegramRuntimeArtifactKind;
|
|
100
|
+
corrupt?: TelegramRuntimeCorruptArtifact;
|
|
101
|
+
ownerPids: number[];
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
const OWNER_FILE_PATTERN = /^owner\.([A-Za-z0-9-]+)\.json$/u;
|
|
105
|
+
const RECLAIM_FILE_PATTERN =
|
|
106
|
+
/^owner\.reclaim\.(\d+)\.([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\.json$/u;
|
|
107
|
+
|
|
108
|
+
function corruption(
|
|
109
|
+
kind: TelegramRuntimeArtifactKind,
|
|
110
|
+
path: string,
|
|
111
|
+
reason: string,
|
|
112
|
+
): ArtifactInspection {
|
|
113
|
+
return { source: kind, corrupt: { kind, path, reason }, ownerPids: [] };
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
function inspectOwners(
|
|
117
|
+
path: string,
|
|
118
|
+
nowMs: number,
|
|
119
|
+
staleHeartbeatMs: number,
|
|
120
|
+
): ArtifactInspection {
|
|
121
|
+
if (!existsSync(path)) return { source: "owners", ownerPids: [] };
|
|
122
|
+
let value: unknown;
|
|
123
|
+
try {
|
|
124
|
+
value = JSON.parse(readFileSync(path, "utf8"));
|
|
125
|
+
} catch {
|
|
126
|
+
return corruption("owners", path, "owners.json is not valid JSON");
|
|
127
|
+
}
|
|
128
|
+
if (!value || typeof value !== "object" || Array.isArray(value)) {
|
|
129
|
+
return corruption("owners", path, "owners.json is not an object");
|
|
130
|
+
}
|
|
131
|
+
const ownerPids = Object.values(value).flatMap((candidate) => {
|
|
132
|
+
const entry = parseTelegramLockEntry(candidate);
|
|
133
|
+
if (
|
|
134
|
+
!entry ||
|
|
135
|
+
(typeof entry.heartbeatMs === "number" &&
|
|
136
|
+
nowMs - entry.heartbeatMs > staleHeartbeatMs)
|
|
137
|
+
) {
|
|
138
|
+
return [];
|
|
139
|
+
}
|
|
140
|
+
return [entry.pid];
|
|
141
|
+
});
|
|
142
|
+
return { source: "owners", ownerPids };
|
|
143
|
+
}
|
|
144
|
+
|
|
145
|
+
function inspectState(path: string): ArtifactInspection {
|
|
146
|
+
if (!existsSync(path)) return { source: "state", ownerPids: [] };
|
|
147
|
+
try {
|
|
148
|
+
JSON.parse(readFileSync(path, "utf8"));
|
|
149
|
+
return { source: "state", ownerPids: [] };
|
|
150
|
+
} catch {
|
|
151
|
+
return corruption("state", path, `${basename(path)} is not valid JSON`);
|
|
152
|
+
}
|
|
153
|
+
}
|
|
154
|
+
|
|
155
|
+
function parseTransactionOwner(path: string): number | undefined {
|
|
156
|
+
const value: unknown = JSON.parse(readFileSync(path, "utf8"));
|
|
157
|
+
if (!value || typeof value !== "object" || Array.isArray(value)) {
|
|
158
|
+
return undefined;
|
|
159
|
+
}
|
|
160
|
+
const pid = (value as Record<string, unknown>).pid;
|
|
161
|
+
const acquiredAtMs = (value as Record<string, unknown>).acquiredAtMs;
|
|
162
|
+
const generation = (value as Record<string, unknown>).generation;
|
|
163
|
+
return typeof pid === "number" &&
|
|
164
|
+
typeof acquiredAtMs === "number" &&
|
|
165
|
+
typeof generation === "string"
|
|
166
|
+
? pid
|
|
167
|
+
: undefined;
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
function inspectTransaction(path: string): ArtifactInspection {
|
|
171
|
+
if (!existsSync(path)) return { source: "transaction", ownerPids: [] };
|
|
172
|
+
try {
|
|
173
|
+
const stat = lstatSync(path);
|
|
174
|
+
let ownerPath: string;
|
|
175
|
+
let expectedOwnerPid: number | undefined;
|
|
176
|
+
if (stat.isDirectory()) {
|
|
177
|
+
const entries = readdirSync(path);
|
|
178
|
+
if (entries.length !== 1) {
|
|
179
|
+
return corruption(
|
|
180
|
+
"transaction",
|
|
181
|
+
path,
|
|
182
|
+
"transaction guard has an unverifiable directory shape",
|
|
183
|
+
);
|
|
184
|
+
}
|
|
185
|
+
const entry = entries[0];
|
|
186
|
+
const ownerMatch = OWNER_FILE_PATTERN.exec(entry);
|
|
187
|
+
const reclaimMatch = RECLAIM_FILE_PATTERN.exec(entry);
|
|
188
|
+
if (!ownerMatch && !reclaimMatch) {
|
|
189
|
+
return corruption(
|
|
190
|
+
"transaction",
|
|
191
|
+
path,
|
|
192
|
+
"transaction guard has an unrecognized owner marker",
|
|
193
|
+
);
|
|
194
|
+
}
|
|
195
|
+
ownerPath = join(path, entry);
|
|
196
|
+
expectedOwnerPid = reclaimMatch ? Number(reclaimMatch[1]) : undefined;
|
|
197
|
+
const ownerPid = parseTransactionOwner(ownerPath);
|
|
198
|
+
if (ownerPid === undefined) {
|
|
199
|
+
return corruption(
|
|
200
|
+
"transaction",
|
|
201
|
+
path,
|
|
202
|
+
"transaction guard owner metadata is invalid",
|
|
203
|
+
);
|
|
204
|
+
}
|
|
205
|
+
if (ownerMatch) {
|
|
206
|
+
const value = JSON.parse(readFileSync(ownerPath, "utf8")) as Record<
|
|
207
|
+
string,
|
|
208
|
+
unknown
|
|
209
|
+
>;
|
|
210
|
+
if (value.generation !== ownerMatch[1]) {
|
|
211
|
+
return corruption(
|
|
212
|
+
"transaction",
|
|
213
|
+
path,
|
|
214
|
+
"transaction guard generation does not match its owner marker",
|
|
215
|
+
);
|
|
216
|
+
}
|
|
217
|
+
}
|
|
218
|
+
return {
|
|
219
|
+
source: "transaction",
|
|
220
|
+
ownerPids: [expectedOwnerPid ?? ownerPid],
|
|
221
|
+
};
|
|
222
|
+
}
|
|
223
|
+
if (!stat.isFile()) {
|
|
224
|
+
return corruption(
|
|
225
|
+
"transaction",
|
|
226
|
+
path,
|
|
227
|
+
"transaction guard has an unsupported filesystem type",
|
|
228
|
+
);
|
|
229
|
+
}
|
|
230
|
+
ownerPath = path;
|
|
231
|
+
const ownerPid = parseTransactionOwner(ownerPath);
|
|
232
|
+
return ownerPid === undefined
|
|
233
|
+
? corruption(
|
|
234
|
+
"transaction",
|
|
235
|
+
path,
|
|
236
|
+
"legacy transaction guard owner metadata is invalid",
|
|
237
|
+
)
|
|
238
|
+
: { source: "transaction", ownerPids: [ownerPid] };
|
|
239
|
+
} catch {
|
|
240
|
+
return corruption(
|
|
241
|
+
"transaction",
|
|
242
|
+
path,
|
|
243
|
+
"transaction guard cannot be inspected",
|
|
244
|
+
);
|
|
245
|
+
}
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
/**
|
|
249
|
+
* Classify disposable runtime corruption without mutating any artifact.
|
|
250
|
+
*
|
|
251
|
+
* Corruption remains recoverable only when neither owners.json nor a
|
|
252
|
+
* verifiable transaction marker identifies a process that is still alive.
|
|
253
|
+
*/
|
|
254
|
+
export function classifyTelegramRuntimeRecovery(
|
|
255
|
+
options: TelegramRuntimeRecoveryClassificationOptions,
|
|
256
|
+
): TelegramRuntimeRecoveryClassification {
|
|
257
|
+
const transactionPath =
|
|
258
|
+
options.transactionPath ?? `${options.ownersPath}.transaction`;
|
|
259
|
+
const inspections = [
|
|
260
|
+
inspectOwners(
|
|
261
|
+
options.ownersPath,
|
|
262
|
+
options.nowMs ?? Date.now(),
|
|
263
|
+
options.staleHeartbeatMs ?? TELEGRAM_BUS_LEADER_STALE_HEARTBEAT_MS,
|
|
264
|
+
),
|
|
265
|
+
...(options.statePaths ?? []).map(inspectState),
|
|
266
|
+
inspectTransaction(transactionPath),
|
|
267
|
+
];
|
|
268
|
+
const artifacts = inspections.flatMap((inspection) =>
|
|
269
|
+
inspection.corrupt ? [inspection.corrupt] : [],
|
|
270
|
+
);
|
|
271
|
+
if (artifacts.length === 0) return { kind: "clean" };
|
|
272
|
+
|
|
273
|
+
const processAlive = options.isProcessAlive ?? defaultIsProcessAlive;
|
|
274
|
+
const ignoredTransactionPids = new Set(
|
|
275
|
+
options.ignoredTransactionPids ?? [],
|
|
276
|
+
);
|
|
277
|
+
const livePids = [
|
|
278
|
+
...new Set(
|
|
279
|
+
inspections.flatMap((inspection) =>
|
|
280
|
+
inspection.ownerPids.filter(
|
|
281
|
+
(pid) =>
|
|
282
|
+
!(
|
|
283
|
+
inspection.source === "transaction" &&
|
|
284
|
+
ignoredTransactionPids.has(pid)
|
|
285
|
+
) && processAlive(pid),
|
|
286
|
+
),
|
|
287
|
+
),
|
|
288
|
+
),
|
|
289
|
+
].sort((left, right) => left - right);
|
|
290
|
+
return livePids.length > 0
|
|
291
|
+
? { kind: "blocked-live-owner", artifacts, livePids }
|
|
292
|
+
: { kind: "recoverable-corruption", artifacts };
|
|
293
|
+
}
|
|
294
|
+
|
|
295
|
+
/**
|
|
296
|
+
* Quarantine classifier-approved disposable corruption under two guards.
|
|
297
|
+
*
|
|
298
|
+
* A dedicated recovery transaction serializes recoverers. The ownership
|
|
299
|
+
* transaction then prevents a new Telegram owner from appearing between the
|
|
300
|
+
* final classification and mutation. Every artifact is renamed within its
|
|
301
|
+
* filesystem; durable config and diagnostics never enter the candidate set.
|
|
302
|
+
*/
|
|
303
|
+
export function recoverTelegramRuntimeState(
|
|
304
|
+
options: TelegramRuntimeRecoveryOptions,
|
|
305
|
+
): TelegramRuntimeRecoveryResult {
|
|
306
|
+
const pid = options.pid ?? process.pid;
|
|
307
|
+
const transactionPath =
|
|
308
|
+
options.transactionPath ?? `${options.ownersPath}.transaction`;
|
|
309
|
+
const recoveryTransactionPath =
|
|
310
|
+
options.recoveryTransactionPath ??
|
|
311
|
+
join(dirname(options.ownersPath), "runtime-recovery.transaction");
|
|
312
|
+
const quarantineRoot =
|
|
313
|
+
options.quarantineRoot ?? join(dirname(options.ownersPath), "recovery");
|
|
314
|
+
const classificationOptions = {
|
|
315
|
+
ownersPath: options.ownersPath,
|
|
316
|
+
statePaths: options.statePaths,
|
|
317
|
+
transactionPath,
|
|
318
|
+
isProcessAlive: options.isProcessAlive,
|
|
319
|
+
nowMs: options.nowMs ?? options.getNowMs?.(),
|
|
320
|
+
staleHeartbeatMs: options.staleHeartbeatMs,
|
|
321
|
+
} satisfies TelegramRuntimeRecoveryClassificationOptions;
|
|
322
|
+
|
|
323
|
+
return withTelegramFileTransaction(
|
|
324
|
+
recoveryTransactionPath,
|
|
325
|
+
() => {
|
|
326
|
+
const initial = classifyTelegramRuntimeRecovery(classificationOptions);
|
|
327
|
+
if (initial.kind === "clean") return { kind: "not-needed" };
|
|
328
|
+
if (initial.kind === "blocked-live-owner") {
|
|
329
|
+
return {
|
|
330
|
+
kind: "blocked-live-owner",
|
|
331
|
+
livePids: initial.livePids,
|
|
332
|
+
};
|
|
333
|
+
}
|
|
334
|
+
|
|
335
|
+
let quarantineDir: string | undefined;
|
|
336
|
+
const recoveredArtifacts: TelegramRuntimeCorruptArtifact[] = [];
|
|
337
|
+
const ensureQuarantineDir = (): string => {
|
|
338
|
+
if (quarantineDir) return quarantineDir;
|
|
339
|
+
quarantineDir = join(
|
|
340
|
+
quarantineRoot,
|
|
341
|
+
`${options.getNowMs?.() ?? Date.now()}-${pid}-${randomUUID()}`,
|
|
342
|
+
);
|
|
343
|
+
mkdirSync(quarantineDir, { recursive: true, mode: 0o700 });
|
|
344
|
+
return quarantineDir;
|
|
345
|
+
};
|
|
346
|
+
const quarantineArtifact = (
|
|
347
|
+
artifact: TelegramRuntimeCorruptArtifact,
|
|
348
|
+
): void => {
|
|
349
|
+
if (!existsSync(artifact.path)) return;
|
|
350
|
+
const destination = join(ensureQuarantineDir(), basename(artifact.path));
|
|
351
|
+
if (
|
|
352
|
+
renameTelegramPathWithRetry(artifact.path, destination, {
|
|
353
|
+
rename: options.quarantineRename,
|
|
354
|
+
retryDelayMs: options.quarantineRenameRetryDelayMs,
|
|
355
|
+
})
|
|
356
|
+
) {
|
|
357
|
+
recoveredArtifacts.push(artifact);
|
|
358
|
+
}
|
|
359
|
+
};
|
|
360
|
+
|
|
361
|
+
for (const artifact of initial.artifacts) {
|
|
362
|
+
if (artifact.kind === "transaction") quarantineArtifact(artifact);
|
|
363
|
+
}
|
|
364
|
+
|
|
365
|
+
return withTelegramFileTransaction(
|
|
366
|
+
transactionPath,
|
|
367
|
+
() => {
|
|
368
|
+
const current = classifyTelegramRuntimeRecovery({
|
|
369
|
+
...classificationOptions,
|
|
370
|
+
ignoredTransactionPids: [process.pid],
|
|
371
|
+
});
|
|
372
|
+
if (current.kind === "blocked-live-owner") {
|
|
373
|
+
return {
|
|
374
|
+
kind: "blocked-live-owner",
|
|
375
|
+
livePids: current.livePids,
|
|
376
|
+
quarantineDir,
|
|
377
|
+
};
|
|
378
|
+
}
|
|
379
|
+
if (current.kind === "recoverable-corruption") {
|
|
380
|
+
for (const artifact of current.artifacts) {
|
|
381
|
+
if (artifact.kind !== "transaction") quarantineArtifact(artifact);
|
|
382
|
+
}
|
|
383
|
+
}
|
|
384
|
+
return recoveredArtifacts.length > 0 && quarantineDir
|
|
385
|
+
? {
|
|
386
|
+
kind: "recovered",
|
|
387
|
+
artifacts: recoveredArtifacts,
|
|
388
|
+
quarantineDir,
|
|
389
|
+
}
|
|
390
|
+
: { kind: "not-needed" };
|
|
391
|
+
},
|
|
392
|
+
options.transactionOptions,
|
|
393
|
+
);
|
|
394
|
+
},
|
|
395
|
+
options.transactionOptions,
|
|
396
|
+
);
|
|
397
|
+
}
|
|
398
|
+
|
|
399
|
+
/** Build the `/telegram-connect` recovery boundary around runtime artifacts. */
|
|
400
|
+
export function createTelegramPollingStartRecoveryHandler(
|
|
401
|
+
deps: TelegramPollingStartRecoveryHandlerDeps,
|
|
402
|
+
): () => Promise<TelegramPollingStartRecoveryDecision> {
|
|
403
|
+
return async () => {
|
|
404
|
+
const ownersPath = deps.getOwnersPath();
|
|
405
|
+
const statePaths = deps.getStatePaths();
|
|
406
|
+
const classification = classifyTelegramRuntimeRecovery({
|
|
407
|
+
ownersPath,
|
|
408
|
+
statePaths,
|
|
409
|
+
});
|
|
410
|
+
if (classification.kind === "clean") return { kind: "unhandled" };
|
|
411
|
+
if (classification.kind === "blocked-live-owner") {
|
|
412
|
+
return {
|
|
413
|
+
kind: "blocked",
|
|
414
|
+
message: `Telegram temporary state is damaged, but owner process ${classification.livePids.join(", ")} is still live. Restart that Pi instance, then run /telegram-connect again.`,
|
|
415
|
+
};
|
|
416
|
+
}
|
|
417
|
+
|
|
418
|
+
try {
|
|
419
|
+
await deps.suspendPolling();
|
|
420
|
+
} catch (error) {
|
|
421
|
+
deps.recordRuntimeEvent?.("recovery", error, {
|
|
422
|
+
phase: "suspend-before-reset",
|
|
423
|
+
});
|
|
424
|
+
return {
|
|
425
|
+
kind: "blocked",
|
|
426
|
+
message:
|
|
427
|
+
"Telegram polling could not stop safely, so temporary state was not reset. Restart this Pi instance and run /telegram-connect again.",
|
|
428
|
+
};
|
|
429
|
+
}
|
|
430
|
+
try {
|
|
431
|
+
await deps.releaseOwnership?.();
|
|
432
|
+
} catch (error) {
|
|
433
|
+
deps.recordRuntimeEvent?.("recovery", error, {
|
|
434
|
+
phase: "release-before-reset",
|
|
435
|
+
});
|
|
436
|
+
}
|
|
437
|
+
try {
|
|
438
|
+
const recovery = recoverTelegramRuntimeState({
|
|
439
|
+
ownersPath,
|
|
440
|
+
statePaths,
|
|
441
|
+
});
|
|
442
|
+
if (recovery.kind === "blocked-live-owner") {
|
|
443
|
+
return {
|
|
444
|
+
kind: "blocked",
|
|
445
|
+
message: `Telegram temporary state changed during recovery and is now protected by owner process ${recovery.livePids.join(", ")}. Restart that Pi instance, then run /telegram-connect again.`,
|
|
446
|
+
};
|
|
447
|
+
}
|
|
448
|
+
return {
|
|
449
|
+
kind: "retry",
|
|
450
|
+
message:
|
|
451
|
+
recovery.kind === "recovered"
|
|
452
|
+
? "Telegram temporary state was damaged after an unclean shutdown and has been reset."
|
|
453
|
+
: "Telegram temporary state was recovered by another Pi instance.",
|
|
454
|
+
};
|
|
455
|
+
} catch (error) {
|
|
456
|
+
deps.recordRuntimeEvent?.("recovery", error, { phase: "runtime-reset" });
|
|
457
|
+
return {
|
|
458
|
+
kind: "blocked",
|
|
459
|
+
message:
|
|
460
|
+
"Telegram temporary-state recovery failed. Restart this Pi instance and run /telegram-connect again.",
|
|
461
|
+
};
|
|
462
|
+
}
|
|
463
|
+
};
|
|
464
|
+
}
|