telstore 0.1.7 → 0.1.9

package/README.md

@@ -28,8 +28,9 @@ npx telstore restore telstore-20260905-7f3a91
 | `telstore <file\|folder\|pattern>...` | Split each file and upload it. Prints the `backupId` you restore with. |
 | `telstore list` | The backups stored in the destination, newest first. |
 | `telstore restore <backup-id>...` | Download every chunk and reassemble the file. Several ids run one after another. |
+| `telstore verify <backup-id>...` | Check that every chunk of a backup is still in the chat. Downloads nothing. |
 | `telstore delete <backup-id>...` | Remove a backup's chunks and manifest from the chat, for good. Several ids are listed and confirmed once. |
-| `telstore status` | Account, destination, and unfinished backups. |
+| `telstore status` | Account, destination, and unfinished uploads and restores. |
 | `telstore config` | Show or change settings. |
 | `telstore token` | Print a session token for a machine you do not trust. |
 | `telstore logout` | Remove the locally stored session. |
@@ -65,7 +66,7 @@ a prompt that does not echo.
 
 Every chunk goes up as a document captioned `📦 <backupId> · 3/12`, followed by a manifest
 carrying a summary card — file name, size, id, date and the restore command. `list` reads
-those cards straight out of the chat, one search and no downloads:
+those cards straight out of the chat, no downloads:
 
 ```
 Destination  https://web.telegram.org/k/#@my_backups
@@ -117,13 +118,44 @@ it, and the run ends with a line per file and a non-zero exit code:
   c.tar  telstore-20260905-9de447  (1 chunk)
 ```
 
+## Checking a backup is still there
+
+A backup is a set of messages in a chat, and messages can be deleted by hand. `list` reads
+the manifest's card and would happily show a backup whose chunks are long gone; `verify` asks
+the chat about every chunk the manifest names:
+
+```
+$ npx telstore verify telstore-20260905-7f3a91
+Backup telstore-20260905-7f3a91
+File   data.tar (21.4 GB, 12 chunks)
+In     https://web.telegram.org/k/#@my_backups
+
+12 chunks present, at the sizes the manifest records.
+This does not download them, so it cannot prove their contents.
+```
+
+It costs one request per hundred chunks and no bandwidth, so it is cheap enough to run on a
+schedule. What it proves is that a restore would find everything it needs — every chunk still
+there, under the file name telstore wrote, at the length the manifest records. It does not
+read the chunks, so it cannot speak for what is inside them; only a restore does that, and a
+restore checks every sha256 before it renames anything into place.
+
+A backup missing chunks is named line by line, and the exit code is 1:
+
+```
+Chunk 3/12 is gone: message 1042 is no longer in @my_backups.
+
+12 chunks checked, 1 damaged. This backup cannot be restored.
+```
+
 ## Several backups at once
 
-`restore` and `delete` take a list of ids the same way, over one connection, with a summary
-and a non-zero exit code if any of them failed:
+`restore`, `verify` and `delete` take a list of ids the same way, over one connection, with a
+summary and a non-zero exit code if any of them failed:
 
 ```bash
 npx telstore restore telstore-20260905-7f3a91 telstore-20260901-9de447
+npx telstore verify telstore-20260905-7f3a91 telstore-20260901-9de447
 npx telstore delete telstore-20260905-7f3a91 telstore-20260901-9de447
 ```
 
@@ -143,7 +175,14 @@ After a batch, run telstore again with **only the files that are left**: the fin
 have had their records cleared, so repeating the whole command would upload them a second
 time as new backups. `npx telstore status` lists what is unfinished.
 
-**Restore keeps no state** — `Ctrl-C` mid-restore saves nothing, running again starts over.
+## Resuming a restore
+
+`Ctrl-C` mid-restore keeps the `<target>.partial` file rather than throwing it away. Running
+the same command again hashes each chunk-sized region of it against the manifest, in order,
+and carries on from the first one that does not match — nothing already on disk is trusted
+just because it is there. `npx telstore status` lists unfinished restores alongside
+unfinished uploads, with a resume command for each.
+
 And `delete` has **no undo**: Telegram is the only copy.
 
 ## Running on a machine you do not trust
@@ -173,7 +212,8 @@ There is no expiry and no revocation: to end a session for good, terminate it un
   protects your login rather than your files.
 - A chunk cannot exceed 1950MB: Telegram accepts at most 4000 parts of 512KB per file.
 - Deleting a chunk message in the Telegram app destroys the backup, and keeping the
-  `backupId` is what saves you hunting for its manifest in the chat by hand.
+  `backupId` is what saves you hunting for its manifest in the chat by hand. `verify` is how
+  you find that out before you need the file rather than after.
 
 Settings and credentials live in `~/.telstore/config.json`, mode 600 — `apiId`, `apiHash` and
 the session at the top level (or a single `sealed` blob after `login --token`), everything

package/bin/telstore.js

@@ -14,9 +14,9 @@ const SIGINT_EXIT_CODE = 130
 let currentCommand = null
 let currentBackupId = null
 
-// A batch clears each finished file's record as it goes, so by the time Ctrl-C lands these
-// are backups no second run should touch. Ctrl-C needs their names to say so.
-const finishedUploads = []
+// A batch clears each finished item's record as it goes, so by the time Ctrl-C lands these
+// are transfers no second run should touch. Ctrl-C needs their names to say so.
+const finished = []
 
 process.on('SIGINT', () => {
   // A passphrase prompt has stdin in raw mode, and process.exit skips readline's own cleanup.
@@ -24,7 +24,7 @@ process.on('SIGINT', () => {
   if (process.stdin.isTTY) process.stdin.setRawMode(false)
 
   process.stderr.write(
-    interruptMessage(currentCommand, { backupId: currentBackupId, done: finishedUploads }),
+    interruptMessage(currentCommand, { backupId: currentBackupId, done: finished }),
   )
   process.exit(SIGINT_EXIT_CODE)
 })
@@ -104,7 +104,7 @@ async function main() {
           currentBackupId = id
         },
         onFileDone: (file) => {
-          if (file.id) finishedUploads.push(file)
+          if (file.id) finished.push(file)
         },
       })
 
@@ -121,8 +121,30 @@ async function main() {
 
       const { runRestores } = await import('../src/commands/restore.js')
 
-      const { failed } = await runRestores(parsed.args, parsed.options)
+      const { failed } = await runRestores(parsed.args, parsed.options, {
+        onBackupId: (id) => {
+          currentBackupId = id
+        },
+        onRestoreDone: (item) => {
+          finished.push(item)
+        },
+      })
+
+      if (failed > 0) process.exitCode = 1
+      return
+    }
+
+    case 'verify': {
+      if (!parsed.args[0]) {
+        throw new Error('Missing backup id. Example: npx telstore verify telstore-20260905-7f3a91')
+      }
+
+      const { runVerifies } = await import('../src/commands/verify.js')
+
+      const { failed } = await runVerifies(parsed.args, parsed.options)
 
+      // A backup that is damaged, and one telstore could not look up at all, both mean the
+      // run did not find what it was asked to check. Whatever runs telstore learns that here.
       if (failed > 0) process.exitCode = 1
       return
     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "telstore",
-  "version": "0.1.7",
+  "version": "0.1.9",
   "description": "Split large files into chunks and store them on Telegram",
   "keywords": [
     "telegram",

package/src/cli.js

@@ -6,6 +6,7 @@ const SUBCOMMANDS = new Set([
   'logout',
   'list',
   'restore',
+  'verify',
   'delete',
   'status',
   'config',
@@ -21,6 +22,7 @@ const OPTIONS = {
   out: { type: 'string' },
   note: { type: 'string' },
   limit: { type: 'string' },
+  search: { type: 'string' },
   verbose: { type: 'boolean' },
   unset: { type: 'boolean' },
   yes: { type: 'boolean' },
@@ -34,9 +36,11 @@ Usage:
   npx telstore login                      Log in to Telegram, only needed once
   npx telstore <file|folder|pattern>...   Split files and upload them to Telegram
   npx telstore list                       List the backups stored in the destination
+  npx telstore list --search <text>       List only the backups that text appears in
   npx telstore restore <backup-id>...     Download the chunks and reassemble the files
+  npx telstore verify <backup-id>...      Check that a backup's chunks are all still in the chat
   npx telstore delete <backup-id>...      Remove backups' chunks and manifests from the chat
-  npx telstore status                     Show the account, the destination and unfinished backups
+  npx telstore status                     Show the account, the destination and unfinished uploads and restores
   npx telstore config                     Show every setting and where its value comes from
   npx telstore logout                     Remove the saved session
 
@@ -50,8 +54,10 @@ backup. A folder means the files one level inside it, and a pattern means the na
 one file is listed and confirmed before the first byte goes out. Run telstore again with only
 the files that are left to carry on after an interruption.
 
-restore and delete take several ids the same way: one connection, one line each, and an exit
-code that reports any that failed. delete shows everything it is about to destroy and asks once.
+restore, verify and delete take several ids the same way: one connection, one line each, and
+an exit code that reports any that failed. delete shows everything it is about to destroy and
+asks once. verify downloads nothing: it asks the chat whether every chunk message is still
+there at the length the manifest records, which is what restore would need.
 
 Settings:
   npx telstore config <name>              Print one setting's value
@@ -80,6 +86,12 @@ Options apply to one run and are never saved. Use config to change a setting for
                               shell hands the words after the first to telstore as more files
                               to upload.
   --limit <n>                 How many backups list shows this run.
+  --search <text>             List only the backups whose file name, note, backup id or
+                              creation day contains this text. Telegram's own index does
+                              the looking, so the whole chat is reached without reading it
+                              message by message — but it matches whole words only:
+                              "projex" finds projex.zip and "proj" finds nothing. A term
+                              with spaces has to be quoted, as --note does.
   --token                     Log in by pasting a session token. It takes no value on
                               purpose: a token written on the command line would sit in
                               "ps" for the whole life of the command, and stay in that
@@ -120,8 +132,38 @@ export function interruptMessage(command, { backupId, done = [] } = {}) {
     )
   }
 
+  // A restore keeps its .partial now, and the next run proves each chunk in it against the
+  // manifest before trusting a byte — so "running again starts over", which this said while
+  // there was nothing to resume from, would now be false.
   if (command === 'restore') {
-    return '\nStopped. Download progress is not saved, running again starts over.\n'
+    // Finished ids have been renamed to their real names and their records removed, so
+    // repeating the whole command line would meet an overwrite prompt and then download
+    // them again from nothing. Name them and ask for the rest, exactly as a batch upload does.
+    if (done.length > 0) {
+      const width = Math.max(...done.map((item) => basename(item.path).length))
+      const finished = done
+        .map((item) => `  ${basename(item.path).padEnd(width)}  ${item.id}`)
+        .join('\n')
+
+      return (
+        `\nStopped. These are finished and need no second run:\n${finished}\n` +
+        'Run telstore again with only the ids that are left — their .partial files are kept, ' +
+        'so those carry on where they stopped. "npx telstore status" shows what is unfinished.\n'
+      )
+    }
+
+    // With onBackupId firing only once the .partial is open, an id here means there is a
+    // file to carry on from. Without one, this run stopped before it wrote anything, and
+    // saying a .partial was kept would be the same lie this message was rewritten to stop
+    // telling — just from the other side.
+    if (!backupId) {
+      return '\nStopped before anything was written. Run the same command again to start.\n'
+    }
+
+    return (
+      `\nBackup ${backupId} kept its .partial file — run the same command again from this ` +
+      'directory to carry on, or "npx telstore status" to see what is left.\n'
+    )
   }
 
   // A delete has already destroyed messages for good by the time Ctrl-C lands, and the

package/src/client.js

@@ -1,8 +1,10 @@
 import { Api, TelegramClient } from 'teleproto'
 import { Logger } from 'teleproto/extensions/index.js'
 import { LogLevel } from 'teleproto/extensions/Logger.js'
+import { returnBigInt } from 'teleproto/Helpers.js'
 import { StringSession } from 'teleproto/sessions/index.js'
 
+import { MANIFEST_TAG } from './caption.js'
 import { manifestFileName } from './manifest.js'
 import { withRetry } from './retry.js'
 import { assertLoggedIn, unlockConfig } from './session.js'
@@ -23,6 +25,26 @@ export function documentFileName(message) {
   return named?.fileName ?? null
 }
 
+// Telegram records a document's length as a BigInteger, and comparing that to the plain
+// number a manifest carries with === is false for every size there is.
+export function documentSize(message) {
+  const size = message?.media?.document?.size
+
+  return size === undefined || size === null ? null : returnBigInt(size).toJSNumber()
+}
+
+// The flat shape both readers of a chat hand back. The raw message is kept alongside it
+// because downloading needs it whole.
+function toDocument(message) {
+  return {
+    id: message.id,
+    fileName: documentFileName(message),
+    caption: message.message ?? '',
+    date: message.date,
+    message,
+  }
+}
+
 // The one place telstore searches a chat. Both callers want documents and nothing else,
 // and getMessages is preferred over a raw Api.messages.Search because it handles offsets,
 // hashes and pagination itself, so we don't hand-build easily mistyped fields. The raw
@@ -34,13 +56,93 @@ export async function searchDocuments(client, peer, { search, limit }) {
     limit,
   })
 
-  return messages.map((message) => ({
-    id: message.id,
-    fileName: documentFileName(message),
-    caption: message.message ?? '',
-    date: message.date,
-    message,
-  }))
+  return messages.map(toDocument)
+}
+
+// The paging both readers share. It is ours rather than iterMessages', for the reasons
+// deleteMessages does not use teleproto's: every page then carries the retry policy and the
+// stall deadline, and a page that fails is retried by itself instead of restarting from the
+// newest message. offsetId is the id of the last message of the page before, and Telegram
+// answers with the messages older than it — a reader that forgot to advance it would fetch
+// the newest page over and over and never reach an older backup.
+//
+// A generator because the caller stops when it has what it wants: a chat of ten thousand
+// chunks costs one request to list the backups at the top of it.
+export const DOCUMENT_PAGE_SIZE = 100
+
+async function* iterMessagePages(client, peer, { search, what, options }) {
+  const {
+    pageSize = DOCUMENT_PAGE_SIZE,
+    max = Infinity,
+    retryOptions = {},
+    stallMs = DEFAULT_STALL_MS,
+  } = options
+
+  let offsetId = 0
+  let read = 0
+
+  while (read < max) {
+    const limit = Math.min(pageSize, max - read)
+
+    const messages = await withRetry(
+      () =>
+        withStallTimeout(
+          client.getMessages(peer, {
+            ...(search === undefined ? {} : { search }),
+            filter: new Api.InputMessagesFilterDocument(),
+            limit,
+            offsetId,
+          }),
+          stallMs,
+          () =>
+            `Telegram stopped answering while reading the ${what} older than message ` +
+            `${offsetId}: nothing back for ${Math.round(stallMs / 1000)}s.`,
+        ),
+      retryOptions,
+    )
+
+    if (!messages || messages.length === 0) return
+
+    for (const message of messages) yield toDocument(message)
+
+    read += messages.length
+    offsetId = messages[messages.length - 1].id
+
+    // A short page is the end of the results. Asking again would cost a request to be told
+    // the same thing.
+    if (messages.length < limit) return
+  }
+}
+
+// What `list` reads the chat with. searchDocuments asks Telegram's text index a question;
+// this asks for the documents themselves, newest first, which is the only answer that was
+// right every time it was measured — a chat's text index can come back empty while the chat
+// is full of backups, and did for a whole day in a channel that had just been created
+// (docs/design/captions.md carries the measurements).
+export async function* iterDocuments(client, peer, options = {}) {
+  yield* iterMessagePages(client, peer, { what: 'documents', options })
+}
+
+// What `list --search` reads the chat with, and the one place that knows how to ask the index
+// for backups. Two things about the query are load-bearing, both measured against a real chat
+// on 2026-09-08 (docs/design/captions.md carries the numbers):
+//
+// The tag is ANDed in because a term alone brings the backup's chunk messages back too — a
+// chunk caption carries the id — and on a backup of a few hundred chunks those would fill
+// every page before a single manifest appeared. `#telstore` rides on the manifest alone, so
+// the results come back manifests only: searching one backup id returned 1 manifest and 1
+// chunk, and the same id with the tag returned the manifest by itself.
+//
+// The tag goes *after* the term, never before. A query that starts with the hash is read as
+// a hashtag lookup and stops ANDing the rest: "#telstore projex" came back empty while
+// "projex #telstore" returned the one manifest. Leading it would turn every search into
+// "no backups found", which is the sentence this project must never say wrongly.
+export async function* iterManifestSearch(client, peer, term, options = {}) {
+  yield* iterMessagePages(client, peer, {
+    search: `${term} ${MANIFEST_TAG}`,
+    what: 'search results',
+    options,
+  })
 }
 
 // How telstore finds a backup's manifest, in one place because restore and delete must not
@@ -70,11 +172,15 @@ export async function readMessageBytes(client, message) {
 //
 // Telegram does not complain about an id that is no longer there, so sending a batch twice
 // costs nothing: a delete interrupted halfway is finished by running it again.
-export const DELETE_BATCH_SIZE = 100
+//
+// The hundred is Telegram's own limit on how many message ids one request may name, and it
+// is the same limit whether the request removes them or asks about them — so getDocuments
+// below counts in the same batches rather than keeping a second opinion about one number.
+export const MESSAGE_BATCH_SIZE = 100
 
 export async function deleteMessages(client, peer, ids, options = {}) {
   const {
-    batchSize = DELETE_BATCH_SIZE,
+    batchSize = MESSAGE_BATCH_SIZE,
     retryOptions = {},
     stallMs = DEFAULT_STALL_MS,
     onBatch,
@@ -110,6 +216,53 @@ export async function deleteMessages(client, peer, ids, options = {}) {
   return deleted
 }
 
+// What verify asks the chat, and the read-only mirror of deleteMessages above: our own
+// batching, one request in flight at a time, under the same retry policy and the same stall
+// deadline as every other network wait in telstore.
+//
+// The answer is a Map rather than a list because the question is "which of these are still
+// there". Telegram reports a message that is gone as MessageEmpty — an object carrying the
+// id it was asked about — so anything that is not a real message is left out here, where the
+// shape is understood, rather than passed on to a caller that would read an empty as a chunk
+// still sitting in the chat.
+export async function getDocuments(client, peer, ids, options = {}) {
+  const {
+    batchSize = MESSAGE_BATCH_SIZE,
+    retryOptions = {},
+    stallMs = DEFAULT_STALL_MS,
+    onBatch,
+  } = options
+
+  const found = new Map()
+
+  for (let start = 0; start < ids.length; start += batchSize) {
+    const batch = ids.slice(start, start + batchSize)
+
+    const messages = await withRetry(
+      () =>
+        withStallTimeout(
+          client.getMessages(peer, { ids: batch }),
+          stallMs,
+          () =>
+            `Telegram stopped answering while looking up messages ${start + 1}-` +
+            `${start + batch.length} of ${ids.length}: nothing back for ` +
+            `${Math.round(stallMs / 1000)}s.`,
+        ),
+      retryOptions,
+    )
+
+    for (const message of messages ?? []) {
+      if (!message || message instanceof Api.MessageEmpty) continue
+
+      found.set(message.id, message)
+    }
+
+    onBatch?.(Math.min(start + batch.length, ids.length), ids.length)
+  }
+
+  return found
+}
+
 // Every command ends by putting the connection down, and a failure there must never
 // swallow the real error already on its way up. Commands that print progress hand in an
 // onWarn to say so; the quieter ones let it pass, because a connection that will not

package/src/commands/delete.js

@@ -1,6 +1,8 @@
+import { promises as fs } from 'node:fs'
+
 import { chatName, describeChat } from '../chat.js'
 import {
-  DELETE_BATCH_SIZE,
+  MESSAGE_BATCH_SIZE,
   closeQuietly,
   connect as realConnect,
   deleteMessages as realDeleteMessages,
@@ -13,7 +15,7 @@ import { manifestFileName, manifestMessageIds, parseManifestJson } from '../mani
 import { formatBytes, formatDuration } from '../progress.js'
 import { assertLoggedIn } from '../session.js'
 import { requireChat, resolveSettings } from '../settings.js'
-import { clearState, findStates } from '../state.js'
+import { clearRestore, clearState, findRestores, findStates } from '../state.js'
 
 // What list prints when a card cannot be read back. A manifest is text off a chat, and a
 // summary is not worth inventing: the numbers below only decorate a decision the backup id
@@ -179,7 +181,7 @@ export async function runDelete(backupId, options = {}, deps = {}) {
       throw new Error('Cancelled on request.')
     }
 
-    const loud = chunkIds.length > DELETE_BATCH_SIZE
+    const loud = chunkIds.length > MESSAGE_BATCH_SIZE
     let removed = 0
 
     try {
@@ -223,6 +225,30 @@ export async function runDelete(backupId, options = {}, deps = {}) {
 
     if (record) await clearState(record.key, configDir)
 
+    // The chunks are gone from the chat, so a restore record pointing at this backup now
+    // names messages nobody can fetch: `status` would keep offering a resume command that
+    // can only fail. Dropped here rather than earlier for the same reason the upload record
+    // is — anything that throws above leaves the way back intact.
+    //
+    // The .partial itself stays. It is the user's data, sometimes gigabytes of it, and this
+    // command removes what was asked for and nothing else. But it can never be completed
+    // now, so it is named on the way out: that is the difference between a file they can
+    // reclaim and one they will never think to look for.
+    const stranded = []
+
+    for (const found of await findRestores(backupId, configDir)) {
+      await clearRestore(found.key, configDir)
+
+      const partial = `${found.record.target}.partial`
+
+      try {
+        await fs.stat(partial)
+        stranded.push(partial)
+      } catch {
+        // Nothing there to tell them about.
+      }
+    }
+
     if (manifestMessage) {
       log(
         `\nDone. Removed ${backupId} from ${chatName(chat)}: ` +
@@ -236,6 +262,13 @@ export async function runDelete(backupId, options = {}, deps = {}) {
       )
     }
 
+    for (const partial of stranded) {
+      log(
+        `${partial} is a half-finished restore of this backup. Nothing can finish it now — ` +
+          'delete it when you want the space back.',
+      )
+    }
+
     return {
       id: backupId,
       chunks: chunkIds.length,