telstore 0.1.8 → 0.1.10

package/README.md

@@ -16,6 +16,9 @@ npx telstore config chat @my_backups           # where backups go, from now on
 npx telstore data.tar                          # split it and send it there
 npx telstore a.tar b.tar c.tar                 # or several: one backup each, one after another
 npx telstore ./backups                         # or a folder: every file one level inside it
+npx telstore a.tar -- tar cf - ./a             # or from a command, with no file on disk first
+npx telstore tarc a.tar.gz ./dir               # or the common case of that: archive and store
+npx telstore tarx telstore-20260905-7f3a91     # restore a backup and extract it with tar
 npx telstore list                              # what is already in the destination
 npx telstore restore telstore-20260905-7f3a91
 ```
@@ -26,13 +29,19 @@ npx telstore restore telstore-20260905-7f3a91
 |---|---|
 | `telstore login` | Log in to Telegram. Add `--token` to log in with a session token instead. |
 | `telstore <file\|folder\|pattern>...` | Split each file and upload it. Prints the `backupId` you restore with. |
+| `telstore <name> -- <command>...` | Run the command and store what it writes, under `<name>`. The manifest goes out only if the command exits 0. |
+| `telstore tarc <name> <path>...` | Archive the paths with `tar` and store the result. Always compresses, so the name ends in `.tar.gz`. |
+| `telstore tarx <backup-id>` | Restore a backup and extract it with `tar`. |
 | `telstore list` | The backups stored in the destination, newest first. |
 | `telstore restore <backup-id>...` | Download every chunk and reassemble the file. Several ids run one after another. |
+| `telstore restore <backup-id> -- <command>...` | Restore onto the command's stdin instead of a file. |
+| `telstore verify <backup-id>...` | Check that every chunk of a backup is still in the chat. Downloads nothing. |
 | `telstore delete <backup-id>...` | Remove a backup's chunks and manifest from the chat, for good. Several ids are listed and confirmed once. |
 | `telstore status` | Account, destination, and unfinished uploads and restores. |
 | `telstore config` | Show or change settings. |
 | `telstore token` | Print a session token for a machine you do not trust. |
 | `telstore logout` | Remove the locally stored session. |
+| `telstore down` | Remove everything telstore keeps on this machine: session, credentials, settings and resume records. Asks once. Deletes nothing from Telegram. |
 
 ## Settings
 
@@ -64,8 +73,10 @@ a prompt that does not echo.
 ## What the chat looks like
 
 Every chunk goes up as a document captioned `📦 <backupId> · 3/12`, followed by a manifest
-carrying a summary card — file name, size, id, date and the restore command. `list` reads
-those cards straight out of the chat, one search and no downloads:
+carrying a summary card — file name, size, id, date and the restore command. A backup made
+from a command's output is captioned `📦 <backupId> · 3`, with no total: it does not learn how
+many chunks there are until the last one has gone out, and the manifest card carries the final
+count. `list` reads those cards straight out of the chat, no downloads:
 
 ```
 Destination  https://web.telegram.org/k/#@my_backups
@@ -117,13 +128,96 @@ it, and the run ends with a line per file and a non-zero exit code:
   c.tar  telstore-20260905-9de447  (1 chunk)
 ```
 
+## A backup made from a command
+
+`--` says the rest of the line is a command for telstore to run, and what that command writes
+to its standard output is the backup. The name in front of `--` is a label for it, not a file
+telstore reads:
+
+```bash
+npx telstore a.tar -- tar cf - ./a
+npx telstore dir.tar.age -- bash -c 'set -o pipefail; tar c ./dir | age -r age1abc...'
+```
+
+**`set -o pipefail` is load-bearing, not tidiness.** A shell exits with the status of the
+*last* command in a pipeline, so without it a `tar` that dies at 50% hands `age` a clean end
+of input; `age` encrypts what it got, exits 0, the shell exits 0, and the guarantee below is
+satisfied by a truncated archive that restores cleanly and matches every sha256. `pipefail`
+is what makes the shell report the producer's failure as its own. `bash` rather than `sh`
+because `sh` is `dash` on Debian and Ubuntu and `dash` has no `pipefail` — it refuses the line
+and exits 2 before writing a byte, which telstore turns into a failed run rather than a bad
+backup, but a shell that refuses is not an example anyone can paste.
+
+Nothing has to exist on disk first, which is the point: `tar` of a 200GB directory would
+otherwise need 200GB free before a single byte reached Telegram, and `pg_dump` leaves no file
+to point telstore at in the first place.
+
+**The manifest is sent if and only if the command's output reached its end *and* the command
+exited 0.** That biconditional is the whole reason telstore runs the command itself instead of
+reading a pipe. When `tar` dies halfway it closes its end of the pipe, and an EOF after a crash
+is byte-for-byte the same event as an EOF after success — a reader on the other end cannot tell
+them apart, and would hand you a manifest for a truncated archive that restores cleanly, matches
+every sha256, and is garbage. A parent process sees the exit code.
+
+So a run that fails **removes the chunks it already sent.** That is the opposite of what a file
+upload does, and deliberately: a file upload keeps its chunks because a second run resumes onto
+them. A stream has no second run — the bytes have gone past, and a later run cuts them
+differently — so a chunk left in the chat by a failed stream is a chunk no manifest will ever
+name. Ctrl-C means the same thing here: telstore asks the run to remove what it sent and waits
+for it, rather than leaving where it stands. If the removal itself cannot finish, the record
+stays on this machine and the run prints the `npx telstore delete <id> --chat <chat>` that
+finishes it by hand. `delete` reads the chat for chunks carrying the backup id rather than
+taking that record's word for what is there, so a chunk that landed after the record stopped
+being written goes too — and when it cannot read far enough back to be sure, it says so
+instead of reporting the backup gone.
+
+There is no shell in between: the command is spawned as an argv, so nothing needs quoting and
+telstore never builds a command string out of your arguments. `-- bash -c 'set -o pipefail;
+...'` is how a pipeline gets in — with `pipefail`, for the reason above: the biconditional is
+only as good as the exit code the shell reports, and a pipeline without it reports the wrong
+one. That is also how your data is compressed or encrypted **before** it reaches Telegram —
+`zstd`, `gpg`, `age` — with telstore holding nobody's passphrase. The command's
+stderr is left as it is, so one that fails explains itself in its own words and telstore adds
+only the exit code and what it did about it.
+
+## Checking a backup is still there
+
+A backup is a set of messages in a chat, and messages can be deleted by hand. `list` reads
+the manifest's card and would happily show a backup whose chunks are long gone; `verify` asks
+the chat about every chunk the manifest names:
+
+```
+$ npx telstore verify telstore-20260905-7f3a91
+Backup telstore-20260905-7f3a91
+File   data.tar (21.4 GB, 12 chunks)
+In     https://web.telegram.org/k/#@my_backups
+
+12 chunks present, at the sizes the manifest records.
+This does not download them, so it cannot prove their contents.
+```
+
+It costs one request per hundred chunks and no bandwidth, so it is cheap enough to run on a
+schedule. What it proves is that a restore would find everything it needs — every chunk still
+there, under the file name telstore wrote, at the length the manifest records. It does not
+read the chunks, so it cannot speak for what is inside them; only a restore does that, and a
+restore checks every sha256 before it renames anything into place.
+
+A backup missing chunks is named line by line, and the exit code is 1:
+
+```
+Chunk 3/12 is gone: message 1042 is no longer in @my_backups.
+
+12 chunks checked, 1 damaged. This backup cannot be restored.
+```
+
 ## Several backups at once
 
-`restore` and `delete` take a list of ids the same way, over one connection, with a summary
-and a non-zero exit code if any of them failed:
+`restore`, `verify` and `delete` take a list of ids the same way, over one connection, with a
+summary and a non-zero exit code if any of them failed:
 
 ```bash
 npx telstore restore telstore-20260905-7f3a91 telstore-20260901-9de447
+npx telstore verify telstore-20260905-7f3a91 telstore-20260901-9de447
 npx telstore delete telstore-20260905-7f3a91 telstore-20260901-9de447
 ```
 
@@ -179,14 +273,25 @@ There is no expiry and no revocation: to end a session for good, terminate it un
   else's infrastructure — the one thing telstore encrypts is a session token, and that
   protects your login rather than your files.
 - A chunk cannot exceed 1950MB: Telegram accepts at most 4000 parts of 512KB per file.
+- **A backup made from a command cannot be resumed**, so a run that fails or is interrupted
+  removes the chunks it had already sent instead of keeping them. Ctrl-C takes a moment longer
+  for that reason, and leaves nothing of that run behind in the chat.
 - Deleting a chunk message in the Telegram app destroys the backup, and keeping the
-  `backupId` is what saves you hunting for its manifest in the chat by hand.
+  `backupId` is what saves you hunting for its manifest in the chat by hand. `verify` is how
+  you find that out before you need the file rather than after.
 
 Settings and credentials live in `~/.telstore/config.json`, mode 600 — `apiId`, `apiHash` and
 the session at the top level (or a single `sealed` blob after `login --token`), everything
 `config` manages under `settings`. Editing it by hand is fine: a value that cannot be used is
 named on the next run, with the file and the key.
 
+`npx telstore down` removes all of that, and `~/.telstore/state/` with it — everything on
+this machine, in one question. It opens no connection: your backups stay in the chat, and
+the session stays alive on Telegram's side until you terminate it under Settings → Devices.
+Unlike `logout` it takes the `api_id` and `api_hash` too, so the next `login` asks for them
+again. A half-finished restore's `.partial` is left where it is and named on the way out —
+it still resumes, and after this nothing else will remind you it is there.
+
 ## License
 
 MIT

package/bin/telstore.js

@@ -1,4 +1,6 @@
 #!/usr/bin/env node
+import { unlinkSync } from 'node:fs'
+
 import { route, HELP, interruptMessage } from '../src/cli.js'
 
 // Each command is imported where it runs, not here. Importing all nine up front pulled
@@ -8,25 +10,194 @@ import { route, HELP, interruptMessage } from '../src/cli.js'
 
 const SIGINT_EXIT_CODE = 130
 
+// How long the process waits for a run to unwind itself before it stops waiting. Only a
+// rollback that cannot reach Telegram ever gets this far, and the alternative to a deadline
+// is a terminal held open by a network that is never coming back, with no way out but a
+// second Ctrl-C nobody has been told to press.
+const CLEANUP_DEADLINE_MS = 60_000
+
 // Which command is running when Ctrl-C arrives — each one tells a different truth
 // about whether progress was saved, so we need to know which to pick the right line.
 // The backup id arrives a moment later, once upload knows which backup this run is.
 let currentCommand = null
 let currentBackupId = null
 
+// Only a stream upload sets these. `streaming` is known from the command line, before the run
+// has said anything, because the wrong message is the file one that promises a resume; the
+// other two arrive once the run has something in the chat and a way to unwind it.
+let streaming = false
+let currentChat = null
+let abortRun = null
+let interrupting = false
+let deadline = null
+
+// The run is over — it finished, or it unwound itself and threw. Nothing after this point is
+// something Ctrl-C can be about: the process is only waiting for its own output to flush.
+let settled = false
+
+// This process is already on its way out, and has already said why.
+let leaving = false
+
+// The chunk file a stream upload is buffering into right now, or null. The run removes its
+// own on every ending it gets to run code for; this exists for the one ending it does not.
+let tempChunk = null
+
+// The command a streaming restore is feeding, if one is running. A stream upload unwinds
+// cooperatively and kills its own child on the way; a restore has nothing in the chat to
+// unwind, so Ctrl-C leaves at once — and leaving without this would let tar go on writing
+// files into somebody's directory after telstore had said it stopped. Synchronous, like
+// dropTempChunk and for the same reason: anything awaited here would be waiting on the run
+// this exit exists to stop waiting for.
+let killChild = null
+
+function stopChild() {
+  const kill = killChild
+
+  if (kill === null) return
+
+  killChild = null
+
+  try {
+    kill()
+  } catch {
+    // Already gone. There is nothing this could do about it and nothing worth saying.
+  }
+}
+
+// What a Ctrl-C that will not wait can still do about that file, and it has to be exactly this
+// shape: local and unawaited, because anything awaited here would be waiting on the very run
+// this exit exists to stop waiting for. `unlinkSync` holds no handle, opens no socket and
+// cannot hang. Removing a file this process still has open is not a problem where it matters
+// either: on POSIX the name goes now and the space comes back as the process dies. A platform
+// that refuses to unlink an open file leaves the user with a file holding up to a whole chunk,
+// so that ending — and only that ending — prints.
+//
+// It is not free, though, and "one syscall" is the wrong number to quote. Three costs, and
+// they are three different numbers: the call removes a name and returns in microseconds; the
+// goodbye line still reaches the terminal in about 1.2ms; and then the extents are freed at
+// the *last close*, which is process teardown. Measured on ext4, SIGINT to exit, file still
+// open: 60-100ms against 29-43ms for a 64MB chunk, 596-658ms against 45-56ms at 512MB, and
+// 1131-1646ms against 45-64ms at the default 1792MB — so at default settings the prompt comes
+// back about a second after the message does. Nothing avoids that second: whoever runs `rm`
+// on the leftover instead pays the same teardown. docs/design/data-integrity.md has the
+// conditions.
+//
+// Silence on success is the point rather than an omission: the rule is that nothing telstore
+// leaves behind goes unnamed, and this leaves nothing behind.
+function dropTempChunk() {
+  const file = tempChunk
+
+  if (file === null) return ''
+
+  tempChunk = null
+
+  try {
+    unlinkSync(file)
+    return ''
+  } catch (err) {
+    // Already gone — the run's own discard won the race — is not something to report.
+    if (err.code === 'ENOENT') return ''
+
+    return (
+      `\nThe chunk telstore was buffering is still on this machine: ${file} ` +
+      `(${err.message}). It holds up to one chunk — remove it by hand.\n`
+    )
+  }
+}
+
 // A batch clears each finished item's record as it goes, so by the time Ctrl-C lands these
 // are transfers no second run should touch. Ctrl-C needs their names to say so.
 const finished = []
 
+// Every exit Ctrl-C leads to comes through here. Through exitWhenFlushed rather than straight
+// to process.exit, because on a pipe stderr is asynchronous, and the line most at risk of
+// being cut in half is the one below carrying the command that removes the leftovers.
+function leave(message) {
+  stopChild()
+
+  // Called twice means Ctrl-C landed again while the first line was still flushing, or the
+  // deadline arrived on top of it. There is nothing more to say, and someone pressing it a
+  // second time is asking to be gone rather than read to.
+  if (leaving) process.exit(SIGINT_EXIT_CODE)
+
+  leaving = true
+
+  // Every exit the handler leads to comes through here, which is why the borrowed disk is
+  // given back here rather than in the second-Ctrl-C arm alone: the deadline running out
+  // ends the process in exactly the same place, and so does a Ctrl-C on a run that had
+  // nothing to unwind. One write, so the two lines cannot be split by a slow pipe.
+  process.stderr.write(message + dropTempChunk())
+  exitWhenFlushed(SIGINT_EXIT_CODE)
+}
+
+// Said when the waiting ends without the rollback having finished — because someone pressed
+// Ctrl-C again, or because the deadline above ran out. Neither knows how far the removal got,
+// so both point at the command that finishes it by hand.
+function leaveNow() {
+  leave(
+    interruptMessage(currentCommand, {
+      backupId: currentBackupId,
+      done: finished,
+      stream: true,
+      again: true,
+      chat: currentChat,
+    }),
+  )
+}
+
 process.on('SIGINT', () => {
   // A passphrase prompt has stdin in raw mode, and process.exit skips readline's own cleanup.
   // Without this, Ctrl-C hands back a shell that no longer echoes what is typed into it.
   if (process.stdin.isTTY) process.stdin.setRawMode(false)
 
+  // Checked before anything else, because every message below describes a run that is still
+  // going. Said about one that has settled they are all false: a removal that is not running,
+  // leftovers that were already removed, a resume for a backup that is finished and valid.
+  // The window is real — exitWhenFlushed waits up to two seconds on a pipe nobody is reading.
+  if (settled) {
+    leave(interruptMessage(null))
+    return
+  }
+
+  // A second Ctrl-C is someone saying they will not wait.
+  if (interrupting) {
+    leaveNow()
+    return
+  }
+
+  // Everything else keeps what it has sent on purpose — a file upload's chunks are what the
+  // next run resumes onto — so there is nothing to unwind and Ctrl-C is the immediate exit it
+  // has always been.
+  if (!abortRun) {
+    leave(
+      interruptMessage(currentCommand, {
+        backupId: currentBackupId,
+        done: finished,
+        stream: streaming,
+      }),
+    )
+    return
+  }
+
+  interrupting = true
   process.stderr.write(
-    interruptMessage(currentCommand, { backupId: currentBackupId, done: finished }),
+    interruptMessage(currentCommand, {
+      backupId: currentBackupId,
+      done: finished,
+      stream: true,
+    }),
   )
-  process.exit(SIGINT_EXIT_CODE)
+
+  // Deliberately not unref'd, and it does two jobs. It is the deadline; it is also the one
+  // handle that keeps this process alive while the rollback runs. Node leaves with 0 the
+  // moment nothing is pending, and a rollback waiting on something that holds no handle
+  // would end the process mid-removal reporting success, with the chunks still in the chat.
+  // The upload arm clears it, so a rollback that finishes normally never meets the deadline.
+  deadline = setTimeout(leaveNow, CLEANUP_DEADLINE_MS)
+
+  // The abort itself only asks; the waiting is done by main, which is still awaiting the run.
+  // Nothing it could reject with matters here — the run reports its own end.
+  Promise.resolve(abortRun()).catch(() => {})
 })
 
 async function main() {
@@ -65,6 +236,13 @@ async function main() {
       return
     }
 
+    case 'down': {
+      const { runDown } = await import('../src/commands/down.js')
+
+      await runDown(parsed.args, parsed.options)
+      return
+    }
+
     case 'list': {
       const { runList } = await import('../src/commands/list.js')
 
@@ -94,6 +272,46 @@ async function main() {
     }
 
     case 'upload': {
+      // `telstore a.tar -- tar cf - ./a`: one name, and the bytes are what the command writes
+      // rather than a file on disk. route has already refused every other shape of that line.
+      if (parsed.childArgv) {
+        const { runStreamUpload } = await import('../src/commands/upload-stream.js')
+
+        streaming = true
+
+        try {
+          await runStreamUpload(parsed.args[0], parsed.childArgv, parsed.options, {
+            onBackupId: (id) => {
+              currentBackupId = id
+            },
+            onAbortable: (abort, { chat } = {}) => {
+              abortRun = abort
+              currentChat = chat ?? null
+            },
+            onTempChunk: (file) => {
+              tempChunk = file
+            },
+          })
+        } catch (err) {
+          // Only the run that stopped because Ctrl-C asked it to, which has already said so
+          // and already put the chat back as it found it. Everything else — a rollback that
+          // could not finish included, since that throws an error of its own — is a failure
+          // the handler below reports.
+          if (!err?.interrupted) throw err
+
+          process.stderr.write('\nStopped. Nothing this run sent was left in the chat.\n')
+          process.exitCode = SIGINT_EXIT_CODE
+        } finally {
+          // Nothing left to unwind, and nothing left to say about it: whichever way the run
+          // ended, it ended. The deadline that was holding the process open goes with it.
+          settled = true
+          abortRun = null
+          if (deadline) clearTimeout(deadline)
+        }
+
+        return
+      }
+
       const { runUploads } = await import('../src/commands/upload.js')
 
       const { failed } = await runUploads(parsed.args, parsed.options, {
@@ -115,6 +333,36 @@ async function main() {
     }
 
     case 'restore': {
+      // The spec's stage 2, and the refusal that stood here is gone: the bytes were asked for
+      // on a command's stdin and that is now where they go.
+      if (parsed.childArgv) {
+        const { runRestoreStream } = await import('../src/commands/restore-stream.js')
+
+        // So Ctrl-C says the restore sentence rather than the upload one.
+        streaming = true
+
+        await runRestoreStream(parsed.args[0], parsed.childArgv, parsed.options, {
+          // Only the alias promises gzip. The general form promises nothing and is asked
+          // nothing, which is how `restore <id> -- tar xf -` stays useful.
+          requireGzipName: parsed.shortcut === 'tarx',
+          onBackupId: (id) => {
+            currentBackupId = id
+          },
+          onTempChunk: (file) => {
+            tempChunk = file
+          },
+          onChild: (kill) => {
+            killChild = kill
+          },
+        })
+
+        // Mirrors the stream-upload branch above: the run has finished, so a Ctrl-C landing in
+        // the output-flush window that follows must not print the "removing what it already
+        // sent" sentence about a restore that already has everything it is going to get.
+        settled = true
+        return
+      }
+
       if (!parsed.args[0]) {
         throw new Error('Missing backup id. Example: npx telstore restore telstore-20260905-7f3a91')
       }
@@ -134,6 +382,21 @@ async function main() {
       return
     }
 
+    case 'verify': {
+      if (!parsed.args[0]) {
+        throw new Error('Missing backup id. Example: npx telstore verify telstore-20260905-7f3a91')
+      }
+
+      const { runVerifies } = await import('../src/commands/verify.js')
+
+      const { failed } = await runVerifies(parsed.args, parsed.options)
+
+      // A backup that is damaged, and one telstore could not look up at all, both mean the
+      // run did not find what it was asked to check. Whatever runs telstore learns that here.
+      if (failed > 0) process.exitCode = 1
+      return
+    }
+
     case 'delete': {
       if (!parsed.args[0]) {
         throw new Error('Missing backup id. Example: npx telstore delete telstore-20260905-7f3a91')
@@ -182,7 +445,12 @@ main().then(
   },
   (err) => {
     process.stderr.write(`Error: ${err.message}\n`)
-    process.exitCode = 1
-    exitWhenFlushed(1)
+
+    // A run the handler asked to stop leaves with 130 whatever it then had to say: a rollback
+    // that could not finish is still an interrupted run, not a command that failed on its own.
+    const code = interrupting ? SIGINT_EXIT_CODE : 1
+
+    process.exitCode = code
+    exitWhenFlushed(code)
   },
 )

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "telstore",
-  "version": "0.1.8",
+  "version": "0.1.10",
   "description": "Split large files into chunks and store them on Telegram",
   "keywords": [
     "telegram",

package/src/caption.js

@@ -49,8 +49,13 @@ function utcMinutes(createdAt) {
   return `${new Date(createdAt).toISOString().slice(0, 16).replace('T', ' ')} UTC`
 }
 
+// A stream knows its chunk number and not its count. "3/?" would be a question mark in the
+// chat forever; the number alone is true at the time it is written, and the manifest card
+// carries the final count.
 export function chunkCaption({ id, number, total }) {
-  return `📦 ${id} · ${number}/${total}`
+  return total === null || total === undefined
+    ? `📦 ${id} · ${number}`
+    : `📦 ${id} · ${number}/${total}`
 }
 
 export function manifestCaption({ id, name, size, chunks, createdAt, note = null }) {